1. VibeVoice-ASR-7B: Speech-to-Text with Diarization#

What it does in one pass: Takes a 60-minute audio file and produces a structured transcript telling you who said what at what time — all without chunking.

The key benchmark metrics:

• cpWER (concatenated minimum-permutation Word Error Rate): 5.8% — nearly 3× better than Whisper-large-v3 on multi-speaker data
• DER (Diarization Error Rate): Industry-leading speaker separation accuracy
• tcpWER (time-constrained cpWER): Maintains accuracy even with tight timestamp constraints

Custom hotwords deserve special mention. If your domain has specialized terminology (medical terms, product names, technical jargon), you can inject a list of hotwords that guide recognition. This dramatically reduces domain-specific errors without fine-tuning.

Fine-tuning code is available — you can adapt the model to your specific domain or language on your own hardware.

Getting started:

1
# Via HuggingFace Transformers
2
from transformers import pipeline
3

4
asr = pipeline("automatic-speech-recognition",
5
               model="microsoft/VibeVoice-ASR-HF")
6
result = asr("podcast.mp3",
7
             return_timestamps=True,
8
             return_speaker_diarization=True)
9
# Output: [{"speaker": "Speaker 1", "start": 0.0, "end": 3.5, "text": "..."}]

2. VibeVoice-TTS-1.5B: Long-Form Multi-Speaker Synthesis#

If ASR is VibeVoice’s production workhorse, TTS is its creative masterpiece. The model can generate 90 minutes of continuous speech with up to 4 distinct speakers — maintaining consistent voices, natural turn-taking, and semantic coherence throughout.

Benchmark results vs. competitors (from the VibeVoice-TTS paper):

Capabilities:

• 90-minute long-form — Generate a full podcast, audiobook chapter, or lecture with consistent voice quality throughout
• Multi-speaker dialogues — 4 speakers with natural turn-taking, interruptions, and overlapping speech
• Expressive speech — Captures conversational dynamics, emotional nuances, and emphasis
• Cross-lingual — Speak English with Mandarin fluency, or vice versa
• Spontaneous singing — Yes, it can sing. Demos include “See You Again” in multi-speaker arrangement

Important note: The TTS code was removed from the public repo in September 2025 after Microsoft identified misuse. The model weights remain available on HuggingFace, and the architecture is fully documented in the ICLR 2026 paper. Researchers can request access through the standard process.

3. VibeVoice-Realtime-0.5B: Streaming TTS on Consumer Hardware#

This is the model for applications: voice assistants, accessibility tools, real-time dubbing, and interactive characters.

At 0.5B parameters, it runs on a laptop GPU or even CPU with reasonable performance. The 300ms first-audio latency means it feels responsive — start speaking within a third of a second of receiving text input.

Experimental voices include multilingual support for 9 languages (German, French, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish) and 11 distinct English style voices (narration, conversational, formal, etc.).

Getting started:

1
# Quick start on Colab or local
2
git clone https://github.com/microsoft/VibeVoice.git
3
cd VibeVoice
4

5
# Install dependencies
6
pip install -r requirements.txt
7

8
# Run real-time TTS demo
9
python examples/realtime_tts.py --text "Hello, this is VibeVoice speaking."

For Transcription (ASR)#

Option A: HuggingFace Transformers (Easiest)

1
# Works with any Transformers >= 4.48.0
2
from transformers import pipeline
3
import torch
4

5
pipe = pipeline(
6
    "automatic-speech-recognition",
7
    model="microsoft/VibeVoice-ASR-HF",
8
    device="cuda" if torch.cuda.is_available() else "cpu"
9
)
10

11
result = pipe(
12
    "meeting_recording.mp3",
13
    chunk_length_s=0,  # Single pass — no chunking
14
    return_timestamps="word",
15
    generate_kwargs={
16
        "custom_hotwords": "MuleSoft,DataWeave,Anypoint Platform"
17
    }
18
)
19

20
for segment in result["segments"]:
21
    print(f"[{segment['speaker']}] {segment['start']:.1f}s: {segment['text']}")

Option B: vLLM (For Production)

1
# vLLM server for production inference
2
python -m vllm.entrypoints.openai.api_server \
3
    --model microsoft/VibeVoice-ASR \
4
    --port 8000

For Speech Synthesis (TTS — Real-time)#

1
# Using the real-time model (open weights, open code)
2
from vibevoice import RealtimeTTS
3

4
tts = RealtimeTTS(model_name="microsoft/VibeVoice-Realtime-0.5B")
5
tts.speak("Welcome to the future of open-source voice AI.")

Demo Links#

• ASR Playground: https://aka.ms/vibevoice-asr
• Realtime TTS Colab: Try now
• Project Page: https://microsoft.github.io/VibeVoice

VibeVoice-ASR vs. OpenAI Whisper#

Winner depends on use case. Whisper supports more languages. VibeVoice is dramatically better for long-form, multi-speaker audio.

VibeVoice-TTS vs. ElevenLabs#

ElevenLabs wins on polish and convenience. VibeVoice wins on control, cost, and long-form generation.

VibeVoice-Realtime vs. Other Real-Time TTS#

For Developers#

• Podcasters: Transcribe 60-minute episodes with speaker labels in one pass
• Meeting note tools: Build real-time transcription with domain-specific hotwords
• Voice assistants: Real-time TTS with 300ms latency, running on consumer hardware
• Accessibility tools: Generate natural-sounding speech for screen readers
• Content creators: Produce multi-speaker audio for videos, podcasts, audiobooks

For Flutter/Mobile Devs#

VibeVoice-Realtime at 0.5B is small enough to consider for mobile deployment with ONNX quantization. No cloud dependency for basic TTS functionality.

For Researchers#

• ASR fine-tuning code available — adapt to specific domains or languages
• TTS architecture documented in ICLR 2026 paper
• Novel 7.5Hz tokenization approach worth studying