OpenSource Check: NVIDIA Nemotron 3.5 Transcribe and Granary v2 for Deployable Multilingual ASR

For teams comparing open-weight speech models, the useful questions are operational: where to download, what can be prompted, which metrics are public, and what adaptation really costs. NVIDIA Nemotron 3.5 Transcribe and Granary v2 answer those questions with a single speech checkpoint, a reusable data pipeline, and fine-tuning recipes that can be evaluated before enterprise use.

It is available on Hugging Face and ModelScope, supports 40 languages, translates 31 of them to or from English, and attributes speakers in 13. The same checkpoint can process recordings up to an hour long and can label up to eight speakers, while developers control transcription, speaker attribution, translation, and audio question answering through natural language prompts.

Prompt design and verification workflow

Instead of shipping separate endpoints, teams can ask the model to change the output form:

• Transcribe this recording and label each speaker.
• Translate this audio into English.
• What decisions were made in this meeting?

The model is shown in action on a classic scene from the 1940 screwball comedy His Girl Friday, chosen because rapid-fire, heavily overlapping dialogue tests whether Nemotron 3.5 Transcribe can still label speakers.

For unfamiliar terms, try context before training. In a healthcare setting, for example:

Transcribe this medical discussion in the original language. Possible terms: HbA1c, metformin. Use these terms only when supported by the audio. Preserve standard medical spelling and write numbers in conventional written form.

The model card provides recommended presets, including reasoning for audio questions, plus the full prompt guide and a task-by-language matrix. Language support differs by task, and audio question answering is trained in English. Word-level timestamps are also supported. As with any speech model, spot-check names, speaker changes, and answers against the recording.

In a classroom demo, Agora used word-level timestamps. Agora’s real-time voice and video platform powers live conversations in apps; with Nemotron 3.5 Transcribe, a recorded class becomes a transcript shown alongside the replay, letting students see what was said and when. Agora’s blog walks through how they built it.

Benchmarks: public evidence for selection

Transcribe leads open-weight models for content accuracy and wins two speaker-attribution benchmarks outright.

Hugging Face Open ASR Leaderboard

Public test sets are scored by word error rate (WER), and the leaderboard has three tracks:

• English: 5.33% average WER, first among open-weight models.
• Multilingual: 4.62% mean WER across seven languages, the lowest among evaluated open-weight models in the model card’s Open ASR comparison.
• Long-form: 10.98% mean WER, also the lowest in that comparison.

Figure 1. With API models excluded, NVIDIA Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR Leaderboard English track at 5.33% average WER; lower WER means fewer transcription errors.

Artificial Analysis Speech-to-Text Leaderboard

Artificial Analysis ranks speech-to-text models using AA-WER. At 2.8% AA-WER, Nemotron 3.5 Transcribe ties for first among open-weight models, with about 4B active parameters versus 24B for the model it ties. On AA-AgentTalk, the held-out voice-agent speech set that makes up 50% of the AA-WER index, Transcribe scores 1.8% WER, versus 2.3% for the next-best open-weight model. Its speech encoder is paired with a hybrid Mamba and Transformer mixture-of-experts language model, so only about 4B of its 30B parameters are active per token. That places it on Artificial Analysis’s open-weight Pareto frontier of accuracy versus active parameters. On the full AA chart, it ranks ahead of several widely used proprietary speech APIs.

Figure 2. About one-sixth the active parameters supports leading AA accuracy: Nemotron 3.5 Transcribe achieves 2.80% AA-WER with approximately 4B active parameters.

Speaker attribution across languages and overlap

Because Sortformer speaker cues reach the language model with the acoustic features, Transcribe can tag each turn during generation rather than relying solely on a later step to attach speakers to words.

David AI DAI-ASR-I18N Leaderboard

On David AI DAI-ASR-I18N Leaderboard, Transcribe ranks first among evaluated open-weight models on the combined transcription-and-speaker-attribution track and second overall. On the benchmark’s 21-language diarization track, which scores who spoke when across 21 languages, it has the lowest diarization error rate of the 14 systems tested, slightly ahead of NVIDIA’s dedicated Nemotron 3 Diarization model.

Figure 3. The top two spots on David AI’s multilingual diarization leaderboard belong to NVIDIA models: Nemotron 3.5 Transcribe posts the lowest overall diarization error rate (DER), 18.26% across 21 languages, followed by Nemotron 3 Diarization at 18.38%. DER measures speaker-time errors; lower is better.

VoiceArena Diarization-Bench

In VoiceArena’s 14-system comparison, Transcribe posts the lowest concatenated minimum-permutation WER (cpWER) of 14 systems and ranks first in both in-person and online settings. It also drops the fewest words: deletions are 8.32% of reference words, versus a median of about 31%.

Figure 4. In VoiceArena’s 14-system comparison, NVIDIA models hold the top two spots: Nemotron 3.5 Transcribe leads at 32.4% cpWER, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. Diarization-Bench v1 covers English full sessions, overlapping speech and 2–8 speakers. cpWER counts word errors after matching speaker labels; lower is better.

Other evaluations and throughput

In Ocular AI’s Converse-STT evaluation, Transcribe is ranked the top open model. Josh Talks’ Voice of India leaderboard reports Transcribe’s results across Indian languages.

Benchmarking showed Transcribe processing 100 hours of audio in under three minutes on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU, using the forthcoming NVFP4 quantized version. Results were measured on 30-second clips at high concurrency.

The available checkpoint can be served today with vLLM. Local deployment with NeMo-Speech.cpp is coming soon alongside the quantized versions, allowing local use on RTX Spark, DGX Spark, or Jetson AGX Thor, or larger deployment on Blackwell data-center GPUs.

Specialized recordings can still call for adaptation to domain vocabulary and transcript conventions. Granary v2 and NeMo Speech fine-tuning address that, starting with the data behind Transcribe.

Granary v2: the data pipeline behind the checkpoint

Granary v2 is part of the Nemotron speech collection on Hugging Face and ModelScope. Its reusable pipeline was built with NVIDIA NeMo Curator and prepared multilingual speech used in Nemotron 3.5 Transcribe’s training, producing aligned transcript variants, task labels, and inspectable quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. These figures describe the dataset release, not Transcribe’s full training mixture or its sampling weights.

For each recording, Granary v2 generates three transcript forms, matching the verbatim and clean styles developers can request from Transcribe with a prompt. Beyond transcription, the pipeline produces short audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing terms, and translations with automatic quality scores.

Training target	Illustrative transcript
Text-normalized, verbatim	I paid, uh, twenty-five euros.
Inverse text normalization, with disfluencies	I paid, uh, €25.
Inverse text normalization, without disfluencies	I paid €25.

Table 1. Granary v2 training targets for the same utterance.

The pipeline is organized into four reusable stages:

1. Audio Preparation (Metadata): raw or long-form audio is segmented, speakers are tracked across the full recording, and acoustic events and language are tagged. Because Granary v2’s sources were already segmented, this release skipped this stage.
2. Transcription (ASR): each language is routed to primary and recovery ASR models, falling back when the primary fails a hallucination check. Human reference transcripts help preserve trusted spellings while adding spoken forms and disfluencies.
3. Task Label Generation (Text): transcript variants are derived, then character rate, hallucination patterns, and model or human-reference agreement are checked. Only accepted records receive contextual-ASR terms and translations.
4. High Fidelity (HiFi): segments that are noisy, clipped, or contain multiple speakers are filtered out, keeping a smaller set of high-quality recordings for uses beyond ASR. For the VoxPopuli dataset, it reduced 354k hours to 111k hours across 22 European languages.

Figure 6. The Granary v2 data curation pipeline follows one sample from raw audio through the Metadata, ASR, and Text stages to a training example.

Developers can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora, or build on the Granary v2 Manifests. The manifests include the three transcript forms as tn_raw, itn_raw, and itn_clean, the labels described above, entity hints, derived instruction pairs, provenance, and a per-record quality flag. Download source audio separately under each corpus’s original license, and follow the release’s published terms.

Domain adaptation case: medical speech

A development-set prompt should specify language, script, and transcript format. In the case study below, naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%. For domain errors that remain, the NeMo Speech fine-tuning workflow pairs representative speech with explicit written targets.

This progression is shown by the NeMo Speech medical ASR case study on Eka Care’s open benchmark of recordings from Indian doctors and medical students, which has 3,619 clips and no designated training split. Adaptation data is built from generated medical text and public medicine catalogs, then combines approximately 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical recordings from MultiMed.

Figure 7. On Eka Care’s Indian medical speech benchmark full test set, a better prompt cuts WER from 30.8% to 23.8%, and fine-tuning brings it to 11.8%, about 62% fewer transcription errors than the model out of the box. The full set includes the 855 tuning clips used to choose the prompt and checkpoints; the held-out results below are the cleaner test.

Synthetic generation separates the written target from its spoken rendering: “HbA1c” is the label, while “H B A one C” is sent to the synthesizer. The audio is broadened with multiple TTS voices, while real recordings add accents and recording conditions. Training text is filtered for exact matches and shared eight-word sequences with benchmark references.

Like Granary v2’s explicit targets, training examples vary digits, unit spacing, company-name expansion, and brand-name hyphens, with instructions specifying the required convention. Some prompts include term lists with near-miss distractors, so the model learns to use a spelling only when the audio supports it. The final system uses one fixed convention prompt and one model pass, without retrieval or transcript post-processing.

LoRA adapts the language model, while the recipe trains the speech encoder and projection. Two runs use different data mixtures; after adapter merging, selected checkpoints are uniformly averaged into one final model. Checkpoint selection uses decoded development-set results, rather than training loss alone. The full adaptation, from text generation through evaluation, took about 14 GPU-hours on one NVIDIA GB300.

Domain gains, measured on held-out speakers

Benchmark splitting by speaker produces 855 tuning clips, used to choose the prompt and checkpoints, and 2,764 held-out clips, decoded once per final system. With Eka Care’s KARMA scoring pipeline, fine-tuning cuts held-out WER from 22.44% for the best-prompt baseline to 12.17%, a 45.8% relative reduction and the clearest evidence that the gains generalize to new speakers. Keyword WER falls from 9.2% to 6.5%, and semantic WER from 10.0% to 7.6%. On the same held-out speakers and scorer, the adapted model’s 12.17% WER beats an open medical-specialist model (13.79%) and a widely used general-purpose ASR model (14.92%).

For general English, LibriSpeech test-clean WER rises slightly, from 1.54% to 1.69%. Exact matches for brand-name drugs and diagnostics are still led by the medical specialist; the study points to real recordings of clinicians saying brand names as the next step. For your own domain, keep at least 100 real, labeled clips for checkpoint selection, and evaluate the entity types your application depends on alongside general-speech retention.

Commands to reproduce each step are included in the case study. The fine-tuning tutorials cover recipes, resource requirements, and tools for training, export, and evaluation, including new-language and code-switching ASR. The accompanying agent skill lets a coding agent run the workflow, from baseline selection and data checks through training validation, adapter merging, and evaluation.

For specialized multimodal applications, ModelScope also enables developers to fine-tune Nemotron 3.5 Transcribe through its ms-swift framework.

Common misconceptions, deployment risks, and next steps

Common misconceptions in open-source speech selection

• Leaderboard rank is not task coverage: language support differs by task, and audio question answering is trained in English.
• Dataset release size is not training mixture size: 463,062 hours, 48 languages, and 16 source corpora describe Granary v2’s public manifest, not Transcribe’s full training mixture or sampling weights.
• Open weights are not automatically local or cheap: quantized versions for RTX Spark, DGX Spark, and Jetson AGX Thor are coming soon, while today’s checkpoint can be served with vLLM.
• Open pipeline does not remove source license obligations: download source audio separately under each corpus’s original license.
• Full-test gains can hide tuning effects: the 855 tuning clips were used to choose prompts and checkpoints, so held-out speaker results are the cleaner comparison.
• The model is built for recorded audio, not live streaming.

Action recommendations for evaluation teams

1. Start with a prompt matrix, then test named entities, speaker changes, and answers against recordings.
2. Try the Hugging Face demo or ModelScope Studios, then download from Hugging Face or ModelScope and serve with the vLLM deployment guide.
3. If self-hosting is out of scope, evaluate hosted endpoints from Baseten, DeepInfra, or Simplismart.
4. Use the Granary v2 Manifests and medical ASR case study as the first domain adaptation path, or use the fine-tuning agent skill with a coding agent.
5. For multimodal customization on ModelScope, use ms-swift.

Additional resources include the NVIDIA Nemotron 3.5 Transcribe checkpoints and interactive demos on Hugging Face and ModelScope; the Granary v2 Manifests and High Fidelity Subset on Hugging Face and ModelScope; the NVIDIA NeMo Speech GitHub, NeMo Curator Granary v2 Pipeline, vLLM Deployment Guide, Speech Fine-Tuning Guide, and Fine-Tuning Agent Skill; partner blogs from Agora, Inflection AI, VoiceArena, Simplismart, and DeepInfra; and local AI updates for NVIDIA RTX PCs.

Written by David Nakamura for AIdataNewsGlobal’s OpenSource column. Editorial contact: editorial@aidatanews.com.