Nemotron 3.5 Transcribe VRAM requirements: GPU and Mac fit

Nemotron 3.5 Transcribe is NVIDIA's open-weight speech model for recorded audio: transcription in 40 languages, speaker labels, translation and audio Q&A from one checkpoint, for recordings up to one hour.^[2][12] NVIDIA published it on Hugging Face on October 28, 2026 under the OpenMDW-1.1 license, and Hugging Face lists 33,697,781,029 parameters.^[1][12] The model card describes it as 34B total / 4.3B active, with a 65,536-token context.^[2]

"4B active" is not "4B of VRAM." It is a mixture-of-experts model: each token only uses about 4.3B parameters, but all 34B have to be loaded. That is why the weights are 67.5 GB in BF16, not 8 GB.^[1][2] It is also a different model from NVIDIA's Nemotron 3.5 ASR Streaming 0.6B, the small real-time model many "run it on a laptop" guides cover.^[22]

Memory needed at each quantization

NVIDIA ships BF16 plus FP8 (about 35 GB) and NVFP4 (about 21 GB) checkpoints for vLLM/SGLang.^[2][4] On October 28 the FP8 and NVFP4 repos linked from the model card did not open publicly for us, so they are not in the table. Everything below is an exact file size from Hugging Face (1 GB = 1,000,000,000 bytes). GGUF rows are AtomicChat's builds for NeMo-Speech.cpp and include the 1.67 GB encoder file every decoder needs.^[5] MLX rows are complete Mac folders.^[8][11]

How much accuracy do the small files lose? AtomicChat measured every decoder from 15.55 GB up to within ±0.18 points of BF16 word error on its English, German, French and Spanish sets. The 13.57 GB AD-Q2_0_M was also within noise there, but +1.5 points character error on Chinese.^[5] For NVIDIA's own checkpoints, the technical report averages a 1.0% relative error increase for FP8 and 3.6% for NVFP4 across 77 test sets.^[4]

Context and audio length add memory

Only 6 of the decoder's 52 layers are attention layers with a growing KV cache. The 23 Mamba-2 layers keep a fixed state instead.^[3] With 2 KV heads and a head size of 128, a BF16 cache takes 2 × 6 × 2 × 128 × 2 bytes = 6,144 bytes per token:

Audio costs 12.5 tokens per second, plus up to 6 per second of transcript. NeMo-Speech.cpp's default 73,728-token limit is what a full hour needs.^[6] Two smaller extras come on top: about 134 KB of encoder output per second of audio (about 0.48 GB for an hour, our multiplication) and about 48 MiB of Mamba state per request.^[6] The KV cache is tiny. The weights decide whether it fits.

Which GPUs and Macs it fits on

Each cell compares one request with a full hour of audio against the device's memory, using the GGUF builds (NeMo-Speech.cpp runs them on CUDA, Metal, Vulkan and CPU).^{[13][14][15][16][17][18][19][20]}

Column headers show one request with a full hour of audio: model files (GGUF decoder + 1.67 GB encoder file) + 0.40 GB BF16 KV cache at 65,536 tokens + about 0.48 GB of encoder output for 3,600 s of audio (134 KB per second, per NeMo-Speech.cpp) + about 48 MiB Mamba state. Engine workspace, the OS/display and batching are not included. GGUF runs through NeMo-Speech.cpp (CUDA, Metal, Vulkan, CPU). Mac MLX builds are covered in the Mac section below.

How to read it: "Fits" means under 90% of memory, "Tight" means 90–100%, and "No" means it doesn't fit on the device alone. The 90% line is our own rule of thumb. Shorter recordings need less: by NVIDIA's settings, 24 GB handles about 29 minutes, not the full hour.^[6] On a Mac, macOS also needs part of the unified memory, so treat Mac rows as optimistic.

NVIDIA's own settings per GPU size

NeMo-Speech.cpp's configuration guide gives these settings (our table above is a cross-check, not a replacement):^[6]

Can an RTX 4090 or RTX 3090 run it?

Yes, with the GGUF build. The official vLLM checkpoints are a different story. NVIDIA's vLLM table lists no 24 GB consumer card: BF16 targets 80 GB parts, FP8 starts at the 48 GB L40S, and NVFP4 needs Blackwell (RTX 5090, RTX Spark).^[2][4] NeMo-Speech.cpp's batched CUDA path runs on any NVIDIA GPU of compute capability 8.0 or newer, which covers RTX 30- and 40-series cards.^[7] Use AD-IQ4_NL (19.68 GB decoder + 1.67 GB encoder) with NVIDIA's 24 GB settings above. That gives about 29 minutes per request, so split longer recordings.^[5][6]

RTX 5090 (32 GB)

This is the one consumer card NVIDIA lists. It runs the NVFP4 checkpoint in vLLM, and in NeMo-Speech.cpp the NVFP4 decoder fits 32 GB at default settings with a full hour per request.^[2][6] AtomicChat's AD-NVFP4 is 19.68 GB, smaller than the 21.79 GB that NVIDIA's converter produces with default settings.^[5]

16 GB GPUs (RTX 4060 Ti 16GB, RTX 5080)

Only for short files. AtomicChat's 13.57 GB AD-Q2_0_M decoder peaked at 15.6 GiB on a 3.7-minute file with max_context 8192, about 7 minutes of audio per request. That was measured on an A100, not on a real 16 GB card, and the packager calls it tight.^[5] Hour-long recordings will not fit in 16 GB.

Can a Mac run it?

Yes. You may see answers claiming it won't run on Apple Silicon, but community MLX ports were up on launch day:

• nativ-community MLX 4-bit: 19.26 GB of files, run with mlx-audio. Neywa Labs measured a 20.37 GB peak MLX allocation, and 53.8 s of audio in 1.59 s (33.8× real time) on an M3 Ultra. BF16 peaked at 65.62 GB.^[8][9]
• mere.run (Sawfwair) MLX 4-bit: 25.09 GB download. mere.run suggests 32 GB minimum / 48 GB recommended (96 / 128 GB for BF16). It measured a 21.83 GiB process footprint, against 61.74 GiB for BF16.^[10][11]
• GGUF on Metal: NeMo-Speech.cpp's Metal path uses the IQ4_NL decoder. AtomicChat's packager suggests AD-Q2_0_L (17.2 GB with the encoder) for 24 GB Macs.^[5][7]

So 32 GB is the comfortable floor and 24 GB is a squeeze. One third-party post also shows a Core ML port on an M5 Pro.^[23]

How to run it

• NVIDIA GPU, 24–32 GB: build NeMo-Speech.cpp and download a decoder plus the Encoder-F16 file from AtomicChat's GGUF repo into one folder. Then run nemo-speech transcribe audio.wav --model <decoder>.gguf. Use --diarize for speaker labels and --translate-to de for translation.^[5]
• 80 GB datacenter GPU: NVIDIA's quick start serves BF16 on one H100 80 GB with vLLM 0.32.0 and NeMo Speech. SGLang-Omni 0.2.0 also works.^[2]
• Mac: mlx-audio with the nativ-community 4-bit folder, or mere.run speech transcribe --backend nemotron.^[9][10]

It is a batch model for recorded files. Even when text streams out, it encodes the whole recording before it starts writing, so it is not a live dictation engine.^[9][12]

Cheapest GPU to rent for Nemotron 3.5 Transcribe

A 32 GB RTX 5090 handles full-hour files. A 24 GB card is fine for recordings up to about half an hour. On-demand Secure Cloud prices listed on Runpod's pricing page:^[21]

Prices checked on 2026-10-28. They change often, so confirm on the provider's page. Runpod's page ("Updated October 8, 2026") was re-checked on October 28, 2026; these rates were unchanged. A 24 GB card runs the IQ4_NL build for recordings up to about half an hour with NVIDIA's settings; 32 GB (RTX 5090) holds a full hour. Rent on Runpod Referral link

See all GPU rental prices for more providers. A marketplace option is Vast.ai (affiliate link).

Frequently asked questions

How much VRAM does Nemotron 3.5 Transcribe need?

About 21–22 GB for one hour-long recording with the 4-bit GGUF (AD-IQ4_NL plus encoder). The official checkpoints are 67.5 GB (BF16), about 35 GB (FP8) and about 21 GB (NVFP4) before runtime memory.^[1][4][5]

Can I run Nemotron 3.5 Transcribe on an RTX 4090 or 3090?

Yes, through NeMo-Speech.cpp with the IQ4_NL GGUF. NVIDIA's 24 GB settings allow recordings up to about 29 minutes per request.^[6][7] The official FP8/NVFP4 vLLM checkpoints don't target these cards.^[2]

Will it run on an RTX 5090?

Yes. It is the consumer card NVIDIA lists for NVFP4, and the NVFP4 GGUF fits 32 GB at default settings with a full hour of audio.^[2][6]

Can it run on a 16 GB GPU?

Only short files. The 13.57 GB 2-bit decoder peaked at 15.6 GiB on a 3.7-minute file in the packager's test.^[5]

Does Nemotron 3.5 Transcribe run on a Mac?

Yes, via community MLX ports (18.3–25.1 GB). Measured peaks were about 20–22 GB at 4-bit, so a 32 GB Mac is comfortable.^[8][9][10]

Why does a "4B active" model need so much memory?

It is a 34B mixture-of-experts model, and every expert must be in memory even though only about 4.3B parameters work per token.^[2]

How we calculated

• Weights are exact Hugging Face file sizes. GGUF rows add the 1.67 GB encoder file.^{[1][5][8][11]}
• KV cache comes from llm_backbone/config.json: 6 attention layers, 2 KV heads, head size 128, BF16.^[3]
• Audio overhead uses NeMo-Speech.cpp's published 134 KB per second of encoder output and 48 MiB of Mamba state per request. The one-hour figures are our own multiplication.^[6]
• Device memory is from NVIDIA and Apple spec pages, compared in decimal GB.
• Not included: engine workspace, the DFlash2 draft model (1.15 GB), batching several requests, and the OS or display.

Sources

All sources checked on 2026-10-28.

1. Hugging Face, nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B (parameter count, 67,496,013,734-byte model.safetensors, OpenMDW-1.1; checked October 28, 2026). https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
2. NVIDIA model card (34B / 4.3B active, 65,536 context, H100 quick start, hardware compatibility table). https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B/blob/main/README.md
3. llm_backbone/config.json (layer types, KV heads, head size). https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B/blob/main/llm_backbone/config.json
4. NVIDIA, Nemotron 3.5 Transcribe Technical Report (FP8 about 35 GB, NVFP4 about 21 GB, quantization error, supported GPUs). https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf
5. Hugging Face, AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF (file sizes, accuracy against BF16, 16 GB settings; checked October 28, 2026). https://huggingface.co/AtomicChat/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-GGUF
6. NVIDIA NeMo-Speech.cpp, SALM configuration (GPU memory table, tokens per second of audio, encoder output, Mamba state). https://github.com/NVIDIA/NeMo-Speech.cpp/blob/main/docs/salm/configuration.md
7. NVIDIA NeMo-Speech.cpp, SALM models and platforms (compute capability 8.0+, CPU/Vulkan/Metal path). https://github.com/NVIDIA/NeMo-Speech.cpp/blob/main/docs/salm/models.md
8. Hugging Face, nativ-community Nemotron-3.5-Transcribe MLX builds (4-bit, MXFP4, 8-bit file sizes; checked October 28, 2026). https://huggingface.co/nativ-community/Nemotron-3.5-Transcribe-30B-A4B-EA-4bit
9. Neywa Labs, "Nemotron Transcribe on Nativ" (M3 Ultra peak memory and speed, measured October 26, 2026; published October 28, 2026). https://www.neywalabs.ai/blog/nemotron-transcribe-nativ
10. mere.run docs, Nemotron 3.5 Transcribe (download sizes, memory budgets, M5 Max footprint). https://docs.mere.run/runtime/nemotron-transcribe
11. Hugging Face, Sawfwair/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-MLX-4bit (file sizes). https://huggingface.co/Sawfwair/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-MLX-4bit
12. Hugging Face blog, NVIDIA Nemotron 3.5 Transcribe and Granary v2 (published October 28, 2026). https://huggingface.co/blog/nvidia/nemotron-transcribe-granary
13. NVIDIA, GeForce RTX 4060 Ti (16 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4060-4060ti/
14. NVIDIA, GeForce RTX 5080 (16 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5080/
15. NVIDIA, GeForce RTX 3090 specs (24 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/
16. NVIDIA, GeForce RTX 4090 specs (24 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4090/
17. NVIDIA, GeForce RTX 5090 specs (32 GB). https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
18. NVIDIA, RTX 6000 Ada Generation (48 GB). https://www.nvidia.com/en-us/products/workstations/rtx-6000/
19. NVIDIA, DGX Spark (128 GB unified memory). https://www.nvidia.com/en-us/products/workstations/dgx-spark/
20. Apple, MacBook Pro tech specs (unified memory options). https://www.apple.com/macbook-pro/specs/
21. Runpod, GPU pricing (page "Updated October 8, 2026"; re-checked October 28, 2026). https://www.runpod.io/pricing
22. Hugging Face, nvidia/nemotron-3.5-asr-streaming-0.6b (the separate 0.6B streaming model). https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
23. @Alex_tra_memory on X, Core ML port on M5 Pro (third-party, October 28, 2026). https://x.com/Alex_tra_memory/status/2115468824392704337