llama.cpp PR adds support for NVIDIA Nemotron-3-Puzzle-75B-A9B, a 75.3B-parameter hybrid Mamba-MoE model

Releases·Model Release

NVIDIA Nemotron Bolt 3.5 74B A9B releases Oct. 15th

NVIDIA is releasing Nemotron Bolt 3.5 74B A9B on October 15, a 74B-parameter mixture-of-experts model with 9B active parameters, packed into 24GB using 2-bit quantization with quantization-aware training. The model targets local hardware, aiming to be faster than popular dense models while smarter than similar-footprint MoE models. A smaller Bolt E4B variant fits in 4GB and follows on October 23.

NVIDIA is releasing Nemotron Bolt 3.5 74B A9B on October 15, a 74B-parameter mixture-of-experts model with 9B active parameters, packed into 24GB using 2-bit quantization with quantization-aware training. The model targets local hardware, aiming to be faster than popular dense models while smarter than similar-footprint MoE models. A smaller Bolt E4B variant fits in 4GB and follows on October 23.

Social·Community

User runs NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B on 2× RTX 3090 with full 262K context

A user successfully runs the NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B model on two RTX 3090 GPUs using a W4A16 quantized version, achieving full 262K context length and N=4 without CPU offload. The setup uses vLLM engine and a patched BAR1-P2P driver. This demonstrates that large 75B models can be run on consumer hardware with careful quantization and multi-GPU configuration.

A user successfully runs the NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B model on two RTX 3090 GPUs using a W4A16 quantized version, achieving full 262K context length and N=4 without CPU offload. The setup uses vLLM engine and a patched BAR1-P2P driver. This demonstrates that large 75B models can be run on consumer hardware with careful quantization and multi-GPU configuration.

Tencent's Hy Team released Hy3, a 295-billion-parameter Mixture-of-Experts (MoE) language model with 21B active parameters and a 3.8B MTP layer. The model outperforms similar-sized models and rivals open-source models with 2-5x parameters. It is available on Hugging Face in full (598GB) and FP8 quantized (300GB) versions.

Releases·Tool Release

Unsloth adds NVIDIA Nemotron 3.5 and Meta Muse Spark Open, text-diffusion LLM training, DeepSpeed, and coding-agent prompt caching

Unsloth shipped an update enabling local runs of NVIDIA's new Nemotron 3.5 models and Meta's new Muse Spark Open, plus training support for text-diffusion LLMs, DeepSpeed training, and improved prompt caching for coding agents. The update matters to practitioners running frontier-scale open models on local hardware, with quantized memory footprints detailed for each model.

Unsloth shipped an update enabling local runs of NVIDIA's new Nemotron 3.5 models and Meta's new Muse Spark Open, plus training support for text-diffusion LLMs, DeepSpeed training, and improved prompt caching for coding agents. The update matters to practitioners running frontier-scale open models on local hardware, with quantized memory footprints detailed for each model.

Releases·Model Release

dots3-note preview: first open-weight dots3 model lands in llama.cpp

A pull request to llama.cpp adds support for dots3-note preview, the first open-weight model in the dots3 family. It is a Mixture-of-Experts model with 280B total parameters, 16B activated, and 512K context, handling text, images, video, and audio.

A pull request to llama.cpp adds support for dots3-note preview, the first open-weight model in the dots3 family. It is a Mixture-of-Experts model with 280B total parameters, 16B activated, and 512K context, handling text, images, video, and audio.