Two ~300B MoE models, each on ONE 128 GB mini PC (AMD Strix Halo): GLM-5.3-Flash at ~580 tok/s prefill, MiMo-V2.6-Flash up to 44 tok/s decode. EXL3 weights + open ROCm engine

Releases·Model Release

Yamz Labs runs Qwen3.8-Flash-Next (125B MoE) on a single Strix Halo mini PC at 44-59 tok/s, open-weights and open engine

Yamz Labs released 95 GB EXL3 quantized weights for Qwen3.8-Flash-Next, a 125B MoE model with 6B active parameters, plus a new open version of its Kyojin inference engine built on ExLlamaV3. On a single AMD Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB), it runs at 44-59 tok/s with speculative decoding and ~1,400 tok/s prefill. Practitioners get a full local inference stack for a large open model on consumer hardware, though Halogen 0.16.2 currently outperforms it on speed.

Yamz Labs released 95 GB EXL3 quantized weights for Qwen3.8-Flash-Next, a 125B MoE model with 6B active parameters, plus a new open version of its Kyojin inference engine built on ExLlamaV3. On a single AMD Strix Halo mini PC (Ryzen AI Max+ 395, 128 GB), it runs at 44-59 tok/s with speculative decoding and ~1,400 tok/s prefill. Practitioners get a full local inference stack for a large open model on consumer hardware, though Halogen 0.16.2 currently outperforms it on speed.

Releases·Tool Release

ExLlamaV3 adds CPU offload of MoE experts, GLM-5.3-Flash and Qwen3.8-Flash-Next quants

Turboderp shipped a batch of ExLlamaV3 updates including CPU offload of MoE experts, ngram disk offload for Qwen3.8-Flash-Next, a GLM-5.3-Flash exl3 quant, and a new self-calibrated optimization technique for quantization. The releases target NVIDIA GPU users running large models locally and expand what fits in limited VRAM.

Turboderp shipped a batch of ExLlamaV3 updates including CPU offload of MoE experts, ngram disk offload for Qwen3.8-Flash-Next, a GLM-5.3-Flash exl3 quant, and a new self-calibrated optimization technique for quantization. The releases target NVIDIA GPU users running large models locally and expand what fits in limited VRAM.

Social·Community

MiMo-V2.6-Flash on vLLM: fixes for "empty responses" with thinking + tools, and a hidden 2,048-token output cap

A developer running MiMo-V2.6-Flash-RL on 2× DGX Spark with vLLM found that reported tool failures trace to serving bugs, not the model. Fixes cover empty streaming responses with thinking on, dropped reasoning in tool loops, and a hidden 2,048-token output cap. After patching, multi-step agentic tool tasks with thinking stopped failing.

A developer running MiMo-V2.6-Flash-RL on 2× DGX Spark with vLLM found that reported tool failures trace to serving bugs, not the model. Fixes cover empty streaming responses with thinking on, dropped reasoning in tool loops, and a hidden 2,048-token output cap. After patching, multi-step agentic tool tasks with thinking stopped failing.

Releases·Tool Release

EXL3-QLoRA: Unofficial ExLlamaV3 fork trains QLoRA on 2-8bpw EXL3 quants, beating bnb NF4 on eval loss

A developer known as UnstableLlama released exl3-qlora, a fork of ExLlamaV3 implementing native QLoRA-style training on 2-8bpw EXL3 quantized models in plain PyTorch. It breaks the previous BitsAndBytes 4-bit lock-in, supports SFT/DPO/KTO across major model families, and reports lower eval loss than bnb NF4 at comparable VRAM. The tool is an unofficial community project already used to produce released models.

A developer known as UnstableLlama released exl3-qlora, a fork of ExLlamaV3 implementing native QLoRA-style training on 2-8bpw EXL3 quantized models in plain PyTorch. It breaks the previous BitsAndBytes 4-bit lock-in, supports SFT/DPO/KTO across major model families, and reports lower eval loss than bnb NF4 at comparable VRAM. The tool is an unofficial community project already used to produce released models.

Products·building

Overspill: disk tier for FreeToken to run oversized MoE models on consumer hardware

Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.

Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.