A developer released a training method that applies LoRA directly over GGUF-quantized models, enabling local QLoRA-style fine-tuning of MoE models that bnb 4-bit does not support. Using GGUF support added in Transformers 5.18, they trained Qwen3.8-Flash-Next (125B-A6B + 51B engram) in 40 GiB VRAM without CPU offload, reaching 200 token/s on Strix Halo. The work opens a path for modifying open-weight MoE models on local hardware, complementing inference advances already achieved with GGUF.
woct0rdho updated the LoRA-over-GGUF training recipe so Qwen3.8-Flash-Next (125B-A6B plus a 51B engram) can be LoRA-trained in 40 GiB VRAM with no CPU offloading, with the engram kept on disk without slowing training. On Strix Halo it hits 9.5 s/it at 2048-token context chunks (200 token/s). The recipe builds on transformers 5.18's newly merged support for modern GGUF, and the author teases GGTensile, an asm-level MMQ kernel optimization that already beats HIP in many cases.
woct0rdho updated the LoRA-over-GGUF training recipe so Qwen3.8-Flash-Next (125B-A6B plus a 51B engram) can be LoRA-trained in 40 GiB VRAM with no CPU offloading, with the engram kept on disk without slowing training. On Strix Halo it hits 9.5 s/it at 2048-token context chunks (200 token/s). The recipe builds on transformers 5.18's newly merged support for modern GGUF, and the author teases GGTensile, an asm-level MMQ kernel optimization that already beats HIP in many cases.
A new recipe enables LoRA training of Qwen3.6-35B-A3B, a 35B MoE model, in just 16 GiB VRAM by using GGUF as the base format with APEX quantization and fused kernels. The approach achieves 6.5 s/i on AMD Strix Halo hardware, replacing bitsandbytes for low-VRAM fine-tuning.
A new recipe enables LoRA training of Qwen3.6-35B-A3B, a 35B MoE model, in just 16 GiB VRAM by using GGUF as the base format with APEX quantization and fused kernels. The approach achieves 6.5 s/i on AMD Strix Halo hardware, replacing bitsandbytes for low-VRAM fine-tuning.
Qwen released Qwen3.8-Flash-Next, an open-weight hybrid model that previews the Qwen 4 architecture. The 125B MoE model reportedly outperforms DeepSeek V4 Flash at half the parameters and is stronger than Opus 4.6. Its use of n-gram tables could enable running 1T+ parameter models on a single server with modest GPUs and system RAM.
Qwen released Qwen3.8-Flash-Next, an open-weight hybrid model that previews the Qwen 4 architecture. The 125B MoE model reportedly outperforms DeepSeek V4 Flash at half the parameters and is stronger than Opus 4.6. Its use of n-gram tables could enable running 1T+ parameter models on a single server with modest GPUs and system RAM.
A community quantizer released mainline-compatible imatrix GGUF quants for Qwen3.8-Flash-Next on Hugging Face after several days of testing. The builds target the same quality band as popular Unsloth and AesSedai Q4 quants while saving roughly 20-30GB of disk and RAM, with perplexity competitive with both. This gives local inference users smaller files without a quality tradeoff.
A community quantizer released mainline-compatible imatrix GGUF quants for Qwen3.8-Flash-Next on Hugging Face after several days of testing. The builds target the same quality band as popular Unsloth and AesSedai Q4 quants while saving roughly 20-30GB of disk and RAM, with perplexity competitive with both. This gives local inference users smaller files without a quality tradeoff.
Unsloth added local-run support for Qwen3.8-Flash-Next and GLM-5.3-Flash with GGUF weights on Hugging Face, then shipped follow-ups enabling multi-token prediction for up to 2x faster inference and a major performance/bugfix release. The latest release switches AMD to Vulkan by default for a 20% prefill/decoding boost over ROCm and includes 200+ bug fixes. Practitioners gain faster local GGUF inference, one-click Hermes support, and MLX fine-tuning on Apple Silicon.
Unsloth added local-run support for Qwen3.8-Flash-Next and GLM-5.3-Flash with GGUF weights on Hugging Face, then shipped follow-ups enabling multi-token prediction for up to 2x faster inference and a major performance/bugfix release. The latest release switches AMD to Vulkan by default for a 20% prefill/decoding boost over ROCm and includes 200+ bug fixes. Practitioners gain faster local GGUF inference, one-click Hermes support, and MLX fine-tuning on Apple Silicon.