Social·Community
Logit penalty for overthinking tokens improves Qwen3.5-4B accuracy across GGUF quantizations, Reddit test finds
A Reddit user extended a Meta arXiv paper on penalizing 'overthinking marker' tokens by testing a −2 logit bias across GGUF quantizations of Qwen3.5-4B in llama.cpp, reporting accuracy gains at every quantization with 11–19% fewer reasoning tokens. A follow-up run by Joseph Murray Adams applied the same −2 bias to Ternary Bonsai 2 27B on MATH-500 and got the opposite result: accuracy dropped from 44/50 to 43/50 and tokens rose ~3%. The technique is thus model-dependent, not a general fix.
677 engagement2 sourcesSun, Sep 27, 2026, 04:29 PM
Developments
Independent retest of the −2 logit bias on Ternary Bonsai 2 27B shows it does not replicate, scoring 43/50 vs 44/50 baseline.
Fri, Oct 2, 2026, 06:13 PM
AI Overview
- A Reddit user tested Meta's overthinking-token logit penalty on 50 random MATH-500 questions using bartowski's Qwen3.5-4B GGUF quantizations
- Accuracy rose at every quantization: BF16 74%→84%, Q8_0 76%→80%, Q4_K_M 60%→66%, Q3_K_M 52%→66%, Q2_K 12%→24%
- Reasoning tokens dropped 11.0–19.4% across quants, with BF16 down 19.4% and Q3_K_M down 17.5%
- The test applied llama.cpp --logit-bias flags to 49 tokens matching the paper's markers, including "wait", "hmm", "alternatively", and "doubt"
Entities
Related
Social·Community
Ternary Bonsai 2 27B (1.75bpw) vs. Gemma 26B-A4B MoE
A developer benchmarked prism-ml's Ternary-Bonsai-2-27B quantized model (1.75 bpw, 5.95 GB) against Gemma 4 26B-A4B on an 8 GB Turing GPU for dialogue speaker attribution in an audiobook pipeline. Despite fitting a 27B-class model fully in VRAM, Bonsai scored 93.3% attribution accuracy but ran 5-10x slower and showed systematic narrator and intonation errors; the tester declined to adopt it.
A developer benchmarked prism-ml's Ternary-Bonsai-2-27B quantized model (1.75 bpw, 5.95 GB) against Gemma 4 26B-A4B on an 8 GB Turing GPU for dialogue speaker attribution in an audiobook pipeline. Despite fitting a 27B-class model fully in VRAM, Bonsai scored 93.3% attribution accuracy but ran 5-10x slower and showed systematic narrator and intonation errors; the tester declined to adopt it.
25 engagement1 sourceredditSat, Sep 19, 2026, 01:54 AM
Social·Community
User experiments with Sol to recover low-bit quant damage in Qwen3.5-4B
A user ran experiments using Sol to recover quality lost in low-bit quantization of Qwen3.5-4B, finding that targeted adjustments to four sensitive tensors recover 3-5% of the KLD damage. The results are preliminary but suggest some quant damage is recoverable with cheap tricks.
A user ran experiments using Sol to recover quality lost in low-bit quantization of Qwen3.5-4B, finding that targeted adjustments to four sensitive tensors recover 3-5% of the KLD damage. The results are preliminary but suggest some quant damage is recoverable with cheap tricks.
17 engagement1 sourceredditTue, Aug 11, 2026, 01:21 PM
Social·Community
Users report where quantized local LLMs break: coding and reasoning degrade first
A Reddit discussion asks practitioners where quantized local LLMs fail, with early responses indicating coding and reasoning tasks expose degradation first. Users report switching back to higher-precision models when quantized versions fail at specific tasks.
A Reddit discussion asks practitioners where quantized local LLMs fail, with early responses indicating coding and reasoning tasks expose degradation first. Users report switching back to higher-precision models when quantized versions fail at specific tasks.
42 engagement1 sourceredditTue, Aug 11, 2026, 04:25 PM
Social·Community
Users report Qwen 3.8 27b performs well even at Q3_xxs quantization
Multiple users report that Qwen 3.8 27b performs strongly even at aggressive quantizations like Q3_xxs, handling complex coding tasks in one shot. This is notable because Q3 quants are typically avoided due to quality degradation, but the model's performance at this level is surprisingly good, making it viable for low-VRAM setups.
Multiple users report that Qwen 3.8 27b performs strongly even at aggressive quantizations like Q3_xxs, handling complex coding tasks in one shot. This is notable because Q3 quants are typically avoided due to quality degradation, but the model's performance at this level is surprisingly good, making it viable for low-VRAM setups.
400348 engagement4 sourcesredditFri, Aug 21, 2026, 01:43 PM
Releases·Tool Release
AgentionAI releases Qwen3.8-Flash-Next AP GGUF quants benchmarked against other high-quality quants
Agention AI released high-precision AP quantized GGUFs of Qwen3.8 Flash on Hugging Face, reporting benchmark results beating other high-quality quants. The team developed a modified KLD measurement with a new dataset because the model's n-gram memorization of Wikipedia skewed standard evaluation. The quants balance precision with prefill performance.
Agention AI released high-precision AP quantized GGUFs of Qwen3.8 Flash on Hugging Face, reporting benchmark results beating other high-quality quants. The team developed a modified KLD measurement with a new dataset because the model's n-gram memorization of Wikipedia skewed standard evaluation. The quants balance precision with prefill performance.
2 engagement1 sourceredditWed, Sep 2, 2026, 10:10 PM