Releases·Tool Release
Dev releases MegaCapybara, an RTX5090 inference engine for Qwen3.8 27B claiming 2x Ninfer decode speed
A developer announced MegaCapybara, a purpose-built inference engine for the RTX 5090 focused on Qwen3.8 27B, with weights on Hugging Face and code on GitHub. It claims roughly twice the decode speed of ninfer, the prior SOTA engine for that GPU, and adds agent-serving, cache management, and benchmark-verified weight metadata. Source code and a weights builder will follow later.
62 engagement1 sourceSat, Oct 3, 2026, 01:33 AM
AI Overview
- MegaCapybara (MIT-licensed, GitHub) targets Qwen3.8 27B on RTX5090, claiming roughly 2x Ninfer decode speed: 500+ t/s single-stream coding, 2600 t/s bursts across 12 slots
- Custom kernels switch dynamically between single/multi and short/long-context workloads; Dflash2, confidence scheduling, and draft trees run together
- Weights ship with metadata benchmarked against BF16 originals, showing KL and top-1% loss per change (4-bit KV, YARN); uncensored and standard versions included
- Agent serving prioritizes throughput across up to 12+ jobs, with unified-context swapping jobs to RAM and resuming in 0.03s without re-prefill
- Two-tier loop guard adjusts repetition penalty dynamically, then fires a stop signal so frontends can recover from agent loops
- Source code and a weights builder for any Qwen3.8 27B BF16 checkpoint are promised for later release; launcher includes autodownloader and interactive docs
Related
Releases·Tool Release
akopytko releases Blackwell-native NVFP4 quant of Qwen3.8 27B, 50% faster than Q4
A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.
A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.
166 engagement1 sourceredditFri, Aug 21, 2026, 09:19 AM
Social·Community
Abliterlitics benchmarks 11 uncensored Gemma 4 12B variants against base
Abliterlitics ran a systematic comparison of 11 uncensored Gemma 4 12B variants from Hugging Face against the official base model, using 165 GPU hours on a single RTX 5090. The evaluation covered weight forensics, KL divergence, 13 benchmark tasks, and HarmBench with 400 behaviors, with every response reviewed by an LLM judge.
Abliterlitics ran a systematic comparison of 11 uncensored Gemma 4 12B variants from Hugging Face against the official base model, using 165 GPU hours on a single RTX 5090. The evaluation covered weight forensics, KL divergence, 13 benchmark tasks, and HarmBench with 400 behaviors, with every response reviewed by an LLM judge.
69 engagement1 sourceredditTue, Aug 25, 2026, 02:39 PM
Social·Communityfeatured
Local LLM inference: Qwen3.8 on 5090s, 7900 XTX at 262k context, dual R9700s, 4x RTX 3060, and Atlas 300I Duo cards
A wave of r/LocalLLaMA posts reports working local deployments of Qwen3.8-27B and Qwen3.8-Flash-Next on hardware from an iPhone-tethered MacBook to Huawei Ascend cards and an eBay 8x V100 server. Common levers are MTP/speculative decoding, low-bit quants, and runtime forks (llama.cpp, vLLM, exllamav3, Ninfer). Together they map where local, full-context agentic inference is actually usable.
A wave of r/LocalLLaMA posts reports working local deployments of Qwen3.8-27B and Qwen3.8-Flash-Next on hardware from an iPhone-tethered MacBook to Huawei Ascend cards and an eBay 8x V100 server. Common levers are MTP/speculative decoding, low-bit quants, and runtime forks (llama.cpp, vLLM, exllamav3, Ninfer). Together they map where local, full-context agentic inference is actually usable.
Updated Fri, Oct 2, 2026, 07:19 PM — iPhone 17 Pro Max offloads layers, KV cache, and Neural Engine decode from a MacBook, speeding Qwen3.8 27B prefill 29-44
542206.2k engagement87 sourcesreddit, rssFri, Oct 2, 2026, 12:00 AM
Developments·Benchmark
2.3x faster Qwen3.8 27B on a 5090: ninfer vs llama.cpp, 4 setups, same prompt - speed and quality tested
A Reddit user benchmarked ninfer against llama.cpp running Qwen3.8 27B on an RTX 5090, generating a billiards game from one identical prompt across four setups. Stock llama.cpp managed ~68 tokens/s while ninfer hit ~147, but enabling MTP in llama.cpp raised it to ~141, suggesting much of ninfer's speed advantage comes from MTP rather than the runtime itself. Quality checks found only the NVFP4 build produced a legally winnable game.
A Reddit user benchmarked ninfer against llama.cpp running Qwen3.8 27B on an RTX 5090, generating a billiards game from one identical prompt across four setups. Stock llama.cpp managed ~68 tokens/s while ninfer hit ~147, but enabling MTP in llama.cpp raised it to ~141, suggesting much of ninfer's speed advantage comes from MTP rather than the runtime itself. Quality checks found only the NVFP4 build produced a legally winnable game.
7 engagement1 sourceredditSat, Oct 3, 2026, 09:11 AM
Social·Communityfeatured
Users report ~2x decode speedup with inco.ai DFlash2 speculative decoding on Qwen3.8 27B
Multiple users tested inco.ai's DFlash2 draft model for speculative decoding with Qwen3.8 27B on RTX 5090 and dual RTX 3090 setups, reporting roughly 2x decode speedups. The technique requires rebuilding llama.cpp with PR #27342 and using a separate draft GGUF, with vLLM integration also demonstrated.
Multiple users tested inco.ai's DFlash2 draft model for speculative decoding with Qwen3.8 27B on RTX 5090 and dual RTX 3090 setups, reporting roughly 2x decode speedups. The technique requires rebuilding llama.cpp with PR #27342 and using a separate draft GGUF, with vLLM integration also demonstrated.
441976 engagement9 sourcesredditTue, Aug 18, 2026, 10:26 PM