Releases·Tool Release
Dev builds Ninfer 4080 to run 27B Qwen GSQ quant at 100k context on 16GB RTX 4080
A software engineer released Ninfer 4080 on GitHub, a fork of the community Ninfer CUDA runtimes tuned for RTX 4080 16GB cards, running ISTA-DASLab-Qwen-3.8-27B-GSQ with DFlash2 speculative decoding at up to 100k context. It reaches 2720 tok/s prefill and up to 262 tok/s decode, far beyond general-purpose engines like llama.cpp or vllm on the same hardware. The work matters to local-LLM practitioners because it shows how much performance specialized per-GPU runtimes can unlock over general inference engines.
15 engagement1 sourceSat, Oct 3, 2026, 06:58 PM
AI Overview
- Reddit user roofkid forks NInfer for RTX 4080 16GB, targeting ISTA-DASLab-Qwen-3.8-27B-GSQ with DFlash2 speculative decoding at 100k+ context
- NInfer 4080 hits 2720 tok/s prefill at 8K and 262 tok/s DFlash2 K=7 decode at 32K, vs MTP3 decode of 151 tok/s at 8K
- Benchmarks with MBPP (90-92%) and HumanEval (95-96%) show accuracy degradations only within measurement noise, attributed to KV quantization (rk4v4-e8)
- Author spent $13 on DeepSeek V4.1 Flash for ~2 billion tokens of agentic coding work, citing low cache-read pricing
- Full 256k F16 context on Qwen 3.8 27B needs 16GB VRAM; 100k context achieved via compressed KV cache
- Runtime released on GitHub as a Docker image, forked from NInfer-4090 by sergiuszm
Related
Social·Community
Reddit users weigh 16GB VRAM local models: DASLab GSQ-RCO, Swift, Qwen 3.8 27B, and new inference servers strata/freetoken
A Reddit user asks which model and harness currently performs best on 16GB VRAM, noting three optimization vectors: quantized model variants, inference optimizations, and new inference servers. They suggest DASLab's GSQ-RCO was among the first models to run well on 16GB and that strata and freetoken revealed idle GPU time in llama.cpp.
A Reddit user asks which model and harness currently performs best on 16GB VRAM, noting three optimization vectors: quantized model variants, inference optimizations, and new inference servers. They suggest DASLab's GSQ-RCO was among the first models to run well on 16GB and that strata and freetoken revealed idle GPU time in llama.cpp.
770 engagement2 sourcesredditSun, Oct 11, 2026, 06:40 PM
Social·Community
User tests Qwen3.8-27B with DFlash2 and MTP on 16GB VRAM for long context
A user on Reddit reports testing Qwen3.8-27B GGUF quantizations on a 16GB RTX 4080, aiming to maximize context and performance for a local Hermes agent. They built llama.cpp with the DFlash2 branch and used ngram-mod to achieve a 131072 context, sharing practical results for running large models on consumer hardware.
A user on Reddit reports testing Qwen3.8-27B GGUF quantizations on a 16GB RTX 4080, aiming to maximize context and performance for a local Hermes agent. They built llama.cpp with the DFlash2 branch and used ngram-mod to achieve a 131072 context, sharing practical results for running large models on consumer hardware.
216 engagement2 sourcesredditWed, Aug 19, 2026, 11:53 PM
Social·Community
Qwen 3.8 27B hits ~96t/s decode with 110k context on a 16GB RTX 5080 via NInfer
A r/LocalLLaMA user reports running Qwen 3.8 27B locally at 90–110 t/s decode on a 16GB RTX 5080 using NInfer v1.5, up from ~75 t/s in their earlier llama.cpp setup. Across 32 Zoo Code coding-session requests the median decode was 96.45 t/s with 110,592-token context allocated and working prompt caching. The results show 27B-class models becoming practical for agentic coding on consumer 16GB GPUs.
A r/LocalLLaMA user reports running Qwen 3.8 27B locally at 90–110 t/s decode on a 16GB RTX 5080 using NInfer v1.5, up from ~75 t/s in their earlier llama.cpp setup. Across 32 Zoo Code coding-session requests the median decode was 96.45 t/s with 110,592-token context allocated and working prompt caching. The results show 27B-class models becoming practical for agentic coding on consumer 16GB GPUs.
9 engagement1 sourceredditTue, Oct 6, 2026, 05:31 PM
Releases·Tool Release
NInfer port hits 45 tok/s Qwen3.8-27B on modded RTX 2080 Ti
A developer ported NInfer, a from-scratch C++/CUDA inference engine originally for Blackwell, to Turing GPUs, achieving 45 tok/s on a 22GB modded RTX 2080 Ti with Qwen3.8-27B using speculative decoding. The port targets sm_75 and is incompatible with unmodded 11GB cards.
A developer ported NInfer, a from-scratch C++/CUDA inference engine originally for Blackwell, to Turing GPUs, achieving 45 tok/s on a 22GB modded RTX 2080 Ti with Qwen3.8-27B using speculative decoding. The port targets sm_75 and is incompatible with unmodded 11GB cards.
21 engagement1 sourceredditFri, Aug 21, 2026, 06:16 AM
Releases·Tool Release
DFlash 2 released for Qwen 3.8 27B and Muse Glimmer with llama.cpp PR
The original authors of DFlash have released a second version, DFlash 2, with GGUF quants for Qwen 3.8 27B and Muse Glimmer. An accompanying llama.cpp pull request (PR #27342) suggests integration into the main inference stack. This update likely brings performance or quality improvements for local inference of these models.
The original authors of DFlash have released a second version, DFlash 2, with GGUF quants for Qwen 3.8 27B and Muse Glimmer. An accompanying llama.cpp pull request (PR #27342) suggests integration into the main inference stack. This update likely brings performance or quality improvements for local inference of these models.
1.4k engagement2 sourcesreddit, xTue, Aug 18, 2026, 09:36 PM