Releases·Tool Release
NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support
A developer released a fork of the NInfer inference engine, adding a from-scratch 4-bit NVFP4 KV cache, YARN long-context support, reliable KV host caching, monitoring, and Jinja support, aimed at real-world use beyond Qwen. They stress-tested it with three concurrent Claude Code sessions for a week. The fork's custom QK matmul kernel differs architecturally from upstream's later NVFP4 path.
49 engagement1 sourceSat, Sep 5, 2026, 10:58 PM
AI Overview
- Fork author implements custom NVFP4 KV cache for QIn3.8-27B: 144 bytes/token/KV-head, 45% VRAM reduction, LongBench matching int8 and 100% needle-in-haystack.
- YaRN extension reaches 555k (multimodal) or 600k (text) context on a 5090 from native 262k, with 8M projected on 96GB+ GPUs.
- HostKVSafetyNet replaces upstream LRU HostKvProvider: two-level prefix matching, session-key fallback, zero re-prefills across 260+ requests at 330k-470k context.
- At 400k+ context with 3 concurrent sessions, fork hits 117 tok/s decode (92% MTP acceptance), 2-16s cached turn turnaround, and 1600 tok/s cold prefill.
- Tolerant tool calling recovers malformed Qwen calls, accepts text/reasoning after tool calls, and adds a Froggeric v22 C++ chat template renderer.
- Fork adds dynamic chat templates, weights profile overrides, OOM recovery, a monitoring dashboard with live KV occupancy, and removes model hash checks.
- E4M3 block-scale MMA kernel quantizes Q/K to NVFP4 with Hadamard rotation for outlier suppression; V dequantizes to BF16 via fused decode kernel.
Related
Releases·Tool Releasefeatured
Developer open-sources NInfer engine achieving 543 tok/s on Qwen3.6-35B-A3B with single RTX 5090
A developer released NInfer, a custom C++/CUDA inference engine optimized for Qwen3.6 checkpoints on a single RTX 5090. The engine sustains 542 tok/s over a full 65,536-token completion, demonstrating extreme single-GPU throughput for large sparse models.
A developer released NInfer, a custom C++/CUDA inference engine optimized for Qwen3.6 checkpoints on a single RTX 5090. The engine sustains 542 tok/s over a full 65,536-token completion, demonstrating extreme single-GPU throughput for large sparse models.
Updated Mon, Jul 27, 2026, 07:17 PM — Windows build reported with 550-720 tok/s and full 250k context support.
110694 engagement3 sourcesreddit, rssMon, Jul 20, 2026, 02:48 PM
Developments·Benchmark
NInfer NVFP4 beats llama.cpp on Qwen3.6-27B: 2.17–3.53x prefill, 1.14–1.28x generation
A user benchmark on an RTX Pro 6000 shows NInfer's NVFP4 engine outperforming llama.cpp's UD-Q4_K_XL for Qwen3.6-27B, with 2.17–3.53x faster prefill and 1.14–1.28x faster generation. NInfer also used ~10% less GPU memory. The engine is optimized for sm_120 (RTX 5090) but performs well on the Pro 6000.
A user benchmark on an RTX Pro 6000 shows NInfer's NVFP4 engine outperforming llama.cpp's UD-Q4_K_XL for Qwen3.6-27B, with 2.17–3.53x faster prefill and 1.14–1.28x faster generation. NInfer also used ~10% less GPU memory. The engine is optimized for sm_120 (RTX 5090) but performs well on the Pro 6000.
1 engagement1 sourceredditFri, Jul 31, 2026, 07:13 PM
Releases·Tool Release
akopytko releases Blackwell-native NVFP4 quant of Qwen3.8 27B, 50% faster than Q4
A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.
A new GGUF quant of Qwen3.8 27B uses Blackwell-native NVFP4 with a prefill-optimized design, achieving 50% faster prefill than a Q4 quant of the same memory footprint and 4-7% faster than other NVFP4 quants on an RTX 5090. It includes a quantized MTP draft head and is available on Hugging Face.
166 engagement1 sourceredditFri, Aug 21, 2026, 09:19 AM
Releases·Tool Release
Community member releases imatrix GGUF quants for Qwen3.8-Flash-Next, 20-30GB smaller than Unsloth builds
A community quantizer released mainline-compatible imatrix GGUF quants for Qwen3.8-Flash-Next on Hugging Face after several days of testing. The builds target the same quality band as popular Unsloth and AesSedai Q4 quants while saving roughly 20-30GB of disk and RAM, with perplexity competitive with both. This gives local inference users smaller files without a quality tradeoff.
A community quantizer released mainline-compatible imatrix GGUF quants for Qwen3.8-Flash-Next on Hugging Face after several days of testing. The builds target the same quality band as popular Unsloth and AesSedai Q4 quants while saving roughly 20-30GB of disk and RAM, with perplexity competitive with both. This gives local inference users smaller files without a quality tradeoff.
39 engagement1 sourceredditSat, Aug 29, 2026, 03:02 PM
Social·Community
CPU hits 100% as bottleneck for agentic coding on a single RTX 5090 decoding ~700 t/s — Kenshi-to-Godot port report
A developer porting Kenshi to Godot with local LLM agents on one RTX 5090 hit a wall: decoding ran at ~700 t/s, but 25 concurrent agents saturated a 9800X3D CPU at 100% on tool calls. The practical lesson for local inference setups is that CPU-side tool execution, not GPU decode or prefill, becomes the limiting factor in agentic workloads.
A developer porting Kenshi to Godot with local LLM agents on one RTX 5090 hit a wall: decoding ran at ~700 t/s, but 25 concurrent agents saturated a 9800X3D CPU at 100% on tool calls. The practical lesson for local inference setups is that CPU-side tool execution, not GPU decode or prefill, becomes the limiting factor in agentic workloads.
13 engagement1 sourceredditMon, Oct 5, 2026, 06:21 PM