Releases·Tool Release
LeDissolution releases Gewell, a Gemma 4 31B inference engine for Blackwell using 0.625x VRAM KV cache
A developer released Gewell, a custom inference engine for running Gemma 4 31B on Blackwell GPUs under high concurrency. It exploits Gemma's tied K/V weights on global attention layers to store KV cache at 0.625x the usual VRAM, mathematically losslessly. It also features a smarter checkpoint eviction policy and a custom mixed-precision quant format, targeting multi-prompt agentic workloads that vLLM's cache handles poorly.
1 engagement1 sourceMon, Sep 21, 2026, 09:56 AM
AI Overview
- LeDissolution released Gewell, a custom inference engine for Gemma 4 31B on Blackwell GPUs, on GitHub and Reddit
- Gewell exploits Gemma's tied K/V weights on global attention layers (RoPE rotating only 25% of K) for a mathematically lossless KV cache at 0.625x VRAM
- Gewell keeps reusable prefixes warm via checkpoint dissolution and explicit cache hints, unlike vLLM's LRU eviction; it targets multi-prompt writer/critic workflows
- Gewell starts in seconds by hardcoding offline-profiled GEMM shapes and skipping Python import overhead
- Its G0 quant (~6bpw, on Hugging Face) biases precision toward attention and global-attention-adjacent MLP; weights + MTP + 250k context fit in 32GB
- Benchmarked on aya_dataset, OpenR1-Math, DocVQA, ChartQA, QASPER, and code_contests; at 64k context even bf16 fails the 16-needle RULER-style NIAH without reasoning
- Developed on Linux with an AMD Pro 6000; samplers limited to temp/top-k/top-p and chat template hardcoded; roadmap includes Gemma 4 26A4 support
Related
Products·concept
Spite: a modular local LLM inference engine where every layer is replaceable
Spite is a customizable inference engine for local LLMs that compiles kernels and optimizations only for your specific GPUs and chosen models. Instead of one engine supporting all hardware, users assemble per-model kernel modules and design optimizations for their own setup. It targets people running local models who want setup-specific performance without touching unrelated layers.
Spite is a customizable inference engine for local LLMs that compiles kernels and optimizations only for your specific GPUs and chosen models. Instead of one engine supporting all hardware, users assemble per-model kernel modules and design optimizations for their own setup. It targets people running local models who want setup-specific performance without touching unrelated layers.
24 engagement1 sourceredditSun, Oct 4, 2026, 09:21 PM
Releases·Tool Release
GGUF/CUDA inference engine with DeepSeek-V2/V3 MLA support, built for cold-start not throughput
A developer released Reflex, a GGUF-native Rust/CUDA inference engine (MIT licensed) built for cold-start latency rather than steady-state throughput. It supports DeepSeek-V2/V3 Multi-head Latent Attention and Qwen3 family models, and benchmarks ~1.3-1.4x faster than llama.cpp cold-start on an A6000. It targets one-shot/serverless embedding, not server deployment.
A developer released Reflex, a GGUF-native Rust/CUDA inference engine (MIT licensed) built for cold-start latency rather than steady-state throughput. It supports DeepSeek-V2/V3 Multi-head Latent Attention and Qwen3 family models, and benchmarks ~1.3-1.4x faster than llama.cpp cold-start on an A6000. It targets one-shot/serverless embedding, not server deployment.
4 engagement1 sourceredditWed, Sep 23, 2026, 06:03 PM
Social·Community
Improved TPS of Gemma 4 31B: the journey and also creating custom patches with VLLM fork
A developer reports improving tokens-per-second throughput for Gemma 4 31B by forking vLLM and applying custom patches to implement their own configuration. They published a Medium article aiming to give beginners a mental model for LLM inference speed optimization. The work underscores the author's view that effective optimization requires digging into model architecture and codebase internals, not just surface-level tuning.
A developer reports improving tokens-per-second throughput for Gemma 4 31B by forking vLLM and applying custom patches to implement their own configuration. They published a Medium article aiming to give beginners a mental model for LLM inference speed optimization. The work underscores the author's view that effective optimization requires digging into model architecture and codebase internals, not just surface-level tuning.
12 engagement1 sourceredditSat, Sep 19, 2026, 08:38 PM
Social·Community
Reddit post argues one-off LLM inference engines will outpace llama.cpp and vLLM
A Reddit post lays out a thesis that specialized inference engines (ninfer, dwarfstar, Splash, llamAmpere, gufo) will proliferate and outpace general engines like llama.cpp and vLLM, because they optimize kernels and compute graphs for a single model/hardware combo. It ties this to AI coding lowering the barrier to building engines, since 'make tok/s go up' is a fully specified task suitable for autonomous implementation. The author also outlines alternative futures: plugin-ified kernels, AI-ified general-engine workflows, or compilers like MLIR/Mojo eliminating hardware-specific tinkering.
A Reddit post lays out a thesis that specialized inference engines (ninfer, dwarfstar, Splash, llamAmpere, gufo) will proliferate and outpace general engines like llama.cpp and vLLM, because they optimize kernels and compute graphs for a single model/hardware combo. It ties this to AI coding lowering the barrier to building engines, since 'make tok/s go up' is a fully specified task suitable for autonomous implementation. The author also outlines alternative futures: plugin-ified kernels, AI-ified general-engine workflows, or compilers like MLIR/Mojo eliminating hardware-specific tinkering.
787 engagement2 sourcesredditTue, Sep 29, 2026, 05:21 PM
Social·Community
User reports 2 of 5 CMP 170HX GPUs dead in 2 weeks, third with defective tensor cores
A Reddit user running NVIDIA CMP 170HX mining cards reports two of five units failing within two weeks of deployment — one drops off immediately when vLLM starts, another throws CUDA errors on boot. A third arrived with defective tensor cores. The user warns current resale prices don't justify the reliability risk for inference workloads.
A Reddit user running NVIDIA CMP 170HX mining cards reports two of five units failing within two weeks of deployment — one drops off immediately when vLLM starts, another throws CUDA errors on boot. A third arrived with defective tensor cores. The user warns current resale prices don't justify the reliability risk for inference workloads.
279 engagement1 sourceredditWed, Sep 2, 2026, 02:00 AM