Releases·Tool Release
GGUF/CUDA inference engine with DeepSeek-V2/V3 MLA support, built for cold-start not throughput
A developer released Reflex, a GGUF-native Rust/CUDA inference engine (MIT licensed) built for cold-start latency rather than steady-state throughput. It supports DeepSeek-V2/V3 Multi-head Latent Attention and Qwen3 family models, and benchmarks ~1.3-1.4x faster than llama.cpp cold-start on an A6000. It targets one-shot/serverless embedding, not server deployment.
4 engagement1 sourceWed, Sep 23, 2026, 06:03 PM
AI Overview
- Lateos AI's open-source Reflex engine (MIT, Rust/CUDA) supports DeepSeek-V2/V3 MLA, Qwen3 dense and MoE, and the Qwen3.5 hybrid Gated DeltaNet mixer.
- Reflex AOT-compiles all kernels with nvcc, uploads weights once with on-GPU dequant, and runs ~1.3-1.4x faster than llama.cpp cold-start on an A6000.
- Reflex is 24-52x faster than vLLM cold-start, though vLLM was tested via safetensors because that build lacked GGUF support.
- Reflex's System1 mode scores multiple candidate continuations in one pass, e.g. Paris 0.997 vs London 0.002, targeting agent decision loops.
- Reflex ships batch_size 1, no server mode, C-FFI plus Python bindings, CUDA-only build with no CPU fallback.
Related
Releases·Tool Release
Truespar open-sources Paddock, a Rust/C++ inference engine with custom CUDA kernels, under MIT/Apache-2.0
The developers of Paddock, a Rust/C++ inference engine with its own CUDA kernels, released the code as open source on schedule, fulfilling an August commitment to ship in September. The public repo is their internal repo, including kernels, and the engine already serves roughly 300B tokens per year in production at their company. Practitioners get a single binary exposing OpenAI- and Anthropic-style APIs that loads GGUF and safetensors weights.
The developers of Paddock, a Rust/C++ inference engine with its own CUDA kernels, released the code as open source on schedule, fulfilling an August commitment to ship in September. The public repo is their internal repo, including kernels, and the engine already serves roughly 300B tokens per year in production at their company. Practitioners get a single binary exposing OpenAI- and Anthropic-style APIs that loads GGUF and safetensors weights.
92 engagement1 sourceredditFri, Sep 4, 2026, 09:15 AM
Products·launchedfeatured
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
260 engagement1 sourcegithubSun, Jul 5, 2026, 08:45 PM
Releases·Tool Release
LeDissolution releases Gewell, a Gemma 4 31B inference engine for Blackwell using 0.625x VRAM KV cache
A developer released Gewell, a custom inference engine for running Gemma 4 31B on Blackwell GPUs under high concurrency. It exploits Gemma's tied K/V weights on global attention layers to store KV cache at 0.625x the usual VRAM, mathematically losslessly. It also features a smarter checkpoint eviction policy and a custom mixed-precision quant format, targeting multi-prompt agentic workloads that vLLM's cache handles poorly.
A developer released Gewell, a custom inference engine for running Gemma 4 31B on Blackwell GPUs under high concurrency. It exploits Gemma's tied K/V weights on global attention layers to store KV cache at 0.625x the usual VRAM, mathematically losslessly. It also features a smarter checkpoint eviction policy and a custom mixed-precision quant format, targeting multi-prompt agentic workloads that vLLM's cache handles poorly.
1 engagement1 sourceredditMon, Sep 21, 2026, 09:56 AM
Social·Community
Reddit post argues one-off LLM inference engines will outpace llama.cpp and vLLM
A Reddit post lays out a thesis that specialized inference engines (ninfer, dwarfstar, Splash, llamAmpere, gufo) will proliferate and outpace general engines like llama.cpp and vLLM, because they optimize kernels and compute graphs for a single model/hardware combo. It ties this to AI coding lowering the barrier to building engines, since 'make tok/s go up' is a fully specified task suitable for autonomous implementation. The author also outlines alternative futures: plugin-ified kernels, AI-ified general-engine workflows, or compilers like MLIR/Mojo eliminating hardware-specific tinkering.
A Reddit post lays out a thesis that specialized inference engines (ninfer, dwarfstar, Splash, llamAmpere, gufo) will proliferate and outpace general engines like llama.cpp and vLLM, because they optimize kernels and compute graphs for a single model/hardware combo. It ties this to AI coding lowering the barrier to building engines, since 'make tok/s go up' is a fully specified task suitable for autonomous implementation. The author also outlines alternative futures: plugin-ified kernels, AI-ified general-engine workflows, or compilers like MLIR/Mojo eliminating hardware-specific tinkering.
787 engagement2 sourcesredditTue, Sep 29, 2026, 05:21 PM
Releases·Tool Release
Flyweight: open-source C++/CUDA engine for running MoE models bigger than your VRAM on one GPU + system RAM. First PyPI release, looking for contributors.
A solo developer released Flyweight, an open-source C++/CUDA GGUF inference runtime that runs MoE models larger than a single GPU's VRAM by keeping experts in system RAM with a hot set cached on-card. It ships OpenAI/Anthropic-compatible APIs, a chat UI, image generation, and quantized KV cache, with benchmarks from a 12 GB laptop GPU. The Apache-2.0 project is on PyPI and actively requesting contributors for hardware and model coverage.
A solo developer released Flyweight, an open-source C++/CUDA GGUF inference runtime that runs MoE models larger than a single GPU's VRAM by keeping experts in system RAM with a hot set cached on-card. It ships OpenAI/Anthropic-compatible APIs, a chat UI, image generation, and quantized KV cache, with benchmarks from a 12 GB laptop GPU. The Apache-2.0 project is on PyPI and actively requesting contributors for hardware and model coverage.
66 engagement1 sourceredditFri, Sep 18, 2026, 07:39 AM