Releases·Tool Release
Truespar open-sources Paddock, a Rust/C++ inference engine with custom CUDA kernels, under MIT/Apache-2.0
The developers of Paddock, a Rust/C++ inference engine with its own CUDA kernels, released the code as open source on schedule, fulfilling an August commitment to ship in September. The public repo is their internal repo, including kernels, and the engine already serves roughly 300B tokens per year in production at their company. Practitioners get a single binary exposing OpenAI- and Anthropic-style APIs that loads GGUF and safetensors weights.
92 engagement1 sourceFri, Sep 4, 2026, 09:15 AM
AI Overview
- Single binary serves OpenAI- and Anthropic-style APIs, loads GGUF and safetensors, and runs ~300B tokens/year in Truespar's production
- Qwen3.8-27B FP8 on one RTX PRO 6000: Paddock beats vLLM in 13/13 configs (1.02x–1.19x) and llama.cpp Q8_0 by 1.5x–37x
- 32 clients at 1024 in/1024 out: 1062 tok/s vs vLLM's 958 and SGLang's 844; SGLang still leads in 2 of 13 configs
- CUDA-only on Windows/Linux, validated on Blackwell and Ampere; Ada kernels require PADDOCK_UNVALIDATED_ARCH=1; no Mac, ROCm, Vulkan, or tensor parallelism
Related
Releases·Tool Releasefeatured
Open-source kernel hits 78,498 output tok/s for Qwen3.6-35B-A3B on 8 AMD MI350X GPUs
A developer team optimized AMD's software stack down to the kernel level for Qwen3.6-35B-A3B on MI350X GPUs, achieving 78,498 output tokens per second across 8 GPUs. The work aims to close the software gap with NVIDIA's CUDA ecosystem, leveraging MI350X's raw compute advantage over B200.
A developer team optimized AMD's software stack down to the kernel level for Qwen3.6-35B-A3B on MI350X GPUs, achieving 78,498 output tokens per second across 8 GPUs. The work aims to close the software gap with NVIDIA's CUDA ecosystem, leveraging MI350X's raw compute advantage over B200.
80 engagement1 sourceredditWed, Aug 26, 2026, 04:12 AM
Releases·Tool Release
TensorSharp benchmarked against llama.cpp, claims on-par performance
A developer released TensorSharp, a new open-source local LLM inference engine supporting Unsloth GGUF models, and published benchmarks showing on-par performance with llama.cpp on CUDA and Vulkan. The engine supports multi-modal models like Gemma4, Qwen3.6, and DiffusionGemma, and offers OpenAI/Ollama-compatible APIs.
A developer released TensorSharp, a new open-source local LLM inference engine supporting Unsloth GGUF models, and published benchmarks showing on-par performance with llama.cpp on CUDA and Vulkan. The engine supports multi-modal models like Gemma4, Qwen3.6, and DiffusionGemma, and offers OpenAI/Ollama-compatible APIs.
4 engagement1 sourceredditSat, Jul 25, 2026, 05:27 PM
Releases·Tool Release
GGUF/CUDA inference engine with DeepSeek-V2/V3 MLA support, built for cold-start not throughput
A developer released Reflex, a GGUF-native Rust/CUDA inference engine (MIT licensed) built for cold-start latency rather than steady-state throughput. It supports DeepSeek-V2/V3 Multi-head Latent Attention and Qwen3 family models, and benchmarks ~1.3-1.4x faster than llama.cpp cold-start on an A6000. It targets one-shot/serverless embedding, not server deployment.
A developer released Reflex, a GGUF-native Rust/CUDA inference engine (MIT licensed) built for cold-start latency rather than steady-state throughput. It supports DeepSeek-V2/V3 Multi-head Latent Attention and Qwen3 family models, and benchmarks ~1.3-1.4x faster than llama.cpp cold-start on an A6000. It targets one-shot/serverless embedding, not server deployment.
4 engagement1 sourceredditWed, Sep 23, 2026, 06:03 PM
Products·launchedfeatured
Rust+CUDA LLM inference engine for RTX 5090, optimized with NVFP4, MoE, and speculative decoding
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
A from-scratch LLM inference engine written in Rust and CUDA, specifically tuned for the RTX 5090 Laptop GPU (Blackwell sm_120a). It supports NVFP4 quantization, mixture-of-experts (MoE), and multi-token prediction (MTP) speculative decoding, achieving up to 1.6x speedup over llama.cpp on target models. Designed for developers who need maximum inference performance on a single high-end laptop GPU.
260 engagement1 sourcegithubSun, Jul 5, 2026, 08:45 PM
Releases·Tool Release
Show HN: PicoLM v1.0-rc1 — C99 LLM inference engine for llama-2, GPT-2, Qwen, Gemma-3n
A developer released PicoLM v1.0-rc1, an LLM inference engine written in C99, on GitHub and announced it on Hacker News. It supports llama-2, GPT-2, Qwen 3.6/3.8 with MoE, and Gemma-3n models, with heavy emphasis on CPU SIMD acceleration and cross-platform portability. For practitioners, it offers a lightweight inference option spanning platforms from DOS to modern OSes.
A developer released PicoLM v1.0-rc1, an LLM inference engine written in C99, on GitHub and announced it on Hacker News. It supports llama-2, GPT-2, Qwen 3.6/3.8 with MoE, and Gemma-3n models, with heavy emphasis on CPU SIMD acceleration and cross-platform portability. For practitioners, it offers a lightweight inference option spanning platforms from DOS to modern OSes.
1 engagement1 sourcehackernewsThu, Sep 3, 2026, 08:19 AM