Releases·Tool Release
Gem16: Codex-written custom engine runs Gemma4 12B & 26B on a 16GB Blackwell GPU at up to 182 t/s decode
A developer released Gem16, a custom single-user inference engine for Gemma4 12B and 26B models, written entirely with Codex and targeting 16GB Blackwell GPUs like the RTX 5080 Laptop. Custom EXL3-like quantization lets the 26B model fit in 16GB VRAM with 220k context, at speeds matching or exceeding vLLM, which failed with MTP on 16GB cards. The engine runs on Linux and Windows with a native GUI.
18 engagement1 sourceSun, Sep 27, 2026, 10:02 PM
AI Overview
- Developer Danmoreng built Gem16, a Codex-written engine matching or beating vLLM speed, since vLLM lacked MTP support on 16GB VRAM
- Custom EXL3-like quantisation lets the Gemma4 26B fit on a 5080 Laptop GPU with 220k context (5,660 t/s prefill, 182 t/s decode)
- Gemma4 12B with audio and vision hits 5,800 t/s prefill and 87 t/s decode and serves two sessions; 26B is single-user
- Audio token recognition in Gemma4 12B breaks after ~8k context, a model issue others reported on Hugging Face, not an engine bug
- Engine runs on Linux and Windows, includes a native GUI, source available on GitHub
Related
Releases·Tool Release
Worklog: faster GEMM/FlashAttention kernels beat vLLM and Llama.cpp TTFT on Gemma4-12B
A developer shared benchmarks showing that compiler-generated GEMM and FlashAttention kernels reduce time-to-first-token (TTFT) on Gemma4-12B with an RTX 5090, beating vLLM and Llama.cpp. The work is experimental and integrated via a vLLM plug-in, which adds overhead and slightly slows token generation.
A developer shared benchmarks showing that compiler-generated GEMM and FlashAttention kernels reduce time-to-first-token (TTFT) on Gemma4-12B with an RTX 5090, beating vLLM and Llama.cpp. The work is experimental and integrated via a vLLM plug-in, which adds overhead and slightly slows token generation.
2 engagement1 sourceredditMon, Aug 3, 2026, 08:06 PM
Releases·Tool Release
TensorSharp benchmarked against llama.cpp, claims on-par performance
A developer released TensorSharp, a new open-source local LLM inference engine supporting Unsloth GGUF models, and published benchmarks showing on-par performance with llama.cpp on CUDA and Vulkan. The engine supports multi-modal models like Gemma4, Qwen3.6, and DiffusionGemma, and offers OpenAI/Ollama-compatible APIs.
A developer released TensorSharp, a new open-source local LLM inference engine supporting Unsloth GGUF models, and published benchmarks showing on-par performance with llama.cpp on CUDA and Vulkan. The engine supports multi-modal models like Gemma4, Qwen3.6, and DiffusionGemma, and offers OpenAI/Ollama-compatible APIs.
4 engagement1 sourceredditSat, Jul 25, 2026, 05:27 PM
Products·launched
PixelUnlockGPU: Android app turning a Pixel into an offline OpenAI-compatible LLM server
An Android app that runs Gemma 4 E4B entirely on-device on a Pixel 10 Pro XL and exposes it as an OpenAI-compatible HTTP server. Any OpenAI client (e.g. TypingMind) can use it via /v1/chat/completions with streaming. It solves the need for private, offline LLM inference without cloud or subscription for Pixel owners.
An Android app that runs Gemma 4 E4B entirely on-device on a Pixel 10 Pro XL and exposes it as an OpenAI-compatible HTTP server. Any OpenAI client (e.g. TypingMind) can use it via /v1/chat/completions with streaming. It solves the need for private, offline LLM inference without cloud or subscription for Pixel owners.
1 engagement1 sourceredditFri, Oct 2, 2026, 07:15 PM
Social·Community
Voice conversations between Gemma4 12B and E2B on GPU and Jetson Orin
A developer demonstrated an open-source voice conversation pipeline running Gemma 4 12B on an RTX PRO 4500 Blackwell and Gemma 4 E2B on a Jetson Orin NX 16GB, powered by Cortexist Little Gemma, a small C-based LLM engine for CUDA devices. The pipeline adds lip sync, expressions, and gestures, and is fully open source.
A developer demonstrated an open-source voice conversation pipeline running Gemma 4 12B on an RTX PRO 4500 Blackwell and Gemma 4 E2B on a Jetson Orin NX 16GB, powered by Cortexist Little Gemma, a small C-based LLM engine for CUDA devices. The pipeline adds lip sync, expressions, and gestures, and is fully open source.
135 engagement1 sourceredditTue, Sep 8, 2026, 04:36 AM
Social·Community
Self-hosters seek GLM 4.5 Air replacement on dual A100 setups as Qwen 3.5 options fall short
A self-hosting operator running GLM 4.5 Air (FP8) on two A100 GPUs reports the model has grown dated and asks the community what to replace it with. They tested Qwen 3.5 122B and Qwen 35B but hit tool-call malformation, SGLang failures, and vLLM MTP bugs. The thread highlights practical friction in serving newer open models on Ampere hardware.
A self-hosting operator running GLM 4.5 Air (FP8) on two A100 GPUs reports the model has grown dated and asks the community what to replace it with. They tested Qwen 3.5 122B and Qwen 35B but hit tool-call malformation, SGLang failures, and vLLM MTP bugs. The thread highlights practical friction in serving newer open models on Ampere hardware.
15 engagement1 sourceredditSun, Aug 30, 2026, 12:56 PM