A developer posted accuracy and speed benchmarks for an open-source CUDA megakernel engine (part of the open-jet repo) that runs Qwen3.8-27B 1.4-1.9x faster than llama.cpp on a single RTX 3090. The update answers accuracy questions with KL-divergence comparisons and lays out quantization support, hardware limits, and contribution asks. It matters to local-inference practitioners as a tuned single-GPU alternative to llama.cpp for this model.
Users share hand-tuned (with AI assistance) inference optimizations for Qwen3.8-27B on consumer hardware. One reports a CUDA megakernel running the model at 140 tok/s code generation on a single RTX 3090, 1.4-1.9x faster than llama.cpp; another details ROCm and llama-server fixes that lift Q8 decode from ~54-61 to 96-110 tok/s on four AMD 7900 XTX GPUs. Both publish code and configs, showing meaningful headroom over stock inference stacks on local rigs.
Users share hand-tuned (with AI assistance) inference optimizations for Qwen3.8-27B on consumer hardware. One reports a CUDA megakernel running the model at 140 tok/s code generation on a single RTX 3090, 1.4-1.9x faster than llama.cpp; another details ROCm and llama-server fixes that lift Q8 decode from ~54-61 to 96-110 tok/s on four AMD 7900 XTX GPUs. Both publish code and configs, showing meaningful headroom over stock inference stacks on local rigs.
A developer released a two-GPU fork of L-Forster's open-jet single-GPU megakernel for running Qwen3.8-27B on RTX 3090s over plain PCIe. It reaches 190 tok/s single-stream (622 tok/s aggregate at 8 requests) in Q4_K_M, roughly 1.6–2x llama.cpp's throughput on the same hardware. It ships with an OpenAI-compatible server and is pending 4-bit KV benchmark follow-ups.
A developer released a two-GPU fork of L-Forster's open-jet single-GPU megakernel for running Qwen3.8-27B on RTX 3090s over plain PCIe. It reaches 190 tok/s single-stream (622 tok/s aggregate at 8 requests) in Q4_K_M, roughly 1.6–2x llama.cpp's throughput on the same hardware. It ships with an OpenAI-compatible server and is pending 4-bit KV benchmark follow-ups.
Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.
Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.
A GitHub release ships 'Bonsai', end-to-end ternary-quantized weights of Qwen3.8-27B for llama.cpp (CUDA, Metal, CPU), cutting the language model from ~54 GB FP16 to ~5.9 GB while retaining 98.2% of FP16 intelligence across 14 thinking-mode benchmarks. It keeps thinking, reasoning, and agentic tool-calling behavior at ~47 tok/s on an Apple M5 Max laptop, targeting GPU-poor users.
A GitHub release ships 'Bonsai', end-to-end ternary-quantized weights of Qwen3.8-27B for llama.cpp (CUDA, Metal, CPU), cutting the language model from ~54 GB FP16 to ~5.9 GB while retaining 98.2% of FP16 intelligence across 14 thinking-mode benchmarks. It keeps thinking, reasoning, and agentic tool-calling behavior at ~47 tok/s on an Apple M5 Max laptop, targeting GPU-poor users.
Following Qwen3.8-27B's release, Mac users are sharing llama.cpp benchmarks. An M4 Pro 24GB runs the Q4_K_M quant at 96.8 tok/s prompt processing and 11.4 tok/s generation, while an M2 Ultra 192GB with Q6_K_XL reaches 360 t/s prompt and 22.38 t/s generation.
Following Qwen3.8-27B's release, Mac users are sharing llama.cpp benchmarks. An M4 Pro 24GB runs the Q4_K_M quant at 96.8 tok/s prompt processing and 11.4 tok/s generation, while an M2 Ultra 192GB with Q6_K_XL reaches 360 t/s prompt and 22.38 t/s generation.