A developer forked the Strata inference engine with heavy modifications to run Qwen3.8-FN UD-Q4_K_XL on an IBM AC922, where llama.cpp previously managed only 130 tok/s prefill and 15 tok/s decode. The fork reaches 7,350 tok/s prefill and ~113 tok/s decode, and the code is published on GitHub with some changes to be contributed upstream.
Strata is a local inference server tool whose `strata serve` command runs a quantized Qwen3.6-35B-A3B MoE model on a single 12 GB RTX 3060 while serving multiple concurrent users. It exposes OpenAI- and Anthropic-compatible APIs with per-key usage logging, so local models can be shared cheaply with a small group. It targets hobbyists who want to split one consumer GPU's capacity among several people.
Strata is a local inference server tool whose `strata serve` command runs a quantized Qwen3.6-35B-A3B MoE model on a single 12 GB RTX 3060 while serving multiple concurrent users. It exposes OpenAI- and Anthropic-compatible APIs with per-key usage logging, so local models can be shared cheaply with a small group. It targets hobbyists who want to split one consumer GPU's capacity among several people.
A Reddit user reports running Qwen3.8-Flash-Next-GSQ-RCO-Abliterated at IQ3_S quantization on modest hardware — 12GB VRAM plus 32GB system RAM and NVMe — achieving 20-30 tok/sec decode and up to ~90,000 tok/sec prefill at 131k context. The setup uses a heavily modified experimental fork of Strata and is published as a GitHub recipe. The poster describes the result as feeling like 'local Opus in some regards' on under $2k of compute.
A Reddit user reports running Qwen3.8-Flash-Next-GSQ-RCO-Abliterated at IQ3_S quantization on modest hardware — 12GB VRAM plus 32GB system RAM and NVMe — achieving 20-30 tok/sec decode and up to ~90,000 tok/sec prefill at 131k context. The setup uses a heavily modified experimental fork of Strata and is published as a GitHub recipe. The poster describes the result as feeling like 'local Opus in some regards' on under $2k of compute.
A user reports that DeepSeek V4 Flash 0731, a GGUF quantized model, achieves 700 tokens/s prompt processing and 18 tokens/s generation on a Threadripper 5965WX with an RTX 5090, using llama-server with specific flags. The --cpu-moe flag and large batch sizes dramatically improve performance over default settings.
A user reports that DeepSeek V4 Flash 0731, a GGUF quantized model, achieves 700 tokens/s prompt processing and 18 tokens/s generation on a Threadripper 5965WX with an RTX 5090, using llama-server with specific flags. The --cpu-moe flag and large batch sizes dramatically improve performance over default settings.
Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.
Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.
A developer shares optimizations for running DSv4-Flash on a dual GH200 system, achieving 10,000 tokens/s prefill and over 300 tokens/s decode on SGLang. The write-up details specific patches and configuration tweaks for vLLM and SGLang, offering practical guidance for high-throughput inference.
A developer shares optimizations for running DSv4-Flash on a dual GH200 system, achieving 10,000 tokens/s prefill and over 300 tokens/s decode on SGLang. The write-up details specific patches and configuration tweaks for vLLM and SGLang, offering practical guidance for high-throughput inference.