llamAmpere: llama.cpp fork claims 90+ TPS on RTX 3090 for 27B models, up to 240K context

Social·Community

Redditor runs Swift-Qwen3.8-27B on four 8GB 3060tis, hitting 120 t/s at 150k context via tensor parallel vLLM

A Reddit user repurposed four 8GB RTX 3060 Ti cards from an old crypto rig into a local inference setup, avoiding a costly GPU upgrade. Using tensor parallelism via vLLM with the HyperQwen Ampere fork, they run Swift-Qwen3.8-27B at ~120 t/s with 150k context at bf16, or full 262k context at ~70 t/s with kv8 quantization. The post is a practical guide for budget multi-GPU local serving.

A Reddit user repurposed four 8GB RTX 3060 Ti cards from an old crypto rig into a local inference setup, avoiding a costly GPU upgrade. Using tensor parallelism via vLLM with the HyperQwen Ampere fork, they run Swift-Qwen3.8-27B at ~120 t/s with 150k context at bf16, or full 262k context at ~70 t/s with kv8 quantization. The post is a practical guide for budget multi-GPU local serving.

Releases·Tool Release

LlamAmpere v0.4 hits 95+ TPS through 100K generated for Qwen3.8 27B at 262K ctx on a single 3090

Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.

Developer JakeATX released LlamAmpere v0.4, a Llama.cpp fork with Ampere-specific optimizations, achieving 95+ tokens/second over 100K generated tokens for a Qwen3.8 27B model at 262K context on one RTX 3090. The update delivers ~10% speed gains and 10%+ more max context versus the prior version, with vLLM as the closest competing runtime. The author also quantifies the quality cost of the TQ5/TQ4 KV cache quantization used.

Products·launched

Qwen3.8-Flash-Next quantization recipe for single GB10/DGX Spark using Intel AutoRound and vLLM

A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.

A community recipe and repo for running Qwen3.8-Flash-Next on a single DGX Spark (GB10) device. It combines Intel AutoRound int4 quantization with additional uncalibrated int8/fp8 quantization of specific layers, served via vLLM with an fp8 ngram table offloaded to SSD or an RDMA server. It lets local-LLM hobbyists fit the model on constrained hardware at usable speeds.

A Reddit user asks what determines local LLM speed, listing token efficiency, token speed, and prefill as factors. The discussion centers on hardware and model selection for running models like Qwen 3.8 27b on a 4090.

Social·Community

User runs Qwen3.6:35B UD Q4_K_M at 80 tok/s on single Nvidia P40

A Reddit user reports achieving 80 tok/s burst speed running Unsloth's Qwen3.6:35B UD Q4_K_M quantized model with 100k context on a single Nvidia P40 (24GB) using TheTom's TurboQuant fork of llama.cpp. The setup disables reasoning to avoid thinking loops and omits the vision component to save memory.

A Reddit user reports achieving 80 tok/s burst speed running Unsloth's Qwen3.6:35B UD Q4_K_M quantized model with 100k context on a single Nvidia P40 (24GB) using TheTom's TurboQuant fork of llama.cpp. The setup disables reasoning to avoid thinking loops and omits the vision component to save memory.