Dotwave runs 56 concurrent Nemotron VoiceChat 11B sessions on one H100, cutting per-session GPU cost ~98%

An arXiv paper introduces DuplexCadence, a runtime technique for serving full-duplex speech models, which must turn each second of input into a second of speech before the next second arrives. The method declares the model's native clocks (per-region counters) to the runtime, enabling demand-sized state allocation at stable addresses and exact-shape graph replay without padding. Across four released models and three decoder architectures with bit-for-bit identical output, it achieves 2.85x stock runtime speed and 38.8% lower peak memory, cutting mean SPEAK time from 14% over the one-second cadence to 2% under it.

AWS, NVIDIA, and Heidi Health published a guide showing NVIDIA Multi-Process Service (MPS) cuts ASR inference costs on Amazon EC2 by 75%. Single ASR requests use only 15-20% of GPU compute, and CUDA's default time-slicing forces sequential access; MPS enables concurrent kernel execution to reclaim idle capacity. The approach matters for practitioners running latency-sensitive, low-utilization GPU workloads like Heidi's AI medical scribe, which processes over 2.4 million encounters.

Releases·Tool Release

Developer's INT8 vLLM fork hits 972 tok/s TG on Qwen3.8 27B with 4x MI100

A developer released a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a GPTQ INT8 quant with DFlash2. On a 4x MI100 rig costing $6.5k, it achieves 972 tok/s time-to-first-token and 5,680 tokens/s per prompt, up from 15 tok/s on stock vLLM. This enables older INT8-centric hardware to run modern models efficiently.

A developer released a complete INT8 serving stack for Qwen3.8 27B based on vLLM, AITER, and a GPTQ INT8 quant with DFlash2. On a 4x MI100 rig costing $6.5k, it achieves 972 tok/s time-to-first-token and 5,680 tokens/s per prompt, up from 15 tok/s on stock vLLM. This enables older INT8-centric hardware to run modern models efficiently.

Social·Community

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Harshul Jain (Audible) and Tanmay Sah lead a YouTube workshop deriving LLM inference-at-scale memory costs from first principles. They show why KV cache sizing determines whether multi-user serving fits on consumer GPUs.

Harshul Jain (Audible) and Tanmay Sah lead a YouTube workshop deriving LLM inference-at-scale memory costs from first principles. They show why KV cache sizing determines whether multi-user serving fits on consumer GPUs.

Social·Community

A practical comparison of llama.cpp, TensorSharp, Ollama, MLX-LM, MLC LLM, vLLM, SGLang, and other open-source inference stacks

A TensorSharp maintainer posted a detailed comparison of nine open-source inference stacks, mapping each to workloads, hardware targets, and language ecosystems. The guide distinguishes local/embedded engines (llama.cpp, Ollama, MLX-LM, MLC LLM, TensorSharp) from high-throughput server stacks (vLLM, SGLang, TensorRT-LLM), and notes TGI is in maintenance mode. For practitioners it offers a selection framework rather than a benchmark, stressing that performance depends on model, quantization, batching, and hardware.

A TensorSharp maintainer posted a detailed comparison of nine open-source inference stacks, mapping each to workloads, hardware targets, and language ecosystems. The guide distinguishes local/embedded engines (llama.cpp, Ollama, MLX-LM, MLC LLM, TensorSharp) from high-throughput server stacks (vLLM, SGLang, TensorRT-LLM), and notes TGI is in maintenance mode. For practitioners it offers a selection framework rather than a benchmark, stressing that performance depends on model, quantization, batching, and hardware.