A Modal blog post describes achieving over 1 billion tokens per minute per GPU by combining a query planner with an inference engine. The approach matters to practitioners running high-throughput LLM serving, where scheduling and engine co-design can raise utilization. The excerpt provides no further technical specifics.
Modal raised a Series D valuing it at $16B after reaching $1B annual recurring revenue, tripling usage and valuation in five months. CTO Akshat credited demand from agent workloads: Sandboxes, launched in 2023, alone hit a $300M run rate and scaled to a million concurrent sandboxes.
Modal raised a Series D valuing it at $16B after reaching $1B annual recurring revenue, tripling usage and valuation in five months. CTO Akshat credited demand from agent workloads: Sandboxes, launched in 2023, alone hit a $300M run rate and scaled to a million concurrent sandboxes.
A new inference engine, Turing Engine, claims to serve 70B-parameter models like LLaMA-3.1-70B and Qwen-2.5-72B on a single 24GB GPU, achieving 3,064 tokens per second with 75% KV cache compression. It supports Unsloth checkpoints, potentially enabling high-throughput inference on consumer hardware.
A new inference engine, Turing Engine, claims to serve 70B-parameter models like LLaMA-3.1-70B and Qwen-2.5-72B on a single 24GB GPU, achieving 3,064 tokens per second with 75% KV cache compression. It supports Unsloth checkpoints, potentially enabling high-throughput inference on consumer hardware.
A research instrument argues that the seven leading AI inference providers (Fireworks, Together, Baseten, Groq, Modal, CoreWeave, Nebius) are not a single market but three: token-serving API specialists, a serverless-compute platform, and bare-metal GPU IaaS/neoclouds. Comparing them on a single axis is a category error; the correct axes differ per group, and inference physics (memory bandwidth as binding constraint, prefill-vs-decode economics) explains the divergence.
A research instrument argues that the seven leading AI inference providers (Fireworks, Together, Baseten, Groq, Modal, CoreWeave, Nebius) are not a single market but three: token-serving API specialists, a serverless-compute platform, and bare-metal GPU IaaS/neoclouds. Comparing them on a single axis is a category error; the correct axes differ per group, and inference physics (memory bandwidth as binding constraint, prefill-vs-decode economics) explains the divergence.
Astorias AI launched an inference platform aimed at affordable pricing and reliable agent tooling, debuting with Qwen 3.8 27B. The founder shared pricing and solicited interest in a planned serverless "Agent Sandbox" feature. GLM 5.3 Flash and other flash models are planned next.
Astorias AI launched an inference platform aimed at affordable pricing and reliable agent tooling, debuting with Qwen 3.8 27B. The founder shared pricing and solicited interest in a planned serverless "Agent Sandbox" feature. GLM 5.3 Flash and other flash models are planned next.
A new paper introduces FastTPS, a method designed to improve throughput and hardware utilization during the token phase of decoder-only LLM inference on AI accelerators. The approach addresses low parallelism and memory overhead issues when processing long sequences.
A new paper introduces FastTPS, a method designed to improve throughput and hardware utilization during the token phase of decoder-only LLM inference on AI accelerators. The approach addresses low parallelism and memory overhead issues when processing long sequences.