Releases·Tool Release
React Native ExecuTorch is now up to 92x faster 🏎️
Software Mansion shipped React Native ExecuTorch v0.10, delivering large on-device inference speedups over v0.9, including a 92x gain on FastSAM instance segmentation and a 3x gain on Qwen3 0.6B with long prompts. The release replaces monolithic native modules with inspectable TypeScript pipelines and makes custom model integration easier across major silicon backends.
32 engagement1 sourceThu, Sep 10, 2026, 05:06 PM
AI Overview
- Software Mansion's React Native ExecuTorch v0.10 runs Qwen3 0.6B over 3x faster on long prompts and FastSAM up to 92x faster, replacing native modules with inspectable TypeScript pipelines.
Related
Products·launched
AutoTuneLLM: Optimizes local LLM performance per device
AutoTuneLLM is a tool that automatically optimizes local LLM inference for a user's specific hardware. It reduces time-to-first-token by 39% and agent wall times by 46%, and prevents memory swaps by dynamically managing KV cache, prefix caching, RAM pressure, context trimming, and KV quantization. It solves the problem of local LLMs being slow or unreliable on consumer devices, targeting developers and power users running models locally.
AutoTuneLLM is a tool that automatically optimizes local LLM inference for a user's specific hardware. It reduces time-to-first-token by 39% and agent wall times by 46%, and prevents memory swaps by dynamically managing KV cache, prefix caching, RAM pressure, context trimming, and KV quantization. It solves the problem of local LLMs being slow or unreliable on consumer devices, targeting developers and power users running models locally.
1 engagement1 sourcehackernewsWed, Jul 22, 2026, 01:48 PM
Releases·Tool Release
Open-source Qwen3-TTS 1.7B hits 34 ms TTFA, 10 RPS on H100
A team open-sourced an optimized Qwen3-TTS 1.7B implementation achieving 10 requests per second and sub-50 ms p95 time-to-first-audio on a single H100, with real-time playback. The project targets local inference speed, claiming significant gains over vLLM-Omni and SGLang-Omni.
A team open-sourced an optimized Qwen3-TTS 1.7B implementation achieving 10 requests per second and sub-50 ms p95 time-to-first-audio on a single H100, with real-time playback. The project targets local inference speed, claiming significant gains over vLLM-Omni and SGLang-Omni.
1 engagement1 sourceredditFri, Aug 21, 2026, 04:01 PM
Releases·Tool Release
FreeToken serves frontier MoE models on consumer GPUs with bandwidth-adaptive execution
FreeToken, a new edge-native MoE serving system, enables frontier models to run on consumer hardware at interactive speeds without extreme quantization. It achieves 3–4× faster decode and 6–30× faster prefill than Ollama, with official checkpoints.
FreeToken, a new edge-native MoE serving system, enables frontier models to run on consumer hardware at interactive speeds without extreme quantization. It achieves 3–4× faster decode and 6–30× faster prefill than Ollama, with official checkpoints.
56 engagement1 sourceredditMon, Aug 24, 2026, 11:38 PM
Releases·Tool Release
Hugging Face integrates native-speed vLLM backend into Transformers library
Hugging Face has upgraded the vLLM pip package to support native-speed inference directly within the Transformers library. The update enables seamless use of vLLM as a backend for Transformer models, leveraging its optimized serving capabilities. This integration allows developers to run models at native speed without switching frameworks.
Hugging Face has upgraded the vLLM pip package to support native-speed inference directly within the Transformers library. The update enables seamless use of vLLM as a backend for Transformer models, leveraging its optimized serving capabilities. This integration allows developers to run models at native speed without switching frameworks.
0 engagement1 sourcerssWed, Jul 8, 2026, 12:00 AM
Products·launched
Run massive MoE models on mobile devices via CPU
BigEdgeOnMoE is an open-source codebase that enables running large Mixture-of-Experts models (up to 120B parameters) on standard smartphones or consumer PCs using only CPU. It achieves 6 tokens/second for Qwen 35B (Q4) on a mid-range phone with 12GB RAM, making large models accessible without specialized hardware.
BigEdgeOnMoE is an open-source codebase that enables running large Mixture-of-Experts models (up to 120B parameters) on standard smartphones or consumer PCs using only CPU. It achieves 6 tokens/second for Qwen 35B (Q4) on a mid-range phone with 12GB RAM, making large models accessible without specialized hardware.
20 engagement1 sourceredditThu, Jul 30, 2026, 03:34 PM