oMLX 0.7.4 can pair a fast Mac as prefill accelerator
Hugging Face hired Jun Kim, creator and maintainer of oMLX, to support the MLX community. oMLX graduates from a side project to a fully funded effort while staying Apache 2.0 under Kim's leadership. Hugging Face aims to streamline moving transformers model definitions onto MLX and upstream work to projects like mlx-lm and mlx-vlm.
Hugging Face hired Jun Kim, creator and maintainer of oMLX, to support the MLX community. oMLX graduates from a side project to a fully funded effort while staying Apache 2.0 under Kim's leadership. Hugging Face aims to streamline moving transformers model definitions onto MLX and upstream work to projects like mlx-lm and mlx-vlm.
A developer running oMLX on an M4 Max reports that Qwen 3.8 Flash Next oq4e-mtp throughput drops from 50-60 tokens/s on a single task to around 20 tokens/s with multiple concurrent agents, with the GPU underutilized. Setting Max Concurrent Requests to 1 works as a workaround when caching reduces prefill, but the user is seeking configurations to fully use the GPU.
A developer running oMLX on an M4 Max reports that Qwen 3.8 Flash Next oq4e-mtp throughput drops from 50-60 tokens/s on a single task to around 20 tokens/s with multiple concurrent agents, with the GPU underutilized. Setting Max Concurrent Requests to 1 works as a workaround when caching reduces prefill, but the user is seeking configurations to fully use the GPU.
A LocalLLaMA user benchmarked five Apple-silicon inference engines — Kapitsa, oMLX, Rapid-MLX, Splash, and MTPLX — on an M3 Max 36 GB across dense and MoE Qwen models plus Flint 40B. Kapitsa 0.2.12 wins dense Qwen3.8-27B-4bit at 41.3 tok/s and ties oMLX on MoE Qwen3.6-35B-A3B (~123 tok/s), becoming the author's new daily default. Flint 40B-A2.5B runs at 134.2 tok/s on a 36 GB laptop under Kapitsa, the post's most surprising result.
A LocalLLaMA user benchmarked five Apple-silicon inference engines — Kapitsa, oMLX, Rapid-MLX, Splash, and MTPLX — on an M3 Max 36 GB across dense and MoE Qwen models plus Flint 40B. Kapitsa 0.2.12 wins dense Qwen3.8-27B-4bit at 41.3 tok/s and ties oMLX on MoE Qwen3.6-35B-A3B (~123 tok/s), becoming the author's new daily default. Flint 40B-A2.5B runs at 134.2 tok/s on a 36 GB laptop under Kapitsa, the post's most surprising result.
Local-LLM users benchmarking newly released models on Apple's M5 Ultra found that raising mlx-vlm's prefill step size to 8192 substantially improves both prompt processing and speculative-generation speed. Early Mimo2.6-Flash runs required a custom mlx-vlm patch and showed usable but underwhelming throughput, with MTP apparently inactive.
Local-LLM users benchmarking newly released models on Apple's M5 Ultra found that raising mlx-vlm's prefill step size to 8192 substantially improves both prompt processing and speculative-generation speed. Early Mimo2.6-Flash runs required a custom mlx-vlm patch and showed usable but underwhelming throughput, with MTP apparently inactive.
Practitioners running two M5U 256GB machines as a Thunderbolt 5 cluster report local inference results for GLM 5.3 and Qwen Flash Next in q6 quants. A later post gives concrete throughput numbers: Qwen Flash Next reaches 200 tok/s decode versus roughly 50 tok/s for GLM 5.3, leading the user to switch models. The posts show community experimentation with clustering software (Tensorfold, oMLX) and harnesses on this hardware.
Practitioners running two M5U 256GB machines as a Thunderbolt 5 cluster report local inference results for GLM 5.3 and Qwen Flash Next in q6 quants. A later post gives concrete throughput numbers: Qwen Flash Next reaches 200 tok/s decode versus roughly 50 tok/s for GLM 5.3, leading the user to switch models. The posts show community experimentation with clustering software (Tensorfold, oMLX) and harnesses on this hardware.