A developer released Pantheon, a dependency-free C LLM inference engine, and Foundry, its underlying ML library, targeting old Kepler GPUs. On a used Tesla K80 it decodes Qwen3.6-35B-A3B and Nex-N2-mini at ~21 tok/s with weights resident on the card. Built-in prefix caching makes multi-turn agent workloads practical, though prefill remains slow for large contexts.
A developer released AgrillaMoE, a llama.cpp fork dedicated to running Qwen3.6-35B-A3B with Unsloth quants on consumer GPUs, exposing OpenAI- and Anthropic-compatible APIs so Claude Code works out of the box. Its runtime 'MoE expansion' routing patch activates more experts per token without retraining. A controlled A/B on HumanEval at 4-bit showed 90.9% pass@1 vs 89.6% stock at ~19% slower decode speed.
A developer released AgrillaMoE, a llama.cpp fork dedicated to running Qwen3.6-35B-A3B with Unsloth quants on consumer GPUs, exposing OpenAI- and Anthropic-compatible APIs so Claude Code works out of the box. Its runtime 'MoE expansion' routing patch activates more experts per token without retraining. A controlled A/B on HumanEval at 4-bit showed 90.9% pass@1 vs 89.6% stock at ~19% slower decode speed.
A developer released a fork of ik_llama.cpp adding bandwidth-adaptive CPU/GPU execution and configurable expert-residency caching for MoE models that don't fit in VRAM, plus Q2_0 quantization support. On an RTX 2060 6GB test setup running Qwen3.6-35B-A3B, the fork delivers roughly 20-30% faster generation (26-30 t/s vs 23 t/s). It only helps when the full MoE doesn't fit in VRAM and the GPU has idle compute.
A developer released a fork of ik_llama.cpp adding bandwidth-adaptive CPU/GPU execution and configurable expert-residency caching for MoE models that don't fit in VRAM, plus Q2_0 quantization support. On an RTX 2060 6GB test setup running Qwen3.6-35B-A3B, the fork delivers roughly 20-30% faster generation (26-30 t/s vs 23 t/s). It only helps when the full MoE doesn't fit in VRAM and the GPU has idle compute.
A solo developer released Flyweight, an open-source C++/CUDA GGUF inference runtime that runs MoE models larger than a single GPU's VRAM by keeping experts in system RAM with a hot set cached on-card. It ships OpenAI/Anthropic-compatible APIs, a chat UI, image generation, and quantized KV cache, with benchmarks from a 12 GB laptop GPU. The Apache-2.0 project is on PyPI and actively requesting contributors for hardware and model coverage.
A solo developer released Flyweight, an open-source C++/CUDA GGUF inference runtime that runs MoE models larger than a single GPU's VRAM by keeping experts in system RAM with a hot set cached on-card. It ships OpenAI/Anthropic-compatible APIs, a chat UI, image generation, and quantized KV cache, with benchmarks from a 12 GB laptop GPU. The Apache-2.0 project is on PyPI and actively requesting contributors for hardware and model coverage.
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
CompiledThoughts released Inferred-Thoughts, an inference engine that runs MoE models too large for VRAM+RAM by streaming routed experts from an NVMe SSD on demand. On a single 16 GB RTX 5060 Ti with 32 GB RAM, it decodes Qwen3.8-Flash-Next 177B NVFP4 (119 GiB) at 9-10 tok/s, roughly double llama.cpp's 4.9 tok/s on the same machine. The team expects a v2 with better SSD streaming to reach ~14-15 tok/s.
A developer released Project Zero, an inference engine written from scratch in pure C99 that runs both 1.58-bit ternary BitNet models and standard dense GGUF models as a single binary with no Python or CUDA. On a Xeon it reports 36 tok/s on BitNet b1.58-2B-4T, about 1.8x faster than bitnet.cpp. It ships with an OpenAI-compatible API and offers a pre-built Linux binary, though some quantization paths still lag llama.cpp.
A developer released Project Zero, an inference engine written from scratch in pure C99 that runs both 1.58-bit ternary BitNet models and standard dense GGUF models as a single binary with no Python or CUDA. On a Xeon it reports 36 tok/s on BitNet b1.58-2B-4T, about 1.8x faster than bitnet.cpp. It ships with an OpenAI-compatible API and offers a pre-built Linux binary, though some quantization paths still lag llama.cpp.