A tool called Berth lets GPU-serving engines like ComfyUI, vLLM, and SGLang offload their weights to system RAM when idle and resume from there instead of reloading from disk, via a park protocol. On an RTX 4090 with 64 GiB RAM, switching to a 21 GB chat model with the file in the page cache takes 5.6 s, versus 13.7 s on the first disk-bound load.
A developer released Berth, an open-source daemon that arbitrates VRAM access among multiple inference engines (llama-swap, llama.cpp, Ollama, ComfyUI) on a single NVIDIA GPU. Engines started via 'berth run' queue for the card, and higher-priority workloads evict current holders via unload URLs or escalating signals, waiting until nvidia-smi confirms memory is free. It targets the common single-GPU failure where concurrent model loads cause CUDA out-of-memory crashes.
A developer released Berth, an open-source daemon that arbitrates VRAM access among multiple inference engines (llama-swap, llama.cpp, Ollama, ComfyUI) on a single NVIDIA GPU. Engines started via 'berth run' queue for the card, and higher-priority workloads evict current holders via unload URLs or escalating signals, waiting until nvidia-smi confirms memory is free. It targets the common single-GPU failure where concurrent model loads cause CUDA out-of-memory crashes.
The developer of Berth, an open-source Linux service that arbitrates a single GPU among AI engines, shipped v0.3.0 with fast-swap engine parking in system RAM and followed with v0.3.4 adding experimental VFIO passthrough so a Windows VM can borrow the card and return it. The tool targets single-24GB-GPU setups where orphaned processes like ComfyUI hold VRAM and block other engines. The author is recruiting testers with large RAM, multi-GPU rigs, and Windows or AMD setups for upcoming features.
The developer of Berth, an open-source Linux service that arbitrates a single GPU among AI engines, shipped v0.3.0 with fast-swap engine parking in system RAM and followed with v0.3.4 adding experimental VFIO passthrough so a Windows VM can borrow the card and return it. The tool targets single-24GB-GPU setups where orphaned processes like ComfyUI hold VRAM and block other engines. The author is recruiting testers with large RAM, multi-GPU rigs, and Windows or AMD setups for upcoming features.
An NVIDIA research preprint demonstrates cross-model KV cache transfer, allowing a switch from Qwen3-14B to Qwen3-32B without re-prefilling. The mapping completes in 278 ms at 32K tokens, replacing a 6.98-second re-prefill. This could reduce latency in multi-model serving.
An NVIDIA research preprint demonstrates cross-model KV cache transfer, allowing a switch from Qwen3-14B to Qwen3-32B without re-prefilling. The mapping completes in 278 ms at 32K tokens, replacing a 6.98-second re-prefill. This could reduce latency in multi-model serving.
TurboFieldfare is an open-source engine that runs large MoE models on Macs by streaming experts from SSD instead of loading them into RAM. This port adds support for Qwen 3.6 35B-A3B, reducing memory usage to ~1.4 GB and achieving 19-23 tok/s on an M5.
TurboFieldfare is an open-source engine that runs large MoE models on Macs by streaming experts from SSD instead of loading them into RAM. This port adds support for Qwen 3.6 35B-A3B, reducing memory usage to ~1.4 GB and achieving 19-23 tok/s on an M5.
Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.
Overspill is an experimental disk-storage tier for the FreeToken inference engine that lets MoE models larger than system RAM run on consumer rigs. It keeps experts on disk and streams them across disk/RAM/VRAM, inspired by Colibri's approach. It targets hobbyists running large MoE models like DeepSeek-V4-Flash on modest GPUs.