El Pulpo 0.1.0 released: local LLM proxy and load balancer

Products·launched

Sleepy LLM: reverse proxy that wakes and shuts down your LLM inference server on demand

Sleepy LLM is a reverse proxy placed between LLM clients (e.g. OpenWebUI, coding harnesses) and a local inference server. It wakes the inference machine when a request arrives and shuts it down after a period of inactivity, saving power and eliminating manual startup. Built as a test of local-model vibe-coding, but useful for anyone running an on-demand local LLM box.

Sleepy LLM is a reverse proxy placed between LLM clients (e.g. OpenWebUI, coding harnesses) and a local inference server. It wakes the inference machine when a request arrives and shuts it down after a period of inactivity, saving power and eliminating manual startup. Built as a test of local-model vibe-coding, but useful for anyone running an on-demand local LLM box.

llmproxy is a modular Python proxy that addresses common LLM integration issues such as timeout errors, dropped tool calls, and routing between local and cloud models. It helps developers building LLM-powered applications achieve more reliable and flexible model interactions.

Social·Community

Developer's 4B local/remote LLM router matched GLM 5.2 performance but broke on GLM 5.3 update

A Reddit developer built a 4B-parameter router splitting inference between local hardware and remote models (DeepSeek v4 Flash 0731, GLM 5.2), matching GLM 5.2 performance at roughly equal OpenRouter token costs. The approach broke with GLM 5.3: the split performed worse than GLM 5.3 Flash, ending the project.

A Reddit developer built a 4B-parameter router splitting inference between local hardware and remote models (DeepSeek v4 Flash 0731, GLM 5.2), matching GLM 5.2 performance at roughly equal OpenRouter token costs. The approach broke with GLM 5.3: the split performed worse than GLM 5.3 Flash, ending the project.

A Medium post examines the current state of running LLMs locally on consumer hardware, noting that while smaller quantized models are feasible, larger models remain challenging. The post highlights ongoing improvements in quantization and hardware efficiency.

Products·launched

TokenFlow Gateway: token-aware load balancer for local LLM stacks

TokenFlow Gateway is a token-aware gateway/load balancer for local LLM stacks like Ollama, llama.cpp, and vLLM. It estimates token counts before dispatch to balance load based on actual GPU memory needs, preventing uneven KV-cache fill, latency spikes, and OOM errors. It solves the problem of nginx/HAProxy treating all requests equally despite varying token costs.

TokenFlow Gateway is a token-aware gateway/load balancer for local LLM stacks like Ollama, llama.cpp, and vLLM. It estimates token counts before dispatch to balance load based on actual GPU memory needs, preventing uneven KV-cache fill, latency spikes, and OOM errors. It solves the problem of nginx/HAProxy treating all requests equally despite varying token costs.