Releases·Tool Release
Custom Models in Oh My Pi: vLLM, llama.cpp, SGLang and More
Oh My Pi adds custom model support for vLLM, llama.cpp and SGLang
41 engagement1 sourceSat, Sep 26, 2026, 12:31 AM
AI Overview
- Oh My Pi adds support for custom models via backends including vLLM, llama.cpp, and SGLang, per doug.sh.
- The feature announcement surfaced on Reddit, where it drew 41 upvotes.
Related
Social·Community
A practical comparison of llama.cpp, TensorSharp, Ollama, MLX-LM, MLC LLM, vLLM, SGLang, and other open-source inference stacks
A TensorSharp maintainer posted a detailed comparison of nine open-source inference stacks, mapping each to workloads, hardware targets, and language ecosystems. The guide distinguishes local/embedded engines (llama.cpp, Ollama, MLX-LM, MLC LLM, TensorSharp) from high-throughput server stacks (vLLM, SGLang, TensorRT-LLM), and notes TGI is in maintenance mode. For practitioners it offers a selection framework rather than a benchmark, stressing that performance depends on model, quantization, batching, and hardware.
A TensorSharp maintainer posted a detailed comparison of nine open-source inference stacks, mapping each to workloads, hardware targets, and language ecosystems. The guide distinguishes local/embedded engines (llama.cpp, Ollama, MLX-LM, MLC LLM, TensorSharp) from high-throughput server stacks (vLLM, SGLang, TensorRT-LLM), and notes TGI is in maintenance mode. For practitioners it offers a selection framework rather than a benchmark, stressing that performance depends on model, quantization, batching, and hardware.
13 engagement1 sourceredditFri, Sep 18, 2026, 04:46 PM
Products·launched
PolyServe: open-source LLM serving autotuner benchmarking configs across vLLM, SGLang, and llama.cpp
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
PolyServe is an open-source autotuner for LLM serving that benchmarks configurations across vLLM, SGLang, and llama.cpp, covering weight precision, KV cache, batching, and speculative decoding. It selects a configuration meeting the user's latency constraints and serves it, replacing fixed tuning heuristics. It targets self-hosters and engineers unsure how to optimize inference for their GPU setup.
23 engagement1 sourceredditThu, Sep 17, 2026, 06:51 AM
Social·Community
Inference engines diverge on Ling-3.0-flash support: SGLang commits day-0, vLLM waits for weights, llama.cpp rejects
SGLang publicly committed to day-0 support for Ling-3.0-flash, describing the model as using KDA + MLA hybrid attention. vLLM said support will come when weights land, while llama.cpp closed a related request as not planned.
SGLang publicly committed to day-0 support for Ling-3.0-flash, describing the model as using KDA + MLA hybrid attention. vLLM said support will come when weights land, while llama.cpp closed a related request as not planned.
73 engagement1 sourceredditMon, Jul 27, 2026, 04:33 PM
Developments·Paper
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
A research paper argues that local inference serving stacks (Ollama, llama.cpp, vLLM, SGLang) materially change measured tool-call fidelity, independent of model behavior. Ollama's default tools= handling gates requests per model via a static template flag, and harnesses that discard rejection metadata can misclassify failures as model non-calls. The authors propose treating serving behavior as part of the evaluation protocol, with a checklist.
A research paper argues that local inference serving stacks (Ollama, llama.cpp, vLLM, SGLang) materially change measured tool-call fidelity, independent of model behavior. Ollama's default tools= handling gates requests per model via a static template flag, and harnesses that discard rejection metadata can misclassify failures as model non-calls. The authors propose treating serving behavior as part of the evaluation protocol, with a checklist.
0 engagement1 sourcearxivTue, Sep 22, 2026, 04:48 PM