Releases·Tool Release
openjev-sglang: Qwen3.6-35B-A3B prefill-only server on SGLang, deployed on Modal B200s with TypeSafe/Jev API
A GitHub project, openjev-sglang, implements the TypeSafe/Jev HTTP API as a prefill-only endpoint running Qwen3.6-35B-A3B on SGLang, deployable on Modal with one B200 per container. It matters to practitioners because it offers an open-model substitute for the proprietary Jev API with one-command Modal deployment and scale-to-zero cost control.
82 engagement1 sourceThu, Sep 17, 2026, 06:11 PM
AI Overview
- Implements the TypeSafe/Jev HTTP API with Qwen3.6-35B-A3B on SGLang 0.5.19's Rust frontend, one B200 per container
- Modal deployment autoscales with no container cap, scales to zero after five idle minutes, and keeps weights in an openjev-huggingface Volume
- Warmups request one unused token probability to dodge SGLang's mixed-logprob batch crash without patching SGLang
- Separate Python FastAPI process exits when SGLang does, letting Modal replace dead containers; CUDA deps stay in SGLang's container
Related
Releases·Tool Release
Build a Jev-Style Decision Engine Locally with Qwen + SGLang
A Medium post describes building a Jev-style decision engine running locally with Qwen models served via SGLang. Instead of generating JSON, the approach scores known valid choices directly.
A Medium post describes building a Jev-style decision engine running locally with Qwen models served via SGLang. Instead of generating JSON, the approach scores known valid choices directly.
0 engagement1 sourcerssSat, Oct 10, 2026, 05:31 AM
Releases·Tool Release
LLM2Jev turns local LLMs into Jev-style structured decision engines with prefill-only binary inference
A new open-source project, LLM2Jev, adapts local Hugging Face-compatible causal language models into Jev-style structured decision engines, emitting typed Choice, Score, and Noul answers with probabilities. It uses prefill-only binary inference — scoring candidates with yes/no logit judgments without generating answer tokens — and leverages SGLang's Radix Cache for shared-prefix efficiency. The project is independent and unaffiliated with Jev or TypeSafe.
A new open-source project, LLM2Jev, adapts local Hugging Face-compatible causal language models into Jev-style structured decision engines, emitting typed Choice, Score, and Noul answers with probabilities. It uses prefill-only binary inference — scoring candidates with yes/no logit judgments without generating answer tokens — and leverages SGLang's Radix Cache for shared-prefix efficiency. The project is independent and unaffiliated with Jev or TypeSafe.
109 engagement1 sourcegithubSat, Sep 19, 2026, 10:59 AM
Social·Community
Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)
A user benchmark on a Qwen3.6-35B-A3B Q6 setup with 64K context on a 24GB RTX 3090 shows that manually tuning MoE expert offload to CPU (spilling eight layers) frees VRAM, allowing larger batch sizes and boosting prompt processing from 564.5 to 1330.0 tok/s (2.36×) while generation speed stays within noise. The result highlights the performance gains available from hand-tuning over auto-fit defaults.
A user benchmark on a Qwen3.6-35B-A3B Q6 setup with 64K context on a 24GB RTX 3090 shows that manually tuning MoE expert offload to CPU (spilling eight layers) frees VRAM, allowing larger batch sizes and boosting prompt processing from 564.5 to 1330.0 tok/s (2.36×) while generation speed stays within noise. The result highlights the performance gains available from hand-tuning over auto-fit defaults.
36 engagement1 sourceredditThu, Aug 6, 2026, 11:52 AM
Releases·Tool Release
Astorias AI launches cheap inference hosting Qwen3.8 27B, plans Agent Sandbox
Astorias AI launched an inference platform aimed at affordable pricing and reliable agent tooling, debuting with Qwen 3.8 27B. The founder shared pricing and solicited interest in a planned serverless "Agent Sandbox" feature. GLM 5.3 Flash and other flash models are planned next.
Astorias AI launched an inference platform aimed at affordable pricing and reliable agent tooling, debuting with Qwen 3.8 27B. The founder shared pricing and solicited interest in a planned serverless "Agent Sandbox" feature. GLM 5.3 Flash and other flash models are planned next.
4 engagement1 sourceredditTue, Sep 22, 2026, 11:51 AM
Releases·Tool Release
dstack toolkit boosts Qwen3.8-27B on MI300X from 311 to 495 tok/s
dstack's open-source agent toolkit optimized Qwen3.8-27B inference on a single AMD MI300X, increasing throughput by 59% from 311 to 495 tok/s at full 1M context. The gains came from linked optimization sessions and source-level patches to SGLang's AITER attention backend, producing a portable preset deployable across AMD clouds, Kubernetes, or bare metal.
dstack's open-source agent toolkit optimized Qwen3.8-27B inference on a single AMD MI300X, increasing throughput by 59% from 311 to 495 tok/s at full 1M context. The gains came from linked optimization sessions and source-level patches to SGLang's AITER attention backend, producing a portable preset deployable across AMD clouds, Kubernetes, or bare metal.
31 engagement1 sourceredditThu, Aug 20, 2026, 07:51 PM