Releases·Tool Release
Dev builds lm-eval-ledger, a benchmark harness with a web app to inspect per-question answers across models
A developer released lm-eval-ledger, a benchmark harness that runs LLM benchmarks and provides a web app for inspecting and comparing each model's answers per question. It addresses the gap where existing harnesses export raw JSONL/Parquet files with headline numbers but no tooling to review responses, saving practitioners from writing custom code to read results.
39 engagement1 sourceSun, Sep 6, 2026, 10:32 PM
AI Overview
- lm-eval-ledger logs per-question prompts, generations, extracted answers, ground truth, stop reasons, and per-benchmark accuracy, tok/s, and runtime to a SQLite DB viewed via a Flask app
- On a single 5090, Qwen3.5-9B scored 0.717 on GPQA Diamond vs Nemotron-3.5-Lightning-30B-A3B's 0.657 and Gemma-4-12B-it's 0.601, but took 2h26m vs 1h34m/1h19m
- On LiveCodeBench, Qwen took 22h57m with 0.713 accuracy vs 9h51m/0.837 and 6h13m/0.820 for Nemotron and Gemma
- The YAML-driven harness supports vLLM, SGLang, HF, and llama.cpp server backends on Linux and HF/llama.cpp on Windows, with pairwise model comparison views
Entities
Related
Developments·Benchmark
Opper benchmarks Kev 4B vs. Jev: matching accuracy, weaker calibration, up to 12x higher short-request cost
Opper ran Jared Palmer's Apache-2.0 Kev 4B, a fine-tune of Qwen3.5-4B, side by side with Jev on the same endpoint using 362 items published after both models shipped. Accuracy landed within 2 points on every task, though Jev showed better calibration and stronger paraphrase detection. Cost differences favored Kev on short requests because Jev adds roughly 257 fixed extra input tokens per request.
Opper ran Jared Palmer's Apache-2.0 Kev 4B, a fine-tune of Qwen3.5-4B, side by side with Jev on the same endpoint using 362 items published after both models shipped. Accuracy landed within 2 points on every task, though Jev showed better calibration and stronger paraphrase detection. Cost differences favored Kev on short requests because Jev adds roughly 257 fixed extra input tokens per request.
148 engagement1 sourceredditFri, Sep 25, 2026, 05:30 PM
Social·Community
Users benchmark Muse Glimmer 30B against Qwen 3.6 and Gemma 4
Local users are benchmarking the new Muse Glimmer 30B against Qwen 3.6 27B and Gemma 4 31B, finding it competitive in VRAM usage and long-context efficiency. Glimmer uses significantly less KV cache at 128K context, enabling full-context inference on a 24GB GPU. While Qwen remains stronger for coding, Glimmer is seen as a compelling agent/general-use model.
Local users are benchmarking the new Muse Glimmer 30B against Qwen 3.6 27B and Gemma 4 31B, finding it competitive in VRAM usage and long-context efficiency. Glimmer uses significantly less KV cache at 128K context, enabling full-context inference on a 24GB GPU. While Qwen remains stronger for coding, Glimmer is seen as a compelling agent/general-use model.
148 engagement2 sourcesredditTue, Aug 11, 2026, 08:10 PM
Releases·Tool Release
Developer releases smokebench, an open-source TUI for quick LLM benchmarking
A developer built smokebench, an open-source TUI for quickly benchmarking LLMs on local setups, addressing the gap between public leaderboards and real-world quantized or fine-tuned models. It supports any OpenAI or Anthropic compatible endpoint and tests across 8 categories including math, code, and creative writing.
A developer built smokebench, an open-source TUI for quickly benchmarking LLMs on local setups, addressing the gap between public leaderboards and real-world quantized or fine-tuned models. It supports any OpenAI or Anthropic compatible endpoint and tests across 8 categories including math, code, and creative writing.
2 engagement1 sourceredditFri, Aug 21, 2026, 09:55 AM
Products·launched
Domain-specific benchmark collection for local LLMs
A website hosting 30+ small, domain-specific benchmarks for evaluating local language models. It allows users to compare model performance on specific tasks like common sense, with results tables and average scores per query.
A website hosting 30+ small, domain-specific benchmarks for evaluating local language models. It allows users to compare model performance on specific tasks like common sense, with results tables and average scores per query.
30 engagement1 sourceredditSat, Aug 1, 2026, 09:20 PM
Products·launched
LLM comparison dashboard for quality, latency, and cost
A dashboard that lets users test LLMs on their own data, comparing quality, latency, and cost side by side. It runs on Nebius Serverless and helps developers choose the best model for their specific use case rather than relying on leaderboards.
A dashboard that lets users test LLMs on their own data, comparing quality, latency, and cost side by side. It runs on Nebius Serverless and helps developers choose the best model for their specific use case rather than relying on leaderboards.
0 engagement1 sourcerssSun, Jul 12, 2026, 05:25 AM