OpenDecider releases Apache-2.0 distilled decision models (400M–80B-A3B): calibration and human agreement improve with size, accuracy doesn't

The ufakzeka lab released ufakzeka-karar, an open Turkish typed-decision model with 182,494,466 parameters, as an arXiv paper with Apache-2.0 weights and code. Instead of generating text, it scores up to ten fixed options in one CPU forward pass, returning per-option probabilities and an expected-error 'not sure' signal. It ranks 7th of 16 on HakemBench v1.0's open set, and its authors disclose that its benchmark numbers were shaped by reading earlier runs' test results.

An arXiv paper examines correctness-gated multi-teacher distillation in a fixed experiment, showing candidate decision correctness and rationale grounding are different objectives. The study reports lost label functionality, decision shifts, and an inconclusive grounding audit across eight experimental arms.

A new arXiv paper studies whether small language models' verbalized confidence can support risk-controlled deferral to humans. Evaluating eleven instruction-tuned models from 0.5B to 14B parameters on ARC-Challenge and TruthfulQA, it proves theoretical limits on what calibration can provide and proposes certified deferral methods.

A new paper proposes Group-Calibrated On-Policy Distillation (GCPD) to fix a mismatch in on-policy distillation for long-context reasoning. Token-level teacher guidance can favor locally plausible but globally wrong responses; GCPD uses group-level calibration to align with task-level rewards. The method is validated on long-context evidence-aggregation tasks.

Social·Community

TrustifAI cuts Jev judge's calibration error 68.1% on TRIVIA+ via human-labelled training

A developer benchmarked calibration of their LLM judge 'Jev' on a 645-example TRIVIA+ test set, reducing expected calibration error from 0.0982 to 0.0313 after learning from human-labelled examples — a 68.1% reduction. Hallucination-detection F1 barely moved (0.5833 to 0.5877), showing the improvement was in confidence alignment, not classification ability. The results inform Typed Evals, their open-source library that builds in calibration rather than trusting raw judge confidence, relevant for practitioners setting production thresholds like auto-approve or escalate-to-human cutoffs.

A developer benchmarked calibration of their LLM judge 'Jev' on a 645-example TRIVIA+ test set, reducing expected calibration error from 0.0982 to 0.0313 after learning from human-labelled examples — a 68.1% reduction. Hallucination-detection F1 barely moved (0.5833 to 0.5877), showing the improvement was in confidence alignment, not classification ability. The results inform Typed Evals, their open-source library that builds in calibration rather than trusting raw judge confidence, relevant for practitioners setting production thresholds like auto-approve or escalate-to-human cutoffs.