Releases·Tool Release
VisTW: a Traditional Chinese vision benchmark that tests whether your VLM can actually read Taiwan
The Twinkle Eval open-source framework added support for VisTW (arXiv 2503.10427), a Traditional Chinese vision-language benchmark from NTU MiuLab testing reading of Taiwanese road signs, receipts, and textbook diagrams. Cross-validation against the official implementation surfaced an answer-extraction bug that had been suppressing scores across five vision benchmarks.
11 engagement1 sourceWed, Sep 16, 2026, 03:20 AM
AI Overview
- VisTW (NTU MiuLab, arXiv 2503.10427) comprises VisTW-MCQ (4,770 multiple-choice questions from real exams across 21 subjects) and VisTW-Dialogue (141 open-ended questions scored 0-10 by an LLM judge).
- Its leaderboard puts o3 top at 0.7769 MCQ / 6.9878 Dialogue, gemma-3-12b-it at 0.4863 / 3.94, unchanged since April 2025.
- Twinkle Eval v2.10.0 added VisTW support; on 253 proportionally sampled questions it scored 81.01% vs the official implementation's 80.56%.
- Twinkle's developers found an extractor bug, triggered by the official prompt's answer format '答案: $字母', that suppressed scores across five vision benchmarks, including a 36-point miss on one subject.
- Twinkle omits the official implementation's LLM-based response-parsing stage, reporting unparsed_rate instead; reasoning VLMs need 8,192 max_tokens to avoid 19-point accuracy losses from truncation.
Related
Developments·Benchmark
OpenTWBench: open Taiwan-domain benchmark suite with 255,854-exam-question eval and leaderboard
Researchers introduce OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in LLMs, comprising tw-exam-bench (253,674 curated questions from fifteen years of national examinations across 148 subjects) and tw-formosa-bench (1,696 difficulty-curated questions on everyday Taiwan). The paper also quantifies how evaluation harness choices—answer extraction, prompt phrasing, judge panels—dominate reported scores. All datasets, manifests, and a leaderboard are released at opentwbench.ai.
Researchers introduce OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in LLMs, comprising tw-exam-bench (253,674 curated questions from fifteen years of national examinations across 148 subjects) and tw-formosa-bench (1,696 difficulty-curated questions on everyday Taiwan). The paper also quantifies how evaluation harness choices—answer extraction, prompt phrasing, judge panels—dominate reported scores. All datasets, manifests, and a leaderboard are released at opentwbench.ai.
0 engagement1 sourcearxivWed, Oct 14, 2026, 09:18 AM
Developments·Benchmark
BEAR-Bench: A Bilingual Enterprise and Academic Reasoning Benchmark for Multimodal Models
Researchers introduce BEAR-Bench, a bilingual benchmark for evaluating multimodal LLMs on reasoning over text-dense professional documents, addressing gaps in existing benchmarks that focus on extraction or require external knowledge. It covers English and Russian, underrepresented in current evaluations.
Researchers introduce BEAR-Bench, a bilingual benchmark for evaluating multimodal LLMs on reasoning over text-dense professional documents, addressing gaps in existing benchmarks that focus on extraction or require external knowledge. It covers English and Russian, underrepresented in current evaluations.
0 engagement1 sourcearxivTue, Aug 18, 2026, 03:29 PM
Developments·Benchmark
MLLMs benchmarked on scientific visualization literacy test
A new benchmark evaluates six multimodal large language models on the Scientific Visualization Literacy Assessment Test, a standardized test of 49 items covering 8 visualization techniques and 11 task types. The study includes three closed-source and three open-source models under a closed-world protocol, aiming to assess MLLMs' understanding of scientific visualizations beyond simple charts.
A new benchmark evaluates six multimodal large language models on the Scientific Visualization Literacy Assessment Test, a standardized test of 49 items covering 8 visualization techniques and 11 task types. The study includes three closed-source and three open-source models under a closed-world protocol, aiming to assess MLLMs' understanding of scientific visualizations beyond simple charts.
0 engagement1 sourcearxivThu, Jul 16, 2026, 04:29 PM
Developments·Paper
arXiv paper identifies 'Visual Insensitivity Gap': VLMs often ignore question-relevant visual evidence
An arXiv paper shows vision-language models frequently answer without using their visual input: blurring the question-relevant image region barely changes the next-token distribution on 40%-97% of samples across six VLMs and three perceptual benchmarks. The authors introduce a per-sample Visual Sensitivity Index (VSI) to diagnose when models rely on visual evidence versus priors. The finding matters to practitioners because aggregate benchmark accuracy can mask samples where models effectively ignore the image.
An arXiv paper shows vision-language models frequently answer without using their visual input: blurring the question-relevant image region barely changes the next-token distribution on 40%-97% of samples across six VLMs and three perceptual benchmarks. The authors introduce a per-sample Visual Sensitivity Index (VSI) to diagnose when models rely on visual evidence versus priors. The finding matters to practitioners because aggregate benchmark accuracy can mask samples where models effectively ignore the image.
0 engagement1 sourcearxivTue, Sep 1, 2026, 08:01 AM
Developments·Benchmark
OmniMapBench benchmark tests LVLMs on visual-centric map reasoning
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
Researchers introduced OmniMapBench, a benchmark of 2,096 QA pairs across 1,603 map documents from nine categories, designed to evaluate visual-centric reasoning in large vision-language models. It addresses the limitation that many document benchmarks allow high performance via text-only cues, requiring genuine visual grounding for tasks from perception to multi-step reasoning.
0 engagement1 sourcearxivFri, Jul 10, 2026, 03:27 AM