A new arXiv paper introduces QuanReview, an open-source system that aligns human and LLM span annotations at character level, applies logged policy decisions to unambiguous cases, and routes conflicts to a browser-based adjudication interface. It targets the practical problem of keeping structured annotation layers trustworthy as language models enter annotation pipelines. On a 4,457-record humanitarian benchmark, it auto-merged or policy-resolved most records and concentrated human review on 3,131 candidate conflicts.
EviStreams is an open-source, no-code web platform where medical review teams control LLM-assisted data extraction across systematic review stages. It fits established protocols (dual independent extraction, adjudication) while keeping an auditable record of every extracted value. It targets the expert-labor bottleneck in the data-extraction step of clinical guideline underpinning reviews.
EviStreams is an open-source, no-code web platform where medical review teams control LLM-assisted data extraction across systematic review stages. It fits established protocols (dual independent extraction, adjudication) while keeping an auditable record of every extracted value. It targets the expert-labor bottleneck in the data-extraction step of clinical guideline underpinning reviews.
A new arXiv paper presents QUORUM (QUality-Optimized Routing Using Multiple annotators), a framework that dynamically assigns each annotation instance to either a human or an LLM under a budget constraint. It targets the reliability gap where LLMs handle simple inputs well but fail on nuanced reasoning. The work matters to practitioners scaling data labeling pipelines who need to balance cost against label quality.
A new arXiv paper presents QUORUM (QUality-Optimized Routing Using Multiple annotators), a framework that dynamically assigns each annotation instance to either a human or an LLM under a budget constraint. It targets the reliability gap where LLMs handle simple inputs well but fail on nuanced reasoning. The work matters to practitioners scaling data labeling pipelines who need to balance cost against label quality.
A research paper proposes using cross-LLM disagreement to surface cases needing expert attention when revising annotation codebooks. In experiments on tutoring-session transcripts, expert feedback elicited via rationale labeling reached 64.9% LLM-labeling accuracy versus 57.8% for an expert-revised codebook. The approach cuts months of codebook development to days, reducing expert workload without sacrificing labeling performance.
A research paper proposes using cross-LLM disagreement to surface cases needing expert attention when revising annotation codebooks. In experiments on tutoring-session transcripts, expert feedback elicited via rationale labeling reached 64.9% LLM-labeling accuracy versus 57.8% for an expert-revised codebook. The approach cuts months of codebook development to days, reducing expert workload without sacrificing labeling performance.
Researchers introduce onPanda, an interactive tool for annotating LLM alignment data and agent trajectories via token-level correction. Annotators fix the first inappropriate token in a model response and let generation continue from the corrected prefix, producing data that stays on-policy while cutting annotation time. They also release Panda-CVL, an annotated dataset, plus a token-level correction benchmark.
Researchers introduce onPanda, an interactive tool for annotating LLM alignment data and agent trajectories via token-level correction. Annotators fix the first inappropriate token in a model response and let generation continue from the corrected prefix, producing data that stays on-policy while cutting annotation time. They also release Panda-CVL, an annotated dataset, plus a token-level correction benchmark.
A new arXiv paper presents ContraVis, a human-in-the-loop visual analytics system for finding contradictions in legal contracts. It represents contracts as typed paragraph graphs that both condition LLM reasoning and serve as the analyst's interactive interface. In controlled tests, graph-conditioned reasoning recovered more injected contradictions than standalone LLM analysis as contract length grew.
A new arXiv paper presents ContraVis, a human-in-the-loop visual analytics system for finding contradictions in legal contracts. It represents contracts as typed paragraph graphs that both condition LLM reasoning and serve as the analyst's interactive interface. In controlled tests, graph-conditioned reasoning recovered more injected contradictions than standalone LLM analysis as contract length grew.