Releases·Tool Release
ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines
A new open-source Python library, ChunkRank, derives text chunk boundaries from a target model's tokenizer and context window and selects answers among candidates produced per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, with only three core dependencies. The paper also reports a negative result on answer ranking and releases code, the model registry, and an evaluation harness.
0 engagement1 sourceThu, Sep 24, 2026, 01:59 PM
AI Overview
- ChunkRank derives chunk boundaries from a target model's tokenizer and context window, shipping a registry of 90 models across 15 providers and six answer-selection methods
- A fidelity study across 11 languages shows token-exact budgets matter beyond English, unlike character-based splitters that overflow or waste context
- A negative result: across NaturalQuestions, TriviaQA and HotpotQA, no content-based ranker reliably beats taking the first non-empty answer, attributing failures to reader abstention
- Chunking matches single-call reading on single-hop questions, so ChunkRank targets small-window and beyond-window settings
Related
Developments·Paper
Right Reset: Chunking by Prefix Removal
A new arXiv paper introduces Right Reset (RR), a method that uses causal language model hidden states to find chunk boundaries by measuring how much the right-hand trajectory changes when left context is removed. On flattened text of topically similar records, RR recovers 47.7% of original records as clean units, outperforming BGE embeddings (25.9%).
A new arXiv paper introduces Right Reset (RR), a method that uses causal language model hidden states to find chunk boundaries by measuring how much the right-hand trajectory changes when left context is removed. On flattened text of topically similar records, RR recovers 47.7% of original records as clean units, outperforming BGE embeddings (25.9%).
0 engagement1 sourcearxivWed, Aug 5, 2026, 01:17 AM
Releases·Tool Release
chunklet-py adds SelfTuningChunker that learns chunk sizes from your data
The developer of chunklet-py, a text/document/code chunker for RAG pipelines and LLM apps, announced a new version featuring SelfTuningChunker. It classifies inputs as document or code, maintains learned per-type profiles, and sizes chunks from learned metrics, removing manual tuning. Practitioners chunking mixed corpora can reuse learned state across runs via an initial_state parameter.
The developer of chunklet-py, a text/document/code chunker for RAG pipelines and LLM apps, announced a new version featuring SelfTuningChunker. It classifies inputs as document or code, maintains learned per-type profiles, and sizes chunks from learned metrics, removing manual tuning. Practitioners chunking mixed corpora can reuse learned state across runs via an initial_state parameter.
2 engagement1 sourceredditThu, Oct 1, 2026, 11:41 PM
Developments·Paper
Paper identifies 'referential dangling' failure in hard prompt compression
A new arXiv paper describes a structural failure mode in hard prompt compression, where independently scoring and retaining text units can split dependent evidence pairs, leaving answers without the context needed to interpret them. The authors demonstrate this 'referential dangling' in Beaver, a compression method using Qwen3-0.6B embeddings, and argue it is a paradigm-level issue.
A new arXiv paper describes a structural failure mode in hard prompt compression, where independently scoring and retaining text units can split dependent evidence pairs, leaving answers without the context needed to interpret them. The authors demonstrate this 'referential dangling' in Beaver, a compression method using Qwen3-0.6B embeddings, and argue it is a paradigm-level issue.
0 engagement1 sourcearxivWed, Aug 5, 2026, 07:58 AM
Social·Community
RAG pipeline tuning: balancing retrieved, reranked, and final chunk counts
A developer reports that optimizing a RAG pipeline requires balancing three chunk-count variables: retrieved, reranked, and final chunks sent to the LLM. Limiting the LLM context to 10 chunks causes failures on global queries like 'List ALL names in the document' because relevant information is cut off.
A developer reports that optimizing a RAG pipeline requires balancing three chunk-count variables: retrieved, reranked, and final chunks sent to the LLM. Limiting the LLM context to 10 chunks causes failures on global queries like 'List ALL names in the document' because relevant information is cut off.
5 engagement1 sourceredditWed, Aug 12, 2026, 06:57 AM
Releases·Tool Release
Developer benchmarks fixed-budget RAG context selection on 250 QASPER questions
A developer built a hosted Context Compiler that ranks and deduplicates retrieved passages under a fixed token budget, and benchmarked it on 250 QASPER questions. The tool aims to prevent silent loss of evidence when retrievers return more material than the model's context window allows. The developer is seeking technical criticism from practitioners running real RAG and agent workflows.
A developer built a hosted Context Compiler that ranks and deduplicates retrieved passages under a fixed token budget, and benchmarked it on 250 QASPER questions. The tool aims to prevent silent loss of evidence when retrievers return more material than the model's context window allows. The developer is seeking technical criticism from practitioners running real RAG and agent workflows.
1 engagement1 sourceredditSun, Aug 9, 2026, 06:04 AM