Releases·Tool Release
chunklet-py adds SelfTuningChunker that learns chunk sizes from your data
The developer of chunklet-py, a text/document/code chunker for RAG pipelines and LLM apps, announced a new version featuring SelfTuningChunker. It classifies inputs as document or code, maintains learned per-type profiles, and sizes chunks from learned metrics, removing manual tuning. Practitioners chunking mixed corpora can reuse learned state across runs via an initial_state parameter.
2 engagement1 sourceThu, Oct 1, 2026, 11:41 PM
AI Overview
- SelfTuningChunker classifies inputs as document or code and keeps separate profiles, updating metrics like sentences per paragraph and header density via Kaufman Adaptive Moving Averages.
- Learned profiles are inspectable via learned_state and reusable: pass a saved baseline as initial_state to skip warm-up, with hard_token_limit capping learned max_tokens.
- Missing metrics fall back to defaults (partial baselines fine), and each instance deep-copies passed state; a token_counter is required or max_tokens is ignored.
Related
Releases·Tool Release
chunklet-py v2.4.0 adds EML and PPTX support, swaps sentence boundary detector
chunklet-py v2.4.0 ships with new EML and PPTX chunking support, and replaces pysbd/sentsplit with the custom yasbd-lib for faster sentence boundary detection. The rule-based library avoids heavy dependencies like tree-sitter.
chunklet-py v2.4.0 ships with new EML and PPTX chunking support, and replaces pysbd/sentsplit with the custom yasbd-lib for faster sentence boundary detection. The rule-based library avoids heavy dependencies like tree-sitter.
2 engagement2 sourcesredditMon, Jul 27, 2026, 06:44 PM
Social·Community
Developer measures 49.7% duplicate embedding work in RAG pipelines, proposes chunk-hash fix
A developer measured embedding costs in a real RAG pipeline over 103 Wikipedia articles and found 49.7% of embedding tokens were duplicate work from unchanged text. Fixed-size chunking made exact-match caches ineffective because single edits shift all subsequent chunk boundaries. The post proposes a content-addressed chunking approach to avoid re-embedding unchanged content.
A developer measured embedding costs in a real RAG pipeline over 103 Wikipedia articles and found 49.7% of embedding tokens were duplicate work from unchanged text. Fixed-size chunking made exact-match caches ineffective because single edits shift all subsequent chunk boundaries. The post proposes a content-addressed chunking approach to avoid re-embedding unchanged content.
9 engagement1 sourceredditWed, Jul 15, 2026, 04:21 PM
Releases·Tool Release
ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines
A new open-source Python library, ChunkRank, derives text chunk boundaries from a target model's tokenizer and context window and selects answers among candidates produced per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, with only three core dependencies. The paper also reports a negative result on answer ranking and releases code, the model registry, and an evaluation harness.
A new open-source Python library, ChunkRank, derives text chunk boundaries from a target model's tokenizer and context window and selects answers among candidates produced per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, with only three core dependencies. The paper also reports a negative result on answer ranking and releases code, the model registry, and an evaluation harness.
0 engagement1 sourcearxivThu, Sep 24, 2026, 01:59 PM
Releases·Tool Release
Dev builds Chunkr, a Rust chunking library claiming ~20x speedups over LangChain and LlamaIndex
A developer released Chunkr, an open-source Rust chunking library supporting strategies like recursive, markdown header, late, and hierarchical chunking, plus native PDF loading. Benchmarks posted on an M4 MacBook show throughputs up to 3,232 MB/s and PDF extraction 15.9x faster than PyPDF, though Chonkie wins on BPE tokenization.
A developer released Chunkr, an open-source Rust chunking library supporting strategies like recursive, markdown header, late, and hierarchical chunking, plus native PDF loading. Benchmarks posted on an M4 MacBook show throughputs up to 3,232 MB/s and PDF extraction 15.9x faster than PyPDF, though Chonkie wins on BPE tokenization.
4 engagement1 sourceredditMon, Oct 5, 2026, 06:07 PM
Products·launched
DocSlicer: fast local document parser & chunker for RAG pipelines
DocSlicer is a lightweight, deterministic document parser and chunker for local agent/RAG pipelines. It converts PDF, DOCX, PPTX, and HTML into chunks, structured tables, and a heading hierarchy, processing 100-page files in about 2 seconds without models or GPU. It solves the parsing bottleneck for developers handling large financial and legal documents.
DocSlicer is a lightweight, deterministic document parser and chunker for local agent/RAG pipelines. It converts PDF, DOCX, PPTX, and HTML into chunks, structured tables, and a heading hierarchy, processing 100-page files in about 2 seconds without models or GPU. It solves the parsing bottleneck for developers handling large financial and legal documents.
1 engagement1 sourceredditTue, Jul 28, 2026, 05:30 PM