LaconIQ open-sources TekMyra, a RAG context compressor that protects citation spans (Apache-2.0)

A developer built a hosted Context Compiler that ranks and deduplicates retrieved passages under a fixed token budget, and benchmarked it on 250 QASPER questions. The tool aims to prevent silent loss of evidence when retrievers return more material than the model's context window allows. The developer is seeking technical criticism from practitioners running real RAG and agent workflows.

MIT researchers released densely, a lossless context compression method for LLM agents that reduces token usage by 2–8x while enabling byte-exact recovery. It re-encodes compressed bytes as 65,536 English words, each costing exactly one token, achieving 16 bits per token versus ~8.8 for lzma+base64.

Social·Community

Developer measures 49.7% duplicate embedding work in RAG pipelines, proposes chunk-hash fix

A developer measured embedding costs in a real RAG pipeline over 103 Wikipedia articles and found 49.7% of embedding tokens were duplicate work from unchanged text. Fixed-size chunking made exact-match caches ineffective because single edits shift all subsequent chunk boundaries. The post proposes a content-addressed chunking approach to avoid re-embedding unchanged content.

A developer measured embedding costs in a real RAG pipeline over 103 Wikipedia articles and found 49.7% of embedding tokens were duplicate work from unchanged text. Fixed-size chunking made exact-match caches ineffective because single edits shift all subsequent chunk boundaries. The post proposes a content-addressed chunking approach to avoid re-embedding unchanged content.

Releases·Tool Release

Secondwind's open-source compression improved by community pull request within a week

Secondwind, a lossless context compression tool for coding agents, went open source and within a week received a community pull request that outperformed the original implementation. The tool compresses all agent-to-model data with cryptographic proof of no data loss, contrasting with typical offloading approaches that add retrieval steps.

Secondwind, a lossless context compression tool for coding agents, went open source and within a week received a community pull request that outperformed the original implementation. The tool compresses all agent-to-model data with cryptographic proof of no data loss, contrasting with typical offloading approaches that add retrieval steps.

Supercompress is a query-aware context compression tool for LLMs and coding agents that reduces input tokens by roughly 65% while preserving evidence. It targets the cost of irrelevant context in RAG chunks, chat history, tool dumps, and logs.