Releases·Tool Release
OpenDCAI releases DataFlow, an open-source tool for pre-training data cleaning and synthesis
OpenDCAI released DataFlow, an open-source tool that automates large-scale data cleaning and synthesis before pre-training. It composes reusable operators into pipelines combining rule-based filters, model-based scoring, and LLM-based synthesis, letting practitioners replace or reorder steps without rewriting workflows.
7 engagement1 sourceFri, Sep 18, 2026, 10:17 AM
AI Overview
- DataFlow chains rule-based filters, model-based evaluation, and LLM-based synthesis via reusable operators storing intermediate results
- Typical pipelines run HTML cleanup, language/length checks, blocklist filtering, MinHash dedup, quality scoring, and QA/SFT sample generation
- Operators can be replaced, reordered, or extended without rewriting the pipeline
Related
Developments·Paper
DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
DataOrchestra is a framework that learns to apply different processing operations per example in pretraining data, rather than using a fixed corpus-level strategy. It decides for each chunk whether to drop, keep, or clean it, and selects specific cleaning operations when needed. This adaptive approach aims to improve downstream LLM performance.
DataOrchestra is a framework that learns to apply different processing operations per example in pretraining data, rather than using a fixed corpus-level strategy. It decides for each chunk whether to drop, keep, or clean it, and selects specific cleaning operations when needed. This adaptive approach aims to improve downstream LLM performance.
0 engagement1 sourcearxivMon, Jul 27, 2026, 05:54 PM
Developments·Paper
CMU's ReScraper: 0.6B Model Replaces Heuristic Scraping and Cleaning for LLM Pretraining Data
A new arXiv paper introduces ReScraper, a 0.6B-parameter language model that unifies web scraping and cleaning for LLM pretraining corpora, replacing hand-written heuristic scrapers and dozens of rule-based filters. Pretraining 400M, 1.4B, and 2.8B models on its curated data improves DCLM Core score by 3.8–4.7% relative over the strongest baseline at each scale. The authors frame it as a demonstration of AI4AI — a small learned model taking over an entire data-curation pipeline stage — and open-source the code.
A new arXiv paper introduces ReScraper, a 0.6B-parameter language model that unifies web scraping and cleaning for LLM pretraining corpora, replacing hand-written heuristic scrapers and dozens of rule-based filters. Pretraining 400M, 1.4B, and 2.8B models on its curated data improves DCLM Core score by 3.8–4.7% relative over the strongest baseline at each scale. The authors frame it as a demonstration of AI4AI — a small learned model taking over an entire data-curation pipeline stage — and open-source the code.
0 engagement1 sourcearxivMon, Sep 28, 2026, 04:31 AM
Developments·Paper
AutoData: Agentic Search for Pre-training Data Selection
Researchers introduce AutoData, an LLM agent that frames pre-training data selection as heuristic engineering over per-document features, searching executable scoring, stratification, and stochastic selection rules rather than domain weights. Guided by proxy-model validation feedback, it discovers a selection algorithm in an overnight search that beats human-designed curation pipelines and transfers to larger scales, improving the CORE metric.
Researchers introduce AutoData, an LLM agent that frames pre-training data selection as heuristic engineering over per-document features, searching executable scoring, stratification, and stochastic selection rules rather than domain weights. Guided by proxy-model validation feedback, it discovers a selection algorithm in an overnight search that beats human-designed curation pipelines and transfers to larger scales, improving the CORE metric.
0 engagement1 sourcearxivThu, Sep 17, 2026, 06:24 AM
Social·Community
From PDF Archives to a Trainable Model: A Practical Open-Source Workflow
A Reddit post lays out a repeatable open-source pipeline for converting PDF archives into supervised fine-tuning data. It uses OpenDCAI/DataFlow for data preparation and OpenDCAI/DataFlex as the training backend, arguing that 'dynamic training' means validated incremental updates, not blind weight updates on upload.
A Reddit post lays out a repeatable open-source pipeline for converting PDF archives into supervised fine-tuning data. It uses OpenDCAI/DataFlow for data preparation and OpenDCAI/DataFlex as the training backend, arguing that 'dynamic training' means validated incremental updates, not blind weight updates on upload.
1 engagement1 sourceredditWed, Sep 30, 2026, 06:54 AM
Developments·Paper
Researchers propose frequency- and length-aware subdocument deduplication for LLM pretraining
A new arXiv paper introduces a scalable subdocument deduplication method for LLM pretraining corpora that is frequency- and length-aware, addressing cross-shard duplicates missed by suffix-array methods and rigid retention policies of hash-based approaches. The method aims to improve data quality and training efficiency.
A new arXiv paper introduces a scalable subdocument deduplication method for LLM pretraining corpora that is frequency- and length-aware, addressing cross-shard duplicates missed by suffix-array methods and rigid retention policies of hash-based approaches. The method aims to improve data quality and training efficiency.
0 engagement1 sourcearxivTue, Aug 4, 2026, 04:02 AM