A new arXiv paper introduces WaterBERT, an encoder model continually pretrained on ~2.97 billion tokens of water treatment literature for semantic representation and structured information extraction. Fine-tuned variants beat general-purpose and domain-specific BERT models on classification, NER, and relation extraction, and the team built a water treatment knowledge graph plus a knowledge-enhanced retrieval system (WaterKERS) at far lower cost than commercial LLMs.
An arXiv paper tests whether continued pretraining on Russian legislative documents improves a Russian ModernBERT encoder on legal text. The adapted model, RuModernBERT-ruLaw, shows lower masked-token loss at long input lengths, though the authors caution the results don't establish practical legal utility. The paper emphasizes transparent evaluation methodology with editable diagrams.
An arXiv paper tests whether continued pretraining on Russian legislative documents improves a Russian ModernBERT encoder on legal text. The adapted model, RuModernBERT-ruLaw, shows lower masked-token loss at long input lengths, though the authors caution the results don't establish practical legal utility. The paper emphasizes transparent evaluation methodology with editable diagrams.
Researchers introduce Climate-ModernBERT, a family of climate-adapted encoder models built by continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. The work targets climate-domain NLP over heterogeneous sources (scientific literature, policy disclosures, synthetic reports) and studies how to combine diverse domain corpora during continued pretraining, an underexplored question.
Researchers introduce Climate-ModernBERT, a family of climate-adapted encoder models built by continued pretraining of ModernBERT-Base on three climate corpora: academic climate text, climate-filtered web data, and synthetic climate documents. The work targets climate-domain NLP over heterogeneous sources (scientific literature, policy disclosures, synthetic reports) and studies how to combine diverse domain corpora during continued pretraining, an underexplored question.
A thesis examines how to select and adapt NLP models for global health literature when annotated data and compute are limited. It benchmarks word embeddings, NER models, and few-shot vs zero-shot topic classifiers, quantifying accuracy-cost trade-offs. The work offers an empirical basis for building document-indexing knowledge systems under resource constraints.
A thesis examines how to select and adapt NLP models for global health literature when annotated data and compute are limited. It benchmarks word embeddings, NER models, and few-shot vs zero-shot topic classifiers, quantifying accuracy-cost trade-offs. The work offers an empirical basis for building document-indexing knowledge systems under resource constraints.
An arXiv paper introduces TransBERT, a framework for pre-training language models exclusively on synthetically translated text, plus TransCorpus, a translation toolkit. Focusing on French life sciences, it shows state-of-the-art downstream performance using only synthetic data. The release targets the scarcity of non-English data in specialized domains.
An arXiv paper introduces TransBERT, a framework for pre-training language models exclusively on synthetically translated text, plus TransCorpus, a translation toolkit. Focusing on French life sciences, it shows state-of-the-art downstream performance using only synthetic data. The release targets the scarcity of non-English data in specialized domains.
A new arXiv paper introduces a contextual token-level semantic graph representation to improve neural topic model interpretability on specialized corpora. The approach addresses embedding-space collapse of domain-specific terms without relying on domain-specific re-training or word-level graph enrichment.
A new arXiv paper introduces a contextual token-level semantic graph representation to improve neural topic model interpretability on specialized corpora. The approach addresses embedding-space collapse of domain-specific terms without relying on domain-specific re-training or word-level graph enrichment.