Releases·Tool Release
Hugging Face releases tokenizers v1 with multi-language support, multi-thread scaling, minimal package size
Hugging Face engineer Aritra announced the release of version 1.0 of the tokenizers library, a Rust-based tokenizer package. The release brings multiple language support, multi-thread scaling, and a minimal package size. For practitioners, v1 signals API stability for a core preprocessing dependency used across model training and inference pipelines.
101 engagement2 sourcesMon, Sep 21, 2026, 03:10 PM
AI Overview
- Hugging Face's Aritra announced tokenizers v1 on Reddit, highlighting multiple language support, multi-thread scaling, and minimal package size
- The tokenizers v1 update is detailed in Hugging Face's blog post at huggingface.co/blog/tokenizers-v1
Entities
Related
Releases·Tool Release
Gigatoken: Open-source tokenizer claims 100x speedup over Tiktoken, 500-1000x over Hugging Face
A new open-source tokenizer called Gigatoken claims to be approximately 100 times faster than OpenAI's Tiktoken and 500 to 1000 times faster than Hugging Face tokenizers. The project is hosted on GitHub and aims to accelerate tokenization for large language model inference and training.
A new open-source tokenizer called Gigatoken claims to be approximately 100 times faster than OpenAI's Tiktoken and 500 to 1000 times faster than Hugging Face tokenizers. The project is hosted on GitHub and aims to accelerate tokenization for large language model inference and training.
157 engagement2 sourcesreddit, hackernewsTue, Jul 21, 2026, 11:08 PM
Developments·Paper
Paper analyzes tokens learned when tokenization is jointly optimized with language modeling
A new arXiv paper investigates what tokens emerge when tokenization is jointly optimized with language modeling, comparing tokenizer-free approaches (SSLMs, H-Nets) against fixed tokenizers across 18 languages. The authors find joint optimization fundamentally alters token structure, with SSLMs recovering morphologically aligned and contextually efficient tokens.
A new arXiv paper investigates what tokens emerge when tokenization is jointly optimized with language modeling, comparing tokenizer-free approaches (SSLMs, H-Nets) against fixed tokenizers across 18 languages. The authors find joint optimization fundamentally alters token structure, with SSLMs recovering morphologically aligned and contextually efficient tokens.
0 engagement1 sourcearxivTue, Aug 18, 2026, 03:34 AM
Releases·Tool Release
Developer builds tokenizer comparison tool for multilingual LLM use
A developer created a tokenizer benchmark tool that compares token counts across languages for different LLM tokenizers. The tool shows that GPT-2 uses 722 tokens for 100 Punjabi words vs 100 for English, while o200k_base uses only 215 for Punjabi, highlighting large disparities in multilingual tokenization efficiency.
A developer created a tokenizer benchmark tool that compares token counts across languages for different LLM tokenizers. The tool shows that GPT-2 uses 722 tokens for 100 Punjabi words vs 100 for English, while o200k_base uses only 215 for Punjabi, highlighting large disparities in multilingual tokenization efficiency.
2 engagement1 sourceredditThu, Jul 16, 2026, 01:16 PM
Releases·Model Release
Community discovers 'grug-27b' model claiming 90% token reduction over Qwen3.6 27B
A user on Reddit spotted a Hugging Face model called 'grug-27b' by ProCreations, which claims to outperform the original Qwen3.6 27B while reducing required tokens by over 90%. If verified, this could significantly improve inference speed on consumer hardware, making a 27B model feel like 30 tokens per second on an old laptop.
A user on Reddit spotted a Hugging Face model called 'grug-27b' by ProCreations, which claims to outperform the original Qwen3.6 27B while reducing required tokens by over 90%. If verified, this could significantly improve inference speed on consumer hardware, making a 27B model feel like 30 tokens per second on an old laptop.
339 engagement1 sourceredditThu, Jul 23, 2026, 06:17 AM
Developments·Paper
Paper shows GPT-2 regex in HuggingFace pre-tokenizer inflates token fertility for abugida scripts
An arXiv paper identifies a structural flaw in byte-level BPE tokenizers using HuggingFace's ByteLevel pre-tokenizer: GPT-2's word regex defines words as Unicode letters (\p{L}+), so abugida vowel signs — combining marks, not letters — split words at every vowel. Because BPE merges only within pre-tokens, the splits persist through training regardless of vocabulary size. The authors formalize this as a training-free lower bound on fertility, affecting all 17 abugidas tested.
An arXiv paper identifies a structural flaw in byte-level BPE tokenizers using HuggingFace's ByteLevel pre-tokenizer: GPT-2's word regex defines words as Unicode letters (\p{L}+), so abugida vowel signs — combining marks, not letters — split words at every vowel. Because BPE merges only within pre-tokens, the splits persist through training regardless of vocabulary size. The authors formalize this as a training-free lower bound on fertility, affecting all 17 abugidas tested.
0 engagement1 sourcearxivWed, Aug 26, 2026, 10:56 PM