A developer released Recursive BitNet N-Gram 102M, an experimental decoder-only model combining ternary BitNet weights, weight-shared recursive transformer layers, and hashed n-gram embeddings, trained on 4.7B tokens for a €100 compute budget. Weights, inference code, and evaluations are on Hugging Face. Results are modest (40.59% zero-shot average) but the author reports better-than-expected performance for the scale.
A developer trained a 250M-parameter LLM from scratch on 30B tokens of FineWeb, quantized to under 2 bits, achieving a 60 MB deployment that runs at 400 tok/s on a laptop CPU. The model uses a novel disk-based KV cache compression for long context, storing older tokens at 1 bit per token.
A developer trained a 250M-parameter LLM from scratch on 30B tokens of FineWeb, quantized to under 2 bits, achieving a 60 MB deployment that runs at 400 tok/s on a laptop CPU. The model uses a novel disk-based KV cache compression for long context, storing older tokens at 1 bit per token.
An independent developer completed the first full pretraining run of WarpState, an experimental non-Transformer language model architecture. The 150.13M-parameter model was trained from scratch on ~300 million English tokens from Ultra-FineWeb L2. The author frames it as a proof of concept, not a claim of superiority over Transformers or state-space models.
An independent developer completed the first full pretraining run of WarpState, an experimental non-Transformer language model architecture. The 150.13M-parameter model was trained from scratch on ~300 million English tokens from Ultra-FineWeb L2. The author frames it as a proof of concept, not a claim of superiority over Transformers or state-space models.
Solo researcher kkuette released Fractale-350M-base, a 386M-parameter base model pretrained from scratch on 10B tokens. The model implements a trained fast-weight memory as an alternative to long context, with full research logs and paper available on GitHub.
Solo researcher kkuette released Fractale-350M-base, a 386M-parameter base model pretrained from scratch on 10B tokens. The model implements a trained fast-weight memory as an alternative to long context, with full research logs and paper available on GitHub.
A 2nd-year B.Tech student posted a 14-page technical monograph on r/mlscaling proposing a 14.2B-parameter architecture (2.4B active per token) combining BitNet b1.58 ternary quantization, multi-head latent attention, fine-grained MoE, and a token-curation scheme. The design targets dense FP16 baseline parity at ~23.3x lower FLOPs and a $22k-$35k training budget. It is a theoretical proposal seeking community feedback, not a trained model or release.
A 2nd-year B.Tech student posted a 14-page technical monograph on r/mlscaling proposing a 14.2B-parameter architecture (2.4B active per token) combining BitNet b1.58 ternary quantization, multi-head latent attention, fine-grained MoE, and a token-curation scheme. The design targets dense FP16 baseline parity at ~23.3x lower FLOPs and a $22k-$35k training budget. It is a theoretical proposal seeking community feedback, not a trained model or release.
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens from FineWeb-Edu, then fine-tuned it with LoRA on OpenHermes to create a chat model, all for about $200. The project, shared on Reddit, demonstrates a low-cost path to building a custom LLM for learning and resume-building.
A developer trained a 1.1B-parameter LLM from scratch on 20B tokens from FineWeb-Edu, then fine-tuned it with LoRA on OpenHermes to create a chat model, all for about $200. The project, shared on Reddit, demonstrates a low-cost path to building a custom LLM for learning and resume-building.