A developer trained Apex-3, a 20.99B-parameter MoE with 3.36B active parameters per token, entirely from scratch on 70B tokens using GH200 GPUs. The training-log analysis found one clear phase transition — an in-context-learning jump around 1.3B–1.5B tokens — while arithmetic gains looked abrupt only due to metric artifacts. Model checkpoints and training notes are public on Hugging Face and GitHub.
A solo developer trained and released Apex-2, a fully MoE decoder-only model with 3.87B total parameters (1.45B active), on Hugging Face. Despite using only 86.5B pretrain tokens versus 18T for Qwen2.5-1.5B, the base model matched Qwen2.5-1.5B on HumanEval+. Knowledge and math scores remain far behind, and DPO training was abandoned after it degraded results.
A solo developer trained and released Apex-2, a fully MoE decoder-only model with 3.87B total parameters (1.45B active), on Hugging Face. Despite using only 86.5B pretrain tokens versus 18T for Qwen2.5-1.5B, the base model matched Qwen2.5-1.5B on HumanEval+. Knowledge and math scores remain far behind, and DPO training was abandoned after it degraded results.
A new arXiv technical report introduces Instella-MoE, a fully open Mixture-of-Experts language model with 16 billion total and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. The work pairs sparse MoE design with architectural and system-level innovations aimed at efficient large-scale training and inference. It matters to practitioners as a fully open reference for MoE training outside the NVIDIA/CUDA ecosystem.
A new arXiv technical report introduces Instella-MoE, a fully open Mixture-of-Experts language model with 16 billion total and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. The work pairs sparse MoE design with architectural and system-level innovations aimed at efficient large-scale training and inference. It matters to practitioners as a fully open reference for MoE training outside the NVIDIA/CUDA ecosystem.
A new paper introduces a 20B parameter looping model that matches or beats Qwen3 Coder 30B while using only 3.5 trillion pre-training tokens (10% of the typical 35 trillion). The model was trained for hundreds of thousands of dollars, far less than the millions usually required, raising hopes for hobbyist-level pretraining. No weights have been released yet.
A new paper introduces a 20B parameter looping model that matches or beats Qwen3 Coder 30B while using only 3.5 trillion pre-training tokens (10% of the typical 35 trillion). The model was trained for hundreds of thousands of dollars, far less than the millions usually required, raising hopes for hobbyist-level pretraining. No weights have been released yet.
A developer trained three LLMs from scratch in raw PyTorch, then post-trained each with SFT and GRPO using identical recipes. Pre-training went as expected, but GRPO hurt two of the three models (V2 and V3) with no clear relationship to scale, highlighting the fragility of RL post-training.
A developer trained three LLMs from scratch in raw PyTorch, then post-trained each with SFT and GRPO using identical recipes. Pre-training went as expected, but GRPO hurt two of the three models (V2 and V3) with no clear relationship to scale, highlighting the fragility of RL post-training.
Ai2 released Olmo-core 3, an upgraded open training framework with a redesigned mixture-of-experts system built on distributed data parallelism instead of FSDP. Benchmarks show 2.7× throughput over the previous implementation and scaling to trillion-parameter configurations. It will underpin the next MoE-based generation of Olmo and is fully open on GitHub.
Ai2 released Olmo-core 3, an upgraded open training framework with a redesigned mixture-of-experts system built on distributed data parallelism instead of FSDP. Benchmarks show 2.7× throughput over the previous implementation and scaling to trillion-parameter configurations. It will underpin the next MoE-based generation of Olmo and is fully open on GitHub.