Ai2 released Olmo-core 3, an upgraded open training framework with a redesigned mixture-of-experts system built on distributed data parallelism instead of FSDP. Benchmarks show 2.7× throughput over the previous implementation and scaling to trillion-parameter configurations. It will underpin the next MoE-based generation of Olmo and is fully open on GitHub.
An arxiv paper (2026-08-28) introduces CE-MoE, a communication-efficient approach to training Mixture-of-Experts language models with expert parallelism. Instead of interleaving MoE layers after every token-mixing layer, it concentrates expert capacity in a few routed MoE layers, cutting all-to-all dispatch/combine communication overhead.
An arxiv paper (2026-08-28) introduces CE-MoE, a communication-efficient approach to training Mixture-of-Experts language models with expert parallelism. Instead of interleaving MoE layers after every token-mixing layer, it concentrates expert capacity in a few routed MoE layers, cutting all-to-all dispatch/combine communication overhead.
NVIDIA's GB300 NVL72 system achieved a world record for mixture-of-experts pre-training, running DeepSeek-V3 671B at 1,648 TFLOPs per GPU. This demonstrates that communication efficiency, not compute, is now the primary bottleneck for scaling MoE models across thousands of GPUs.
NVIDIA's GB300 NVL72 system achieved a world record for mixture-of-experts pre-training, running DeepSeek-V3 671B at 1,648 TFLOPs per GPU. This demonstrates that communication efficiency, not compute, is now the primary bottleneck for scaling MoE models across thousands of GPUs.
A new arXiv technical report introduces Instella-MoE, a fully open Mixture-of-Experts language model with 16 billion total and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. The work pairs sparse MoE design with architectural and system-level innovations aimed at efficient large-scale training and inference. It matters to practitioners as a fully open reference for MoE training outside the NVIDIA/CUDA ecosystem.
A new arXiv technical report introduces Instella-MoE, a fully open Mixture-of-Experts language model with 16 billion total and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. The work pairs sparse MoE design with architectural and system-level innovations aimed at efficient large-scale training and inference. It matters to practitioners as a fully open reference for MoE training outside the NVIDIA/CUDA ecosystem.
An arXiv paper introduces Mixture of Reused Experts (MoRE), a hybrid of Mixture-of-Experts and recurrent weight sharing that cuts MoE memory footprints while keeping routing capacity. Expert pools are shared across groups of adjacent layers, with per-layer routers and learnable depth embeddings. Experiments across 114M-1.15B parameters show better perplexity and downstream performance than standard MoEs at matched budgets.
An arXiv paper introduces Mixture of Reused Experts (MoRE), a hybrid of Mixture-of-Experts and recurrent weight sharing that cuts MoE memory footprints while keeping routing capacity. Expert pools are shared across groups of adjacent layers, with per-layer routers and learnable depth embeddings. Experiments across 114M-1.15B parameters show better perplexity and downstream performance than standard MoEs at matched budgets.
A new arXiv survey organizes the evolution of Mixture-of-Experts (MoE) architectures in large language models along five coupled dimensions: expert granularity, expert topology, routing freedom, load-balancing scope, and execution structure. It synthesizes primary papers, technical reports, and prior surveys into a dependency graph of eight architectural milestones with six mainline developments. The framework helps practitioners understand MoE design trade-offs beyond a chronological list of model releases.
A new arXiv survey organizes the evolution of Mixture-of-Experts (MoE) architectures in large language models along five coupled dimensions: expert granularity, expert topology, routing freedom, load-balancing scope, and execution structure. It synthesizes primary papers, technical reports, and prior surveys into a dependency graph of eight architectural milestones with six mainline developments. The framework helps practitioners understand MoE design trade-offs beyond a chronological list of model releases.