Releases·Model Release
Training a 210M text-to-image DiT from scratch on one GPU: what I measured [P]
Ivan Mikhnenkov trained a 210M-parameter text-to-image diffusion transformer from scratch in 3.5 days on one RTX PRO 6000, then published the write-up, weights, code, and demo. The post shares three measurements on attention sinks, flow-matching loss as a health signal, and inference timestep shifting that matter to anyone training small DiTs.
13 engagement1 sourceFri, Sep 11, 2026, 01:00 PM
AI Overview
- Dev trained a 210M cross-attention DiT from scratch in 3.5 days on one RTX PRO 6000, using 4.2M images at 256² and a 400k-step run
- Two learned key/value slots absorb ~90% of cross-attention mass at mid-noise while the EOS sink token drops to ~4%; register vectors reach 4–13× image-token norms
- Flow-matching loss moved only 0.805→0.754 while held-out FID improved 33.7→27.0 and object accuracy rose 65%→90%, making it a health signal not a quality signal
- At 20 steps, timestep shift 2.8 (from the SD3/RAE rule) gives FID 27.0 vs 27.3 with no shift — worth more than doubling to 50 steps (FID 26.6)
- Data mix: Pexels 2.8M (60%), quality-filtered 1.2M FLUX-Reason-6M slice (25%), GPT-4V-captioned COCO (15%), with frozen flan-t5-base text encoder
- Code, write-up, weights, and demo released via the tinydit GitHub repo and Hugging Face; torch.compile gave 2.4× speedup over eager
Related
Developments·Paper
Abra scaling study: diffusion models scale predictably but need far more data
A new arXiv paper presents Abra, a family of flow-matching transformers trained across three orders of magnitude of compute, to derive compute-optimal scaling laws for text-to-image diffusion models. The study finds diffusion models scale as predictably as language models but require significantly more data for optimal training.
A new arXiv paper presents Abra, a family of flow-matching transformers trained across three orders of magnitude of compute, to derive compute-optimal scaling laws for text-to-image diffusion models. The study finds diffusion models scale as predictably as language models but require significantly more data for optimal training.
0 engagement1 sourcearxivTue, Aug 18, 2026, 02:36 AM
Developments·Paper
Researchers analyze when downscaled image training preserves diffusion gradients
A new arXiv paper analyzes when training diffusion transformers on downscaled images yields the same gradients as full-resolution training. The authors decompose the gradient signal change into a noise-dependent term and a σ-independent term, providing theoretical justification for reduced-resolution training.
A new arXiv paper analyzes when training diffusion transformers on downscaled images yields the same gradients as full-resolution training. The authors decompose the gradient signal change into a noise-dependent term and a σ-independent term, providing theoretical justification for reduced-resolution training.
0 engagement1 sourcearxivWed, Aug 5, 2026, 04:53 AM
Developments·Paper
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
An arXiv paper examines how image tokenizers shape the visual language of unified multimodal models, arguing that isolated or generation-only evaluations miss tokenizer behavior when tokens are modeled jointly with text. The authors build a pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining.
An arXiv paper examines how image tokenizers shape the visual language of unified multimodal models, arguing that isolated or generation-only evaluations miss tokenizer behavior when tokens are modeled jointly with text. The authors build a pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining.
0 engagement1 sourcearxivTue, Sep 8, 2026, 05:57 PM
Developments·Paper
Researchers learn CFG guidance schedules adversarially for text-to-image diffusion
A new arXiv paper proposes learning classifier-free guidance (CFG) schedules as a function of diffusion time, conditioning, and noise, rather than using a static global scale. The approach aims to reduce artifacts and improve image fidelity and text alignment in text-to-image diffusion models.
A new arXiv paper proposes learning classifier-free guidance (CFG) schedules as a function of diffusion time, conditioning, and noise, rather than using a static global scale. The approach aims to reduce artifacts and improve image fidelity and text alignment in text-to-image diffusion models.
0 engagement1 sourcearxivFri, Aug 14, 2026, 07:30 AM
Developments·Paper
CMuon optimizer accelerates and stabilizes Diffusion Transformer training
A new optimizer, CMuon, addresses the suboptimal late-stage convergence of Muon when applied to Diffusion Transformers (DiTs). It uses chunked momentum orthogonalization to handle fused tensors in DiT architectures, accelerating training and improving stability. The paper identifies the root cause as fused weights in AdaLN and QKV layers.
A new optimizer, CMuon, addresses the suboptimal late-stage convergence of Muon when applied to Diffusion Transformers (DiTs). It uses chunked momentum orthogonalization to handle fused tensors in DiT architectures, accelerating training and improving stability. The paper identifies the root cause as fused weights in AdaLN and QKV layers.
0 engagement1 sourcearxivMon, Aug 3, 2026, 05:04 PM