Releases·Tool Release
LARA trains low-rank residual adapters for frozen LLMs, enabling composable behaviors and inference-time routing
A developer shared LARA (Lightweight Additive Residual Adaptation), a research project and PyTorch library enabling modular post-training for frozen language models. It trains low-rank residual adapters at selected layers that can be loaded, removed, blended, or routed at inference time. A Mixture of Behaviors demo shows one frozen model serving coding, math, medical, and summarization behaviors via a token-level soft router.
73 engagement1 sourceWed, Sep 16, 2026, 01:28 PM
AI Overview
- LARA (Lightweight Additive Residual Adaptation) trains low-rank residual adapters at selected layers of frozen models instead of modifying weights
- A Mixture of Behaviors demo uses a soft router to select or combine behaviors token-by-token, letting one model serve coding, maths, medical and summarization
- The GitHub repo includes a LoRA comparison, writing-style behaviors trained on Hemingway, Fitzgerald and Gertrude Stein, plus training code and reproduction instructions
Related
Developments·Paper
LARC: Low-Rank Adaptive Residual Connections for Learning in Frozen Models
An arXiv report introduces Low-Rank Adaptive Residual Connections (LARC), a method that lets frozen models learn from feedback via a compact low-rank numerical state. A slow state learns starting factors across tasks while a fast private state adapts and resets. The authors demonstrate the approach on a frozen MiniCPM5-1B-SFT substrate with a rank-4 input residual using only 12,288 trainable parameters.
An arXiv report introduces Low-Rank Adaptive Residual Connections (LARC), a method that lets frozen models learn from feedback via a compact low-rank numerical state. A slow state learns starting factors across tasks while a fast private state adapts and resets. The authors demonstrate the approach on a frozen MiniCPM5-1B-SFT substrate with a rank-4 input residual using only 12,288 trainable parameters.
0 engagement1 sourcearxivWed, Sep 30, 2026, 04:21 PM
Developments·Paper
READ: Read-Only LoRA Composition Lets New Skills Read but Never Write Old Ones
An arXiv paper introduces READ (Read-only Expansion of Adapter Deltas), a method for combining independently trained LoRA adapters into a single model without interference. It canonicalizes each adapter's factorization and lets new skills read old skills' input subspaces without writing into their output subspaces, improving on published baselines by over twenty points on SuperGLUE.
An arXiv paper introduces READ (Read-only Expansion of Adapter Deltas), a method for combining independently trained LoRA adapters into a single model without interference. It canonicalizes each adapter's factorization and lets new skills read old skills' input subspaces without writing into their output subspaces, improving on published baselines by over twenty points on SuperGLUE.
0 engagement1 sourcearxivFri, Sep 25, 2026, 05:54 PM
Developments·Paper
MoE$^2$-LoRA: New PEFT method for MoE models uses MoE-style adapters
MoE$^2$-LoRA introduces a parameter-efficient fine-tuning method that applies MoE-style low-rank adaptation to MoE models, coupling pretrained expert specialization with task-specific adapters. It addresses limitations of existing PEFT methods for MoE, such as ignoring router priors or using static expert selection.
MoE$^2$-LoRA introduces a parameter-efficient fine-tuning method that applies MoE-style low-rank adaptation to MoE models, coupling pretrained expert specialization with task-specific adapters. It addresses limitations of existing PEFT methods for MoE, such as ignoring router priors or using static expert selection.
0 engagement1 sourcearxivFri, Jul 24, 2026, 04:50 AM
Developments·Paper
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
Latent-LoRA introduces a method for continual learning in LLMs by storing low-rank adapters in a compressed latent space and using a gradient-free routing mechanism to select the correct adapter at inference. This avoids catastrophic forgetting without requiring task identity or introducing trainable parameters that themselves suffer from forgetting.
Latent-LoRA introduces a method for continual learning in LLMs by storing low-rank adapters in a compressed latent space and using a gradient-free routing mechanism to select the correct adapter at inference. This avoids catastrophic forgetting without requiring task identity or introducing trainable parameters that themselves suffer from forgetting.
0 engagement1 sourcearxivSun, Jul 26, 2026, 08:42 PM
Developments·Paper
Developer proposes Mixture-of-Attention architecture with centroid routing
A developer shared a new language model architecture called Mixture-of-Attention, which routes tokens to experts via centroid clustering to address attention dilution and dead experts. The author seeks feedback on novelty, fearing prior publication. The architecture is detailed in a GitHub repository.
A developer shared a new language model architecture called Mixture-of-Attention, which routes tokens to experts via centroid clustering to address attention dilution and dead experts. The author seeks feedback on novelty, fearing prior publication. The architecture is detailed in a GitHub repository.
20 engagement1 sourceredditMon, Aug 17, 2026, 08:44 PM