Releases·Tool Release
ParaRNN open-sources PyTorch/Triton library for parallel training of non-linear RNNs via Newton scans
A developer open-sourced ParaRNN, a PyTorch/Triton library that parallelizes training of non-linear recurrent architectures (sLSTM, RWKV-7, CfC, Titans) across time using Newton-Raphson iterations mapped to associative prefix scans. It targets the historical bottleneck of slow sequential unrolls while preserving the O(1) inference advantage of non-linear RNNs. Reported speedups reach ~200x over sequential BPTT, making time-parallel training of such architectures practical on consumer GPUs.
1 engagement1 sourceMon, Sep 7, 2026, 10:31 AM
AI Overview
- ParaRNN reports ~200x speedup over sequential BPTT (CfC at T=2048: 2.79ms vs 643ms on RTX 3060) and flat Newton iteration budgets (K*<=3) up to 131,072 tokens.
- Fused Triton kernels support sLSTM, RWKV-7, CfC, Titans, Modern Hopfield, and M2RNN, with torch.compile fullgraph support and an out-of-tree vLLM plugin.
Related
Developments·Paper
NeurIPS 2026 Spotlight: Parallel-in-Time RNN Training Combining DEER and GTF for Chaotic Dynamical Systems Reconstruction
Researchers present a NeurIPS 2026 spotlight combining DEER with generalized teacher forcing to train nonlinear RNNs on chaotic time series over 100x faster, enabling stable training on sequences longer than 10^6. The method outperforms Mamba and other state space models on dynamical systems reconstruction.
Researchers present a NeurIPS 2026 spotlight combining DEER with generalized teacher forcing to train nonlinear RNNs on chaotic time series over 100x faster, enabling stable training on sequences longer than 10^6. The method outperforms Mamba and other state space models on dynamical systems reconstruction.
105 engagement1 sourceredditThu, Oct 1, 2026, 01:12 PM
Developments·Paper
RWKV demonstrates RNN architecture with Transformer-level LLM performance and parallelization
RWKV is a novel recurrent neural network architecture that achieves large language model performance comparable to Transformers while being parallelizable during training. The project, hosted at rwkv.com, aims to combine the efficiency of RNNs with the scalability of Transformers.
RWKV is a novel recurrent neural network architecture that achieves large language model performance comparable to Transformers while being parallelizable during training. The project, hosted at rwkv.com, aims to combine the efficiency of RNNs with the scalability of Transformers.
2 engagement1 sourceredditWed, Jul 15, 2026, 06:21 PM
Developments·Paper
Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling
An arXiv paper introduces RiccatiSSM, a nonlinear state space model whose state dimensions follow input-conditioned Riccati differential equations. Because each step's flow is a Möbius transformation, which is closed under composition, the full nonlinear trajectory can be evaluated exactly with a single associative parallel scan — no iterative linearization. It matches nonlinear baselines' predictive performance while cutting runtime by 22–33% versus LrcSSM.
An arXiv paper introduces RiccatiSSM, a nonlinear state space model whose state dimensions follow input-conditioned Riccati differential equations. Because each step's flow is a Möbius transformation, which is closed under composition, the full nonlinear trajectory can be evaluated exactly with a single associative parallel scan — no iterative linearization. It matches nonlinear baselines' predictive performance while cutting runtime by 22–33% versus LrcSSM.
0 engagement1 sourcearxivMon, Sep 28, 2026, 03:40 PM
Developments·Paper
SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization
SwiftQK is a multi-GPU RMSNorm kernel that reduces cross-GPU communication in Query-Key Normalization (QK-Norm) under Tensor Parallelism by exchanging only scalar normalization statistics. It overlaps peer-to-peer reduction with independent computation, cutting QK-Norm latency by 81.4–93% on recent LLMs.
SwiftQK is a multi-GPU RMSNorm kernel that reduces cross-GPU communication in Query-Key Normalization (QK-Norm) under Tensor Parallelism by exchanging only scalar normalization statistics. It overlaps peer-to-peer reduction with independent computation, cutting QK-Norm latency by 81.4–93% on recent LLMs.
0 engagement1 sourcearxivMon, Aug 10, 2026, 06:16 AM
Developments·Benchmark
DataKernelBench: New benchmark tests LLMs on GPU database query kernels
Researchers introduced DataKernelBench, a benchmark that translates SQL into PyTorch TorchPlan programs and evaluates LLMs optimizing database-style operators in CUDA or Triton. It targets irregular, data-movement-heavy operators that existing LLM kernel benchmarks ignore, testing ten models on TPC-H SF10 with an H100 GPU.
Researchers introduced DataKernelBench, a benchmark that translates SQL into PyTorch TorchPlan programs and evaluates LLMs optimizing database-style operators in CUDA or Triton. It targets irregular, data-movement-heavy operators that existing LLM kernel benchmarks ignore, testing ten models on TPC-H SF10 with an H100 GPU.
0 engagement1 sourcearxivTue, Aug 25, 2026, 06:57 PM