Fixing GRPO's credit assignment problem without evaluating every step (arXiv paper)
A new paper introduces Le Critique, a method that uses privileged value functions to improve LLM reinforcement learning. It addresses limitations of group-relative methods like GRPO by providing token-level credit assignment and reducing variance without large rollout groups.
A new paper introduces Le Critique, a method that uses privileged value functions to improve LLM reinforcement learning. It addresses limitations of group-relative methods like GRPO by providing token-level credit assignment and reducing variance without large rollout groups.
A new arXiv paper demonstrates that dense per-step prediction rewards, intended to improve memory in long-horizon LLM agents, instead cause catastrophic policy collapse under group-normalized RL (GRPO). Across Qwen3-1.7B/4B/8B on ALFWorld, the reward drives agents into a degenerate state where prediction accuracy reaches 1.0 but task success drops to 0, with episode length pinned at the horizon. The authors identify the cause via single-factor ablation and propose alternative reward designs that actually work.
A new arXiv paper demonstrates that dense per-step prediction rewards, intended to improve memory in long-horizon LLM agents, instead cause catastrophic policy collapse under group-normalized RL (GRPO). Across Qwen3-1.7B/4B/8B on ALFWorld, the reward drives agents into a degenerate state where prediction accuracy reaches 1.0 but task success drops to 0, with episode length pinned at the horizon. The authors identify the cause via single-factor ablation and propose alternative reward designs that actually work.
A new arxiv paper proposes Process Reward Informed Tree Rollout (PRITR), a method that improves multi-turn reinforcement learning for LLM agents by organizing trajectory groups as trees and using process rewards to guide exploration. This addresses inefficiencies in uniform rollout strategies like GRPO/RLOO, which waste budget on dead-end attempts in long-horizon tasks.
A new arxiv paper proposes Process Reward Informed Tree Rollout (PRITR), a method that improves multi-turn reinforcement learning for LLM agents by organizing trajectory groups as trees and using process rewards to guide exploration. This addresses inefficiencies in uniform rollout strategies like GRPO/RLOO, which waste budget on dead-end attempts in long-horizon tasks.
A new arXiv paper introduces Group Entropy-Controlled Policy Optimization (GECPO), which addresses the challenge of heterogeneous entropy needs across different tasks in LLM reinforcement learning. The method replaces global or token-level entropy regulation with per-group control, and also corrects a bias in GRPO-style advantage normalization that arises from entropy differences across prompt groups.
A new arXiv paper introduces Group Entropy-Controlled Policy Optimization (GECPO), which addresses the challenge of heterogeneous entropy needs across different tasks in LLM reinforcement learning. The method replaces global or token-level entropy regulation with per-group control, and also corrects a bias in GRPO-style advantage normalization that arises from entropy differences across prompt groups.
A controlled study on 4B-8B language and vision-language model web agents found that GRPO reinforcement learning, across 18 runs varying learning rate, KL weight, seed, initialization, and clipping, never credibly improved success rate over a strong supervised baseline. The paper suggests RL mostly reshapes existing behavior rather than adding new skill.
A controlled study on 4B-8B language and vision-language model web agents found that GRPO reinforcement learning, across 18 runs varying learning rate, KL weight, seed, initialization, and clipping, never credibly improved success rate over a strong supervised baseline. The paper suggests RL mostly reshapes existing behavior rather than adding new skill.