Researchers from the University of Science and Technology of China have developed a up-to-date reinforcement learning (RL) framework that helps train enormous language models (LLMs) on intricate agentic tasks beyond well-defined problems such as math and coding.
Their frames, Agent-R1is compatible with popular RL algorithms and shows significant improvement for inference tasks that require multiple search steps and multi-turn tool interactions.
The framework is based on a redefinition of the RL paradigm that takes into account the animated nature of agent-based applications that require interactions with evolving environments and imperfect information. This framework is much more similar to real-world applications and can have crucial applications for agent tasks in enterprise settings.
A up-to-date approach to reinforcement learning for agents
RL has become the cornerstone of LLM training in well-defined reasoning tasks. In areas like math and coding, the model receives a clear signal: the answer is either right or wrong. This makes it relatively effortless to reward or punish his behavior.
However, this approach cannot cope with agent-based tasks that require models to operate in interactive environments, develop animated memory during conversations, perform multi-step reasoning, and respond to unpredictable feedback. Training agents with RL skills for these scenarios poses unique challenges, especially in multi-turn interactions where designing effective rewards is intricate and the trained agent often fails to generalize to the cluttered, unpredictable nature of real-world environments.
To address these challenges, researchers from the University of Science and Technology have re-examined the basic RL framework known as Markov decision process (MDP). MDP models the decision-making process using four key components: the state space (the set of possible states an agent can be in); action space (what the agent can do); state transition probability (the state to which an action is likely to lead); and the reward function (regardless of whether the outcome is good or bad). This article proposes an extension of this framework to better suit LLM agents.
In the up-to-date formulation, the state space is expanded not only to include the current state (the current sequence of tokens generated by the model), but also by the entire history of interactions and feedback with the environment. Actions are still essentially text generation, but specific text sequences can now trigger external tools, such as an API call. State transitions become unpredictable or “stochastic” because the outcome depends not only on the tokens predicted by the model, but also on the response of the environment, which depends on external factors. Finally, the reward system becomes more detailed and includes intermediate “process rewards” for successfully completing individual stages, rather than just a single reward at the very end. This provides more repeated and precise guidance to the agent during training.
This last bit is particularly crucial and addresses the “sparse reward” problem that most RL frameworks struggle with. When an agent receives a single reward signal based on the final outcome, it does not learn from the good and bad intermediate steps it took along the way. Process rewards solve this problem by providing feedback signals at these intermediate stages, making the learning process much more competent.
“These extensions are critical to enabling learning algorithms by enhancing the training of sophisticated agents capable of complex, multi-step reasoning and interaction in dynamic environments,” the researchers wrote in their paper.
Agent-R1 structure
Scientists have developed an expanded definition of MDP Agent-R1a versatile and user-friendly RL-based LLM agent training platform. It extends time-honored single-turn RL structures to support the multi-turn, interactive nature of agent tasks, enabling seamless integration into a variety of environments.
The most significant difference is in the “deployment phase” where the agent generates responses. In the case of single-turn RL, the model generates the response once. In multi-turn RL, the process involves a series of intricate back and forth interactions.
Agent-R1 achieves this versatile, multi-turn implementation with two core modules: Tool and ToolEnv. The Tool module acts as a performer of specific actions, such as calling an API or accessing a database. When invoked, the tool performs its action and returns a direct, raw result. Whereas the ToolEnv module is the coordinator and interpreter. Takes the output from the Tool and determines how this output affects the state of the agent and the overall progress of the task. ToolEnv manages state transitions, computes reward signals based on tool results, and packages up-to-date state information for the agent.
In brief, when the action completes, the tool reports “what happened”, while ToolEnv dictates “what this result means for the agent and the task”.
Agent-R1 in action
Researchers tested Agent-R1 on a challenging multi-hop question answering task that requires intricate reasoning, searching for information across multiple documents, and multi-step decision-making. They trained Qwen2.5-3B-Instruct on QA datasets and evaluated its performance on the platform Hotpot AND 2WikiMultihopQA data sets. They also tested this on the Musique dataset, which was outside the domain of the tasks the agent was trained on.
They compared different RL algorithms trained with Agent-R1 against two benchmarks: Naive RAG, a single-pass search method in which LLM responds based on a single set of retrieved documents, and Base Tool Call, which leverages the native ability to call model functions without specialized RL training.
The results showed that all RL-trained agents significantly outperformed baseline. GRPO, an RL algorithm used in advanced reasoning models such as DeepSeek-R1provided the best overall performance.
“These results solidly support the effectiveness of Agent-R1 in training powerful LLM agents via end-to-end RL, demonstrating consistent, significant gains over baseline across a variety of datasets and RL algorithms,” the researchers write.
These findings may be relevant to the enterprise where there is a powerful emphasis on the employ of RL and reasoning beyond well-defined domains. A framework designed to handle cluttered, multi-turn interactions with users and animated environments could pave the way for up-to-date agents capable of solving intricate problems in real-world settings.
“We hope that Agent-R1 will provide a foundation for future work on scalable and standardized RL training for LLM agents,” the researchers conclude.