TLDR: Microsoft has launched Agent Lightning, an open-source AI framework designed to enable Reinforcement Learning (RL)-based training for Large Language Models (LLMs) within any AI agent. This framework allows for the optimization of multi-agent systems without requiring extensive rewrites of existing agent stacks, by separating training from execution and introducing a unified trace format and a hierarchical method called LightningRL.
Microsoft’s AI team has introduced Agent Lightning, an innovative open-source AI framework aimed at revolutionizing the training of Large Language Models (LLMs) for various AI agents through Reinforcement Learning (RL). Released on October 29, 2025, Agent Lightning addresses the challenge of converting real agent traces into RL transitions to enhance policy LLMs without necessitating changes to existing agent infrastructures. This framework is designed to optimize multi-agent systems by making reinforcement learning accessible for any AI agent without requiring extensive rewrites.
At its core, Agent Lightning operates by disaggregating training from execution and defining a unified trace format. A key component is LightningRL, a hierarchical method that transforms complex agent runs into transitions that can be optimized by standard single-turn RL trainers. The framework models an agent as a decision process, formalizing it as a partially observable Markov decision process (POMDP). In this model, the observation is the current input to the policy LLM, the action is the model call, and the reward can be either terminal or intermediate. It meticulously extracts only the calls made by the policy model, along with their inputs, outputs, and associated rewards, thereby eliminating extraneous framework noise and yielding clean transitions for training.
LightningRL is instrumental in performing credit assignment across multi-step episodes, subsequently optimizing the policy using a single-turn RL objective. The research team highlights its compatibility with existing single-turn RL methods, noting that teams commonly utilize trainers implementing algorithms like PPO or GRPO, such as VeRL, which seamlessly integrate with this interface.
The system architecture of Agent Lightning employs Training Agent Disaggregation. This design involves a Lightning Server responsible for running training and serving, which exposes an OpenAI-like API for the updated model. Concurrently, a Lightning Client operates within the existing agent runtime, capturing traces of prompts, tool calls, and rewards, and streaming them back to the server. This architectural separation ensures that tools, browsers, shells, and other dependencies remain close to production environments, while GPU-intensive training is confined to the server tier. The runtime supports tracing paths, including a default path that leverages OpenTelemetry spans.
Also Read:
- GitHub Introduces Agent HQ: A Unified Platform for AI-Powered Coding Collaboration
- Eclipse LMOS AI Platform Unveils Agent Definition Language for Enterprise AI
Furthermore, Agent Lightning introduces Automatic Intermediate Rewarding (AIR), a feature that converts runtime signals, such as tool return status, into dense feedback. This mechanism is crucial for mitigating issues related to sparse rewards in lengthy workflows. The framework is designed for broad compatibility, allowing existing agents built with popular tools like LangChain, OpenAI Agents SDK, AutoGen, or CrewAI to connect with minimal code changes, effectively serving as a practical bridge between agent execution and reinforcement learning without requiring a complete framework overhaul.
