TLDR: LinkedIn has developed a new notification optimization system using Decision Transformers (DT) to improve user engagement and manage fatigue. This DT-based approach reframes policy learning as return-conditioned supervised learning, offering greater stability, scalability, and flexibility compared to previous reinforcement learning methods like Conservative Q-Learning (CQL). Key innovations include a multi-reward design, a quantile regression method for dynamic return-to-go prompting, and a production system using a circular buffer for efficient real-time inference. Online experiments showed a 0.72% increase in user sessions and reduced notification volume, demonstrating the effectiveness of DT in balancing notification utility and user experience at scale.
Notifications are a crucial way for platforms like LinkedIn to deliver timely and relevant information to users. However, optimizing these notifications is a complex challenge: sending too many can lead to user fatigue, while sending too few might mean users miss valuable updates. Finding the right balance requires sophisticated models that understand how users interact over time and can optimize for long-term goals.
Historically, notification systems often relied on simpler supervised learning models that predicted immediate actions, like clicks. While effective for short-term metrics, these approaches sometimes overlooked the broader impact on user experience and platform value. More recently, reinforcement learning (RL) methods, such as Conservative Q-Learning (CQL), have been used to optimize notification policies for cumulative platform value. However, these value-based RL methods can be difficult to manage at scale, often suffering from training instability, reproducibility issues, and challenges in explaining their decisions in complex recommendation environments.
Introducing Decision Transformers for Notification Optimization
A new approach, based on the Decision Transformer (DT) paradigm, offers a more stable and flexible alternative. Decision Transformers reframe the problem of learning a policy (how to act) as a supervised learning task, where the model is conditioned on desired future outcomes. This makes the learning process more robust, scalable, and easier to interpret compared to traditional RL methods.
Researchers at LinkedIn have designed and deployed a DT-based framework for notification decision-making. Their key innovations include a multi-reward design suitable for continuous, non-episodic tasks (unlike typical RL scenarios with clear start and end points), a quantile regression method for conditioning on desired future returns (return-to-go), and a production-ready system that uses a circular buffer for efficient, near-real-time processing of user interaction sequences.
How the System Works
The system operates in a continuous environment where notifications are generated throughout the day. At any given moment, the decision-making agent observes the current user state (including past notification history, user profile, and contextual signals), selects an action (send as a badge, send as a push, or don’t send), and receives a vector of rewards. These rewards are multi-objective, balancing predicted notification value (like click likelihood), actual user visits, and penalties for excessive notification volume to manage user fatigue.
Instead of manually defining how different rewards should be combined, the Decision Transformer learns an implicit way to balance these competing objectives. It also uses a novel quantile regression approach to predict the ‘return-to-go’ – the sum of future rewards – at inference time. This allows the system to be prompted with a desired level of future return, giving fine-grained control over the policy’s behavior without needing to retrain the model for every adjustment.
Deployment at Scale
To handle LinkedIn’s massive scale (averaging 100K queries per second, with peaks up to 150K QPS), the system integrates with existing LinkedIn platforms like Air Traffic Controller (ATC) for nearline stream processing and Venice for distributed data storage. A crucial component is the circular buffer mechanism, which efficiently stores and updates historical sequence features for each user. This allows the Decision Transformer to access a fixed-length history of user interactions, enabling sequence-aware modeling while minimizing storage costs and ensuring fast updates.
The model serving architecture is decoupled from the application logic, allowing independent scaling and tailored hardware configurations for different models. This isolation also facilitates safe experimentation with multiple model variants.
Real-World Impact
Extensive offline and online A/B tests were conducted at LinkedIn. Compared to the previous CQL-based agent, the DT-based approach achieved a significant +0.72% increase in user sessions while simultaneously reducing notification volume by 1.68%, with no statistically significant change in click-through rate (CTR). This demonstrates that the new system effectively improves notification utility and overall session activity while minimizing user fatigue.
Further analysis showed that incorporating learned return-to-go prompts helped maintain session activity with fewer notifications, and extending the historical context length provided additional gains. The ability to tune the target quantiles for specific rewards (prompt tuning) also offered direct control over policy objectives, allowing for the generation of diverse notification strategies.
Also Read:
- Agentic Reinforcement Learning: Empowering LLMs as Autonomous Decision-Makers
- Smart Planning for LLM Agents: Balancing Speed and Expense
Key Lessons from Deployment
The transition to Decision Transformers brought several practical benefits:
- Feature Scalability and Training Stability: Unlike CQL, which struggled with high-dimensional feature sets, DT maintained stable training and achieved significant performance gains with a tenfold increase in feature dimensionality.
- Robustness and Model Efficiency: DT produced much more consistent results across training runs, reducing the number of candidate models needed and accelerating the experimentation cycle.
- Improved Policy Tuning Flexibility and Interpretability: The return-conditioned nature of DT made policy behaviors more understandable and tunable, allowing for direct alignment of system outcomes with business goals.
In conclusion, this work presents a robust and scalable Decision Transformer-based approach for multi-objective notification optimization. By reframing policy learning as return-conditioned supervised learning and introducing innovative components for reward modeling and production serving, LinkedIn has successfully enhanced user engagement and operational efficiency. For more technical details, you can read the full research paper here.
