TLDR: A new study challenges the perceived superiority of Decision Transformer (DT) in offline reinforcement learning, especially for sparse-reward environments. It introduces Filtered Behavior Cloning (FBC), a simpler method that filters out low-performing data and applies basic behavior cloning. Experiments show FBC often outperforms DT, is more efficient, and uses less data, suggesting that DT may not always be the preferred choice for raw-state robotic tasks and that simpler methods can be more effective.
In the rapidly evolving field of artificial intelligence, the Transformer architecture, initially a breakthrough in natural language processing, has found its way into various domains, including reinforcement learning. Among these applications, the Decision Transformer (DT) has garnered significant attention, particularly in offline reinforcement learning. DT approaches policy learning as a sequence modeling task, a novel perspective that has intrigued researchers.
Recent work by Bhargava et al. (2024) suggested that DT exhibited superior performance compared to more conventional MLP-based offline reinforcement learning algorithms like Behavior Cloning (BC) and Conservative Q-Learning (CQL), especially in environments with sparse rewards and low-quality data. This claim sparked further investigation into whether DT truly holds an advantage in such challenging scenarios.
A new research paper, titled “Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?” by Yumi Omori, Zixuan Dong, and Keith Ross, delves deeper into this question. The authors conducted extensive experiments on robotic manipulation tasks (Robomimic) and locomotion benchmarks (D4RL) to re-evaluate DT’s performance in sparse-reward settings.
The paper introduces a straightforward yet highly effective MLP-based algorithm called Filtered Behavior Cloning (FBC). FBC operates on a simple premise: it first filters out low-performing trajectories from the dataset and then applies standard behavior cloning on the remaining, higher-quality data. This approach is not only simple but also computationally efficient, requiring less training data and fewer parameters compared to the complex Transformer architecture.
The experimental results presented in the paper challenge the notion of DT’s consistent superiority. For the sparsified D4RL datasets, FBC outperformed DT in 7 out of 9 datasets, leading to an aggregate performance improvement of approximately 4%. In the Robomimic sparse-reward tasks, FBC also showed competitive or slightly better performance than DT. Interestingly, a variant called Filtered Decision Transformer (FDT), which trains DT on the same filtered dataset as FBC, generally performed worse than FBC, suggesting that DT might benefit from both high and low-quality trajectories, but not necessarily to its advantage in sparse-reward settings.
The authors discuss that DT’s sophisticated architecture, while appealing, might not always translate to concrete advantages in sparse-reward domains. Factors such as overfitting or poor credit assignment could hinder its performance compared to simpler, more direct methods like FBC. Furthermore, based on prior literature and the findings from Bhargava et al. (2024) for dense-reward environments, the paper argues that DT is arguably not preferable for dense-reward datasets either, especially for raw-state robotic tasks.
Also Read:
- PARS: A New Algorithm for Stable Reinforcement Learning with Offline Data
- CL3R: A New Framework for Smarter Robotic Manipulation Through 3D Understanding
In conclusion, while transformer-based models hold immense potential, this research highlights that increased algorithmic and architectural complexity does not inherently guarantee improved performance in offline reinforcement learning, particularly in sparse-reward environments where learning signals are limited. The study suggests that simpler, well-tuned methods like FBC can often be more effective and efficient. This work encourages continued research into tailoring transformer architectures to the specific nuances of sequential decision-making tasks to truly unlock their power. You can read the full research paper here.
