TLDR: This research demonstrates that Large Language Models (LLMs) can learn genuinely new, complex skills through Reinforcement Learning (RL) by combining simpler, existing abilities. Using a controlled string transformation task, the study shows that RL training on compositional problems enables LLMs to generalize to more difficult, unseen compositions and even transfer these compositional skills to entirely different tasks, provided the basic skills for the new task are already present. This challenges the idea that RL merely re-ranks existing knowledge, highlighting its potential for true skill acquisition and generalization in LLMs.
A central debate in the world of large language models (LLMs) revolves around whether reinforcement learning (RL) truly teaches these powerful AI systems new skills, or if it merely activates and refines abilities they already possess from their vast pre-training. This new research offers compelling evidence that LLMs can indeed acquire genuinely new skills during RL by learning to combine existing ones, a process that mirrors how humans develop new cognitive abilities.
The study, titled FROMf(x)ANDg(x)TOf(g(x)): LLMSLEARNNEW SKILLS INRLBYCOMPOSINGOLDONES, was conducted by a team of researchers including Lifan Yuan, Weize Chen, Yuchen Zhang, Ganqu Cui, Hanbin Wang, Ziming You, Ning Ding, Zhiyuan Liu, Maosong Sun, and Hao Peng. To rigorously investigate this question, the researchers developed a unique synthetic framework. This framework allowed them to precisely control task complexity and avoid issues like data contamination, which can often obscure true learning in LLMs.
In this setup, a “skill” was defined as the ability to predict the output of a string transformation function, like f(x), given an input x. The experiments revealed a fascinating insight: if an LLM had already learned individual functions f and g prior to RL, the RL process enabled it to learn how to solve unseen compositions of these functions, such as h(x) = g(f(x)). What’s more, this newfound compositional ability wasn’t limited to simple combinations; it generalized to even more complex problems involving the composition of more than two functions that the model had never encountered during its RL training.
Perhaps one of the most surprising findings was the transferability of this compositional skill. The ability acquired on one type of task (string transformations) could be applied to a completely different target task, like the Countdown problem (a mathematical puzzle). This transfer occurred even though the model had never been trained on any compositional problems within the Countdown task itself. The only prerequisite was that the model had already acquired the basic, or “atomic,” skills for the target task before its RL training on the source task.
The researchers also conducted a qualitative analysis, which showed that RL fundamentally alters the reasoning behaviors of the models. Instead of simply improving accuracy, RL training on compositional problems led to a significant reduction in errors related to ignoring or misunderstanding composition. Failures shifted primarily to errors in predicting the atomic functions themselves, indicating that the models had genuinely learned to parse and execute compositional plans. This behavioral transformation suggests a deeper acquisition of compositional skills rather than just a superficial performance boost.
These findings stand in stark contrast to traditional next-token prediction (NTP) training, where none of these compositional learning or transfer effects were observed with the same data. This highlights the unique role of RL in fostering advanced skill acquisition.
Also Read:
- Unlocking Personalized AI: Learning Directly from User Conversations
- UniAPL: Unifying Language Model Training for Enhanced Instruction Following
Challenging the “Reranking Illusion”
The study also addresses a recent pessimistic view that RL with verifiable rewards (RLVR) merely “reranks” existing reasoning patterns in base models, rather than teaching new abilities. This view often stems from observations that the performance gap between RL-trained and base models narrows as more samples (k) are considered in pass@k evaluations. The authors argue that this “reranking illusion” might arise from evaluating and training RL on tasks where base models already perform well, thus giving RL little incentive to learn genuinely new skills.
By focusing on challenging compositional problems where the base model’s performance was near zero, the research demonstrated that RL substantially improved performance, with the gap between RL-trained and base models widening significantly as k increased. This divergence provides clear evidence of new skill acquisition, suggesting that RL can indeed expand performance limits when properly incentivized on difficult problems.
The implications of this research are significant for LLM development. It suggests a valuable strategy: first, build base models with the necessary fundamental skills through extensive pre-training, and then apply RL with appropriate incentives to help them acquire more advanced, compositional skills that generalize effectively to complex and novel problems. This work provides an optimistic outlook on RL’s critical role in post-training, emphasizing its potential for easy-to-hard generalization and cross-task transfer in the ongoing evolution of large language models.
