TLDR: A study trained small language models (LLMs) to deliver Acceptance and Commitment Therapy (ACT) using Supervised Fine-Tuning (SFT) and Odds Ratio Policy Optimization (ORPO), both with and without Chain-of-Thought (COT) reasoning. ORPO-trained models significantly outperformed SFT and base models in ACT fidelity and empathy. COT improved SFT performance but offered no benefit to ORPO or instruct models, suggesting its utility depends on the training method. ORPO’s success is attributed to learning the therapeutic ‘process’ over ‘content’.
The field of artificial intelligence is rapidly expanding into new domains, and one area showing significant promise is mental health. A recent study explores how different training methods impact the ability of large language models (LLMs) to deliver Acceptance and Commitment Therapy (ACT), a modern form of cognitive behavioral therapy.
Understanding Acceptance and Commitment Therapy (ACT)
ACT is a “third-wave” cognitive behavioral therapy that focuses on increasing psychological flexibility rather than just reducing symptoms. Its goal is to help individuals change their relationship with difficult thoughts and feelings, rather than trying to eliminate them. ACT principles, such as cognitive defusion and values clarification, are valuable tools for navigating challenging situations. However, access to therapy, including ACT, is often limited by factors like cost, scarcity of providers, and social stigma. This is where LLMs could play a crucial role in expanding access to mental health support.
The Study’s Approach to Training AI Therapists
The research, conducted by Talha Tahir, investigated how post-training methodologies and explicit reasoning influence a small, open-weight LLM’s capacity to deliver ACT. The base model used was Llama-3.2-3b-Instruct. The study employed two main training approaches: Supervised Fine-Tuning (SFT) and Odds Ratio Policy Optimization (ORPO). Each approach was tested both with and without an explicit Chain-of-Thought (COT) reasoning step. COT involves the model articulating its intermediate thought processes before generating a final response.
To train these models, 50 sets of synthetic ACT transcripts were generated by Mistral-Large, then validated by DeepSeek-Chat and manually reviewed. Performance was evaluated in simulated therapy sessions using a ‘patient agent’ (gpt-5-mini) and an LLM judge (Mistral-Large, fine-tuned on human evaluations). The judge assessed the models on two key metrics: ACT Fidelity Measure (ACT-FM), which measures adherence to ACT principles, and the Therapist Empathy Scale (TES).
Key Findings: ORPO’s Edge and Conditional Benefits of Reasoning
The study’s findings revealed significant differences in therapeutic performance. The models trained with ORPO consistently outperformed both the SFT and the base Instruct models in both ACT fidelity and therapeutic empathy. This suggests that ORPO is particularly effective at instilling ACT competencies in LLMs.
Interestingly, the impact of Chain-of-Thought (COT) reasoning was highly conditional. COT provided a significant benefit to SFT models, improving their ACT-FM scores. However, it offered no discernible advantage to the superior ORPO-trained models or the instruct-tuned variants. This suggests that for models already optimized for preference, explicit reasoning might not be necessary, and in some cases, could even hinder performance by making responses too didactic.
The researchers posit that ORPO’s superiority comes from its ability to learn the therapeutic ‘process’ – the stylistic and relational qualities of effective therapy – rather than merely imitating the ‘content’ of training examples. ACT, being a process-focused therapy, benefits greatly from this approach. For SFT models, COT acts as a necessary “scaffold,” compelling the model to deliberate on therapeutic stance and values before responding, which is crucial when training is primarily based on imitation.
Furthermore, the study found a strong positive correlation between therapeutic empathy and adherence to ACT principles, indicating these qualities often reinforce each other. Exploratory analyses also highlighted that patient characteristics, such as psychological mindedness and primary clinical issue, significantly influenced which model performed best, suggesting a need for personalized AI therapy approaches.
Also Read:
- Small Language Models Show Promise in Formal Logic Reasoning for Ontology Engineering
- Assessing Foundation Models for Planning Assistance
Implications and Future Directions
This research establishes that preference-aligned policy optimization, like ORPO, can effectively train small LLMs to deliver ACT. The conditional utility of explicit reasoning underscores that the effectiveness of AI training techniques is highly dependent on the underlying paradigm. The study’s use of relatively small models and datasets further emphasizes the quality of ORPO as a training signal for this task.
Future work could explore how model size and data diversity might alter these findings. The observed patient-specific performance variations suggest an exciting future direction: developing an “LLM router.” This meta-controller could dynamically select the most appropriate AI therapist variant and reasoning setting based on a patient’s initial profile, mirroring how human clinicians adapt their approach. This could lead to more cost-effective and personalized AI mental health interventions. You can read the full research paper here: The Thinking Therapist Research Paper.
