TLDR: A research paper benchmarks leading LLMs (Gemini, ChatGPT, Meta AI, Mistral AI, Grok) on managerial decision-making in a dynamic 12-month business simulation. It evaluates their performance on financial metrics, strategic coherence, and adaptability. Findings show Gemini models, especially PRO, outperformed others, demonstrating better long-term strategy and adaptability, while most LLMs struggled with consistency and profitability compared to human players. The study highlights current limitations of LLMs as autonomous business managers.
A recent study delves into the capabilities of Large Language Models (LLMs) in complex managerial decision-making within dynamic business simulations. This research addresses a significant gap in current AI benchmarking, which often focuses on short-term or discrete tasks, rather than the integrated, multi-step strategic decisions required to run an entire business over an extended period.
The paper, titled “AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulations” by Berdymyrat Ovezmyradov, introduces a novel, reproducible framework for evaluating LLMs. It utilizes an open-access management simulator, developed as a comprehensive spreadsheet model, to put five leading LLMs to the test: Gemini, ChatGPT, Meta AI, Mistral AI, and Grok. These models were tasked with managing a simulated retail company, “Retailer One,” over a 12-month period. Each month, the LLMs received a detailed business report and had to make crucial strategic decisions, including pricing, order size, marketing budget, hiring, dismissals, loans, training, R&D expenses, and sales and income forecasts.
The methodology allowed for a comparison of LLMs based on quantitative metrics such as profit, revenue, and market share, as well as qualitative aspects like strategic coherence, adaptability to market changes, and the rationale behind their decisions. This approach moves beyond simple performance metrics to assess long-term decision-making in a business context.
The findings revealed significant differences in performance among the LLMs. Gemini models, particularly Gemini PRO, emerged as the strongest performers in terms of financial outcomes. Gemini PRO demonstrated a balanced approach, utilizing the full range of available decisions, including loans, which other models often neglected. It also showed a higher degree of strategic coherence and adaptability, adjusting dynamically to market conditions without experiencing catastrophic failures where sales revenue dropped to zero.
In contrast, ChatGPT and Grok were among the worst performers. Many LLMs, except for Gemini, struggled with long-horizon coherence and adaptability, often reacting belatedly or counterproductively to market shocks. Their decision-making was frequently volatile, with initial high-price strategies leading to low sales, followed by sharp price cuts that, while recovering some ground, resulted in unstable financial performance. Forecasting quality was also generally poor across most models outside of Gemini.
The study highlights that while LLMs can follow complex, multi-turn prompts and act as virtual managers, their strategic coherence and adaptability are still limited as of 2025. None of the LLMs managed to end the year with a net profit, and human players consistently outperformed all LLM comparators in profitability and sales. This suggests that the “AI CEO” era is still a distant possibility, emphasizing the need for more nuanced benchmarking that evaluates an AI’s ability to navigate the dynamic and interconnected functions of a business.
Also Read:
- The Power of Collaboration: Orchestrating LLMs for Better Performance
- Beyond a Single Roll: Why Repetitions Are Key to Reliable LLM Evaluations
The research contributes a valuable open-source simulator file, encouraging future studies to test new models against a common standard. This work is crucial for understanding the practical capabilities and limitations of LLMs in managerial decision-making, informing the development of AI-driven decision support systems and the future of business education. For more details, you can read the full paper here.
