TLDR: The AI Productivity Index (APEX) is a new benchmark designed to assess how well frontier AI models perform high-economic-value knowledge work in domains like investment banking, management consulting, law, and primary medical care. Developed with industry experts, APEX-v1.0 features 200 test cases with expert-created prompts and evaluation rubrics. Initial evaluations show GPT 5 as the top performer, but a significant gap remains between AI and human expert capabilities, highlighting the need for more economically relevant AI benchmarks.
A new benchmark called the AI Productivity Index (APEX) has been introduced to evaluate whether advanced AI models can perform high-economic-value knowledge work. This initiative addresses a significant inefficiency in AI research, where many benchmarks fail to test capabilities relevant to real-world economic impact, outside of coding.
The first version, APEX-v1.0, comprises 200 test cases across four critical domains: investment banking, management consulting, law, and primary medical care. The development process involved three key steps: first, sourcing experts with top-tier experience from fields like Goldman Sachs for investment banking; second, these experts created prompts reflecting high-value tasks from their daily work; and third, they developed detailed rubrics for evaluating model responses.
Twenty-three frontier AI models were evaluated on APEX-v1.0 using an LM judge. GPT 5 (with ‘Thinking = High’ settings) achieved the highest mean score at 64.2%, followed by Grok 4 (61.3%) and Gemini 2.5 Flash (with ‘Thinking = On’) at 60.4%. Qwen 3 235B emerged as the best-performing open-source model, ranking seventh overall. Despite these top performances, a substantial gap remains between even the best models and human experts, underscoring the need for improved measurement of AI’s ability to produce economically valuable work.
Each prompt in APEX-v1.0 represents a knowledge task that would typically take an expert between 1 and 8 hours to complete, requiring sophisticated reasoning. These prompts are accompanied by evidence sources and a rubric of prompt-specific quality criteria. The rubrics are used to autograde responses with a panel of judge LMs, which has shown a high correlation with human grades. A public-facing leaderboard for APEX-v1.0 is available, showcasing results for the 23 models from 13 providers.
The dataset is evenly split with 50 cases per domain. Experts involved in creating the cases had extensive industry experience, with a mean of 7.25 years. For instance, investment bankers had a mean of 8.7 years of experience from firms like Goldman Sachs, and lawyers had a mean of 5 years from Big Law firms.
The domains varied in difficulty for the models. Law proved to be the easiest, with a mean score of 56.9% across all models, while medicine and investment banking were the most challenging, with mean scores of 47.5% and 47.6% respectively. Management consulting fell in between at 52.6%. Interestingly, GPT 5 (Thinking = High) consistently ranked as the best-performing model across all four domains.
A moderate performance gap was observed between closed-source and open-source models, with closed-source models achieving a mean score of 55.2% compared to 45.8% for open-source models. However, some open-source models like Qwen 3 235B and DeepSeek R1 showed strong performance. The study also noted that more powerful or newer versions of models are not always necessarily better at these real-world tasks. For example, Gemini 2.5 Flash (Thinking = On) slightly outperformed Gemini 2.5 Pro (Thinking = On) in mean score.
The research paper also discusses limitations, such as potential measurement error due to rubric creation and annotator calibration, and the fact that rubrics do not penalize incorrect or irrelevant claims. The scores on APEX-v1.0 are not necessarily correlated with real-world value, as a 60% score might still be effectively useless in a live setting. Future expansions of APEX aim to include a wider range of roles, integrate tool use and data rooms, and apply fine-grained tags for more detailed loss analysis.
Also Read:
- VitaBench: A New Standard for Evaluating LLM Agents in Real-World Scenarios
- Evaluating Language Models on Complex Physics Problems with the CritPt Benchmark
For more in-depth information, you can read the full research paper here.
