TLDR: A new research paper evaluates 15 foundation models across 79 problems and eight academic domains on supercomputing, cloud, and university clusters. It finds that reasoning quality is model-intrinsic, not infrastructure-dependent. Key discoveries include a ‘parameter efficiency paradox’ where smaller, well-trained models outperform larger ones, and a ‘transparency-correctness trade-off’ where models either excel at showing reasoning steps or getting the final answer, but rarely both. The study offers practical guidelines for selecting models based on specific application needs.
Understanding how advanced AI models think and solve problems is crucial as they become more integrated into our daily lives. A recent research paper, “Cross-Platform Evaluation of Reasoning Capabilities in Foundation Models”, delves into this complex area, offering a comprehensive look at the reasoning abilities of various foundation models across different computing environments. The study, conducted by J. de Curtòa, I. de Zarzà c, Pablo GarcÃab, and Jordi Cabotc, establishes a new benchmark that is independent of the underlying computing infrastructure.
The researchers set out to evaluate 15 foundation models on 79 problems spanning eight academic domains, including Physics, Mathematics, Chemistry, Economics, Biology, Statistics, Calculus, and Optimization. This extensive evaluation was carried out across three distinct computational setups: a high-performance computing (HPC) supercomputer (MareNostrum 5), cloud platforms (Nebius AI Studio), and university clusters (equipped with H200 GPUs). This multi-platform approach was designed to ensure the reproducibility and generalizability of their findings.
Key Insights from the Evaluation
One of the most significant findings is the confirmation that a model’s reasoning quality is largely intrinsic to the model itself, rather than being dependent on the infrastructure it runs on. The study found minimal performance variance (less than 3%) when identical models were evaluated on different platforms. This is a crucial step towards democratizing AI research, allowing researchers without access to supercomputing facilities to conduct reliable evaluations.
The paper also challenges the conventional wisdom that bigger models are always better. This is termed the “parameter efficiency paradox.” For instance, Hermes-4-70B, a model with 70 billion parameters, achieved a higher score (0.598) than its much larger 405B counterpart (0.573) and Meta’s LLaMA 3.1-405B (0.560). Similarly, a dense smaller model like Phi-4-mini (14B parameters) outperformed a larger, sparse Mixture-of-Experts (MoE) architecture like Phi-3.5-MoE (42B parameters). These results suggest that the quality of training data and architectural design are more critical for reasoning capabilities than simply increasing model size.
Another fascinating discovery is the “transparency-correctness trade-off.” The evaluation revealed a fundamental tension between a model’s ability to show its step-by-step reasoning (transparency) and its ability to arrive at the correct final answer. DeepSeek-R1, for example, demonstrated exceptionally high step-accuracy (0.716), meaning its intermediate reasoning steps were very clear, but its final scores were only moderate (0.457). In contrast, Qwen3 models showed very low correlation between step-accuracy and final correctness, suggesting they might use “shortcut learning” to reach answers without explicit, transparent reasoning chains. This trade-off has practical implications: for educational or safety-critical applications, transparent reasoning might be prioritized, while for production systems needing consistent correct answers, models like Qwen3 or Hermes-4 might be preferred.
The research also highlighted domain-specific evolution and the potential of diverse architectures. Calculus, for instance, showed a dramatic improvement in model performance over time, becoming a top-ranked domain. Optimization, however, remained universally challenging across all models and generations. The study also validated that non-transformer architectures, such as the state-space model Falcon-Mamba-7B, can achieve competitive reasoning performance, sometimes even matching traditional transformer models like LLaMA-3.1-8B, with superior consistency.
Also Read:
- Assessing AI’s Mind: A Look into Large Language Models’ Reasoning Abilities
- A New Way for Large Language Models to Self-Verify Their Reasoning
Implications for Model Selection
Based on these findings, the paper identifies three archetypal model profiles: Traditional models (like Hermes-4-70B and Qwen3-235B) are optimized for final correctness, making them suitable for production systems. Reasoning-focused models (such as DeepSeek-R1 and GPT-OSS) prioritize transparent reasoning, ideal for educational or safety-critical applications. Balanced models (like Phi-4-mini) offer strong performance in both metrics, especially on less challenging problems, making them good for general-purpose reasoning tasks.
This research provides valuable guidelines for selecting foundation models across various contexts, from education and research to real-world production deployments. It underscores that future advancements in AI reasoning may depend more on sophisticated data curation and architectural design than on simply scaling up model parameters.
