TLDR: AInstein is a new framework that tests if large language models (LLMs) can solve AI research problems using only their internal knowledge. It extracts problems from research papers and uses LLM agents to propose and refine solutions through iterative critique. The study found that while LLMs can rediscover feasible solutions and sometimes create novel ones, their problem-solving is sensitive to problem framing, and the internal model’s strength is key. LLMs often generate valid, original alternatives rather than perfectly replicating human solutions.
Large language models (LLMs) have shown remarkable abilities across many tasks, but it’s often debated whether this success comes from genuine reasoning or just remembering information. A new framework called AInstein has been introduced to explore this question, specifically testing if LLMs can create valid solutions to AI research problems using only their existing knowledge, without any special fine-tuning or external help.
The AInstein framework operates in two main phases. First, a “Problem Extraction Phase” distills complex scientific abstracts from high-quality research papers into clear, concise research challenges. The goal here is to capture the essence of the problem without revealing the original solution. This ensures that the LLM agents tasked with solving these problems aren’t just paraphrasing existing work.
Following this, the “Solution Generation Phase” involves specialized “solver agents” – which are also LLMs – proposing and refining technical solutions. This process mimics the real-world scientific cycle of proposing ideas, getting them reviewed, and then revising them based on feedback. The framework uses iterative critique loops, both internal (self-correction by the LLM) and external (review by a stronger LLM), to refine the problem statements and solutions.
To evaluate AInstein, researchers used a dataset of 1,214 papers from the ICLR 2025 conference, categorized by their acceptance tier (Oral, Spotlight, Poster). An “LLM-as-a-judge” paradigm was employed, where an LLM (GPT-OSS-120B) assessed the generated solutions against a structured rubric. This was complemented by targeted manual checks to ensure reliability. The performance was measured using three key metrics: Success Rate (whether the solution addresses the problem), Rediscovery (how well it aligns with human-proposed methods), and Novelty (if it offers valid, original approaches).
The findings from the AInstein study are quite insightful. While LLMs can indeed rediscover feasible solutions and sometimes propose creative alternatives, their problem-solving abilities are still somewhat fragile and highly dependent on how the problem is framed. A significant discovery was that the capability of the “internal model” (the LLM doing the self-correction) was the most crucial factor for success. For instance, the GPT-OSS-120B internal agent consistently outperformed others, achieving a 74.05% success rate in strict evaluations when solving problems from the GPT-OSS-120B generalizer.
The study also highlighted a clear difference between rediscovering an existing solution and creating a novel, valid one. Under a relaxed evaluation, LLMs showed high “Rediscovery” rates, meaning they often came up with solutions conceptually similar to human ones. However, when the criteria were tightened to demand functional equivalence, these rates dropped significantly. This suggests that perfect rediscovery is rare. Interestingly, the “Novel & Valid” metric remained high and stable, indicating that when LLMs don’t perfectly match human solutions, they often generate alternative, scientifically sound approaches.
Another interesting result was that the LLM’s ability to generate valid solutions wasn’t significantly affected by the perceived difficulty or prestige of the original paper (Oral, Spotlight, or Poster tier). This suggests a consistent problem-solving capability across different levels of research complexity.
The research also included human-verified competitive rankings, where human evaluators compared solutions. This confirmed the superior performance of the GPT-OSS-120B self-play configuration, ranking it second only to ground-truth human abstracts. Quantitative text analysis further supported these findings, showing strong conceptual alignment and higher technical complexity in solutions from more capable models.
Also Read:
- Targeted Expert Guidance Enhances LLM Reasoning and Exploration
- Enhancing LLM Performance with Evolving Contexts: Introducing Agentic Context Engineering (ACE)
In conclusion, the AInstein framework provides the first large-scale evidence on the extent to which LLMs can act as autonomous scientific problem-solvers. It reveals both their potential for scientific creativity and their current limitations, emphasizing that while they can generate novel and valid solutions, their reasoning remains sensitive to framing. This work advances our understanding of LLMs not just as tools for research, but as potential independent problem-solvers. You can read the full paper here.
