TLDR: A new research paper introduces a highly clean Japanese recipe dataset with detailed ingredient state annotations to help Large Language Models (LLMs) better understand cooking processes. The dataset enables three novel tasks—Step Reference Resolution, Ingredient Usage Prediction, and Ingredient Tracing—to evaluate LLMs’ ability to track ingredient transformations. Experiments show that larger LLMs (over 70B parameters) significantly improve their understanding of recipes after fine-tuning on this dataset, with Large Multimodal Models (LMMs) showing even stronger performance, though challenges remain in handling complex ingredient manipulations like discarding or splitting.
Large Language Models (LLMs) have learned from a vast amount of text, including cooking recipes. However, they don’t directly observe real-world phenomena. This creates a challenge in understanding recipes, as intermediate states of ingredients (like whether an egg is raw, whisked, or boiled) are often not explicitly mentioned. This makes it difficult for LLMs to accurately track ingredient states and fully comprehend cooking procedures.
To address this, researchers Mashiro Toyooka, Kiyoharu Aizawa, and Yoko Yamakata have applied a method called ‘state probing’ to the domain of cooking. State probing is used to evaluate how well a language model understands the real world. They propose a new task and dataset specifically designed to assess how well LLMs can recognize these intermediate ingredient states during cooking.
The team constructed a new Japanese recipe dataset using ‘NHK Recipes,’ which are known for being very clean, well-structured, and consistently normalized. This high-quality dataset allowed them to focus on the complex task of annotating ingredient state changes at each step of the cooking process. The dataset includes annotations describing how ingredients transform and their state at every stage.
Evaluating LLM Understanding
Using this unique dataset, three novel tasks were designed to evaluate LLMs’ ability to track ingredient state transitions and identify ingredients present at intermediate steps:
-
Step Reference Resolution Task: This task involves predicting masked step numbers within recipe instructions. LLMs need to infer ingredient states and understand what operations can be performed on them to correctly identify the referenced step.
-
Ingredient Usage Prediction Task: Here, the model predicts whether a specific ingredient from the initial list remains unused at the end of a given cooking step. This requires the LLM to determine which ingredients are involved in each step and trace their usage.
-
Ingredient Tracing Task: In this task, the model must identify which intermediate ingredients at a certain step contain a specified initial ingredient. This demands simulating the cooking process and understanding the input-output relationships of ingredients through various transformations.
Key Findings from Experiments
Experiments were conducted with widely used LLMs such as Llama3.1-70B, Qwen2.5-72B, and even Large Multimodal Models (LMMs) like Qwen-2.5-VL. The results showed a clear correlation between model size and performance on these tasks, indicating that the proposed tasks are effective in evaluating LLM and LMM capabilities.
Models with fewer than 10 billion parameters generally performed poorly, and their accuracy sometimes even decreased after fine-tuning. This suggests that a certain model size (at least 10 billion parameters) is necessary for LLMs to effectively learn how to infer states from recipe texts. In contrast, larger models (over 70 billion parameters) showed substantial performance improvements after fine-tuning, even with a relatively small training dataset. This improvement is attributed to the cleanliness and consistent, high-quality annotations of the dataset.
Interestingly, the 72B Large Multimodal Model (LMM) outperformed LLMs of the same size in task accuracy. The researchers hypothesize that providing an image of the completed dish helps the LMM clarify the overall cooking trajectory, aiding in the inference of ingredient states. Pretraining on datasets with step-by-step images might also enable models to visually learn intermediate ingredient states, further enhancing recipe understanding.
Also Read:
- Enhancing Multimodal AI: A New Approach to Visual Understanding in Context
- Refining Language Model Training Data with Influence Functions
Challenges and Future Work
Despite the improvements, qualitative analysis revealed that even highly capable LLMs and LMMs still struggle with certain aspects of recipe understanding. Common failure cases include scenarios involving ingredient discarding (where an ingredient is explicitly removed), splitting (where a single ingredient is divided and used in different ways), and later recombination. These challenges highlight the difficulty models face in reasoning about subtraction or duplication of ingredients and recognizing logically parallel actions.
The dataset used in this research is publicly available on Hugging Face Hub, allowing other researchers to further explore and enhance the state probing capabilities of LLMs and LMMs in the context of cooking recipes. You can find more details about this research in the paper: A Highly Clean Recipe Dataset with Ingredient States Annotation for State Probing Task.
