TLDR: CrochetBench is a new benchmark evaluating multimodal AI models’ ability to perform detailed, low-level procedural reasoning in crochet. It shifts focus from describing visuals to generating executable crochet procedures using a domain-specific language, CrochetPARADE. The benchmark includes tasks like stitch recognition, instruction selection, instruction generation, and natural language to DSL translation. Findings show that while AI models can recognize stitches, their performance significantly declines when required to generate compilable instructions, exposing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. This highlights a critical gap between surface-level understanding and executable precision in creative domains.
A new benchmark called CrochetBench has been introduced to evaluate how well multimodal large language models (VLMs) can perform detailed, low-level procedural reasoning in the craft of crochet. This benchmark shifts the focus from simply describing visual information to enabling AI models to actively ‘do’ by generating compilable crochet procedures. The research paper, titled “CROCHET BENCH : C AN VISION -LANGUAGE MODELS MOVE FROM DESCRIBING TO DOING IN CROCHET DOMAIN ?”, was authored by Peiyu Li, Xiaobao Huang, and Nitesh V. Chawla from the University of Notre Dame.
Traditional benchmarks for multimodal AI often focus on high-level descriptions or visual question answering. However, crochet patterns are unique because they combine structured symbolic language (stitch abbreviations), long-form natural language for context, and visual evidence like photographs. Successfully creating crochet patterns requires not just understanding across these different types of information, but also step-by-step reasoning that maintains procedural accuracy, similar to program synthesis.
CrochetBench addresses this gap by focusing on instructional fidelity. It asks whether models can not only recognize and generate, but also produce step-wise, compilable instructions that respect symbolic, numerical, and topological structures. The benchmark uses CrochetPARADE, a domain-specific language (DSL), as an intermediate representation. This allows for structural validation and functional evaluation through execution, meaning the AI’s output can be automatically checked for correctness, unlike crafts like cooking which often require real-world execution.
The dataset for CrochetBench is extensive, comprising 6,085 crochet patterns from the Yarnspirations website, covering 55 distinct project categories. These patterns are converted into a machine-readable JSON format, with nearly all entries including an associated product image for multimodal supervision. The dataset also categorizes patterns by skill level (beginner, easy, intermediate, experienced), allowing for evaluation across different complexity tiers.
Also Read:
- FaithAct: A Framework for Verifying AI’s Visual Reasoning Steps
- A New Benchmark for Evaluating AI in Electronic Health Records: Introducing EHRStruct
Benchmark Tasks and Findings
CrochetBench includes four main tasks designed to test models across progressively challenging stages:
-
Stitch Recognition (Task A): This task evaluates a model’s ability to identify different crochet stitch types from an image of a finished product. Models are expected to output a comma-separated list of standard U.S. crochet abbreviations. While commercial models like Claude Sonnet 4 showed strong precision, and DeepSeek-VL had high recall among open-source models, there’s a clear difference in performance between commercial and open-source VLMs.
-
Instruction Selection (Task B): Here, models must align a finished crochet product image with the correct natural-language instruction from a set of four multiple-choice options. Distractor instructions are intentionally similar to increase difficulty. GPT-4o achieved the highest accuracy, significantly outperforming open-source models, indicating the challenge of fine-grained visual-textual alignment.
-
Instruction Generation (Task C): This task requires models to generate natural-language crochet instructions directly from an image. Outputs are expected to follow domain-specific syntax and stitch logic. Gemini 2.5 Flash-Lite performed best, but overall scores were low, highlighting the difficulty in generating structurally faithful instructions that capture both visual details and procedural logic.
-
Instruction-to-DSL Translation (Task D): This is the most demanding task, requiring models to translate natural language instructions into compilable CrochetPARADE DSL code. It has two variants: step-level (translating single instructions) and project-level (generating a complete program for an entire pattern). Evaluation is based on Compilation Success Rate (CSR) and Partial Executable Rate (PER). Even top closed-source models rarely exceeded 6% project-level compilation success, while Qwen2-VL surprisingly achieved 21%, suggesting that some open-source models might be better at symbolic generalization under execution constraints. Errors often involved syntax issues or undefined stitches.
The research reveals that while current models can perform low-level recognition and mid-level comprehension, their performance drops significantly when asked to generate or formalize instructions into compilable programs. This highlights limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. The findings underscore the critical gap between surface-level understanding and executable precision in real-world creative domains.
The authors suggest future work should focus on developing models that can translate free-form natural language into the formal CrochetPARADE DSL, treating crochet as a program synthesis problem. This could involve incorporating explicit state-tracking mechanisms, multimodal pretraining with procedural and topological domains, and hybrid evaluation pipelines that combine compilation checks with visual render comparisons. For more details, you can refer to the full research paper.
CrochetBench and CrochetPARADE represent a significant step forward in multimodal learning, program synthesis, and digital fabrication, paving the way for AI models that can reason in structured 3D spaces and potentially automate textile production from human intent.
