TLDR: Geometric Red-Teaming (GRT) is a new framework that automatically generates “CrashShapes”—subtly deformed objects—to expose critical failure modes in robotic manipulation policies. It uses a simulator-in-the-loop optimization and can be guided by AI models. These CrashShapes not only diagnose vulnerabilities but can also be used to fine-tune policies, significantly improving robot robustness in both simulation and real-world tasks.
A new research paper titled “Geometric Red-Teaming for Robotic Manipulation” introduces an innovative framework designed to uncover vulnerabilities in robotic systems. Authored by Divyam Goel, Yufei Wang, Tiancheng Wu, Guixiu Qiao, Pavel Piliptchak, David Held, and Zackory Erickson, this work addresses a critical gap in robotic evaluation: understanding how systems fail under realistic variations in object geometry.
Traditional methods for evaluating robotic manipulation policies often rely on carefully selected, in-distribution test sets. While these benchmarks show how well a robot performs under ideal conditions, they offer limited insight into how policies might break down when faced with subtle, yet plausible, changes in an object’s shape. Such geometric shifts can unexpectedly alter how a robot interacts with an object, disrupting contact dynamics and leading to task failure.
The researchers propose Geometric Red-Teaming (GRT), a framework inspired by cybersecurity practices where systems are proactively tested for weaknesses. GRT automatically generates what they call “CrashShapes”—structurally valid, user-constrained mesh deformations of objects that are specifically designed to trigger catastrophic failures in pre-trained manipulation policies. Imagine a robot that can perfectly grasp a standard screwdriver, but completely fails when the screwdriver handle has a slight, almost unnoticeable, bulge or indentation. GRT aims to find these critical, failure-inducing deformations.
The core of GRT involves a sophisticated process. It integrates a Jacobian field-based deformation model, which allows for smooth and physically plausible changes to an object’s 3D mesh. This is combined with a gradient-free, simulator-in-the-loop optimization strategy. This means the system iteratively deforms an object in a simulated environment, observes the robot’s performance, and then refines the deformation to maximize policy failure, all without needing direct gradient information from the robot’s policy or the simulator.
To make the deformation process more intelligent and task-relevant, GRT can leverage a vision-language model (VLM) like ChatGPT-4o. This VLM helps in selecting “handle points” and “anchor points” on an object’s surface. Handle points are areas that can be actively displaced, while anchor points remain fixed, ensuring that deformations are semantically coherent and physically plausible. For instance, the VLM can identify the critical contact regions of a USB plug for insertion, guiding the deformation to those specific areas most likely to cause failure.
GRT was rigorously tested across various robotic manipulation tasks, including high-precision industrial insertion, articulated drawer manipulation, and rigid-object grasping. In all these scenarios, GRT consistently discovered deformations that significantly degraded policy performance, revealing brittle failure modes that static benchmarks typically miss.
Beyond just identifying failures, the research also introduces “blue-teaming.” This process involves fine-tuning the robot’s policies on the discovered CrashShapes. The results were remarkable: fine-tuning on individual CrashShapes improved task success by up to 60 percentage points on those specific deformed shapes, while crucially preserving performance on the original, undeformed object. This demonstrates that CrashShapes are not just diagnostic tools but also actionable signals for targeted policy refinement.
The findings were further validated with real-world robotic arms. Fabricated CrashShapes for tasks like USB insertion on an xARM 6 and grasping on a Franka arm showed that simulated CrashShapes reduced task success from 90% to as low as 22.5%. Impressively, blue-teaming recovered performance to up to 90% on the corresponding real-world geometries, closely matching simulation outcomes. This real-world transferability underscores the practical utility of GRT.
Also Read:
- Simulating Intelligent Attacks to Strengthen Robot Patrol Security
- Designing Robots on Demand: An AI Co-Pilot for Tailored Manipulators
In conclusion, Geometric Red-Teaming offers a scalable and policy-agnostic framework for structured, object-centric robustness evaluation in robotic manipulation. By automatically generating physically plausible, failure-inducing object geometries, GRT provides a powerful tool for diagnosing vulnerabilities and, through blue-teaming, offers a practical pathway to enhance the robustness of robotic systems in the real world. You can find more details about this research at the project website: https://arxiv.org/pdf/2509.12379.
