TLDR: This research introduces a new method called Selective Contrastive Learning for Weakly Supervised Affordance Grounding (WSAG). It helps AI models identify functional parts of objects for specific actions (e.g., a cup’s handle for holding) using limited data. The method adaptively learns from both object and part-level cues, distinguishing relevant areas from backgrounds and other actions. It uses “prototypical” and “pixel” contrastive learning, along with a map calibration step, to improve localization accuracy and generalize better to new objects, outperforming previous methods.
Understanding how humans interact with objects is a fundamental challenge in artificial intelligence, especially for robots that need to perform actions in the real world. A key aspect of this is “affordance grounding,” which involves identifying which parts of an object allow for specific actions. For example, knowing that a cup’s handle is for holding, or a knife’s blade is for cutting.
Traditionally, teaching AI models this skill requires extensive, detailed annotations, like drawing precise outlines around every “affordable” part in an image. This is incredibly time-consuming and expensive. To overcome this, researchers are exploring “weakly supervised affordance grounding” (WSAG), where models learn from less precise information, mimicking how humans intuitively grasp functional parts without needing pixel-perfect instructions.
A recent research paper, “Selective Contrastive Learning for Weakly Supervised Affordance Grounding”, introduces a novel approach to tackle the limitations of existing WSAG methods. Previous models often struggled because they tended to focus on general patterns associated with an object class (e.g., the whole bicycle for “riding”) rather than the specific, often subtle, parts that enable an action (like the seat or handlebars). This happens because affordance-relevant clues aren’t always clearly distinguishable, leading models to rely on broader classification cues.
The core innovation in this paper is the introduction of “selective prototypical and pixel contrastive objectives.” This means the model learns adaptively, focusing on affordance-relevant cues at both the part level (e.g., the handle) and the object level (e.g., the entire cup), depending on how clear the information is. If a specific part is reliably identified as affordable, the model learns to distinguish it from other irrelevant parts. If not, it shifts its focus to distinguishing the entire target object from the background, ensuring it still learns something useful.
The process begins by identifying action-associated objects in both “egocentric” (object-focused, first-person view) and “exocentric” (third-person view, showing human-object interaction) images. This is done using a powerful AI model called CLIP, which helps create an “object affinity map” highlighting relevant objects. These object-level clues are then refined to pinpoint precise part-level affordance clues in each perspective.
The “prototypical contrastive learning” component helps the model learn from the exocentric images. Unlike simpler methods that just try to match representations, this approach teaches the model to not only align egocentric and exocentric views but also to differentiate between prototypes of affordable parts/objects, background information, and prototypes of other action classes. This makes the model’s understanding of each action class more precise.
To further enhance precision, “pixel contrastive learning” is employed. This directly uses the identified affordance-relevant clues in egocentric images to teach the model to distinguish individual pixels that are part of an affordable region from those that are not. This fine-tunes the model’s ability to localize affordable parts with high accuracy.
Finally, the researchers introduced a “calibration process” for the Class Activation Map (CAM), which is the model’s output localization map. CAMs can sometimes spread activations beyond the actual object boundaries. By combining the CAM prediction with the binarized object affinity map, the model’s attention is limited only to the salient, relevant parts, leading to more accurate and refined localization.
Also Read:
- Beyond Localization: Invert4TVG Improves AI’s Grasp of Actions in Videos
- SkillNav: Teaching AI to Navigate by Mastering Atomic Skills
Experimental results on datasets like AGD20K and HICO-IIF demonstrate that this new method significantly outperforms previous approaches, especially in “unseen scenarios” where the model encounters novel objects it hasn’t been specifically trained on. This is a crucial step towards developing more robust and adaptable AI systems that can interact with the world in a human-like manner.
