TLDR: Capybara is a new AR-based, AI-powered visual programming tool that lets children create and animate 3D characters and program their interactions with the physical world using speech-to-3D generative AI, body motion tracking, and object detection. User studies show it empowers creativity and teaches AI literacy, though challenges like AI output alignment and potential distractions exist.
Artificial Intelligence (AI) and Augmented Reality (AR) are rapidly changing how we interact with technology, but often, children are just consumers of these experiences. A new research paper introduces Capybara, an innovative tool designed to transform children from passive users into active creators of AI-enabled AR experiences. This system empowers young learners to build, customize, and program 3D characters that interact with their physical surroundings.
What is Capybara?
Capybara is a visual programming environment built for mobile Augmented Reality. It uses a block-based coding system, similar to popular tools like Scratch, making it intuitive and accessible for children. The core idea is to allow children to overlay their own custom 3D characters onto the real world and define how these characters behave and interact with physical objects.
How Capybara Works
The system offers three novel functionalities that significantly enhance creative expression:
First, children can customize 3D characters and accessories using generative AI. By simply describing their desired character or item using speech, the system employs a text-to-3D generative AI model to create it. For instance, a child could say, “a cute panda character wearing a wizard hat,” and Capybara would generate it. The system also includes a moderation step to ensure the content is appropriate for young audiences.
Second, Capybara allows for animation customization through a technique called puppeteering. After a character is generated, the system automatically prepares it for animation. Children can then use their own body motions to control the character’s movements, effectively puppeteering their virtual creations. These custom animations can be recorded and replayed programmatically within their AR experiences.
Third, the tool enables programming interactions between virtual characters and the physical world. Using vision-based AI models, Capybara can recognize physical objects (like a banana or a laptop) or user-defined zones in the environment. Children can then program their virtual characters to react to these real-world elements. For example, a character could wave its arms to keep balance when stepping on a physical banana, or start a typing animation when touching a physical keyboard.
The programming interface uses five categories of drag-and-drop code blocks: Event Blocks (to trigger actions), Motion Blocks (for movement), Looks Blocks (for visual elements and animations), Control Blocks (for loops and conditions), and Sensing Blocks (for detecting objects and zones in the physical world). This clear separation between building programs on-screen and triggering actions with physical gestures simplifies the creative process.
Real-World Examples and User Feedback
To demonstrate its capabilities, Capybara has been used to create diverse AR experiences, such as a tiger character preparing for the day by putting on a cowboy hat and interacting with a laptop, or a language learning scenario where a character wears different accessories (like an apple or banana) when touching specific zones representing letters.
User studies conducted with 20 children aged 7 to 16 in the United States and Argentina showed promising results. Participants found Capybara easy to use, especially those familiar with Scratch, and enjoyed the unique 3D coding environment. They particularly appreciated the creative freedom offered by AI-enabled customization, allowing them to personalize characters and animations to reflect their own ideas and movements. Many expressed a desire for even more customization, including sound effects and virtual environments.
However, the studies also highlighted some challenges. Children were aware of the AI’s presence and sometimes experienced frustration when the AI-generated results didn’t perfectly align with their intentions (e.g., a generated capybara with “really, really big teeth”). There were also instances where children became overly focused on refining their prompts, potentially distracting them from their broader creative goals. Despite these challenges, the experience was largely seen as educational and enjoyable, fostering computational thinking and AI literacy by allowing children to explore AI’s capabilities and limitations firsthand.
Also Read:
- LL3M: Crafting 3D Models with Language Models and Code
- Adapting Introductory Programming Courses for AI Tools
Looking Ahead
The development of Capybara marks a significant step towards empowering children as creators in the evolving landscape of AI and AR. Future research will focus on enhancing the system’s understanding of the physical world for more intricate interactions, balancing AI automation with children’s creative agency, and providing structured educational support to cultivate deeper computational thinking and AI literacy. The full research paper can be found here: Empowering Children to Create AI-Enabled Augmented Reality Experiences.
