TLDR: Tenma is a lightweight diffusion-transformer model that enables robots to perform complex manipulation tasks robustly across different robot types and diverse environments. It achieves this by integrating multimodal data (vision, proprioception, language) into a shared latent space using a cross-embodiment normalizer, a Joint State–Time encoder for efficient observation learning, and a diffusion action decoder for stable action generation. Tenma significantly outperforms existing methods in success rates, especially under object and scene variations, demonstrating strong generalization with moderate data and model size.
Robotics is taking significant strides, especially with the integration of advanced AI models like Transformers and diffusion models. However, building lightweight, adaptable robot systems that can learn across different types of robots and handle various kinds of data simultaneously has been a persistent challenge. A new research paper introduces a novel solution called Tenma, a diffusion-transformer designed to make robot manipulation more robust and versatile across different robot designs.
The core problem Tenma addresses lies in two main areas: multimodal learning and cross-embodiment heterogeneity. Multimodal learning refers to the challenge of processing diverse data inputs, such as visual information (RGB video), robot movement data (proprioception and control signals), and language commands, all at once. Traditional methods often struggle to integrate these varied data types effectively without losing crucial details or becoming computationally expensive. Cross-embodiment heterogeneity, on the other hand, deals with the fact that robots come in many shapes and sizes, with different ways of controlling their movements (e.g., joint angles, end-effector poses) and varying numbers of arms. A truly intelligent robot policy needs to understand and operate across these diverse physical forms.
Tenma tackles these complexities with a clever architecture. It integrates multiview RGB images, proprioception (the robot’s sense of its own body position), and language instructions. A key innovation is its cross-embodiment normalizer, which translates the unique state and action spaces of different robots into a shared, common language. This allows the system to learn from a wide range of robot data without interference. Another crucial component is the Joint State–Time Encoder, which helps the robot learn from observations that are aligned in time, significantly boosting inference speed. Finally, a diffusion action decoder is optimized for stable training and enhanced learning capabilities, enabling the robot to generate precise and robust actions.
The model’s design is particularly efficient. It uses a T5-small language encoder and ViT-S-sized transformer blocks, making it parameter-efficient. For stability, it incorporates RMSNorm and QKNorm throughout its non-frozen modules. The input data is tokenized using modality-specific methods: DINOv2-S for RGB vision, T5-S for language, and a standardization process for proprioception. This ensures all inputs are converted into a compatible format for the transformer.
In experiments, Tenma was evaluated on bi-manual arm control tasks, specifically four food-preparation tasks: Pour, PickCup, PlacePan, and Stirring. These tasks were chosen to test different aspects of manipulation, from precise rotations to robust grasping and long-horizon control. The evaluation considered three axes of generalization: in-distribution performance (on familiar tasks), object-shift (with novel objects), and scene-shift (with varied backgrounds and lighting). Tenma was compared against other state-of-the-art diffusion-transformer policies like DiT-Policy, Diffusion Policy, and Octo.
The results were impressive. Tenma achieved an average success rate of 88.95% in-distribution, significantly outperforming baseline policies, whose best average was 18.12%. Even under challenging object and scene shifts, Tenma maintained strong performance, with success rates of 72.56% and 81.13% respectively. This demonstrates Tenma’s remarkable robustness and generalization capabilities. For instance, in the PickCup task, Tenma achieved 100% success in-distribution and under scene-shift, and 87.50% under object-shift, indicating reliable grasping even with different cup appearances.
The researchers attribute Tenma’s success to several architectural factors. Its use of pretrained DINOv2-S for vision provides richer perceptual features compared to the ResNet-based encoders used by baselines. The cross-embodiment standardizer creates a unified representation for diverse robots, ensuring stable training. Furthermore, the Joint State–Time encoder combined with a DiT-based decoder allows for fine-grained, temporally consistent action generation. Unlike some baselines that compress observations into a single conditioning token, Tenma processes multiple observation tokens per timestep, avoiding information bottlenecks.
Despite using moderate data scale and a compact policy, Tenma delivers robust manipulation and generalization. This suggests that thoughtful multimodal representation design and targeted data curation can yield greater benefits than simply scaling up data or model size. The work highlights the potential of multimodal and cross-embodiment learning strategies to further enhance transformer-based imitation learning policies. For more technical details, you can refer to the full research paper here.
Also Read:
- Safe and Efficient Robot Skill Acquisition Through Self-Augmented Trajectories
- Probabilistic Structure Integration: A New Framework for Controllable World Models
In conclusion, Tenma represents a significant step forward in developing scalable, robust, and adaptable robotic policies, bringing embodied AI closer to practical deployment in real-world scenarios, particularly for complex tasks like food preparation.
