TLDR: VOTE is a novel framework for Vision-Language-Action (VLA) models that significantly enhances efficiency and generalization in robotic manipulation. It introduces a tokenizer-free fine-tuning method for parallel action prediction, drastically reducing computational overhead and accelerating inference. Additionally, an ensemble voting strategy improves model performance and robustness by aggregating historical action predictions. Experimental results show VOTE achieves state-of-the-art performance with 35x faster inference and lower training costs, making it highly suitable for real-time edge deployment.
Robotic systems are becoming increasingly sophisticated, capable of understanding natural language commands and performing complex tasks. This is largely thanks to Vision-Language-Action (VLA) models, which combine visual understanding, language processing, and robotic control. However, a significant challenge remains: these models often struggle when faced with new objects or unfamiliar environments not seen during their training. Furthermore, many existing solutions that try to improve this adaptability often add a lot of computational burden, making them slow and inefficient for real-world use.
A new framework called VOTE (Vision-Language-Action Optimization with Trajectory Ensemble Voting) has been introduced to tackle these limitations. VOTE aims to make VLA models more efficient and general, allowing robots to adapt more easily to novel situations without sacrificing speed.
Streamlined Action Prediction
One of VOTE’s core innovations is a novel fine-tuning approach that eliminates the need for a traditional ‘tokenizer’ for actions. In previous VLA models, actions were often broken down into many small ‘tokens’ that the model had to predict sequentially. This process was computationally intensive and slowed down inference. VOTE simplifies this by introducing a special token, <ACT>, which represents an entire chunk of actions. The model directly predicts continuous action values from the hidden state of this single token, enabling parallel computation and significantly reducing the number of tokens generated. This ‘tokenizer-free’ method drastically cuts down computational overhead and speeds up the inference process, making it much faster to fine-tune the model for new tasks with less data.
Ensemble Voting for Enhanced Reliability
Another key feature of VOTE is its unique ‘ensemble voting’ strategy for action sampling. Traditionally, robots would execute an action based solely on the most recent prediction from the VLA model. This can lead to instability if the current prediction is slightly off. VOTE addresses this by creating an ‘action sampling committee’ that considers not only the current action prediction but also relevant historical action predictions from previous steps. By comparing these predictions using a similarity measure (cosine similarity) and then ‘voting’ on the most consistent actions, VOTE can filter out unreliable predictions and arrive at a more robust and accurate final action. This adaptive voting mechanism helps to correct errors implicitly, leading to a more stable and reliable robotic trajectory.
Also Read:
- EC-Flow: Enabling Robots to Learn Complex Tasks from Unlabeled Videos
- Enhancing Multi-Task Reinforcement Learning Through Cross-Task Policy Guidance
Impressive Performance and Efficiency
Experimental results demonstrate VOTE’s significant advantages. It achieves state-of-the-art performance in robotic manipulation tasks, showing improved generalization across various benchmarks. For instance, VOTE boosts the average success rates of OpenVLA by over 20% across four LIBERO task suites and surpasses CogACT’s average success rate on the SimplerEnv WidowX Robot. More impressively, VOTE delivers a remarkable 35 times faster inference speed and achieves a throughput of 145 Hz on an A6000 GPU, and 42 Hz on an NVIDIA Jetson Orin edge device. This makes it highly suitable for real-time robotic control where quick responsiveness is crucial. The framework also boasts lower training costs and minimal memory overhead compared to other methods.
By combining a tokenizer-free fine-tuning approach with an intelligent ensemble voting strategy, VOTE offers a powerful and efficient solution for developing general-purpose robotic policies. Its ability to achieve high performance with significantly faster inference and reduced training costs marks a notable step forward in making advanced robotic manipulation more practical and widely deployable. The code for VOTE is available for further exploration and development. You can find more details in the research paper: VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting.
