TLDR: This research introduces a method to efficiently train small Vision Language Models (VLMs) for predicting product attributes in e-commerce. By using Parameter-Efficient Fine-Tuning (PEFT) and Direct Preference Optimization (DPO) on abundant unlabeled product images, the approach significantly improves accuracy over traditional supervised methods, overcoming the high costs of manual data annotation.
In the fast-paced world of e-commerce, accurately predicting product attributes from images is a critical task. This helps with everything from automating product listings to ensuring quality control. However, using large, powerful Vision Language Models (VLMs) for this purpose comes with significant challenges. These models are computationally expensive, require a lot of memory, and fine-tuning them with manually labeled data is impractical due to the sheer volume of products in online catalogs. Even using APIs for automatic labeling can become prohibitively expensive at scale.
A new research paper, “Efficient Learning for Product Attributes with Compact Multimodal Models,” explores a smart solution to these challenges. Authored by Mandar Kulkarni from Flipkart Data Science, this work focuses on making VLM training more efficient and scalable by leveraging the vast amounts of readily available unlabeled product images. You can read the full paper here: Efficient Learning for Product Attributes with Compact Multimodal Models.
The Challenge of Scale in E-commerce
Traditional methods often rely on supervised fine-tuning, where models learn from meticulously labeled datasets. But for e-commerce, with millions of items across diverse categories like “sandal,” “kurta,” or “suitcase,” manual labeling is simply not feasible. Seller-provided data can also be noisy, requiring extensive quality checks. This highlights the need for methods that can learn effectively without relying heavily on expensive, human-annotated data.
A Novel Approach: Compact VLMs with Unlabeled Data
The researchers investigated the viability of smaller VLMs, specifically Qwen-family models (2-3 billion parameters), for image-based attribute prediction. To minimize computational overhead, they employed Parameter-Efficient Fine-Tuning (PEFT). PEFT allows for updating only a small fraction of the model’s parameters (low-rank adapter weights) while keeping the core VLM frozen, making the training process much more efficient.
The core innovation lies in their semi-supervised fine-tuning strategy, which utilizes Direct Preference Optimization (DPO) with unlabeled product listings. The process begins with a small, initial labeled dataset, generated using a GPT-4o-mini-based pipeline. This seed data is used to fine-tune the PEFT adapters.
Leveraging Self-Consistency with Direct Preference Optimization (DPO)
For the abundant unlabeled data, the method generates multiple “reasoning-and-answer” chains for each product image. These chains are then categorized as “preferred” or “dispreferred” based on their self-consistency – essentially, how often a particular answer appears across the generated chains. The most consistent prediction is considered the “consensus” label.
The model is then fine-tuned using DPO loss, which directly aligns the model’s predictions with these preferred outputs. This iterative process involves regenerating pseudo-labels with the updated model and then further fine-tuning. To ensure quality, only predictions exceeding a certain confidence threshold (50% in their experiments) are used for training, filtering out highly uncertain pseudo-labels.
Key Findings and Advantages
The experimental results were compelling. DPO-based fine-tuning on pseudo-labeled data led to significant improvements in attribute-prediction accuracy. For instance, a PEFT-adapted model saw its accuracy jump from 0.751 to 0.857 after DPO training, a substantial gain. Even directly fine-tuning a pre-trained model with unlabeled data showed an impressive 11% accuracy improvement.
Crucially, the study found that accuracy with DPO training consistently improved as the amount of unlabeled data increased. This indicates that DPO can effectively harness large pools of unlabeled samples to enhance performance, a stark contrast to other methods like Self-Learning (SL), which showed degraded accuracy in small VLM settings due to potential model collapse and reinforcement of early mistakes.
The research also highlighted the importance of generating an explicit reasoning trace during training. Models trained to produce both reasoning and the final answer significantly outperformed those trained only on the answer, even for seemingly straightforward attribute tasks.
Also Read:
- Enhancing VLM Performance Through Smart Data Filtration
- TransPrune: Boosting Efficiency in Large Vision-Language Models Through Token Transition Analysis
Conclusion
This paper demonstrates a powerful and efficient way to improve small VLMs for e-commerce attribute prediction. By cleverly using Direct Preference Optimization with self-consistency on unlabeled data, the need for costly manual annotations is eliminated. Combined with Parameter-Efficient Fine-Tuning, this approach offers a scalable and computationally efficient solution, proving that large unlabeled datasets can be effectively leveraged to enhance VLM performance without sacrificing stability. This methodology can be applied to improve any supervised fine-tuned VLM model that produces objective outputs.
