TLDR: A new research paper introduces a human-in-the-loop AI framework for accelerating advertisement localization. It integrates scene text detection, Stable Diffusion-based inpainting, machine translation (Azure AI Translator), and geometry/typography-aware text reimposition (using LLMs for styling). The system significantly reduces localization time from days to minutes while maintaining high visual fidelity and semantic accuracy across multiple languages and locales.
In today’s global marketplace, advertisements need to resonate with diverse audiences across different cultures and languages. This process, known as ad localization, goes far beyond simply translating text. It involves maintaining the visual appeal, spatial arrangement, and overall style of an advertisement while adapting its linguistic content. Traditionally, this has been a labor-intensive and costly endeavor, relying heavily on human experts to manually review and adjust visual elements.
A new research paper introduces an innovative framework that combines the power of automated AI components with essential human oversight to streamline and accelerate the complex process of advertisement localization. This work is notable for integrating several advanced technologies—scene text detection, image inpainting, machine translation, and text reimposition—into a unified system designed specifically for evaluating ad localization workflows.
The Challenge of Ad Localization
Companies strive to make their products relatable to target audiences, and effective ad localization is crucial for aligning with local preferences and norms. While machine translation (MT) has made significant strides in handling text and speech, applying and evaluating translated content in visual advertisements has remained a challenge. Existing methods often involve extensive manual processing, leading to long turnaround times and high costs. Previous research has focused on individual aspects like optical character recognition (OCR) or image manipulation, but a comprehensive, scalable system for ad localization has been lacking.
A Step-by-Step AI-Powered Approach
The proposed system tackles these bottlenecks by breaking down ad localization into a modular, stepwise pipeline. This framework leverages deep learning and large language models (LLMs) to enhance quality and efficiency at each stage:
First, a deep learning-based scene text detection engine identifies text within the ad images. This engine incorporates a “human-in-the-loop” mechanism, allowing human annotators to refine detected text regions and categorize them (e.g., Brand, Translatable, Certification). This collaboration significantly boosts accuracy and reduces the average annotation time from 40 minutes to just 15 minutes.
Once text regions are identified, the system moves to background inpainting. This crucial step involves removing the original scene text without introducing any visual distortions, ensuring a clean canvas for the new translated text. The framework utilizes an automated pipeline based on Stable Diffusion, a powerful generative AI model. It intelligently regenerates masked text regions and selectively recombines them to seamlessly restore the background, eliminating the need for manual retouching.
For linguistic conversion, the system employs Azure AI Translator to translate the identified “translatable” text elements into the target languages. This service supports over 100 languages and provides accurate, context-aware translations, making it suitable for scalable ad localization needs.
The final and perhaps most intricate step is localized text reimposition. After translation and inpainting, the system renders the new text back into the image while preserving the original ad’s visual identity. This involves accurately recovering typographic attributes like font family, size, color, and style, as well as geometric layout properties such as position, rotation, and alignment. A deep classifier, based on the EfficientNet-B3 architecture, predicts the font family. Font size is adaptively estimated to ensure the translated text fits comfortably within the original bounding box, even if its length changes. For font color and style, large language models such as Claude 3.5 and GPT-4o are leveraged, taking context from the ad image to provide brand-consistent typographic cues. The system then renders the translated text onto a transparent canvas, applying rotation and precise alignment to maintain visual fidelity.
Also Read:
- Boosting Efficiency in UI Code Generation with Smart Token Compression
- Precise Image Generation with Layout Conditions Using Structured Masking
Impressive Results and Future Outlook
The framework was evaluated across six locales, demonstrating its effectiveness. For text recognition, the model achieved a high F1-score of 0.90, indicating reliable multilingual text detection. The visual quality of the localized outputs was assessed using the Learned Perceptual Image Patch Similarity (LPIPS) metric, which showed an average score of 0.067. This low score suggests that the inpainting and reimposition processes introduced minimal perceptual distortion, meaning the localized ads look very similar to the originals to the human eye.
Crucially, this AI-driven automation dramatically reduces the overall handling time for ad localization from hours or even days to just minutes. This significant efficiency gain makes the system highly suitable for industrial deployment.
This research represents a significant step forward in making advertisement localization faster, more accurate, and more scalable. The modular framework, detailed in the paper Human + AI for Accelerating Ad Localization Evaluation, lays the groundwork for future enhancements, including improved support for curved and stylized text, and the integration of vision-language models for even deeper semantic alignment.
