TLDR: This research introduces ‘Reverse Browser,’ a novel image-to-code generator that uses vector images instead of bitmaps as input. By creating large datasets and a new image quality metric (MSPS), the author, Zoltan Toth-Czifra, trained a large language model (Llama 3.2 90B) to convert vector designs into HTML+CSS. The results show that this vector-based approach achieves competitive accuracy compared to bitmap-based methods, suggesting a promising direction for automating UI development, despite current limitations in commercial viability.
The process of converting user interface (UI) designs into functional code, often referred to as image-to-code or image-to-UI, is a significant challenge in software engineering. Current methods, which typically rely on bitmap images, struggle to achieve high fidelity to the original design.
A New Perspective: Vector Images for Code Generation
This research introduces a novel approach by utilizing vector images as input for machine learning models, rather than traditional bitmaps. Vector images, composed of mathematical descriptions of shapes, text, and embedded images, offer a distinct advantage. They possess lower intrinsic dimensionality and inherently contain explicit structural information, which is crucial for generating accurate UI code like HTML and CSS. This contrasts with bitmaps, where structural information must be inferred, often leading to inaccuracies.
Building the Foundation: Datasets and Metrics
To support this new approach, the author, Zoltan Toth-Czifra, created several large datasets for training machine learning models. These datasets were generated by rendering web pages in a headless browser and constructing vector images (specifically SVG files) paired with their corresponding HTML and CSS markup. The process involved simplifying and compressing both the web pages and SVG files, with quality control checks using screen captures.
A significant contribution of this work is the development and release of a new Image Quality Assessment (IQA) metric called Multi-Scale Pixel Similarity (MSPS). While existing metrics like LPIPS are useful for tracking training loss, MSPS provides a stricter, pixel-level comparison, essential for filtering out imperfect conversions during dataset generation. MSPS calculates the mean squared error of pixel values across progressively reduced sampling densities of an image, ensuring robust quality assessment even against small translations.
Training the “Reverse Browser” Model
The research involved proof-of-concept training experiments on smaller models using synthetic data, demonstrating that transformer models could learn the translation between simple vector images and markup. For more complex, real-world web data, the Llama 3.2 model family was chosen as the base. The largest variant, a 90B parameter model, was fine-tuned on a public web dataset, which was filtered for length due to computational constraints. This fine-tuning process, taking over 392 hours on a single H100 GPU, aimed to create a robust vector-image-to-code model, dubbed “Reverse Browser.”
Also Read:
- Measuring Quality in Text-to-SVG: Introducing SVGauge, a Human-Aligned Metric
- Evaluating AI-Generated 3D Models: A Quantitative Human-in-the-Loop Framework
Performance and Future Outlook
While the final model’s accuracy scores (0.9530 MSPS and 0.3012 LPIPS) indicate it’s not yet ready for commercial real-world software engineering workflows, the results are promising. When benchmarked against existing bitmap-to-webpage tasks, the “Reverse Browser” model achieved accuracy scores comparable to, and in some cases significantly better than, large commercial multimodal Foundation Models. This suggests that vector images indeed offer a more appropriate input choice for UI image-to-code tasks.
The paper acknowledges limitations, including the model’s current speed and the need to address responsive design, functional aspects (like distinguishing buttons from text inputs), and code quality. Future work will explore reinforcement learning with verifiable rewards, larger and higher-quality training data, and potentially multimodal models that combine both vector and bitmap inputs. For more in-depth technical details, you can read the full research paper here: Reverse Browser: Vector-Image-to-Code Generator.
