TLDR: GLYPH-SR is a new image super-resolution method that simultaneously improves both the visual quality of images and the legibility of text embedded within them. Unlike previous methods that often sacrifice text clarity for overall image sharpness or vice-versa, GLYPH-SR uses a vision-language guided diffusion model with a specialized Text-SR Fusion ControlNet and a “ping-pong” scheduler to achieve high-fidelity text recovery alongside high-quality image reconstruction, making it crucial for applications where reading scene-text is vital.
Image super-resolution (SR) is a crucial technology that reconstructs high-resolution images from low-resolution inputs. It’s vital for many applications, from autonomous driving to document analysis, where clear details are paramount. However, a significant challenge in this field has been the accurate recovery of “scene-text”—text embedded in natural images like signs, product labels, or storefronts. While conventional SR methods often make images look sharper overall, they frequently fail to make this embedded text truly legible, leading to errors in tasks like optical character recognition (OCR).
The problem stems from two main biases in existing SR models. Firstly, a “metric bias” means that standard quality metrics tend to focus on the overall image, largely ignoring small text regions. This results in character-level errors being weakly penalized. Secondly, an “objective bias” causes training processes to treat text as generic high-frequency texture rather than distinct semantic units. This often leads to two common failure modes: either the model “hallucinates” sharp but incorrect characters, or it performs “conservative restoration,” preserving blurry input to avoid artifacts, which limits the actual improvement in image quality.
To address this overlooked challenge, researchers have introduced GLYPH-SR, a novel vision–language-guided diffusion framework. GLYPH-SR is designed to tackle what they call a “bi-objective problem”: simultaneously optimizing for both high visual quality and high text legibility. This means creating images that not only look right but also read right.
At the heart of GLYPH-SR is a component called the Text-SR Fusion ControlNet (TS-ControlNet). This system is guided by OCR data, which provides specific information about text strings and their positions within the image, alongside a general scene caption. This dual guidance allows the model to inject complementary restoration cues specifically for text while maintaining the overall generative quality of the image. During training, the text-specific branch of the TS-ControlNet is fine-tuned on a specially designed synthetic corpus, ensuring targeted text restoration without disrupting the broader image super-resolution capabilities.
Another innovative feature is the “ping-pong scheduler.” This scheduler dynamically alternates between text-centric and image-centric guidance during the image reconstruction process. This ensures that the model pays attention to precise glyph cues during text-focused phases and stabilizes global structure and appearance during image-focused phases, effectively balancing the two objectives.
The researchers conducted extensive experiments across various challenging scene-text benchmarks. GLYPH-SR demonstrated significant improvements in OCR F1 scores, a key metric for text legibility, by up to +15.18 percentage points over existing diffusion and GAN-based baselines. Crucially, it achieved these gains while maintaining competitive scores in perceptual quality metrics like MANIQA, CLIP-IQA, and MUSIQ. This indicates that GLYPH-SR successfully avoids the trade-off seen in other methods, where improving one aspect often degrades the other.
The results highlight GLYPH-SR’s robustness, especially under severe degradation conditions, such as an ×8 magnification scale. It consistently produces coherent and legible results where other models might hallucinate incorrect characters or yield overly blurry text. This balanced approach makes GLYPH-SR a significant advancement for applications where both visual realism and accurate text recognition are critical.
Also Read:
- Improving Text-to-Image Models with Adaptive Text Embeddings
- Guiding Text-to-Image Models Towards Safer Content Without Retraining
For more in-depth information, you can read the full research paper here.
