TLDR: A new self-supervised learning framework called “Mask & Match” improves handwritten mathematical expression recognition (HMER) by pretraining an image encoder with global and local contrastive losses and introducing a novel self-supervised attention network trained via progressive spatial masking. This approach reduces reliance on labeled data and outperforms existing methods on CROHME benchmarks, demonstrating enhanced structural understanding of math expressions.
Handwritten Mathematical Expression Recognition (HMER) is a complex task that involves converting handwritten math into digital formats like LaTeX. This process is crucial for various real-world applications, including checking exam papers, developing math OCR tools, and supporting online learning platforms. Unlike regular text, mathematical expressions have a two-dimensional structure with varying symbol scales and intricate spatial relationships, making HMER particularly challenging.
Traditionally, HMER models relied heavily on large amounts of labeled data for training. However, acquiring and annotating such data, especially for complex mathematical structures, is both time-consuming and expensive. To overcome this hurdle, researchers have been exploring self-supervised learning (SSL), which allows models to learn from unlabeled images.
A new research paper, “Mask & Match: Learning to Recognize Handwritten Math with Self-Supervised Attention”, introduces an innovative self-supervised learning framework designed to improve HMER performance without the need for extensive labeled data. The core of their approach involves two key self-supervised components.
Self-Supervised Pretraining of the Image Encoder
The first component focuses on pretraining an image encoder. This encoder learns to understand both the overall structure and the fine details of handwritten math expressions. The authors use a technique called momentum contrastive learning, which involves a dual-contrastive loss. This loss combines a global objective, focusing on the entire image, with a local objective, which analyzes smaller patches of the image. This dual approach helps the model capture both broad features and subtle, fine-grained patterns that are essential for accurately recognizing mathematical symbols and their relationships.
Self-Supervised Attention Mechanism with Progressive Masking
The second major contribution is a novel self-supervised attention network. This attention mechanism is trained using a unique “progressive spatial masking” strategy. Imagine gradually revealing parts of an image to the network. Initially, the network sees only small, scattered regions, forcing it to learn global context. As training progresses, more of the image is revealed, allowing the model to refine local details while maintaining a coherent global understanding. This progressive masking curriculum helps the attention network learn to identify semantically important regions, such as operators, exponents, and nested notations, without any explicit supervision. This makes the network robust even when parts of the input are missing or obscured.
The complete system integrates these self-supervised components with a Transformer decoder, which is responsible for generating the final LaTeX sequences. The pretrained encoder and attention module feed into this decoder, enhancing its ability to handle complex spatial layouts and improve decoding accuracy.
Experimental Results and Impact
The researchers conducted extensive experiments on the CROHME benchmarks, which are standard datasets for HMER. Their method demonstrated superior performance compared to existing self-supervised and even fully supervised baseline models. The improvements were particularly notable in error-tolerant metrics, suggesting that the approach significantly enhances the structural understanding of mathematical expressions rather than just fixing isolated symbol errors. While the model showed strong gains on CROHME 2014 and 2016 datasets, its performance on CROHME 2019, which contains more complex and longer expressions, was competitive but slightly below the best prior self-supervised method in some aspects, though ablations showed progressive masking recovered some of this gap.
This research highlights the effectiveness of combining global and local contrastive learning with a progressive masking strategy for attention. It offers a promising direction for developing robust HMER systems that can learn effectively from unlabeled data, potentially reducing the high costs associated with data annotation in this challenging field.
Also Read:
- LiLoRA: A New Approach to Efficient Continual Learning in Multimodal AI
- Unlocking Deeper Insights: How FMCE-Net++ Improves Neural Network Training
Future Directions
The current model processes static images. For future work, the authors plan to incorporate “online” stroke information, such as pen trajectories and pressure, alongside images. This cross-modal fusion aims to further disambiguate structural elements and improve recognition rates by providing more dynamic context to the system.
