TLDR: A study explored using multimodal large language models (MLLMs) to detect and measure hailstone diameters from crowd-sourced images. They found that models like GPT-4o, especially with a two-stage prompting strategy that utilizes reference objects like hands, can estimate hailstone sizes with an average error of 1.12 cm. This approach offers a promising way to complement traditional weather sensors by extracting valuable information from social media imagery for faster and more detailed assessments of severe weather events.
Hailstorms are a significant natural phenomenon, causing billions of dollars in damage annually across the globe. Accurately measuring hailstone size is crucial for various sectors, including agriculture, insurance, and meteorological forecasting. However, traditional methods like ground sensors and weather radar often fall short, providing limited spatial coverage and sometimes inaccurate measurements, especially for smaller hailstones.
A recent study, titled Detection and Measurement of Hailstones with Multimodal Large Language Models, explores a novel approach to overcome these limitations. Researchers Moritz Alker, David C. Schedl, and Andreas Stöckl from the Digital Media Lab at the University of Applied Sciences Upper Austria investigated the potential of using everyday social media and news images, combined with advanced artificial intelligence, to detect and measure hailstones.
Leveraging Crowd-Sourced Data and AI
The core idea behind this research is to tap into the vast amount of crowd-sourced imagery available online. People often share photos of hail events on social media, which can provide a rich, spatially dense dataset far exceeding what traditional sensor networks can offer. The challenge, however, lies in extracting quantitative measurements from these unstructured images, which vary widely in quality, viewing angles, and the presence of reference objects.
This is where Multimodal Large Language Models (MLLMs) come into play. These sophisticated AI models can process both visual and textual information, making them ideal for understanding and reasoning about images. Unlike older computer vision techniques that require extensive, specialized training, MLLMs can leverage their pre-trained knowledge to perform complex visual tasks, even without specific fine-tuning for hailstone measurement.
The Study’s Approach
The researchers compiled a dataset of 474 crowd-sourced hailstone images from documented events in Austria between January 2022 and September 2024. These images featured hailstones with maximum diameters ranging from 2 to 11 cm, with precise ground-truth measurements available for comparison. Each image was also manually annotated to note the presence of reference objects (like hands, coins, or rulers) and the viewing distance.
Four state-of-the-art MLLMs were evaluated: OpenAI’s GPT-4o and GPT-4o-mini, Anthropic’s Claude-Sonnet 4, and Google’s Gemini 2.5 Flash Lite. Two main prompting strategies were tested to guide the AI models in their estimations:
- Single-Stage Prompting (P1): The model was directly asked to estimate the maximum hailstone diameter.
- Two-Stage Prompting (P2): In the first step, the model identified any reference objects in the image. In the second step, it used the known dimensions of these objects (e.g., an average human hand size) to estimate the hailstone diameter. If no reference object was present, the model relied on contextual cues.
Key Findings and Impact
The results were highly encouraging. The study demonstrated that off-the-shelf MLLMs could indeed extract quantitative information about hailstones with surprising accuracy. The best-performing model, GPT-4o, when used with the two-stage prompting strategy (P2), achieved a mean absolute error of just 1.12 cm in diameter estimation and showed a strong correlation with actual measurements.
A significant finding was the effectiveness of the two-stage prompting strategy. It reduced the overall error by an average of 18.6% compared to the single-stage approach and drastically cut down the number of instances where the models failed to provide an estimate. The presence of clear reference objects, particularly a human hand, substantially improved accuracy, reducing the error to 0.75 cm. This highlights how crucial interpretable scale information is in crowd-sourced images.
While all models showed a slight tendency to underestimate hailstone size, this consistent bias suggests a shared limitation in interpreting three-dimensional scales from two-dimensional images, possibly due to a conservative approach when visual information is ambiguous.
These findings suggest that current MLLMs, even without specialized fine-tuning, can significantly enhance traditional hail sensors. By extracting detailed and spatially rich information from social media imagery, these models can enable faster and more precise assessments of severe hail events. This capability holds immense potential for operational meteorology, helping to address the increasing socio-economic risks posed by climate change-driven increases in hailstorm intensity.
Also Read:
- AI Streamlines Radiology Report Analysis for Image Classification
- AI Agents Navigate Weather Data for Advanced Forecasting
Future Directions
The researchers acknowledge limitations, such as the dataset being confined to Austria, and the need for automated, real-time image harvesting from social media for practical applications. Future work will focus on expanding the dataset’s geographic diversity, exploring techniques to correct the underestimation bias, and developing a fully automated pipeline to integrate these AI-driven measurements into real-time weather forecasting systems.
