TLDR: LPI-RIT’s research at LeWiDi-2025 focuses on improving AI models’ ability to predict human annotator disagreement. They enhanced the DisCo neural architecture by incorporating annotator metadata (like age, gender, nationality), updating sentence encoders, and modifying loss functions to better align with evaluation metrics. These changes led to significant improvements in predicting soft label distributions and individual annotator behavior across sarcasm, irony, and paraphrase detection datasets, demonstrating the value of modeling human disagreement rather than simplifying it.
In the evolving landscape of machine learning, systems are increasingly involved in critical decision-making processes, from social interactions to legal judgments. A key challenge in developing these systems is aligning them with human values, especially when human opinions diverge. Traditional machine learning often simplifies this by aggregating human annotations into a single ‘ground truth,’ typically through a majority vote. However, this approach can inadvertently suppress valuable minority perspectives, particularly in subjective or contentious areas like hate speech detection or moral judgment.
The LeWiDi (Learning With Disagreements) 2025 shared task directly addresses this issue. Its goal is to encourage the development of models that can predict not just a single label, but a ‘soft label distribution’ that reflects the full spectrum of annotator disagreement. Furthermore, it challenges models to approximate individual annotator behavior in a ‘perspectivist’ setting, acknowledging that different people may have different valid interpretations.
A team from the Rochester Institute of Technology, LPI-RIT, participated in LeWiDi-2025, adapting and enhancing a neural architecture called DisCo (Distribution from Context). DisCo is designed to jointly model label distributions at both the item level (how all annotators view a specific piece of content) and the annotator level (an individual annotator’s typical labeling patterns across items). The original DisCo model used simple annotator IDs, which limited its ability to understand the nuances of annotator characteristics.
LPI-RIT’s Enhancements to DisCo
For the post-evaluation phase of the LeWiDi-2025 task, LPI-RIT introduced several significant improvements to the DisCo model:
- Incorporating Annotator Metadata: The team modified DisCo to utilize rich annotator metadata, such as age, nationality, gender, and education. This information was converted into natural language descriptions and then into numerical embeddings, allowing the model to learn more nuanced annotator representations.
- Expanded Preprocessing: DisCo’s data preprocessing capabilities were extended to handle a wider variety of data formats.
- Updated Sentence Encoders: The underlying sentence transformer models, which DisCo relies on for understanding text, were updated to more powerful versions.
- Modified Loss Functions: Crucially, the loss functions used during training were adjusted to better align with the specific evaluation metrics of the LeWiDi task, including Wasserstein distance for soft label prediction and Mean Absolute Error for perspectivist modeling. A combined loss function, weighting both objectives, proved particularly effective.
- Failure Mode Analysis: The team also conducted in-depth analysis of the model’s errors to understand where improvements occurred.
These enhancements were tested across three datasets from the LeWiDi-2025 task: the Conversational Sarcasm Corpus (CSC), MultiPico (MP), and Paraphrase (Par). These datasets represent diverse challenges, from detecting sarcasm to judging semantic similarity, all with varying degrees of human disagreement.
Also Read:
- How Large Language Models Can Better Express Their Uncertainty to Humans
- Enhancing AI Content Moderation: A New Benchmark and Human-Centric Strategy
Results and Insights
While LPI-RIT’s initial submission placed tenth, the post-evaluation improvements led to substantial gains across all three datasets. For instance, on the Conversational Sarcasm Corpus, the Wasserstein distance (a measure of how well predicted distributions match human ones) decreased significantly, indicating a much closer approximation to the gold label distributions. Similarly, the MultiPico dataset saw a reduction in Manhattan distance and improved recall for the ‘IRONIC’ class, along with better confidence calibration.
The Paraphrase dataset showed the largest improvement in soft-label matching, with a significant drop in Wasserstein distance. This suggests that the enhanced DisCo model can more effectively capture annotator-specific variations in judging paraphrase strength. The analysis also revealed that the new model was more robust to uncertainty in ambiguous cases, particularly in CSC and MP, and better reflected the inherent subjectivity in human judgments.
In conclusion, the work by LPI-RIT demonstrates that by thoughtfully incorporating annotator metadata, refining input representations, and adapting loss functions to disagreement-aware objectives, machine learning models can achieve consistent improvements in modeling the complexity of human-annotated data. This research contributes to a growing understanding of how to build systems that reflect, rather than obscure, the richness of human disagreement. You can find the full research paper here: LPI-RIT at LeWiDi-2025: Improving Distributional Predictions via Metadata and Loss Reweighting with DisCo.
