TLDR: TalentCLEF 2025 is the first evaluation campaign for NLP applications in Human Capital Management, addressing the lack of public benchmarks for skill and job title intelligence. It features two tasks: Multilingual Job Title Matching and Job Title-Based Skill Prediction, using real, anonymized job data. The campaign revealed that effective training strategies, especially with encoder-based models and LLM-driven data augmentation, are more impactful than model size alone, and demonstrated progress in developing fair and robust language technologies for the labor market.
The global job market is undergoing a significant transformation, driven by rapid technological advancements like artificial intelligence and automation. This shift means that the skills needed for jobs are constantly changing, and many new roles are emerging that didn’t exist before. Both individuals and organizations face the challenge of adapting to these changes, with professionals needing to reskill and companies needing to identify and attract new talent.
Natural Language Processing (NLP) and large language models (LLMs) are playing a crucial role in Human Capital Management (HCM) by helping with tasks like finding suitable candidates, developing employee upskilling programs, and planning for future workforce needs. However, a major hurdle has been the lack of public data and standardized benchmarks to properly evaluate and improve these smart systems.
To address this critical gap, a new initiative called TalentCLEF 2025 was launched. It’s the first evaluation campaign specifically focused on skill and job title intelligence in the human resources domain. TalentCLEF 2025 introduced two main challenges, or tasks, designed to test the capabilities of NLP models in real-world scenarios.
Task A: Multilingual Job Title Matching
This task aimed to develop systems that can identify and rank job titles that are similar to a given one, even across different languages. The dataset included English, Spanish, German, and Chinese. The goal was to help systems recognize when different job titles, such as ‘Software Engineer’ and ‘Ingeniero de Software’, refer to the same underlying role. This is vital for building accurate and fair job recommendation systems in a global market.
The evaluation for Task A covered several scenarios: matching job titles within the same language (monolingual), matching job titles across different languages (cross-lingual, specifically English queries to Spanish and German job titles), and an optional Chinese language track. A key aspect was also evaluating gender bias, assessing how sensitive models were to grammatical gender in job titles, particularly in Spanish and German.
Task B: Job Title-Based Skill Prediction
Recognizing the increasing importance of a skill-centric approach to talent management, Task B focused on predicting relevant skills for a given job title. With more than half of Europe’s workforce estimated to require reskilling due to technological innovations, models that can identify skills aligned with specific job positions are essential. This helps organizations pinpoint skill gaps, create targeted training programs, and plan their workforce based on competencies.
This task used an English-only dataset, where systems were evaluated on their ability to identify and rank skills from a knowledge base that were most relevant to a specific job title.
Data and Evaluation
The data for both tasks was built from real job applications, carefully anonymized, and manually annotated to reflect the complexity and diversity of the actual labor market. This included linguistic variations and gender-marked expressions. The primary evaluation metric for both tasks was Mean Average Precision (MAP), which measures how well a system returns highly relevant items at the top of a ranked list. For gender bias in Task A, the Rank Biased Overlap (RBO) metric was also used to see how consistent rankings remained despite gender variations in job titles.
Key Findings and Insights
The TalentCLEF 2025 campaign saw significant participation, with many teams submitting diverse system configurations. For Task A, the AlexU-NLP team achieved the best overall multilingual performance, while pjmathematician and AlexU-NLP shared the top spot in cross-lingual scenarios. Notably, some systems, like those from NLPnorth and Ixa, achieved perfect scores in gender bias evaluation, demonstrating their ability to produce consistent rankings regardless of gendered language.
In Task B, the pjmathematician team achieved the highest MAP score, followed by moali. A significant insight from both tasks was that fine-tuning encoder-based models with contrastive learning techniques often outperformed larger, more computationally intensive decoder-based embedding models. This suggests that the training strategy and methodological choices are more critical for performance than simply the size of the model alone.
Furthermore, the use of large language models (LLMs) for data augmentation proved highly beneficial, leading to significant performance improvements by providing more diverse training data. While LLMs were not always the core model, their supportive role in generating synthetic data or re-ranking results was a common and successful strategy among top-performing teams.
Also Read:
- Text Anomaly Detection: Unveiling Performance with LLM Embeddings
- TREC 2022 Deep Learning Track: Advancing Reusable Test Collections and Neural Retrieval
Looking Ahead
TalentCLEF 2025 has successfully established the first community evaluation framework for NLP applications in Human Capital Management. By providing shared, open, and publicly available evaluation methods and datasets, it addresses a crucial need in this evolving field. The benchmarks created will not only support ongoing research but also help guide the responsible development and use of language technologies in areas that profoundly impact people’s professional lives. You can read the full research paper here: TalentCLEF 2025 Research Paper.
