TLDR: FERA (FEncing Referee Assistant) is a prototype AI referee system for foil fencing that integrates pose-based multi-label action recognition and rule-based reasoning. It extracts 2D joint positions from video, computes kinematic features, uses a Transformer for move and blade classification, and applies a distilled language model to determine priority and scoring with explanations. Trained on a new dataset of two-fencer exchanges, FERA achieves a macro-F1 score of 0.549, demonstrating a promising direction for automated referee assistance and coaching in fencing.
The fast-paced and intricate sport of foil fencing has long relied on human referees to make critical decisions, often under immense pressure. These decisions, which involve evaluating complex blade interactions, interpreting right-of-way rules, and identifying who initiated an attack, can be subjective and prone to human error. This challenge is even more pronounced in club settings where trained referees might be scarce.
Addressing these issues, a new research paper introduces FERA (FEncing Referee Assistant), a groundbreaking prototype AI referee system designed specifically for foil fencing. FERA aims to bring fairness, consistency, and objectivity to refereeing by integrating advanced artificial intelligence techniques.
How FERA Works: A Four-Stage Approach
FERA operates through a sophisticated four-stage pipeline, transforming raw video footage into clear referee decisions and explanations:
1. Pose Estimation: The system first extracts 2D skeleton poses of the fencers from video. It accurately identifies the two active fencers and tracks their movements across frames, even when parts of their bodies are temporarily hidden.
2. Feature Engineering: Once poses are identified, FERA goes beyond simple joint positions. It computes a comprehensive set of 101 kinematic features for each frame. These features capture crucial temporal and geometric dynamics, such as distances between key body parts, joint angles, torso orientation, arm extension, and even velocities and accelerations of movements. This rich data provides a deeper understanding of the fencers’ actions.
3. Move and Blade Detection (FERA-MDT): The engineered features are then fed into FERA-MDT, an encoder-only Transformer model. This model is trained to perform multi-label classification, meaning it can identify multiple actions occurring simultaneously (e.g., a ‘step forward’ combined with a ‘beat’). It also classifies blade positions. A dynamic windowing mechanism allows FERA to adapt to varying move lengths, ensuring accurate detection of actions like lunges, parries, and hits.
4. Rule-Based Reasoning (FERA-LM): The final stage, FERA-LM, is a distilled language model. It takes the detected moves and blade positions from both fencers, combines them with encoded right-of-way rules, and then generates a concise decision (e.g., ‘Left fencer has priority’) along with a clear explanation for that decision. This ensures transparency and helps validate the AI’s judgment.
Building a Better Dataset
A significant hurdle for AI in fencing has been the lack of comprehensive datasets. Existing ones often focus on single fencers, limited footwork, and fixed camera angles, making them unsuitable for real-world bouts. To overcome this, the researchers created a new dataset from professional Grand Prix competition videos. This dataset captures two-fencer exchanges under varied camera angles and includes detailed frame-by-frame annotations of moves and blade positions, inferred from actual referee decisions and score changes.
Promising Performance
In evaluations, FERA-MDT demonstrated strong performance, outperforming several baseline models like Temporal Convolutional Networks (TCNs) and BiLSTMs in recognizing fencing moves. It achieved an average macro-F1 score of 0.549, indicating its effectiveness across both common and rarer fencing actions. This is crucial for refereeing, where accurate recognition of all actions, not just the frequent ones, is vital. The system also showed reasonable calibration, meaning its confidence in predictions is well-aligned with its accuracy.
Also Read:
- Automated Assessment for Nursing Competency Using Video-Language Models
- Enhancing Robotic Action Recognition with Knowledge-Infused AI
Future Potential and Limitations
While FERA is currently a prototype, its results pave a promising path for automated referee assistance and new opportunities in AI applications for sports, such as coaching. The researchers acknowledge limitations, including the need for more diverse data (especially for rare moves and different fencer skill levels), explicit blade detection capabilities, and further refinement of the language model’s reasoning. Future work will also explore 3D joint extraction to reduce camera-angle sensitivity.
This innovative system represents a significant step towards bringing AI into the heart of competitive sports, offering a glimpse into a future where technology can enhance fairness and provide valuable insights for athletes and coaches alike. You can read the full research paper here.
