TLDR: This research introduces an automated pipeline to generate synthetic audio-text data with grammatical errors and disfluencies, addressing the scarcity of spoken grammatical error correction (SGEC) datasets. The pipeline uses a reverse GEC module to introduce errors, a disfluency addition module, and an F5-TTS system for high-quality audio generation. Objective metrics are proposed to evaluate data quality, ensuring speaker identity and assessment score preservation. Experiments show that this augmented data significantly improves performance for both written GEC and end-to-end SGEC models, demonstrating a promising solution for data-scarce spoken language tasks.
The field of grammatical error correction (GEC) has seen significant advancements, particularly for written text, thanks to robust benchmark datasets. However, a major challenge persists in Spoken Grammatical Error Correction (SGEC): the scarcity of high-quality, annotated spoken datasets. Spoken language inherently presents complexities like noise, hesitations, repetitions, and incomplete sentences, making error detection and correction far more difficult than in written text. These factors also make manual annotation of spoken data time-consuming and expensive.
To address this critical data shortage, researchers Penny Karanasou, Mengjie Qian, Stefano Bann`o, Mark J.F . Gales, and Kate M. Knill have proposed a novel, fully automated method to generate audio-text pairs that include grammatical errors and disfluencies. Their work, detailed in their research paper, aims to create an augmented dataset that enriches existing corpora without altering the language assessment scores of second language (L2) learners. This augmented data is designed to maintain the original data’s textual and acoustic characteristics while introducing new types of errors, proving beneficial for both written GEC and SGEC tasks.
The Automated Data Augmentation Pipeline
The core of their solution is a sophisticated, model-based data augmentation pipeline that works in reverse of traditional SGEC systems. Instead of correcting errors, it introduces them into clean text and then generates corresponding audio. The pipeline consists of three main modules:
First, the Reverse GEC Module takes clean, grammatically correct text and introduces errors. It utilizes a BART-large model, previously used for GEC, but with its input and output reversed. By training specifically on utterances that contain errors, the model learns to generate diverse and meaningful grammatical mistakes, significantly increasing the number of new errors in the augmented data. Sanity checks confirm that the generated errors align well with expected types and can be corrected by existing GEC systems.
Next, the Disfluencies Addition Module integrates common spoken disfluencies such as hesitations, repetitions, and false starts into the text. This module aligns the disfluencies with their original positions in the source corpus, preserving their natural frequency and minimizing changes to the text. This careful approach is hypothesized to ease the subsequent voice cloning process.
Finally, the Text-to-Speech (TTS) Module generates aligned audio from the augmented text. The crucial requirement for this module is its ability to preserve the original speaker’s voice and accent while accurately synthesizing the modified transcription. After evaluating several multi-lingual, multi-speaker TTS systems with zero-shot voice cloning capabilities, the F5-TTS model was selected. F5-TTS stood out for its natural and expressive zero-shot capabilities, faster inference, and superior audio quality compared to alternatives like VoiceCraft, VALL-E-X, YourTTS, XTTS-v2, and Orpheus-TTS, which faced issues with speed, editing limitations, voice transfer, or audio quality.
Evaluating the Generated Data
To ensure the quality and suitability of the augmented spoken data, the researchers propose four objective metrics:
- Speaker Verification: This metric uses speaker embeddings to calculate the cosine distance between the original and generated speech. A lower distance indicates better preservation of speaker identity. F5-TTS demonstrated superior performance here, showing greater similarity to the original speaker’s voice.
- ASR Output (Word Error Rate – WER): An ASR system is used to transcribe the generated audio, and its WER is compared to a baseline. A low WER indicates good audio-text alignment, crucial for SGEC training. F5-TTS produced the lowest WER, closest to the original baseline, despite a slight increase in insertions due to prompt words.
- BERT Text-based SLA Grader: This metric assesses whether the augmented text alters learner assessment outcomes. Scores from the generated text are compared to original scores, showing strong positive correlations, indicating that the augmented data maintains assessment integrity.
- Wav2vec Audio-based SLA Grader: Similar to the text-based grader, this evaluates the acoustic characteristics of the generated speech. Strong positive correlations between generated and original audio scores confirm that the augmented data preserves the original acoustic features and does not impact assessment.
Also Read:
- Assessing LLM Code Generation: Introducing ReCatcher for Regression Testing
- Navigating the AI Frontier: Large Language Models in Social Simulation
Impact on Grammatical Error Correction
The augmented datasets were rigorously evaluated for both written GEC and end-to-end SGEC tasks. When the reverse GEC system was applied to generate augmented written data, it significantly improved the performance of GEC models, outperforming baselines across various cascaded and semi-cascaded SGEC pipelines. This demonstrates the effectiveness of the generated errors in enriching the training data.
For end-to-end SGEC, audio-text augmented data generated with F5-TTS was used to train a Whisper model. The results showed that incorporating this augmented data improved performance. Further gains were observed when filtering the generated data based on speaker similarity and when fine-tuning the model on the original S&I Corpus data. This highlights the value of high-quality synthetic spoken data for training robust SGEC systems.
In conclusion, this research presents a significant step forward in addressing the data scarcity for Spoken Grammatical Error Correction. By providing an automated pipeline for generating realistic audio-text pairs with grammatical errors and disfluencies, along with objective metrics for quality evaluation, the authors offer a powerful tool for advancing SGEC models. The positive experimental results underscore the potential of data augmentation to enhance both text-based and spoken GEC tasks. You can read the full paper here.
