TLDR: FED-PISA is a novel federated learning framework for voice cloning that addresses high communication costs and insufficient personalization in existing methods. It introduces a disentangled Low-Rank Adaptation (LoRA) mechanism, where a private ID-LoRA captures speaker timbre locally, and only a lightweight Style-LoRA is transmitted for collaborative learning. A personalized aggregation strategy, inspired by collaborative filtering, allows clients to learn from stylistically similar peers. This approach significantly improves style expressivity, naturalness, and speaker similarity while maintaining low communication costs and preserving privacy.
Voice cloning, a technology that allows Text-to-Speech (TTS) systems to generate speech in a target speaker’s voice from any text, is becoming increasingly sophisticated. The goal is to create speech that not only matches a speaker’s unique vocal timbre but also captures their expressive style and prosodic patterns. While significant progress has been made, especially with high-fidelity zero-shot personalization and efficient on-device deployment, these methods often operate in isolation. This creates ‘style silos,’ where a client’s model is limited by its own data, unable to benefit from the diverse stylistic variations across a wider community.
This challenge highlights a growing need for collaborative style learning among different clients, all while strictly preserving local data privacy. Federated Learning (FL) emerges as a promising solution, offering a privacy-preserving framework where multiple clients can collaboratively train a model without their private data ever leaving their local devices.
However, existing FL-based TTS frameworks, such as FedSpeech and Federated Dynamic Transformer, face their own set of hurdles. They often incur substantial computational and communication costs due to complex model modifications or large-scale parameter exchanges, making them difficult to deploy on resource-constrained devices. More critically, these approaches tend to suppress the rich stylistic heterogeneity found in speech data, such as variations in emotion and prosody, in an effort to preserve individual speaker timbre. This suppression limits the model’s ability to learn diverse and expressive styles, leading to insufficient personalization.
Introducing FED-PISA: A Novel Approach
To overcome these limitations, researchers have proposed FED-PISA, which stands for Federated Personalized Identity-Style Adaptation. This innovative framework is designed to enable efficient and personalized federated TTS by effectively learning and utilizing heterogeneous styles across clients while maintaining low communication costs. FED-PISA builds upon Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically leveraging Low-Rank Adaptation (LoRA).
FED-PISA incorporates two key components:
1. Disentangled LoRA Mechanism: To enhance efficiency and robustly preserve a speaker’s unique timbre, FED-PISA introduces a decoupled LoRA mechanism. A private ID-LoRA (Identity-LoRA) is trained for each client locally and then permanently frozen. This ID-LoRA captures the speaker’s unique vocal timbre and channel characteristics and is never uploaded or aggregated to the server, ensuring privacy. In contrast, a lightweight Style-LoRA is used for communication. This federated, globally shared adapter modulates expressive variations and is collaboratively updated across clients. By only transmitting the Style-LoRA, communication costs are significantly reduced.
2. Personalized Aggregation Strategy: To fully harness stylistic heterogeneity, FED-PISA employs a personalized aggregation strategy inspired by collaborative filtering, a technique commonly used in recommendation systems. This strategy ensures that each client benefits most from other clients with similar speaking styles. The server computes attention scores based on the similarity of the Style-LoRA matrices from different clients. These scores are then used to create a custom-aggregated style model for each client, prioritizing updates from stylistically similar peers. This personalized model is then sent back to the client, while the private ID-LoRA remains securely on the device.
Also Read:
- FocalCodec-Stream: Real-time Low-Bitrate Speech Compression for Live Applications
- MICA: Intelligent AI Assistants for Modern Industrial Operations
Experimental Validation and Key Findings
Experiments were conducted on four publicly available datasets with emotion annotations, unified into 10 distinct style categories. FED-PISA was compared against several baselines, including zero-shot voice cloning, local fine-tuning, and existing federated methods like FedSpeech and Federated Dynamic Transformer.
The results demonstrated that FED-PISA significantly improves style expressivity, naturalness, and speaker similarity, consistently outperforming both non-federated and federated baselines. Notably, FED-PISA achieved higher speaker similarity scores than purely local fine-tuning, indicating that its collaborative learning paradigm allows clients to learn richer stylistic variations from their peers, overcoming data scarcity limitations. Compared to existing federated baselines, which often suppressed stylistic heterogeneity, FED-PISA effectively preserved and leveraged it, leading to substantially better performance in terms of speech quality and accuracy.
Furthermore, FED-PISA exhibited strong efficiency advantages. Due to its use of LoRA, its trainable parameter count was approximately one-fifth that of some baselines, and its communication cost was significantly lower – about one-tenth that of Federated Dynamic Transformer and one-third that of FedSpeech.
Ablation studies confirmed the critical importance of FED-PISA’s design choices. Removing either the private ID-LoRA or the collaborative Style-LoRA led to a significant degradation in speaker similarity and naturalness, highlighting that a single adapter struggles to balance preserving identity and learning diverse styles. The personalized aggregation strategy also proved superior to naive aggregation (like FedAvg), which tended to average out styles and harm both speaker identity and expressive quality.
In conclusion, FED-PISA offers an effective solution to the central challenge in federated voice cloning: leveraging stylistic heterogeneity across clients without incurring high communication costs or degrading speaker identity. By disentangling identity and style learning through LoRA and employing a novel personalized aggregation strategy, FED-PISA paves the way for more expressive, natural, and private voice cloning systems. For more details, you can refer to the original research paper.
