TLDR: This research evaluates how different prompt translation strategies impact the performance of Large Language Models (LLMs) in multilingual classification tasks, particularly in Retrieval-Augmented Generation (RAG) systems. It finds that translating identity statements and task rules into the user’s native language often improves performance, while translating candidate lists can be detrimental, especially for low-resource languages like Hindi. The optimal strategy depends heavily on the specific LLM and the language’s resource availability.
Large Language Models (LLMs) have made significant strides in understanding and generating human language. However, their performance can vary considerably across different languages and tasks. This is particularly true in multilingual systems that use Retrieval-Augmented Generation (RAG), where knowledge bases are often in high-resource languages like English, but user queries might come in other languages.
When a user’s query is in a different language than the retrieved information, system designers face a choice: either pre-translate parts of the prompt to create a single-language input, or use cross-lingual prompting for direct inference. The impact of these choices on LLM performance has been unclear, leading to the research explored in this paper.
Understanding the Research
A team of researchers from Amazon, including Aman Gupta, Yingying Zhuang, Zhou Yu, Ziji Zhang, and Anurag Beniwal, systematically evaluated how different prompt translation strategies affect classification tasks when using RAG-enhanced LLMs in multilingual setups. Their findings suggest that an optimized prompting strategy can significantly improve how knowledge is shared across languages, ultimately boosting performance on downstream tasks like intent classification.
The study advocates for greater use of multilingual resource sharing and cross-lingual prompt optimization, especially for non-English and low-resource languages. This work is crucial for developing conversational AI systems that can serve a wider range of communities.
How the Study Was Conducted
The researchers focused on the task of intent classification in dialogue systems. This involves a two-step RAG process: first, a retrieval model identifies a list of possible intents, and then an LLM selects the final intent from this list. For this study, the prompt given to the LLM included four key components:
- An identity statement: Describing the overall task.
- Task rules: Instructions on how to perform the task and format the output.
- A candidate list: Possible intents to choose from.
- The speaker utterance: The user’s query for which the intent needs to be detected.
The study used a dataset of 6,000 utterances each in French and Hindi. French was chosen as a high-resource language with linguistic similarities to English, while Hindi represented a low-resource language with greater dissimilarity. The user utterance was always kept in the source language (French or Hindi), while other prompt components were selectively translated from English into the source language.
Six translation strategies were evaluated:
- Baseline (B): All components remained in English.
- Candidates (C): Only the candidate list was translated to the source language.
- Identity (I): Only the identity statement was translated.
- Rules (R): Only the task rules were translated.
- Identity & Rules (I&R): Both identity and rules were translated.
- All (A): All prompt components were translated to the source language.
Five state-of-the-art multilingual LLMs were used for evaluation: Llama-3.1-8B, Qwen2.5-7B-Instruct, BLOOMZ-7b1, Mistral-Nemo-Instruct-2407, and BLOOMZ-7b1-mt. These models were selected to represent diverse architectural designs and pre-training approaches.
Key Findings and Insights
The experiments revealed several important insights:
- Overall, the absolute baseline performance for intent detection was relatively low across all models, partly because the intent detection instances were out-of-domain for the LLMs and LLMs sometimes struggle with verbatim copying.
- The impact of translation strategies varied significantly across different LLMs. For instance, Llama-3.1-8B and Mistral-Nemo-Instruct-2407 generally saw performance decreases when any component was translated into the source language.
- In contrast, the BLOOMZ-based models showed consistent improvements in performance across most translated prompt configurations. Qwen2.5-7B-Instruct had mixed results, improving with Identity, Rules, or both translated, but declining when candidates or all components were translated.
- From a language perspective, models generally struggled with generating Hindi text (in configurations where candidates or all components were translated), leading to performance declines. For French, however, native language generation proved beneficial, but primarily for the BLOOMZ models.
- A general trend observed was that translating the Identity statement, Rules, or both into the respective source language often led to some performance improvements.
Also Read:
- Unlocking Smarter AI Responses: The SemRAG Approach to Knowledge Integration
- Large Language Models Transform Recommender Systems: A Deep Dive into New Capabilities and Remaining Hurdles
Recommendations for Optimal Prompting
The researchers offer several key recommendations based on their findings:
- English Generation is Often Better: Most models perform poorly when forced to generate responses in the source language, especially for low-resource languages like Hindi. This is likely due to the vast amount of English training data available to LLMs.
- Translate Identity and Instructions: Framing the task by translating the identity statement and rules into the user’s native language can help the model better understand the context and intent of the query. This suggests a hybrid approach where task framing is in the source language, but generation remains in English.
- Model Choice Matters: The architectural design and pre-training objectives of an LLM significantly influence its cross-lingual capabilities. Models specifically trained for multilingual tasks, like the BLOOMZ variants, tend to perform better with translated prompts.
- Consider Language Resources and Task Needs: Optimal strategies differ between high-resource (French) and low-resource (Hindi) languages. For tasks like intent classification, translating identity and rules proved beneficial, but this might not apply to all tasks.
In conclusion, this research highlights that there is no one-size-fits-all solution for cross-lingual prompting. The optimal strategy depends heavily on the specific language, its resource availability, and the chosen LLM. By carefully considering which parts of a prompt to translate, developers can significantly enhance the performance of multilingual AI systems, fostering greater inclusivity and accessibility. You can read the full research paper for more details here.
