TLDR: A newly identified vulnerability in Google Gemini for Workspace allows attackers to embed hidden, malicious instructions within emails. When users prompt Gemini to summarize these emails, the AI assistant can be tricked into generating fake security alerts or phishing messages, leveraging the AI’s perceived trustworthiness to facilitate credential theft and social engineering. Mozilla’s Marco Figueroa reported the flaw, while Google maintains its robust defenses against such prompt injection attacks.
A significant security vulnerability has been uncovered in Google Gemini, the artificial intelligence tool integrated into Google Workspace applications such as Gmail, Docs, and Drive. This flaw enables malicious actors to execute sophisticated phishing and social engineering attacks by manipulating Gemini’s summary generation feature, effectively turning the AI assistant into an unwitting accomplice.
The attack vector, termed ‘indirect prompt injection’ or ‘prompt injection attack,’ relies on embedding invisible commands within the body of an email. These hidden instructions, often utilizing techniques like white-on-white text or specific HTML tags such as <admin> tags, are imperceptible to the human eye when reading the email normally. However, when a user activates Gemini’s ‘Summarize this email’ function, the AI processes these concealed directives.
Upon processing, Gemini is tricked into generating and displaying fabricated security warnings, password reset prompts, or requests for sensitive information, which appear as legitimate AI-generated content. This method bypasses traditional security measures designed to detect suspicious links or attachments, as the malicious content originates from the trusted AI tool itself. For instance, a demonstration showed Gemini displaying a message informing a victim that their Gmail password was compromised and instructing them to call a specific phone number to reset it, leading to potential credential theft.
Marco Figueroa, the Mozilla GenAI Bug Bounty Programs Manager, reported this critical issue through Mozilla’s 0din program, which focuses on generative AI vulnerabilities. Figueroa highlighted that similar attacks were first reported last year, and despite Google publishing detailed mitigations in June, the technique remains viable today.
Google has acknowledged the report, stating there has been no indication of Gemini manipulation as discovered by Figueroa. A Google spokesperson emphasized the company’s commitment to security, stating, ‘We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks.’ However, the vulnerability is rated as a moderate risk, scaling with bulk spam, and requires user interaction to be exploited.
The implications extend beyond email summaries. Figueroa noted that other parts of Google Workspace, including Docs, Slides, and Drive search, where Gemini processes third-party content, could be susceptible to similar prompt injection attacks. This broadens the potential attack surface to include newsletters, customer relationship management systems, and automated ticketing emails, potentially ‘turning one compromised SaaS account into thousands of phishing beacons,’ as Figueroa warned.
Also Read:
- Grok-4 AI Breached Days After Launch by Novel Dual-Method Jailbreak
- Generative AI Reshapes Cybersecurity Landscape: A Dual-Edged Sword in Digital Defense
Security experts advise users to exercise caution and treat AI-generated summaries as helpful but not authoritative, urging manual verification of any critical alerts. For organizations, the consensus is that ‘security teams must treat AI assistants as part of the attack surface and instrument them, sandbox them, and never assume their output is benign,’ according to Figueroa.
