TLDR: A new study evaluates how major AI companies have fulfilled their 2023 voluntary commitments to the White House regarding AI safety, security, and trust. It found significant variation in compliance, with an average score of 52%. While OpenAI scored highest, performance was notably poor in model weight security and third-party reporting. The research recommends that future voluntary commitments be more precise, targeted, and include mechanisms for public verification to enhance accountability.
In the rapidly evolving landscape of artificial intelligence, governments worldwide are grappling with how to ensure the safe and responsible development of powerful AI systems. A recent study delves into the effectiveness of voluntary commitments made by major AI companies to the White House in 2023, offering a critical look at how these pledges translate into action.
The research paper, titled “Do AI Companies Make Good on Voluntary Commitments to the White House?”, was authored by Jennifer Wang, Kayla Huang, Kevin Klyman, and Rishi Bommasani. It highlights that while voluntary commitments are a cornerstone of international AI governance, their actual implementation by companies varies significantly. For a deeper dive into their findings, you can read the full paper here: Research Paper.
The study developed a detailed rubric based on the eight voluntary commitments made by 16 leading AI companies to the White House. These commitments broadly covered three principles: product safety, system security, and public trust. Researchers scored companies based on their publicly disclosed behavior across 30 specific indicators, gathering information up to December 31, 2024. This rigorous approach aimed to provide a clear, verifiable assessment of company adherence.
Key Findings on Company Performance
The analysis revealed a wide range of performance among the companies. OpenAI emerged as the highest-scoring company with an impressive 83% overall, demonstrating strong adherence to the commitments. In stark contrast, Apple scored the lowest at 13%. The average score across all companies was a modest 52%, indicating a general gap between stated commitments and verifiable actions.
A notable trend observed was that companies belonging to the Frontier Model Forum (a non-profit industry association focused on AI safety) and those who were earlier signatories to the commitments tended to score higher. The six highest-scoring companies were all members of the Frontier Model Forum, achieving at least 66%.
Areas of Concern: Model Weight Security and Third-Party Reporting
Two specific commitments showed particularly poor performance across the board. The commitment to “model weight security” had an average score of just 17%, with 11 out of 16 companies scoring 0%. This highlights a significant vulnerability, as securing model weights is crucial for preventing misuse and theft of advanced AI systems. Despite its global emphasis in AI policy, companies are largely failing to publicly demonstrate their efforts in this area.
Similarly, “third-party reporting”—which involves establishing bug bounties or contests to incentivize external discovery of vulnerabilities—also scored low, averaging 34.4%. This suggests that companies are not adequately engaging with external researchers to identify and address potential safety issues, potentially stifling valuable independent scrutiny.
Conversely, “content provenance and watermarking” received the highest average score at 92.2%. This high score was often due to companies not developing audio or visual models, making the commitment vacuously met. However, even among companies that do develop such models, adherence was strong, often through participation in industry standards like the Coalition for Content Provenance and Authenticity (C2PA).
Also Read:
- When Less is More: Enhancing Trust in AI by Obscuring Less Factual Content
- The Hidden Truth: LLMs Deceive Even Without Prompts
Recommendations for Future AI Governance
Based on their findings, the researchers put forth three key recommendations to policymakers to improve the design and effectiveness of future voluntary AI commitments:
First, commitments should be **precise and specific**. The vague wording of the 2023 commitments led to ambiguity, making it difficult to determine what constitutes satisfactory compliance. Future commitments should clearly define goals and the evidence required to demonstrate completion.
Second, commitments should be **targeted**. A one-size-fits-all approach is ineffective, as companies occupy different roles in the AI supply chain. Commitments should be tailored to specific companies or specific layers of the supply chain to ensure relevance and applicability.
Third, commitments should **enable public verification**. The study found a significant lack of public transparency regarding how companies are meeting their pledges. Future initiatives should include accountability mechanisms, such as mandatory transparency reports, to allow the public to verify compliance and build trust.
This research underscores a critical structural shortcoming in current voluntary AI governance. For these commitments to truly advance responsible AI development and ensure public accountability, companies must proactively disclose their actions, and policymakers must design commitments that are clear, targeted, and verifiable.
