TLDR: The IAB Tech Lab has launched its LLM Content Ingest API Initiative, a new framework designed to address the significant revenue and traffic losses experienced by publishers due to unchecked AI content scraping. The initiative aims to establish a ‘fair value exchange’ by providing technical standards for controlling AI access to content, enabling monetization through mechanisms like a ‘cost per crawl’ metric, and ensuring proper attribution and representation of brand information. This move comes as publishers report substantial traffic declines, with IAB Tech Lab CEO Anthony Katsur highlighting decreases of 15% or more.
New York, NY – July 14, 2025 – The Interactive Advertising Bureau (IAB) Tech Lab has introduced a critical new framework, the LLM Content Ingest API Initiative, aimed at tackling the escalating challenges of AI content scraping and the resulting revenue and traffic declines faced by publishers. This proactive step seeks to establish a ‘fair value exchange’ between content creators and the rapidly expanding ecosystem of large language models (LLMs) and AI agents.
The initiative comes in response to growing concerns within the publishing community, where AI-driven search summaries and chat interfaces are increasingly displacing traditional search engines, leading to significant reductions in referral traffic. Anthony Katsur, CEO of IAB Tech Lab, highlighted the severity of the situation, stating, ‘It is clear that AI agents powered by large language models are shifting how users engage with content. While this is a promising new way for people to access information, we have also seen data showing publisher traffic decreases at 15% or higher, and revenue is down.’ Some reports suggest that publishers are experiencing 20-60 percent less traffic from search as AI bots consume content directly, rather than driving users to publisher sites.
The core of the LLM Content Ingest API Initiative is a technical specification designed to empower publishers and brands with greater control over how their content is accessed, monetized, and represented by AI systems. The framework proposes several key mechanisms:
1. Controlled Bot Access: Enabling publishers to manage and restrict access for internet bots, ensuring only authorized AI systems can ingest content.
2. Efficient Content Delivery: Facilitating LLM-friendly content discovery and delivery, allowing authorized bots to access and represent content accurately and efficiently.
3. Monetization Framework: Introducing a ‘cost per crawl’ (CPCr) metric, which could evolve into a biddable, real-time marketplace for content usage. This aims to compensate publishers for the value derived from their intellectual property by AI platforms.
4. Brand Control and Attribution: Providing structured approaches for brands to manage how their information is accessed and interpreted by AI systems, addressing concerns about misrepresentation and ensuring proper attribution.
IAB Tech Lab has been actively consulting with the industry, recently closing a consultation on the framework. The goal is to rapidly deploy this framework within a few months. Katsur emphasized the urgency, noting, ‘Let’s not let perfection be the enemy of good… we need to move now.’ He also acknowledged the potential hurdle that major AI platforms might not willingly pay for content without legal enforcement, underscoring the need for an enforceable path grounded in technical standards.
The initiative was formally proposed at the IAB Tech Lab’s annual summit in early June 2025, and the organization is actively inviting publishers, brands, LLM platforms, and AI agent developers to participate in its development. A workshop is planned to collaboratively explore solutions to the challenges posed by rogue AI content ingestion and to develop practical, enforceable tools.
Also Read:
- IAB Europe Unveils Landmark Whitepaper on AI’s Role in European Digital Advertising
- Malaysia Considers Mandatory ‘AI Generated’ Labels for Online Content Under New Safety Act
This move by the IAB Tech Lab signifies a pivotal moment for the digital advertising and publishing industries, as they grapple with the profound economic shifts brought about by generative AI. The success of this initiative could redefine the relationship between content creators and AI developers, ensuring a more equitable and sustainable future for the open web.
