There’s so much to think about regarding AI right now: tools, ethics, regulation, partnerships. But if I could make one recommendation for immediate action, it’s this: Measure and manage the bots that are scraping your sites. Please. 

Why? Because if you let AI companies take your content for free, you’re undermining any future business model for licensing or compensation.

As I wrote in an INMA report on working with AI companies earlier this year: Robots.txt does not cut it; it’s a gentleman's agreement at best. Many companies, such as Perplexity, will tell you they don’t scrape. Which is true. But it is also true that they buy data from third parties that do. 

And yes there are a LOT of third parties. Check out this list of AI bots that may be scraping your site. 

You may have seen Axios’ recent piece that laid out the stark reality of click-throughs as you can see on the chart below, showing the number of Web pages crawled per visitor referral.

These numbers should stop any news organisations in their tracks. The scraper bots are coming, and they’re taking a lot more than they give back.

In a recent INMA webinar, Robert Hahn from The Guardian shared how they are approaching AI licensing — and made it clear how surprised he was that so few publishers are acting on this. On that call, 59% of attendees said they are not blocking scraper bots.

Let’s be blunt: If you allow scrapers to crawl your content, you’re giving it away for free. And if it’s free, why would anyone pay for it?

But here is what you can do.

Companies such as Cloudflare, Tollbit, ScalePost, Dark Visitors, and Miso are all building solutions for this:

• Cloudflare is working on a tool to manage this at a large scale with blocking by default.
• Tollbit, ScalePost, and Dark Visitors are helping news publishers track and monetise AI usage for free.
• And Miso even has a tool that lets you see, right now, how many scrapers are crawling your site. (Warning: The results may surprise you.)

If you’re looking for more context or next steps, you can check out this earlier post. But the headline remains: This is the single most important first move for protecting your content in an AI world.

And if this resonated, please consider joining us at Media, Tech & AI week, where we’ll spend some time really diving into this.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .