Over the past few months, I have been fairly vocal about scrapers, but they aren’t a zero-sum game. Should you block the bots? Probably. But as AI rapidly evolves, it’s not always that straightforward. 

For news organisations, the decision sits at the intersection of protecting intellectual property, enabling discovery, and exploring new monetisation paths.

Below is a framework to help you navigate the trade-offs.

1. Start with the “who” 

Not all scrapers are equal. This isn’t one-size-fits-all. Treat each scraper as its own negotiation.

• Which scrapers are accessing your content? Big AI players? Aggregators? Niche apps?

• Do you have an existing deal? If there’s a commercial or licensing arrangement in place, that changes the calculation.

• Could allowing access benefit your brand? In some cases, broader visibility can drive awareness or subscription funnels.

2. Understand the business impact

Before defaulting to “block,” map out potential downstream effects. This decision isn’t just about being defensive, it’s increasingly tied to distribution.

• Core business risk → Could unrestricted scraping undermine your subscription, advertising, or audience strategy?

• Monetisation potential → Is there a clear path to value, whether via licensing, commerce, or attribution?

• Strategic positioning → Could allowing certain bots help your content surface in AI-driven discovery engines?

3. Be selective about what’s scraped 

The conversation doesn’t need to be all or nothing. Instead:

• Decide which verticals or products within your content pool you’re comfortable exposing.

• Atomise your content. Allow some pieces of some content, restrict others.

• Treat access like you would any marketing channel. Open up where the upside outweighs the risk.

4. Set boundaries then test and measure

If you allow limited scraping, set up a measurement period:

• Monitor traffic referrals, attribution, and downstream conversions.

• Compare against your closed-off content to assess real-world impact.

• Use data, not assumptions, to decide whether to maintain, expand, or revoke access.

Conclusion

Scraping decisions are now strategic, not purely technical. AI has shifted the stakes. The same bots you block today might become powerful distribution engines tomorrow — or new revenue streams.

The right framework blends protection, experimentation, and negotiation. Start small, measure impact, and keep iterating as the ecosystem evolves.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .