AI bots and crawlers: publishers build their own defences

The bosses of eight UK national newspapers, from The Guardian to The Sun, urged the Prime Minister to back the Automated Online Software (Access and Transparency) Bill, which would require bots to declare who operates them and why they are taking content, with the News Media Association claiming 19 of the 20 biggest UK news sites can currently be scraped by stealth bots despite paywalls. People Inc CEO Neil Vogel told The Rebooting his company's default is to "block every single bot unless we permission it, obviously except Google", and that the phone started ringing within 30 days of doing so. Simulmedia's Dave Morgan, writing in MediaPost, asked when the industry will be willing to keep malicious bots out of the emerging agentic ad ecosystem, where code consumes impressions no human ever sees.

The crux

Without legal force behind robots.txt, publishers are concluding the only leverage they have is technical and political: block the bots, then name the ones that matter.

Sell-side read

Every publisher's real, addressable audience is being diluted by unidentifiable bot traffic that inflates hosting costs and suppresses the value of legitimate inventory, so bot transparency is a direct yield and margin issue, not a policy nicety. The Hinds bill, as the letter puts it, "simply gives website owners the information they need to identify and assess potentially abusive traffic, then take action to block it or negotiate fair payment."

Watch

The bill's second reading and whether Prime Minister Andy Burnham backs it, plus New York's Stealth Crawler Prohibition Act awaiting the governor and the US Congress's Stealth Bot Prohibition Act.