Google, OpenAI, Anthropic and Microsoft pledge support for new AI scraping control signal

Google, OpenAI, Anthropic and Microsoft have pledged to support a new common robots.txt signal that lets websites block scraping of their content for AI training.

Read the full story at Engadget ↗

Timeline · 1 report

2026-10-31 10:34 · Engadget
Google joins OpenAI and Anthropic in promising new scraping controls

More stories

OpenAI, Google, Microsoft and Anthropic to honor unified robots.txt AI training opt-out — OpenAI News
AI giants pledge unified way for publishers to block content scraping — TechXplore AI & ML
Major AI companies back new standard for publishers to opt out of scraping — The Verge AI
Forward Deployed Engineer: the job AI borrowed from Palantir — DEV Community — AI
Mistral releases weights for trillion-parameter 'Le Chonk' model — Mistral AI News
Get ChatGPT, Claude, and Gemini in one lifetime AI toolkit for $99.99 — Mashable AI
I Benchmarked 41 AI Agent Harness Patterns in Isolated VMs: How to Cut Tokens by 90% Without Losing Accuracy — DEV Community — AI
What we got wrong building an MCP server for AI visibility data — DEV Community — AI

Get the daily brief of stories like this at 6:30 every morning →