robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)
This story is from 2026-10-22. It is preserved in the archive; the latest stories are on the live feed.
Every few months this request lands in my inbox: "can we block the AI crawlers but stay in Google?" The answer is yes, and it takes about 20 lines of robots.txt — but only if you stop treating robots.txt as one switch. This is the walkthrough I run on WordPress sites. 1. Separate the three decision…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-10-22 08:30 · DEV Community — AI
robots.txt patterns that let search crawlers in and keep AI training crawlers out (without wrecking indexing)