How to charge AI crawlers per page with HTTP 402 (and keep your site free for people)
If you run a website, a growing share of your traffic isn't people. It's AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Bytespider and a long tail of others, fetching pages to train models or answer questions. Until now you had two choices: Let them in for free. You pay the bandwidth; they get the…
Read the full story at DEV Community — AI ↗
Timeline · 1 report
- 2026-11-15 04:16 · DEV Community — AI
How to charge AI crawlers per page with HTTP 402 (and keep your site free for people)