AINewsnow

Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

Short answer: in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawle…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-02 11:23 · DEV Community — AI
    Which blogging platforms let AI search bots read your posts? (robots.txt and raw HTML, checked 31 Oct 2026)

More stories

  1. Alphabet launches eurobond in test of AI debt markets — Semafor Technology
  2. AI widens gap between Meta's ad growth and Google's search slowdown — CNBC Technology
  3. ISTA DeepSeek-V4.1-Flash-1017 GSQ-RCO quants comparison — r/LocalLLaMA
  4. Brazil courts data centers with tax breaks and cheap power as US opposition grows — Fortune AI
  5. Big Tech returns smallest share of cash to shareholders in 15 years amid AI spending — Reuters AI
  6. Google, OpenAI, Microsoft and Anthropic to honor shared robots.txt AI-training opt-out — Techmeme
  7. Reddit moves toward logged-in-only access as Google AI Overviews cut search traffic — Business Insider AI
  8. Google's Gemini 4 Carbon tops independent AI leaderboards with restricted access — Latent Space

Get the daily brief of stories like this at 6:30 every morning →