AINewsnow

Cloudflare Says Perplexity Ignored Robots.txt. We Checked the Record.

This story is from 2026-09-16. It is preserved in the archive; the latest stories are on the live feed.

Originally published on LetsLaunch . Our own AI crawler cheat sheet says the named crawlers "have an established track record of compliance" with robots.txt . That line needs a footnote, and the footnote is Perplexity. In a blog post published in August 2025, Cloudflare accused Perplexity of "steal…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-16 08:20 · DEV Community — AI
    Cloudflare Says Perplexity Ignored Robots.txt. We Checked the Record.

More stories

  1. Getting more accurate results - personalizations — r/ArtificialInteligence
  2. Opti 27B: Qwen3.8-27B in 11.8 GB at 3.47 bpw, within 0.5% of FP16 perplexity and matching Q4_K_M at 30% fewer bytes. Patched llama.cpp runtime, source public, reproduce with one command — r/LocalLLM
  3. claude, chatgpt, kimi or perplexity subscription (end of 2026) — r/ArtificialInteligence
  4. Perplexity’s AI agents helped build a database. They weren’t allowed to run it. — The New Stack AI
  5. Google Joins OpenAI, Anthropic, Meta in Disclosing AI Hacks — Bloomberg AI
  6. AI's role in building AI surging? Anthropic says Claude now leads 26% of its R&D — Mint AI
  7. Introducing Astra for Law — OpenAI News
  8. Novo Nordisk Will Use Anthropic’s Claude for Drug Research — Wall Street Journal Technology

Get the daily brief of stories like this at 6:30 every morning →