AINewsnow

Most AI-crawler blocking doesn't happen in robots.txt

This story is from 2026-09-19. It is preserved in the archive; the latest stories are on the live feed.

Most AI-crawler blocking doesn't happen in robots.txt Numbers re-derived from the live Agent Web Index aggregate on 2026-09-19 (47,314 domains with a measured verdict). Dataset CC BY 4.0: Zenodo DOI, GitHub DeusAcc/agent-web-index, Hugging Face DeusHorizon/agent-web-index. Live hub: https://shop.lu…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-09-19 20:19 · DEV Community — AI
    Most AI-crawler blocking doesn't happen in robots.txt

More stories

  1. Deploy Hugging Face models on Amazon SageMaker AI with coding agents — AWS Machine Learning Blog
  2. I literally build the jev architecture one year back and made it open-sourced — r/reinforcementlearning
  3. A quick Minimax H3 news round-up - 17th September 2026 — r/comfyui
  4. Hugging Face Hack Shows Humans Can Keep AI In Check — AI Now Institute
  5. Made a tool that tells you which GGUF quants will fit your GPU/Mac, with the llama.cpp command to run them — r/LocalLLM
  6. An Adversary Capable of Defeating — r/artificial
  7. ‘Godfather of AI’ Geoffrey Hinton warns humans running out of time to control Artificial Intelligence – ‘maybe a year’ — Mint AI
  8. Your AI agents are isolated. Your infrastructure isn’t — InfoWorld AI

Get the daily brief of stories like this at 6:30 every morning →