AINewsnow

How the top 1,000 .jp sites treat AI bots: weekly robots.txt census

Whether an AI search answer can draw on a site depends partly on which crawlers that site lets in. For English-language sites there are several public counts of who blocks GPTBot or ClaudeBot. For Japanese sites there is very little public data. So every week we fetch robots.txt and llms.txt from t…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-03 03:58 · DEV Community — AI
    How the top 1,000 .jp sites treat AI bots: weekly robots.txt census

More stories

  1. Anthropic reports $17B quarterly revenue, $900B spending commitments ahead of IPO — Reuters AI
  2. OpenAI appoints former US Army Secretary Dan Driscoll as Senior Counselor for National Security Modernization — OpenAI News
  3. Anthropic Unveils Paperwork for Blockbuster IPO — Wall Street Journal Technology
  4. Marvell to acquire Lumentum in $115 billion all-stock AI data-centre deal — DigiTimes Asia
  5. OpenAI cuts funding target to about $20 billion as tech stocks sell off — Economic Times Tech
  6. Anthropic SEC filing warns of imminent threshold for AI-driven research acceleration — TechCrunch AI
  7. OpenAI to merge ChatGPT's Chat and Work tabs into a single desktop experience — TechCrunch AI
  8. DeepSeek launches Creator Mode for building custom plugins via natural language — Techmeme

Get the daily brief of stories like this at 6:30 every morning →