AINewsnow

A third of my Common Crawl index lookups get no answer, and 04:00 UTC is when it is worst

This story is from 2026-10-12. It is preserved in the archive; the latest stories are on the live feed.

Since 15 May a small Cloudflare Worker of mine asks the Common Crawl index, once a day, whether about a dozen URLs of my site are in the current crawl. It is a cheap, boring check: one request to get the list of collections, then one CDX lookup per URL, and a row in a table with the answer for each…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-10-12 06:05 · DEV Community — AI
    A third of my Common Crawl index lookups get no answer, and 04:00 UTC is when it is worst

More stories

  1. Microsoft CEO Nadella calls for AI 'emergency brake' and trust architecture review — CNBC Technology
  2. Nvidia in talks to acquire US ‘open’ model start-up Reflection AI — Financial Times AI
  3. Qwen Image 2.1 Turbo and MiniMax H3 Gain Community Support — r/StableDiffusion
  4. Microsoft president calls for global AI regulation, citing lack of oversight — BBC Technology
  5. Former HSBC executive launches AI-driven wealth management startup — South China Morning Post Tech
  6. Ex-DeepMind scientist David Silver launches startup Ineffable Intelligence — Financial Times AI
  7. Centre for the Governance of AI Announces 2027 Winter Fellowships — Centre for the Governance of AI
  8. Qualcomm CEO Says AI Firms Seek 100B-Parameter Models on Phones by 2028 — Techmeme

Get the daily brief of stories like this at 6:30 every morning →