AINewsnow

I checked my nginx log for AI crawlers. A third of the “Googlebots” were fake

I went through two weeks of my site's nginx access log and counted every bot that said it was an AI crawler or a search engine. I only kept requests whose IP address really belongs to the company behind the bot. AI crawlers made 1,590 verified requests. All search engines together made 563. Meta's…

Read the full story at DEV Community — AI ↗

Timeline · 1 report

  1. 2026-11-09 08:50 · DEV Community — AI
    I checked my nginx log for AI crawlers. A third of the “Googlebots” were fake

More stories

  1. Meta chief strategy officer Dave Wehner to join AI startup Fireworks as CFO — CNBC Technology
  2. Meta adds custom 'Spaces' to its Muse AI app — The Verge AI
  3. Meta launches specialized AI models with Active Memory and Active Focus features — Techmeme
  4. Meta pauses planned updates to commercial and AI ad terms — Reuters AI
  5. AI Daily Digest November 7, 2026: AI Cracks Legacy RSA, Gemini Omni Gets Longer Clips, First Humanoid IPO — DEV Community — AI
  6. Meta Brings Muse Wonder's Memory and Focus to Everyone: What Active Memory and Active Focus Actually Change — DEV Community — AI
  7. Physics-Augmented Diffusion Modeling for deep-sea exploration habitat design in hybrid quantum-classical pipelines — DEV Community — AI
  8. CyberTiel35BA3B working on Blender Rigging via MCP. 8GB VRAM, 16GB RAM. llama.cpp. — r/LocalLLaMA

Get the daily brief of stories like this at 6:30 every morning →