Website operators have spent the last two years deciding whether to welcome AI crawlers, block them, or measure them as a new source of visits. A fresh discussion around Known Agents pushed a harder operational point into view: a request that calls itself ClaudeBot, Googlebot or another familiar crawler in the User-Agent header is not proof that it came from that crawler. It may be an ordinary vulnerability scanner borrowing a respectable name.

Firewall dashboard verifying suspicious AI crawler User-Agent headers The immediate trigger was a 12 August 2026 Hacker News thread linking to Known Agents, titled “Someone is running mass vulnerability scans, spoofing AI bots like ClaudeBot”. The thread matters less as a courtroom record than as a snapshot of what administrators already see: noisy probes for /wp-login.php, .env, admin panels, API routes and test tools now arrive under labels that sound like search engines or AI assistants. Known Agents' public Agentic Web Index gives useful context. It says its measurements come from more than 5,000 websites using its analytics products, and it presents automated traffic as a measurable part of the web rather than an edge case.

What is confirmed, and what should not be overstated

The careful version is this: Known Agents tracks automated agents and classifies traffic types including AI assistants, AI agents, AI coding agents, AI data scrapers, AI search crawlers, search engines, security scanners, generic scrapers and undocumented AI agents. Its insights page reported bot traffic at 35 percent of measured traffic, “Agentrification” at 29 percent of bot traffic, AI chat referral volume at 0.1 percent of human visits, and robots.txt effectiveness at 98.5 percent at research time. Those numbers are product-derived measurements, not a universal census of the entire internet, so they should be read as a live operational signal from a large sample rather than a global law.

The Hacker News item around the mass-scan claim had hundreds of points and comments at the research check. Several participants made the practical observation that User-Agent strings are commonly spoofed and that network ownership, reverse DNS and hosting context matter before trusting a bot label. Others argued that mass scanning is not new at all; internet-facing systems have been probed constantly for years. Both points can be true. The novelty is not that scanners exist. The novelty is that legitimate AI crawling, AI-assisted retrieval and AI referral measurement create a new cover story for scans that already existed.

It is also important not to imply that Anthropic, OpenAI or Google are performing the malicious scans simply because their bot names appear in logs. Official crawler documentation describes legitimate bots and robots.txt controls; a malicious client can copy a string without being that company. The distinction between “ClaudeBot” and “traffic claiming to be ClaudeBot” is the whole security issue.

Why User-Agent is a label, not an identity

A User-Agent header is easy to send. It was designed as client self-description, not as cryptographic identity. Any script, scanner, proxy or compromised host can put ClaudeBot/1.0, Googlebot, ChatGPT-User, a browser string, or an invented agent name into the header. Logs will faithfully record the label, but the label alone does not tell you who controls the connection.

That weakness was tolerable when the stakes were basic analytics or browser compatibility. It is much less tolerable when teams use the same field to decide whether to permit large-scale crawling, classify AI referrals, relax rate limits, exempt traffic from bot challenges, or suppress security alerts. A spoofed name can pollute dashboards, hide probes in trusted buckets, and make incident response start from the wrong assumption.

Search-engine operators have long published verification guidance because impersonation is old. Google's guidance for verifying Googlebot relies on reverse DNS and matching DNS resolution, not on the string alone. Cloudflare's bot documentation similarly distinguishes verified bots from unverified automated traffic and documents categories for AI bots. The practical pattern is consistent: identity requires corroboration from the network path, provider documentation and behavior, not just an HTTP header.

The AI practice problem

AI adoption changes the operational environment in three ways. First, more legitimate automated fetchers are hitting websites. Search crawlers, assistant fetchers, model-training crawlers, coding-agent browsers and data scrapers have different purposes and should not all be governed by one rule. Second, business teams increasingly want to count AI chat referrals and visibility in answer engines. Third, attackers benefit from the confusion because a bot name that used to look suspicious can now sound strategically important.

That creates a governance trap. If a publisher blocks every AI-labelled request, it may reduce scraping and load but also lose useful assistant visibility or product integrations. If it allows every request that claims to be a known AI crawler, it gives scanners a free costume. If it treats robots.txt as access control, it mistakes a voluntary convention for an enforcement layer. Robots.txt remains useful for communicating policy to cooperative crawlers; it does not stop a scanner looking for secrets.

For SaaS companies the risk is not only content scraping. Fast internal tools, AI-generated dashboards, preview deployments and half-forgotten admin panels are now common. A scanner using a friendly crawler name may probe exactly the paths that reveal weak operational discipline: exposed environment files, debug routes, old WordPress endpoints, test APIs, temporary uploads and unauthenticated health checks. The AI label can make those requests look like analytics noise rather than hostile reconnaissance.

A practical taxonomy for logs

Do not start with allow or block. Start by classifying the traffic. An AI assistant fetcher retrieves a page because a user asked a system to open or summarize it. An AI search crawler indexes content for answer products. A model-training or data-scraping crawler collects content at scale. A coding-agent browser may fetch documentation, package pages, GitHub files or application previews as part of a development workflow. A security scanner probes for known weaknesses. A malicious impersonator borrows names from any of the previous groups.

Each category deserves a different policy. Assistant fetchers may be acceptable at moderate rate when they respect robots.txt and identify themselves clearly. Training scrapers may need stricter licensing and rate limits. AI search crawlers may sit between marketing opportunity and content-control concern. Security scanners may be permitted only when they are your own vendor, bug-bounty partner or known research organization. Unknown traffic claiming famous names belongs in the suspicious bucket until verified.

The classification should be visible in logs. Add fields for raw User-Agent, source IP, ASN, hosting provider, reverse DNS result, forward-confirmed DNS result when available, country or region if relevant, request path, status code, rate, robots.txt behavior, challenge result and final policy decision. Without those fields, post-incident analysis becomes a debate over anecdotes.

What teams should do now

First, stop writing rules that trust names alone. A WAF rule that says “allow if User-Agent contains ClaudeBot” is too soft. Replace it with a rule that checks vendor-documented identity where possible, source network reputation, request rate, path sensitivity and behavior. If full verification is not possible, use a lower-trust lane: allow public content at conservative rates, challenge unusual behavior and block sensitive paths.

Second, separate robots policy from security policy. Robots.txt can express whether cooperative crawlers may access training, search or assistant surfaces. It should not be used as the only control for private routes, staging hosts, admin tools or API endpoints. Sensitive paths require authentication, network controls, deny rules and monitoring whether or not a bot says it respects robots.txt.

Third, maintain purpose-specific allowlists. A list for search-engine crawlers is not the same as a list for AI assistant fetchers, security vendors or analytics tools. Record why each entry exists, who owns it internally, which documentation proves identity, which paths it may access, and what rate is acceptable. Expire entries that no one can justify.

Fourth, protect the boring paths. Rate-limit probes against .env, /admin, /wp-login.php, /debug, /api/user, staging hostnames and forgotten test tools. Return consistent denials. Do not leak stack traces. Make sure preview deployments and AI-built prototypes inherit the same basic controls as production. Many incidents begin not with an advanced exploit but with a scanner finding a convenience endpoint that was never meant to be public.

Fifth, keep AI referral analytics separate from bot identity. It is useful to know whether ChatGPT, Claude, Perplexity or another assistant sent a human visitor. It is a different problem to decide whether an automated request is allowed to crawl. Mixing those dashboards invites bad decisions: marketing wants visibility, security wants confidence, and the header string satisfies neither requirement by itself.

What not to do

Do not block all AI traffic as a reflex unless your business model supports that choice. Some sites want citations, assistant retrieval and developer traffic. A blanket block can be strategically correct for private documentation, paid archives or fragile infrastructure, but it should be an explicit decision rather than a panic reaction.

Do not allow famous names by default. That is the mistake the spoofing story exposes. Treat famous names as claims that need verification. Do not suppress alerts merely because the label says Googlebot, ClaudeBot or ChatGPT-User. Do not give those labels access to staging systems, internal search, admin paths or unpublished APIs.

Do not expect one vendor product to solve the policy problem. WAFs, bot scores, reverse DNS checks, verified-bot feeds, rate limits and SIEM enrichment are useful, but the organization still needs to decide which automated uses are welcome. The policy should be written in business language first and implemented in technical controls second.

The future: signed agents, not polite names

The long-term answer is stronger identity for automated agents. The web has several possible paths: signed crawler requests, verified-agent registries, mTLS-like relationships for high-value APIs, token-based access for partners, browser or platform attestations, and standardized disclosure of agent purpose. Each path has trade-offs. Strong identity can reduce spoofing, but it can also centralize power, raise barriers for small crawlers and push the open web toward permissioned access.

Until those standards mature, site owners must use layered heuristics. That is not elegant, but it is practical. Verify the network when documentation exists. Challenge or rate-limit ambiguous traffic. Separate public content from sensitive systems. Keep logs rich enough to revisit decisions. Make the business trade-off explicit: visibility in AI answers and assistant workflows has value, but so do uptime, security, licensing and trust.

A compact operating checklist

For the next week, review five things. In logs, add User-Agent, IP, ASN, reverse DNS and final bot decision as searchable fields. In the firewall, review rules for known crawlers, sensitive paths, high-rate requests, failed challenges and suspicious hosting providers. In robots.txt, separate AI training, AI search and generic crawling where the vendors support those distinctions. In analytics, separate human AI referrals from automated bot visits. In engineering, make sure preview deployments, internal dashboards and AI-generated tools require authentication before they appear on the public internet.

The practical lesson is simple but easy to ignore: AI traffic is now ordinary infrastructure traffic. That makes it worth measuring and governing. It also makes it attractive camouflage. If a crawler's name can be typed by anyone, the name is not a passport. Treat it as a claim, verify what can be verified, and design the rest of the system so a forged label cannot become a security decision.

Sources

Known Agents Agentic Web Index and agent pages for ClaudeBot and ChatGPT-User; Hacker News item 49272569; Anthropic crawler documentation; OpenAI bots documentation; Google Search Central robots.txt and Googlebot verification guidance; Cloudflare bot, verified bot and AI bot documentation.