{"schema_version":"1.0","service":"Publicasta","type":"article","id":316,"slug":"ai_crawler_spoofing_user_agent_verification_2026_08_13","title":"不能再只凭名称信任 AI 爬虫","excerpt":"伪造的爬虫名称正在把 AI 流量治理变成安全、分析和运维问题。","language":"zh","default_language":"en","canonical_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh","image":{"url":"https://publicasta.com/storage/projects/8/pages/316/2026/08/d11c21ad-1744-438f-a67a-81c60f3db0b6.webp","alt":"防火墙面板正在验证可疑 AI 爬虫的 User-Agent 请求头"},"publisher":{"id":8,"slug":"ai_practice","name":"AI Practice","url":"https://publicasta.com/ai_practice"},"author":{"name":"Anton R"},"published_at":"2026-08-13T10:12:40+00:00","updated_at":"2026-08-13T10:13:15+00:00","content_markdown":"网站和 SaaS 团队正在学习如何处理 AI 爬虫：有的希望获得问答引擎可见度，有的担心内容被抓取，有的只想把日志里的自动流量分清。Known Agents 相关讨论把一个更具体的问题摆到台面上：请求头里写着 `ClaudeBot`、`Googlebot` 或 `ChatGPT-User`，并不等于它真的来自这些服务。\n\n ![防火墙面板正在验证可疑 AI 爬虫的 User-Agent 请求头](https://publicasta.com/storage/projects/8/pages/316/2026/08/d11c21ad-1744-438f-a67a-81c60f3db0b6.webp) 2026 年 8 月 12 日，Hacker News 上关于 Known Agents 的帖子引发讨论，标题提到有人冒充 ClaudeBot 等 AI bot 进行大规模漏洞扫描。更稳妥的理解是：这不是证明某家公司在扫描，而是说明攻击者可以把熟悉的 bot 名称当作伪装。管理员在日志中看到的 `.env`、`/wp-login.php`、管理入口、测试 API 和临时部署探测，可能只是披着 AI 爬虫外衣的普通扫描。\n\n ## 已确认的信息\n\n Known Agents 的 Agentic Web Index 把自动化流量分成 AI 助手、AI 代理、编程代理、数据抓取器、AI 搜索爬虫、搜索引擎、安全扫描器、普通抓取器和未记录 AI 代理等类别。研究时页面显示：样本站点中 bot traffic 为 35%，其中 29% 与 AI 相关，AI chat referral volume 为 human visits 的 0.1%，robots.txt effectiveness 为 98.5%。这些数字来自其产品覆盖的 5,000 多个网站，不应当被说成整个互联网的普遍比例。\n\n ## 名称为什么不是身份\n\n `User-Agent` 是客户端自报家门，不是签名。任何脚本、代理、扫描器或被入侵主机都可以发送一个熟悉字符串。日志会记录这个标签，但它无法证明来源组织、网络所有权、抓取目的或可信程度。过去这主要影响浏览器兼容和基础分析；现在同一字段可能被用于放行爬虫、降低速率限制、分类 AI referral 或压低安全告警，风险就大得多。\n\n Google 对 Googlebot 的验证依赖反向 DNS 和正向解析匹配；Cloudflare 也区分 verified bots、AI bots 和未验证自动化流量。共同原则很清楚：要结合 IP、ASN、DNS、网络声誉、供应商文档和行为，而不是只看请求头。\n\n ## 对 AI 实践的影响\n\n AI 采用让合法自动访问变多：用户让助手打开网页，AI 搜索抓取内容，训练抓取器大规模收集数据，编程代理访问文档和预览环境。它们的目的不同，策略也应不同。把所有 AI 字样都封掉，可能损失引用和可见度；把所有知名名称都放行，则给扫描器一件免费外衣。\n\n robots.txt 仍然有价值，但它不是访问控制。它表达给合作爬虫看的规则，不能保护 staging、管理后台、私有 API 或临时工具。真正敏感的路径需要认证、网络限制、拒绝规则和监控。\n\n ## 具体做法\n\n 日志应记录原始 `User-Agent`、源 IP、ASN、托管商、反向 DNS、请求路径、状态码、速率、robots.txt 行为、challenge 结果和最终策略决定。没有这些字段，事后分析只能靠猜。\n\n WAF 规则不要写成“包含 ClaudeBot 就允许”。更安全的做法是检查供应商文档、网络信誉、请求频率、路径敏感度和行为模式。无法完全验证时，把流量放到低信任通道：只给公共内容、保守限速、异常挑战、敏感路径拒绝。\n\n 允许列表要按目的维护。搜索引擎、AI 助手、安全供应商、内部工具和分析服务不应共用同一张表。每项都要有内部负责人、允许原因、验证依据、可访问路径和速率限制。没人能解释的条目应过期。\n\n ## 不该做什么\n\n 不要把 AI referral 分析和 bot 身份混在一起。知道一个人是否从 ChatGPT、Claude 或 Perplexity 来到网站，是受众归因；判断一个自动请求能否抓取，是安全策略。也不要因为标签看起来有名就降低告警优先级，更不要让它访问 staging、内部搜索、管理入口和未发布 API。\n\n ## 下一步\n\n 更长期的答案可能是签名 crawler 请求、verified-agent registry、面向高价值 API 的 mTLS 关系、合作伙伴 token 或平台证明。这些机制能减少伪造，但也可能提高小型爬虫门槛，让开放 Web 更接近许可访问。在标准成熟前，现实方案就是分层判断。\n\n ## 简短清单\n\n 本周可以检查五件事：日志字段是否足够；防火墙是否保护 `.env`、`/admin`、`/wp-login.php`、debug 和测试 API；robots.txt 是否区分训练、搜索和通用抓取；分析系统是否区分人类 referral 与自动访问；preview deployment、内部 dashboard 和 AI 生成工具是否默认需要认证。\n\n 核心结论很简单：AI 流量已经是普通基础设施流量，值得测量和治理。但它也成了好用的伪装。如果名称任何人都能打出来，它就不是护照，只是一个待验证的声明。\n\n ## 来源\n\n Known Agents Agentic Web Index、ClaudeBot 和 ChatGPT-User 页面；Hacker News item 49272569；Anthropic、OpenAI、Google Search Central 与 Cloudflare 关于 bots、robots.txt、验证和 AI crawlers 的文档。","available_translations":[{"language":"ar","title":"لم يعد اسم زاحف الذكاء الاصطناعي كافيًا للثقة","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=ar","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=ar","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=ar","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=ar"},{"language":"de","title":"KI-Crawler sind am Namen nicht mehr verlässlich","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=de","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=de","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=de","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=de"},{"language":"en","title":"AI crawlers can no longer be trusted by name","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=en","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=en","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=en","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=en"},{"language":"es","title":"Ya no basta confiar en el nombre de un crawler de IA","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=es","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=es","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=es","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=es"},{"language":"fr","title":"Le nom d’un crawler IA ne suffit plus","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=fr","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=fr","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=fr","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=fr"},{"language":"pl","title":"Crawlerom AI nie można już ufać po nazwie","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=pl","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=pl","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=pl","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=pl"},{"language":"ru","title":"AI-краулерам больше нельзя верить по имени","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=ru","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=ru","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=ru","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=ru"},{"language":"zh","title":"不能再只凭名称信任 AI 爬虫","html_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh","markdown_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=zh","json_url":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=zh","api_url":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh"}],"_links":{"self":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=zh","api":"https://publicasta.com/api/public/v1/channels/ai_practice/articles/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh","html":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh","canonical":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13?lang=zh","markdown":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.md?lang=zh","json":"https://publicasta.com/ai_practice/ai_crawler_spoofing_user_agent_verification_2026_08_13.json?lang=zh","channel":"https://publicasta.com/api/public/v1/channels/ai_practice","channel_articles":"https://publicasta.com/api/public/v1/channels/ai_practice/articles","search":"https://publicasta.com/api/public/v1/search","documentation":"https://publicasta.com/api-docs#reading-publicasta","openapi":"https://publicasta.com/api-docs/openapi.json","llms":"https://publicasta.com/llms.txt"}}