本站重新定位为
分享技术和经验的个人博客

CF里显示GOOGLE BOT一小时爬10K的次数。。这正常吗

RT……………我做英文站的

顺便分享一下网上找的很好的屏蔽bad BOT的规则,这个适用做英文站的哈,因为屏蔽了中国的一些bot
(http.user_agent contains "Yandex") or (http.user_agent contains "muckrack") or (http.user_agent contains "Qwantify") or (http.user_agent contains "Sogou") or (http.user_agent contains "BUbiNG") or (http.user_agent contains "knowledge") or (http.user_agent contains "CFNetwork") or (http.user_agent contains "Scrapy") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "AhrefsBot") or (http.user_agent contains "Baiduspider") or (http.user_agent contains "python-requests") or (http.user_agent contains "crawl" and not cf.client.bot) or (http.user_agent contains "Crawl" and not cf.client.bot) or (http.user_agent contains "bot" and not http.user_agent contains "bingbot" and not http.user_agent contains "Google" and not http.user_agent contains "推特" and not cf.client.bot) or (http.user_agent contains "Bot" and not http.user_agent contains "Google" and not cf.client.bot) or (http.user_agent contains "Spider" and not cf.client.bot) or (http.user_agent contains "spider" and not cf.client.bot)

下图是我的CF介面,好BOT(浅蓝allow)的流量远超BAD BOT

天津网友:楼上的规则会屏蔽谷歌吗
贵州网友:

has 发表于 2020-12-6 18:46
楼上的规则会屏蔽谷歌吗

不会的,我使用了,很好用,
contains "Google" and not cf.client.bot,这个是屏蔽假google bot
上海网友:为什么屏蔽 yandex sougo呢
北京网友:做啥站,我的都没bot来爬
西藏网友:如果有的蜘蛛 没被 cloudflare收录呢

赞(0)
这篇文章为天际博客(天朝世纪)原创,转载请注明。天际部落 » CF里显示GOOGLE BOT一小时爬10K的次数。。这正常吗

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址