國(guó)外有幾個(gè)爬蟲(chóng)，是真的很煩人，畢竟他沒(méi)有多少用戶(hù)，還抓取抓取中文網(wǎng)頁(yè)，對(duì)于網(wǎng)站來(lái)講意義不大。于是昨天因?yàn)榭蛻?hù)服務(wù)器上卡頓，所以索性幫他進(jìn)行了屏蔽。

今天看日志，semrush爬蟲(chóng)果然是老實(shí)了，如下圖所示：

網(wǎng)站的訪(fǎng)問(wèn)量，下降很多，爆贊！

主要封禁的是：MJ12bot、SemrushBot、AhrefsBot。robots.txt文件如下，大家可以做個(gè)參考：

User-agent: MJ12bot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SemrushBot-BM
Disallow: /
User-agent: SemrushBot-SEOAB
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Uptimebot
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Researchscan
Disallow: /
User-agent: DnyzBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: YandexBot
Disallow: /