nginx配置屏蔽蜘蛛访问-舍利云

vps云服务器云主机新闻中心

NEWS CENTER

nginx配置屏蔽蜘蛛访问

发布时间:2021-06-01 16:48






 if ( $http_user_agent ~* "SemrushBot|semrush|petalsearch|PetalBot|dotbot|AhrefsBot|ahrefs|Googlebot" ) {

return 403 ;

}   

#禁止Scrapy等爬虫工具的抓取

if ($http_user_agent ~* "Scrapy|Sogou web spider") {

  return 403;

}

#禁止指定UA及UA为空的访问

if ($http_user_agent ~ "FeedDemon|SemrushBot|semrush|AhrefsBot|ahrefs|dotbot||Googlebot|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|YisouSpider|HttpClient|MJ12bot|heritrix|EasouSpider|LinkpadBot|Ezooms|^$" )

{

  return 403;

}

寻求帮助?

我们很高兴能为您服务

您可以通过工单等方式联系我们~

订阅我们的邮件

与我们保持联系,获取最新消息和折扣优惠券