三十而立

nginx下的文章

对于做国内站的我来说,我不希望国外蜘蛛来访问我的网站,特别是个别垃圾蜘蛛,它们访问特别频繁。这些垃圾流量多了之后,严重浪费服务器的带宽和资源。通过判断user agent,在nginx中禁用这些蜘蛛可以节省一些流量,也可以防止一些恶意的访问。

1、进入nginx的配置目录,例如cd /usr/local/nginx/conf

2、添加agent_deny.conf配置文件 vim agent_deny.conf

加入以下

#禁止Scrapy等工具的抓取
if ($http_user_agent ~* (Scrapy|Curl|HttpClient)) {
  return 403;
}
#禁止指定UA及UA为空的访问
if ($http_user_agent ~ "FeedDemon|JikeSpider|Indy Library|Alexa Toolbar|AskTbFXTV|AhrefsBot|CrawlDaddy|CoolpadWebkit|Java|Feedly|UniversalFeedParser|ApacheBench|Microsoft URL Control|Swiftbot|ZmEu|oBot|jaunty|Python-urllib|lightDeckReports Bot|YYSpider|DigExt|HttpClient|MJ12bot|heritrix|EasouSpider|LinkpadBot|Ezooms|^$" )
{
  return 403;
}
#禁止非GET|HEAD|POST方式的抓取
if ($request_method !~ ^(GET|HEAD|POST)$) {
  return 403;
}

还有加一些针对特殊的user_agent的访问

if ($http_user_agent ~ "Mozilla/4.0\ \(compatible;\ MSIE\ 6.0;\ Windows\ NT\ 5.1;\ SV1;\ .NET\ CLR\ 1.1.4322;\ .NET\ CLR\ 2.0.50727\)") { return 404; }

这个是如何得出是频繁访问的user_agent呢,通过分析nginx的日志可以得出

tail -n 1000 /usr/local/nginx/logs/access.log | awk -F\" '{A[$(NF-1)]++}END{for(k in A)print A[k],k}' | sort -n | tail 

分析访问次数
执行以上命令可以得出访问最多的user_agent,通过人为判断是否正常来屏蔽

然后在nginx.conf的location中加入include agent_deny.conf;

平滑重启nginx

/usr/local/nginx/sbin/nginx –s reload

然后测试一下 设置是否成功

curl -I -A "BaiduSpider" www.test.com
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 09 Feb 2015 03:37:20 GMT
Content-Type: text/html; charset=UTF-8 Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/5.5.19 Vary: Accept-Encoding, Cookie
Cache-Control: max-age=3, must-revalidate
WP-Super-Cache: Served supercache file from PHP
curl -I -A "JikeSpider" www.test.com
HTTP/1.1 403 Forbidden
Server: nginx Date: Mon, 09 Feb 2015 03:37:44 GMT
Content-Type: text/html
Content-Length: 162 Connection: keep-alive

到这里,nginx通过判断User-Agent屏蔽蜘蛛访问网站就已经完成,可以根据实际情况对agent_deny.conf中的蜘蛛进行增加、删除或者修改。

参考原文:https://blog.csdn.net/qq_22929803/article/details/50724662

将以下代码添加到配置文件中即可,比如网站路径:root /home/wwwroot/www;下面

if ($scheme = http ) {
    return 301 https://$host$request_uri;
}
    add_header Strict-Transport-Security max-age=15768000;

modoer nginx伪静态,目录结尾形式

if ($uri !~ "^.*(.css|.js|.gif|.png|.jpg|.jpeg|.ico)$"){
set $rule_0 1$rule_0;
}
if (!-f $request_filename){
set $rule_0 2$rule_0;
}
if (!-d $request_filename){
set $rule_0 3$rule_0;
}
if ($rule_0 = "321"){
rewrite ^/(.*)$ /index.php?Pathinfo=$1 last;
}

modoer nginx伪静态,.html结尾形式

rewrite
 /(mylist|article|card|comment|coupon|exchange|fenlei|item|link|member|index|party|product|space|tuan|ucenter|ask|pay|graph|ask|review|shop|group|weixin|mobile).html$ /index.php?Rewrite=$1.html last;
    rewrite ^/(mylist|article|card|comment|coupon|exchange|fenlei|item|link|member|index|party|product|space|tuan|ucenter|ask|pay|graph|ask|review|shop|group|weixin|mobile)-(.+).html$ /index.php?Rewrite=$1-$2.html last;

nginx 301重定向

if ($http_host ~ "^abc.com"){
set $rule_0 1$rule_0;
}
if ($rule_0 = "1"){
rewrite ^/(.*)$ http://www.abc.com/$1 permanent;
}//把aa.com301重定向到www.abc.com

Apache 301跳转

RewriteCond %{HTTP_HOST} ^abc.com
RewriteRule ^(.*)$ http://www.abc.com/$1 [R=permanent,L]

Apache 伪静态

RewriteEngine On
# / 标识Modoer是在主域名或者二级域名下面,如果你的访问地址是www.modoer.com/modoer,就需要把 / 改成 /modoer
RewriteBase /
# 这是HTML伪静态部分
RewriteRule ^(mylist|article|card|comment|coupon|exchange|fenlei|item|link|member|index|party|product|space|tuan    |ucenter|ask|pay|graph|ask|review|shop|group|weixin|mobile)\.html$ index.php?Rewrite=$1.html [L]
    RewriteRule ^(mylist|article|card|comment|coupon|exchange|fenlei|item|link|member|index|party|product|space|tuan|ucenter|ask|pay|graph|ask|review|shop|group|weixin|mobile)\-(.+)\.html$ index.php?Rewrite=$1-$2.html [L,NC]

这是目录形式的URL改写

RewriteCond %{REQUEST_URI} !^.*(\.css|\.js|\.gif|\.png|\.jpg|\.jpeg|\.ico)$
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?Pathinfo=$1 [L]

apache转nginx伪静态工具:
http://www.anilcetin.com/convert-apache-htaccess-to-nginx/