HOWTO: Using Nginx - best practices!

paix

Verified User
Joined
Oct 31, 2006
Messages
55
Location
ua

This instruction describes how to use nginx as reverse proxy for apache, also I'll show some tips how use nginx for heavy loaded sites.
Majority of my servers are freebsd boxes, so this howto oriented on freebsd, but I'll show some tips for linux too.


Nginx is a free, open-source, high-performance HTTP server and reverse proxy.
It extremely fast HTTP server, really! It can serve many thousand concurrent connections!

http://nginx.net/
http://wiki.codemongers.com/


Using nginx as reverse proxy for apache.
==========================================

For understanding how this works please see:
http://blog.kovyrin.net/2006/05/18/nginx-as-reverse-proxy/lang/en

Using reverse proxy(nginx) behind big web server(apache) is classic two-layer web server architecture ( frontend + backend web servers)

Most advantages is that small frontend can wait very long time while client will receive his content and will close connection - backend server will not consume resources for such long time.

In advanced configurations nginx runs behind your big web server and handle all requests to static content and to pass all dynamic requests to primary web-server. With this solution your big server will spawn additional threads/processes only for dynamic pages and it will return answers to small frontend very fast and then can free resources to use them to handle another queries.

Lets go.
NOTE: Any changes in apache configuration are not required !!
If your backend scripts are using user IP addresses for some purposes, you will need to install mod_rpaf (apache2) or mod_realip(apache1) module to use X-Real-IP header provided by nginx.

1) installing nginx:
Code:
freebsd # make install clean -C /usr/ports/www/nginx

optinos:
Code:
freebsd # less /var/db/ports/nginx/options
# This file is auto-generated by 'make config'.
# No user-servicable parts inside!
# Options for nginx-0.6.31
_OPTIONS_READ=nginx-0.6.31
WITHOUT_DEBUG=true
WITHOUT_GOOGLE_PERFTOOLS=true
WITH_HTTP_MODULE=true
WITHOUT_HTTP_ADDITION_MODULE=true
WITH_HTTP_DAV_MODULE=true
WITHOUT_HTTP_FLV_MODULE=true
WITH_HTTP_PERL_MODULE=true
WITH_HTTP_REALIP_MODULE=true
WITH_HTTP_REWRITE_MODULE=true
WITH_HTTP_SSL_MODULE=true
WITH_HTTP_STATUS_MODULE=true
WITHOUT_HTTP_SUB_MODULE=true
WITHOUT_MAIL_MODULE=true
WITHOUT_MAIL_IMAP_MODULE=true
WITHOUT_MAIL_POP3_MODULE=true
WITHOUT_MAIL_SMTP_MODULE=true
WITHOUT_MAIL_SSL_MODULE=true
WITH_WWW=true

for linux you can install nginx using packet manager
# apt-get install nginx # or get nginx sources and build package
# yum install nginx

also you can compile nginx from sources as described here:
http://directadmin.com/forum/showthread.php?t=20133

2) configuring nginx
In freebsd nginx configs are installed into /usr/local/etc/nginx/
Here is my simplified nginx.conf but I strongly reccomended also see default config.

Code:
# /usr/local/etc/nginx/nginx.conf

user  www apache;
worker_processes  2;   # how many processors or how many hard disks you have

error_log  /var/log/nginx/error.log;

events {
    worker_connections  8192;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                      '"$status" $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    tcp_nopush     on;
    keepalive_timeout  75 20;
    gzip  on;

    server_names_hash_bucket_size 64;
    reset_timedout_connection on;

     client_max_body_size 100m;

#------------------
#fullproxy for all
    server {
        listen       _SERVERIP_:81 default rcvbuf=8192 sndbuf=16384 backlog=32000 accept_filter=httpready;
        server_name  _DEFAULTDOMAIN_  _ ;     # "_" is for handle all hosts that are not described by server_name
        charset off;

        access_log     off;
        #access_log /var/log/nginx/access.log  main;

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_pass              http://_SERVERIP_;    # apache here

                client_max_body_size       16m;
                client_body_buffer_size    128k;

                proxy_buffering     off;
                #proxy_buffering     on;  

                proxy_connect_timeout      90;
                proxy_send_timeout         90;
                proxy_read_timeout         120;
               #proxy_buffer_size          8k;
               #proxy_buffers              32 32k;
               #proxy_busy_buffers_size    64k;
               #proxy_temp_file_write_size 64k;

                error_page              502 503 /usr/local/www/nginx-dist/50x.html;
        }

        location /nginx_status {
            stub_status on;
            access_log   off;
            allow _STUFFIP_;
            allow _SERVERIP_;
            allow 127.0.0.1;
            deny all;
    }

    }

  #---------------------------
  # Heavy loaded virtualhosts
  #---------------------------
        include /usr/local/etc/nginx/vhosts/*.conf;     # this is optional
}

some comments:
proxy_buffering is very efficient, but I have some troubles, when clients requested lage files (600mb for example), apache tried sent them to nginx as fast as he can, and nginx tried receive this files from apache....and this caused some overload. So I turn this feature off. But anyway try end test it!

nginx_status is feature like apache httpd_status, optional.

Attention! You should replase _SERVERIP_ _STUFFIP_ _DEFAULTDOMAIN_ to you own.

Also you should create /var/log/nginx and /usr/local/etc/nginx/vhosts if you dont have this dirs.
Also add ' nginx_enable="YES" ' to rc.conf.

Every time when we change nginx configuration run `nginx -t` to test config:
Code:
#nginx -t
2008/08/13 10:15:40 [info] 75959#0: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
2008/08/13 10:15:40 [info] 75959#0: the configuration file /usr/local/etc/nginx/nginx.conf was tested successfully

All ok, now you could start nginx:
Code:
# /usr/local/etc/rc.d/nginx start

After this nginx should listen 81 port

Code:
freebsd # sockstat -4l | grep 81
linux # netstat -ntpl |grep 81

Now, you even cat to test nginx, connecting to
http://your_sample_domain:81
and should see you sample_domain :) .

Fine!

3) Logrotating:
#FreeBSD:

Code:
# newsyslog.conf
/var/log/nginx/*.log            root:wheel      640  7     *    @T00    GJ        /var/run/nginx.pid 30

# linux:
Code:
#vi /etc/logrotate.d/nginx
/usr/local/nginx/logs/*.log {
daily
missingok
rotate 9
compress
delaycompress
notifempty
postrotate
kill -USR1 `cat /usr/local/nginx/logs/nginx.pid`
endscript
}

4) configuring apache
As I said early you should have
mod_rpaf (for apache2) or mod_realip(for apache1).

Code:
freebsd # make install clean -C /usr/ports/www/mod_realip
or  freebsd # make install clean -C /usr/ports/www/mod_rpaf2
of course if you have apache installed through ports.

Else, follow this instructions:
mod_rpaf: http://stderr.net/apache/rpaf/

mod_realip:
download module:
wget http://sysoev.ru/mod_realip/mod_realip-2.0.tar.gz
unpack and compile it:
# apxs -i -a -c path_to/mod_realip.c
then copy builded mod_realip.so to dir where are your other apache modules.

Then you need add this to httpd.conf:
Code:
<IfModule mod_realip.c>
        RealIP localhost xfwd
        RealIP _SERVERIP_ xfwd
</IfModule>
where _SERVERIP_ is you own!

That all! just restart apache.

5) Putting nginx and apache working together

After you have successfully tested some domains on http://exampledomain:81 it is time to make nginx handling request from users.

I'm using pf firewall on freebsd, so simple redirect here:
Code:
# requests via ngnix
rdr on $ext_if proto tcp from any to $me port 80 -> $me port 81
where $me is macros to my _SERVERIP_ and $ext_if - is network interface (em0 in my case)

Now all request coming to server are redirected to nginx to port 81, than nginx handle it and pass to backend server apache.

If you have linux, use iptables forwarding:
Code:
iptables -t nat -A PREROUTING -p tcp -s ! _SERVERIP_ --dport 80 -j REDIRECT --to-ports 81

if you will need to turn off nginx for some reasone, you could simply delete this iptables rule, and all requests will come to apache.


That is all :) Enjoy.

Here some my freebsd graphics from mrtg after starting nginx.
Loadaverage:
loadavg-week.png

Memory:
mem-month.png

6) Optional 1: improvements of handling requests for heavy loaded domains.
For example you have some big domains that have many static files and uniq hosts.
So you want to put nginx serving requests to static files on this domain directly.
Create config for you domain and put it into /usr/local/etc/nginx/vhosts:

Code:
# less /usr/local/etc/nginx/vhosts/heavyloadeddomain.conf
#
 server {
        listen     _SERVERIP_:81;
        server_name  _HEAVYLOADEDDOMAIN_ www._HEAVYLOADEDDOMAIN_;
        charset off;
        access_log  /var/log/nginx/_HEAVYLOADEDDOMAIN_.access.log  main;

        # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }

        # proxy the PHP scripts to Apache
        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_pass              http://217.20.175.140;
                proxy_buffering         off;
                error_page              502 503 /usr/local/www/nginx-dist/50x.html;
        }

        #static files
        location ~* ^.+.(nrg|htm|html|jpg|jpeg|gif|png|ico|css|zip|7z|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|avi|mp3|mp4|mpg|iso|djvu|dmg|flac|r70|mdf|chm|sisx|sis|flv|thm|bin)$ {

           root /home/_USER_/domains/_HEAVYLOADEDDOMAIN_/public_html/;
        }
    }

test nginx: nginx -t
and reload them.

Now nginx serving static files for this domain directly!

7) Optional 2: improvements of handling requests for heavy loaded domains.
If you have very very busy domain, you may want completely turn off apache.
In this case you should proxy all php request to php-fastcgi server.
It's described in this howto: http://directadmin.com/forum/showthread.php?t=20133
where php-fastcgi server statrs using lighttpd spawn manager.

I recomended php-fpm as fasctcgi server. Today it is most advansed PHP fastcgi process manager. Please see http://php-fpm.anight.org/
for freebsd you could download port from my site: http://paix.org.ua/sk/php5_fpm_526.tar.gz
small instruction here: http://paix.org.ua/node/11 (sorry, but only on russian)

So, in case of such setup you should have config like this:
# less /usr/local/etc/nginx/vhosts/heaveloadeddomain2.conf
Code:
   server {
        listen       _SERVERIP_:81;
        server_name  ${domainame} www.${domainame};

        access_log  /var/log/nginx/${domainame}.access.log  main;
        location / {
            root   /home/vhosts/${domainame}/public_html;
            index  index.html index.php;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/local/www/nginx-dist;
        }
        location ~ \.php$ {
            #fastcgi_pass   127.0.0.1:1026;      # connect to php-fastcgi sever via tcp port
            fastcgi_pass unix:/tmp/php-fpm/${domainame}.sock;  # connect to php-fastcgi sever via unix socket. I preffer this way.

            root  /home/vhosts/${domainame}/public_html;
            fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            
            fastcgi_index  index.php;
            include        fastcgi_params;
        }
}

Now you have very strong setup! Enjoy! :)
 
bandwidth

This is very good how to.

If we made some web sites work with nginx for different contens, will DA calculate correct bandwidths for this web sites?
 
This is very good how to.

If we made some web sites work with nginx for different contens, will DA calculate correct bandwidths for this web sites?

As far as I know, DA uses the logs of apache to do calculation.

In case of configuration as reverse proxy for many domains all traffic will be counted, because all files still pass through apache (Nginx just serving files for slow clients, etc.)

Unfortunately, I don't know clearly how DA counts traffic for different services, and how I can tell DA to calculate traffic from non-apache logs.

If you want make a custom setup for some busy domains to put nginx serving static files directly ( nginx do this over and over effectively than apache) you should also configure access logs for this domains (this described in my howto, plz see Optional 1 and Optional 2 sections). Some log parsers such as Webalizer, analog etc, may be configured to show statistics from nginx logs for certain domains.
But in this case DA dosn't count traffic for these domains (at least while you don't make some hack to tell DA count traffic from nginx logs).
For example it may be symlink for access log on this domain to apache log dirs...etc.


PS. I'm using nginx on different projects, not only on hosting. In majority for heavy loaded domains with many thousand of uniq visitors. I'm very happy of using nginx, it's wonderful, so I just wanted to show some possibilites of using nginx on DA hosting boxes.
But please keep in mind that nginx is not panacea and you should use it if you understand what you can win and what you can lose. After all, I think, once you will started to using nginx you never back again :)


Thank you for reply!
 
As i see in graph , when u start using nginx your server load go up ! doen't ?
 
ehsanch, no. graph show that LA drop after installing nginx.
Please read graph from right-to-left. Do you see red triangle ? it show you direction of graph
 
i install nginx with this config , but when i visit sampledomain:81 i see default apache page for that ip!
 
Last edited:
Hi,

one question. Should nginx not run on port 80 and forward all to apache (port 81)? you have written, that apache runs on port 80 and forward to nginx.

can i run nginx -> apache too? because i want to make some special things to block ddos, and there is nginx a good option.
i have tried to run nginx on port 80 and apache on port 81, it was no problem. but when i call a site (ip/~username) there stands
Not Found

The requested URL /user was not found on this server.

Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
Apache/2 Server at xx Port 80
the forwarding is working, but why i cant reach sites?

i hope you understand me, sorry for my bad english.
 
Hi,

one question. Should nginx not run on port 80 and forward all to apache (port 81)? you have written, that apache runs on port 80 and forward to nginx.

can i run nginx -> apache too? because i want to make some special things to block ddos, and there is nginx a good option.
i have tried to run nginx on port 80 and apache on port 81, it was no problem. but when i call a site (ip/~username) there stands

Main idea is all request should come to nginx, then (if needed) to apache.

In my howto nginx is listen port 81, and apache listen port 80
All incoming requests are forwarded via firewall to port 81.

Code:
#freebsd
rdr on $ext_if proto tcp from any to $me port 80 -> $me port 81

# linux
iptables -t nat -A PREROUTING -p tcp -s ! _SERVERIP_ --dport 80 -j REDIRECT --to-ports 81


In case such setup no changes in Control Panel (Directadmin for example) are needed.

Another way is put nginx to listen port 80 and reconfigure apache (and all virtualhosts) to listen some other port. In this case you shouldn't use firewall forwarding.
 
Quest. about iptables

I have 2 IPs: IP1 (server ip) & IP2 (addon ip)
I need:
IP1:80->IP2:81 (nginx)->Apache
IP2:80 (Apache, no port redirect)

In your exmpl:
# linux
iptables -t nat -A PREROUTING -p tcp -s ! _IP1_ --dport 80 -j REDIRECT --to-ports 81

IP2:80 redirects to IP1:81 too. What can I do to get only IP1:80 redirection?
 
[...]
What can I do to get only IP1:80 redirection?
Code:
iptables -t nat -A PREROUTING -p tcp -s ! _IP1_ -d _IP1_ --dport 80 -j REDIRECT --to-ports 81
This will catch any packet, originating from any address different then IP1, with destination IP1, to port 80, redirecting it to port 81.
 
Hi, in case if you have apache listening on multiple IP's you could use next setup:

firewall redirects:
Code:
IP1:80 -> IP1:81 (nginx here)
IP2:80 -> IP2:81 (nginx here)

nginx.conf (all additional settings are skipped):
Code:
# IP1
    server {
        listen       _SERVER_IP1_:81;
        server_name  _SERVER_IP1_  _ ;     
        
        location / {
                proxy_pass    http://_SERVER_IP1_;  # apache here on IP1:80
        }
    }

# IP2
    server {
        listen       _SERVER_IP2_:81;
        server_name  _SERVER_IP2_  _ ;     
        
        location / {
                proxy_pass    http://_SERVER_IP2_; # apache here on IP2:80
        }
    }

Please note "server_name _" is for handle all hosts that are not described by server_name.
 
I would like to use NGINX with something like CARP for load balancing and failover. The questions I would have are:
- How well does it maintain sessions?
- How well does it work with SSL?
- Does anyone have any configs for using this with CARP for failover?

My thoughts are to using something like this with FreeBSD using this type of architecture: http://siag.nu/pen/vrrpd-linux2.shtml
 
Last edited:
I would like to use NGINX with something like CARP for load balancing and failover. The questions I would have are:
- How well does it maintain sessions?
- How well does it work with SSL?
- Does anyone have any configs for using this with CARP for failover?

My thoughts are to using something like this with FreeBSD using this type of architecture: http://siag.nu/pen/vrrpd-linux2.shtml

Hi,
first, I recommend you ask nginx-related info in the nginx maillist ( nginx at sysoev.ru ).

You could save sessions in the memcache server http://en.wikipedia.org/wiki/Memcached
or save sessions in the Database server. This application-level job.
SSL works fine under nginx.

Also you could look at the http://highscalability.com for review of system architecture for HA projects.

CARP is good for failover the firewalls (i.e. core gates for your network).
If you want to create a HA web cluster, look at http://en.wikipedia.org/wiki/High-availability_cluster

First thing you should keep in mind is synchronizing the content between servers. More expensive solution is shared disk over network http://en.wikipedia.org/wiki/Storage_area_network
Common cheap (software) solution is using DRBD http://en.wikipedia.org/wiki/DRBD

Please google for "heartbeat + drbd" for building HA cluster. In this case you should have at least 2 PC. But this solution (drbd) works only on linux.
 
Last edited:
CARP is good for failover the firewalls (i.e. core gates for your network).
I'm thnking more like pf + pfsync + carp on the server. Then all you do is add servers.

Couple questions:
1. How does this affect .htaccess?
2. How does this affect mod_rewrite?

Thanks!
 
Couple questions:
1. How does this affect .htaccess?
2. How does this affect mod_rewrite?

nginx doesn't support .htaccess because this model has big overhead. Htaccess slows down request processing.

nginx has it's own mod_rewrite. There are a big amount of different rewrites (wordpress, drupal, etc) for nginx.
But you really need mod_rewrite when you will use simple nginx + php-fastcgi backend.
This scheme works on really heavy loaded sites.

For shared hosting (or systems with multidomains) you should rather prefer nginx as reverse proxy + apache backend.
 
I wonder how this would work the other way around. Have mod_proxy push off the image processing to Nginx. That way people could maintain the functionality of Apache for their web apps, and off-load the images to Nginx.
 
Last edited:
I wonder how this would work the other way around. Have mod_proxy push off the image processing to Nginx. That way people could maintain the functionality of Apache for their web apps, and off-load the images to Nginx.

Proxying only dynamic web apps to apache is a really good idea. But you will get a good effect (performance gain) from this if you have a heavy loaded domain with a lot of static and hosts\visitors.

So for this domains you should use a custom scheme, that described in the section
6) Optional 1 of howto.

I.e. you should divide the static content and the dynamic webapps if you really need this.
 
Proxying only dynamic web apps to apache is a really good idea.
No, I mean proxying only static pages to nginx would be a good idea. If NGINX is to be useful with DirectAdmin, it must be able to efficiently offload Apache's static content without reducing Apache's functionality and PHP performance and reliability. PHP high performance and reliability comes from CLI. FastCGI fills in for a lack of CLI on these servers and only works acceptably for sites that are light on dynamic content and heavy on static content. NGINX and similar servers are efficient for static content not because they are technological marvels, but because they lack 90% of Apache's functionality. That's similar to removing the body from a car to get better performance and fuel efficiency.

Unless NGINX can efficiently offload Apache's static content without reducing Apache's functionality, there can be no net gain, only a trade-off that very few DirectAdmin customers that can benefit from. A net gain requires a gain in performance and efficiency with no loss in functionality. If NGINX can deliver that, that's perfect! If not, time would be better spent looking into something like Apache MPM Event.
 
Back
Top