nginx reverse proxy http2 causes problem with DA

d.dobrev · Mar 13, 2021

Here's what I observed:

- My websites traffic is passed through nginx reverse proxy outside of the DA server.
- When http2 is enabled in the reverse proxy, Safari on mac and iphone stops opening the websites with an error "cannot parse response" (NSURLErrorDomain:-1017)
- As soon as I disable http2 in the reverse proxy - the sites start working normally.

This behavior only occurs on sites hosted on DA, if I host on a "normal" apache somewhere else using http2 in the reverse proxy - all is OK, meaning - this is something related to the way DA works (what exactly - not sure), not a bug in nginx.

Hopefully someone knows.

Regards,
D

smtalk · Mar 13, 2021

Please make sure you have no custom templates in /usr/local/directadmin/data/templates/custom.

d.dobrev · Mar 13, 2021

I don't have any files/folders in "/usr/local/directadmin/data/templates/custom/"

Zhenyapan · Mar 13, 2021

compare all request paths - on DA server, and nonDA server. Maybe something with cert's, or ports, or syntax for http2 activation.

d.dobrev · Mar 13, 2021

How to compare request paths when Safari on iOS and MacOS simply stops requesting and spits an error? I'm unable to get any requests previously made in order to check request/response headers and what not.

The cert on the DA server is self-signed, which doesn't affect anything because the nginx proxy doesn't check for certificate validity.
The cert on the "frontend" server (reverse proxy) is generated by Let's Encrypt and works w/o any problems.

Something on DA's side screws things up when http2 is enabled on the reverse proxy.
It cannot be reverse proxy (nginx) fault because http2 works fine if I "host" the same site on plain apache with the same self-signed certificate on port 443 - it works fine. It doesn't on DA.

d.dobrev · Mar 13, 2021

Can someone try to reproduce by installing the latest nginx version (I think it's 1.19.8) with a valid cert (or let's encrypt generated) and placing it in front of a DA-hosted site with self-signed certificate (backend) in reverse-proxy mode ?

d.dobrev · Mar 13, 2021

I can also provide a hosted domain if anyone wants to test..

Zhenyapan · Mar 13, 2021

I mean you need to check virtualhost configs at any step, at frontend, at backend, proxypath, ports, headers in this configs etc. all chain that request must follow from 443 port at reverse proxy - till docroot at backend. also http2 must be configured at backend

d.dobrev · Mar 14, 2021

Why would http2 need to be configured on the backend, when nginx reverse proxy doesn't support http2 connections to backends ?

smtalk · Mar 14, 2021

Check version of DirectAdmin (to make sure it's latest), and also "Custom HTTPd Configuration" for that specific domain.

d.dobrev · Mar 15, 2021

This happens on *all* domains hosted on the DA server, none of them have any custom httpd configurations.
The DA version is 1.61.5

When I access the site with http2 enabled on the reverse proxy, the following occurs in the logs:
- reverse proxy logs - 3 GET / requests with HTTP 200 response code
- DA domain logs - 3 GET / requests with HTTP 200 response code

The site doesn't render on Safari (iOS / macOS).

When I disable http2 on the reverse proxy - the site starts working normally.

d.dobrev · Mar 15, 2021

Here is the reverse proxy config:

NGINX:

server {
    listen REVPROXY-FRONTEND-IP:443 ssl http2;

    server_name domain.com;
    server_name *.domain.com;

    access_log /var/log/domain.com_HTTPS.log full;

    client_max_body_size     128M;
    client_body_buffer_size  128k;
    keepalive_timeout        60;
    client_body_timeout      600;
    client_header_timeout    600;
    send_timeout             600;
    proxy_connect_timeout    60s;
    proxy_read_timeout       60s;

    ssl_certificate      /usr/local/etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key  /usr/local/etc/letsencrypt/live/domain.com/privkey.pem;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AE128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:EDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-CM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;
    ssl_dhparam /usr/local/openssl/certs/dhparams.pem;

    location / {
        proxy_pass         https://DA-BACKEND-IP:443;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   CF-Connecting-IP $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto https;
        proxy_set_header   Host $host;
        proxy_pass_header  Set-Cookie;
        proxy_pass_header  P3P;
        proxy_redirect off;
    }
    location /.well-known {
        root /usr/local/etc/nginx/lecr/domain.com/;
    }

    location ~ /\.ht {
        deny all;
    }
}

d.dobrev · Mar 16, 2021

Anyone able to reproduce ?

smtalk · Mar 16, 2021

Why do you proxy 443 to 443? Check:

Code:

/usr/local/directadmin/directadmin c | grep ^port_

d.dobrev · Mar 17, 2021

# /usr/local/directadmin/directadmin c | grep ^port_
port_80=80
port_443=443
port_8080=8080
port_8081=8081

smtalk · Mar 17, 2021

Strange, because this line should be different:

Code:

proxy_pass         https://DA-BACKEND-IP:443;

Are you sure it's :443? If yes, try rewrite_confs or check if that file doesn't have "chattr +i" set (using lsattr).

d.dobrev · Mar 19, 2021

Yes, I am sure it is proxy_pass https://DA-BACKEND-IP:443;
To what file are you referring to?

# ./build rewrite_confs
Checking to ensure /etc/httpd/conf/ssl.crt/server.ca is set.
Using XXX,XXX,XXX,XXX for your server IP
Installing Comodo Rule Set for ModSecurity...
Updating to latest CWAF client version
current version is up to date
update process finished!
Defaulting to Comodo WAF SecDefaultAction...
Installation of ModSecurity Rule Set has been finished.
PHP has been secured.
Restarting php-fpm73.
Restarting php-fpm72.
Restarting php-fpm71.
Defaulting to Comodo WAF SecDefaultAction...
Restarting apache.

-- Still the same problem

d.dobrev · Mar 22, 2021

So.. anyone able to reproduce?

d.dobrev · Mar 29, 2021

Can someone provide me with *default* DA configs (da/options/apache/etc/etc), so I can compare against the configs on my server and possibly find out what is causing the problem?

kheush · Mar 30, 2021

d.dobrev said:

Here is the reverse proxy config:

NGINX:

server {
    listen REVPROXY-FRONTEND-IP:443 ssl http2;

    server_name domain.com;
    server_name *.domain.com;

    access_log /var/log/domain.com_HTTPS.log full;

    client_max_body_size     128M;
    client_body_buffer_size  128k;
    keepalive_timeout        60;
    client_body_timeout      600;
    client_header_timeout    600;
    send_timeout             600;
    proxy_connect_timeout    60s;
    proxy_read_timeout       60s;

    ssl_certificate      /usr/local/etc/letsencrypt/live/domain.com/fullchain.pem;
    ssl_certificate_key  /usr/local/etc/letsencrypt/live/domain.com/privkey.pem;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AE128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:EDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-CM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;
    ssl_dhparam /usr/local/openssl/certs/dhparams.pem;

    location / {
        proxy_pass         https://DA-BACKEND-IP:443;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   CF-Connecting-IP $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto https;
        proxy_set_header   Host $host;
        proxy_pass_header  Set-Cookie;
        proxy_pass_header  P3P;
        proxy_redirect off;
    }
    location /.well-known {
        root /usr/local/etc/nginx/lecr/domain.com/;
    }

    location ~ /\.ht {
        deny all;
    }
}

Hi,
You could try and add proxy_hide_header Upgrade; to your nginx config. Worked for me on a different hosting provider.
See this post for more details about this issue: https://trac.nginx.org/nginx/ticket/915
Hope this helps someone.

nginx reverse proxy http2 causes problem with DA

Verified User

Administrator

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Administrator

Verified User

Verified User

Verified User

Administrator

Verified User

Administrator

Verified User

Verified User

Verified User

New member