Solved Websites often cannot be accessed suddenly.

fianbiasa · Mar 17, 2024

hi everyone,

I ask for help, I use a Dedicated Server.
Direct Admin Panel runs smoothly without any problems, very quickly accessed.

but, sometimes the website can't be accessed suddenly.
It's like the web server is dead, when I restart httpd all the websites can be accessed normally again.

The question is, what solution should I do?

ericosman · Mar 17, 2024

Well, some more info would be needed when you go to
Admin Tools --> Service monitor

Is everything up and running?

What error do you get when trying to visit your site?
Did you do something on the server? e.g. update or change?

Does your firewall block maybe? Maybe the firewall at the hosting company?

johannes · Mar 17, 2024

If it happens again, check from any webproxy with another IP, to rule out firewall or DA block.

fianbiasa · Mar 17, 2024

ericosman said:
Well, some more info would be needed when you go to
Admin Tools --> Service monitor

Is everything up and running?

What error do you get when trying to visit your site?
Did you do something on the server? e.g. update or change?

Does your firewall block maybe? Maybe the firewall at the hosting company?

everything is up and running.

ericosman · Mar 17, 2024

Can you send me a domain in pm?

fianbiasa · Mar 17, 2024

johannes said:
If it happens again, check from any webproxy with another IP, to rule out firewall or DA block.

it's down from my friend PC and my computer.

error like "taking too long respond"

fianbiasa · Mar 17, 2024

ericosman said:
Can you send me a domain in pm?

Currently all normal websites can be accessed, it's just that sometimes the error takes too long to respond.

ericosman · Mar 17, 2024

What are the specs of your server?
Do you run your own dns?
Maybe your server gets attacked sometimes?

fianbiasa · Mar 17, 2024

ericosman said:
What are the specs of your server?
Do you run your own dns?
Maybe your server gets attacked sometimes?

This is my Server Specs

If the server gets attacked, I don't think so.
because the DirectAdmin page is normal and very fast to access.

Richard G · Mar 17, 2024

Do you have swap space enabled? If yes, how much?
Also, when it happens, directly afterwards, check your logfiles to see what could be the cause.
Like in RHEL alike systems the /var/log/messages and /var/log/php-fpmXX.log files where the XX stands for the php-fpm version you're running.
And maybe /var/log/httpd/acces_log and error_log to see what is going on.

Could be many things, also you and your friends ISP or something else. Would be a good thing to use some monitoring software. There are some free options with Betteruptime for example. This way it's monitored from different country's if a server is responding to ping or not for example.

fianbiasa · Mar 17, 2024

Richard G said:
Do you have swap space enabled? If yes, how much?
Also, when it happens, directly afterwards, check your logfiles to see what could be the cause.
Like in RHEL alike systems the /var/log/messages and /var/log/php-fpmXX.log files where the XX stands for the php-fpm version you're running.
And maybe /var/log/httpd/acces_log and error_log to see what is going on.

Could be many things, also you and your friends ISP or something else. Would be a good thing to use some monitoring software. There are some free options with Betteruptime for example. This way it's monitored from different country's if a server is responding to ping or not for example.

In /var/log/messages Too many code like this :

Code:

Mar 18 03:40:41 server kernel: [43526.099044] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=162.243.138.58 DST=37.27.101.118 LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=54321 PROTO=TCP SPT=54817 DPT=512 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 03:40:46 server kernel: [43531.429917] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=37.44.238.81 DST=37.27.101.118 LEN=40 TOS=0x00 PREC=0x00 TTL=246 ID=54321 PROTO=TCP SPT=43451 DPT=8728 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 03:41:12 server kernel: [43556.749762] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=107.170.234.29 DST=37.27.101.118 LEN=40 TOS=0x00 PREC=0x00 TTL=239 ID=54321 PROTO=TCP SPT=51122 DPT=6667 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 03:41:18 server kernel: [43562.660537] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=45.137.201.9 DST=37.27.101.118 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=27668 PROTO=TCP SPT=45117 DPT=28075 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 18 03:41:22 server kernel: [43566.774519] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=162.216.149.37 DST=37.27.101.118 LEN=44 TOS=0x00 PREC=0x00 TTL=60 ID=54321 PROTO=TCP SPT=57340 DPT=224 WINDOW=65535 RES=0x00 SYN URGP=0
Mar 18 03:41:41 server kernel: [43586.267466] Firewall: *TCP_IN Blocked* IN=enp5s0 OUT= MAC=08:bf:b8:18:23:26:80:db:17:80:99:db:08:00 SRC=45.137.201.9 DST=37.27.101.118 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=904 PROTO=TCP SPT=45117 DPT=15277 WINDOW=1024 RES=0x00 SYN URGP=0

in /var/log/php-fpm74.log too many code like this :

Code:

[18-Mar-2024 03:16:34] NOTICE: [pool umarriau] child 228254 started
[18-Mar-2024 03:16:35] NOTICE: [pool umarriau] child 227179 exited with code 0 after 648.880532 seconds from start
[18-Mar-2024 03:16:35] NOTICE: [pool umarriau] child 228257 started
[18-Mar-2024 03:16:45] NOTICE: [pool kosmdn] child 227469 exited with code 0 after 548.229572 seconds from start
[18-Mar-2024 03:16:45] NOTICE: [pool kosmdn] child 228260 started
[18-Mar-2024 03:20:26] NOTICE: [pool kosmdn] child 227593 exited with code 0 after 673.203683 seconds from start
[18-Mar-2024 03:20:26] NOTICE: [pool kosmdn] child 228409 started
[18-Mar-2024 03:20:27] NOTICE: [pool kosmdn] child 227590 exited with code 0 after 676.531239 seconds from start
[18-Mar-2024 03:20:27] NOTICE: [pool kosmdn] child 228412 started

and currently, the website cannot be accessed again.
there is an error 502 bad gateway.

now when I restart the httpd service,

Code:

service httpd restart

the website returns to normal.

please help.

Richard G · Mar 17, 2024

fianbiasa said:
In /var/log/messages Too many code like this :

The log files don't show any strange things.
The exit 0 on the php-fpm is normal.
So these don't show odd things and the server is Hetzner, which is a good datacenter.

Does the /var/log/httpd/access_log or error_log give any insight?

Is it plain httpd? No nGinx or proxy stuff or something?

fianbiasa · Mar 17, 2024

Richard G said:
The log files don't show any strange things.
The exit 0 on the php-fpm is normal.
So these don't show odd things and the server is Hetzner, which is a good datacenter.

Does the /var/log/httpd/access_log or error_log give any insight?

Is it plain httpd? No nGinx or proxy stuff or something?

some of /var/log/httpd/access_log like this :

Code:

89.40.227.126 - - [18/Mar/2024:04:04:55 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Linux; Android 10; SM-N960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Mobile Safari/537.36"
185.160.182.220 - - [18/Mar/2024:04:04:55 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"
103.190.242.4 - - [18/Mar/2024:04:05:00 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 5.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
103.190.242.4 - - [18/Mar/2024:04:05:04 +0700] "POST /xmlrpc.php HTTP/1.0" 404 3231 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0"
46.242.246.220 - - [18/Mar/2024:04:05:09 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Linux; Android 10; SM-G960U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Mobile Safari/537.36"
2a0b:7280:200:0:4a9:5cff:fe00:d7b - - [18/Mar/2024:04:05:17 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Mobile Safari/537.36"
195.231.52.213 - - [18/Mar/2024:04:05:19 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (iPad; CPU OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/91.0.4472.80 Mobile/15E148 Safari/604.1"
198.54.114.12 - - [18/Mar/2024:04:05:25 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
136.243.163.181 - - [18/Mar/2024:04:06:08 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0"

because httpd often turns itself off, I used nginx_apache reverse proxy : https://docs.directadmin.com/webservices/nginx_apache/general.html

and are these tips useful?
- https://serverfault.com/questions/7...he-if-it-stops-responding-and-socket-is-still

Richard G · Mar 17, 2024

fianbiasa said:
"POST /xmlrpc.php

Oke these are attacks and those can cause some servers to get a very high load. Which might cause the timeout.
If you restart httpd, then those connections are gone too, but can get back again.

The tips you're pointing to are probably not usefull as they are checking timeout connections, while your log is logging 404 errors.

I'll be honest I don't know about these proxys and nGinx, I don't use that. But you can do something about the xmlrpc.php attacks.
Have a look at this page:

How to block access to xmlrpc.php serverwide on Directadmin?

What is WordPress XML-RPC and how to stop an attack with Directadmin serverwide?

help.poralix.com

There is also a regex option you could try in csf firewall to block these, but the access blocking from the above link is already very helpfull.

fianbiasa · Mar 18, 2024

Richard G said:
I'll be honest I don't know about these proxys and nGinx, I don't use that. But you can do something about the xmlrpc.php attacks.
Have a look at this page:

How to block access to xmlrpc.php serverwide on Directadmin?

What is WordPress XML-RPC and how to stop an attack with Directadmin serverwide?

help.poralix.com

There is also a regex option you could try in csf firewall to block these, but the access blocking from the above link is already very helpfull.

oke... thank you so much... i'll try...

can you share tips about regex option in csf firewall ?

fianbiasa · Mar 18, 2024

Richard G said:
.

Is this okay?

Code:

PHP has been secured.
Restarting php-fpm74.
Restarting php-fpm72.
Restarting php-fpm80.
Restarting php-fpm82.
2024/03/18 13:47:53  info executing task            task=action=rewrite&value=nginx
md3: write failed, user block limit reached.
Restarting nginx.
2024/03/18 13:48:08  info executing task            task=action=directadmin&value=reload

Richard G · Mar 18, 2024

fianbiasa said:
Is this okay?

No. Mostly this is because some user's quota failed, user is over quota. However I don't know how to fix as it looks as if this is root or admin.
Let's call in some help. @Zhenyapan or @jamgames2 any clue on where he should look in this case?

fianbiasa said:
can you share tips about regex option in csf firewall ?

Can't guarantee if this works, but I found this one somewhere couple of years ago.

Code:

# XMLRPC attack
if (($globlogs{CUSTOM1_LOG}{$lgfile}) and ($line =~ /^(\S+).*POST.*(wp-login\.php|xmlrpc\.php).* (200|401)/)) {
    return ("Forbidden",$1,"WPXMLRPCATTACK","7","80,443","604800");
    }

You can change the "Forbidden" and "WPXMLRPCATTACK" to whatever you want, also the blocktime of 604800 can be changed.

You also see here the CUSTOM1_LOG file. You have to adjust that one in the /etc/csf.conf file. I have it like this:

Code:

CUSTOM1_LOG = "/var/log/httpd/domains/*"

after the change, restart csf and lfd.

Zhenyapan · Mar 18, 2024

md3: write failed, user block limit reached. - same on my servers with cloudlinux where we have global limit (default user) with limited I/O speed.
but better to check Inode/quota limits for users too.

fianbiasa · Mar 18, 2024

It looks like someone is trying to infiltrate, in the logs I see this activity.

Code:

45.79.223.93 - - [18/Mar/2024:21:32:29 +0700] "GET /LahG HTTP/1.0" 404 1015 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"
173.230.128.57 - - [18/Mar/2024:21:32:30 +0700] "GET /ZAwJ HTTP/1.0" 404 481 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"
173.230.128.57 - - [18/Mar/2024:21:32:30 +0700] "GET /rp5F HTTP/1.0" 404 481 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"
45.79.223.93 - - [18/Mar/2024:21:32:30 +0700] "GET /Pml4 HTTP/1.0" 404 1015 "-" "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)"

fianbiasa · Mar 18, 2024

Richard G said:
Oke these are attacks and those can cause some servers to get a very high load. Which might cause the timeout.
If you restart httpd, then those connections are gone too, but can get back again.

But you can do something about the xmlrpc.php attacks.
Have a look at this page:

How to block access to xmlrpc.php serverwide on Directadmin?

What is WordPress XML-RPC and how to stop an attack with Directadmin serverwide?

help.poralix.com

There is also a regex option you could try in csf firewall to block these, but the access blocking from the above link is already very helpfull.

I have tried this method, but the logs still appear in my DirectAdmin.

Code:

37.148.212.176 - - [18/Mar/2024:21:17:13 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"
87.248.153.52 - - [18/Mar/2024:21:17:14 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0"
2600:3c00::f03c:93ff:fe00:77c0 - - [18/Mar/2024:21:17:18 +0700] "POST /xmlrpc.php HTTP/1.0" 404 3231 "-" "Mozilla/5.0 (Linux; Android 10; SM-A205U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Mobile Safari/537.36"
192.180.81.33 - - [18/Mar/2024:21:17:19 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
80.71.144.37 - - [18/Mar/2024:21:17:21 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0"
102.164.38.82 - - [18/Mar/2024:21:17:23 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 5.1; rv:30.0) Gecko/20100101 Firefox/30.0"
137.135.71.44 - - [18/Mar/2024:21:17:27 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"
157.230.108.140 - - [18/Mar/2024:21:17:29 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0"
45.156.187.48 - - [18/Mar/2024:21:17:29 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
5.101.157.166 - - [18/Mar/2024:21:17:30 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
185.107.112.243 - - [18/Mar/2024:21:17:37 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0"
23.105.208.201 - - [18/Mar/2024:21:17:42 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
80.71.144.37 - - [18/Mar/2024:21:17:46 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
80.71.144.37 - - [18/Mar/2024:21:17:50 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0"
109.199.112.156 - - [18/Mar/2024:21:17:55 +0700] "POST /xmlrpc.php HTTP/1.0" 404 814 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36

Solved Websites often cannot be accessed suddenly.

Verified User

Verified User

Verified User

Verified User

Attachments

Verified User

Verified User

Verified User

Verified User

Verified User

Attachments

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Verified User

Attachments

Verified User

Verified User

Verified User

Verified User