Log for tracking traffic

P

patrickkasie

Verified User
Joined
Sep 21, 2021
Messages
241
Location
Een echte Hollander
Dear DirectAdmin forums,

I'm receiving the following load messages multiple times per day:

Code: 
This is an automated message notifying you that the 5 minute load average on your system is 62.24.
This has exceeded the 10 threshold.

One Minute      - 189.14
Five Minutes    - 62.24
Fifteen Minutes - 23.43

top - 09:34:15 up 7 days, 20:13,  0 users,  load average: 186.09, 63.71, 24.11
Tasks: 678 total,  69 running, 607 sleeping,   0 stopped,   2 zombie
%Cpu(s): 66.6 us, 32.9 sy,  0.0 ni,  0.0 id,  0.1 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem :  8008420 total,   121504 free,  7403880 used,   483036 buff/cache
KiB Swap:  1048572 total,        0 free,  1048572 used.   258500 avail Mem

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
1196 mysql     20   0 2869960   1.1g    984 S  39.4 15.0   1556:29 /usr/sbin/mysqld
11827 user1 20   0  506208  53652   8348 R  20.2  0.7   0:02.37 /usr/sbin/httpd -DFOREGROUND
12817 user2 20   0  354620  26880   3924 R  11.7  0.3   0:00.48 php-fpm: pool user2
12724 user3  20   0  351188  24420   2944 R  10.6  0.3   0:00.39 php-fpm: pool user3
12380 user4 20   0  419444  44068   3656 R   6.4  0.6   0:00.57 /usr/sbin/httpd -DFOREGROUND
12515 user5  20   0  413548  39688   3444 R   6.4  0.5   0:00.63 /usr/sbin/httpd -DFOREGROUND
12121 user5  20   0  412968  39772   3092 R   5.3  0.5   0:00.40 /usr/sbin/httpd -DFOREGROUND
12235 user4 20   0  421356  45984   2992 R   5.3  0.6   0:00.59 /usr/sbin/httpd -DFOREGROUND
12331 user2 20   0  353844  24800   2524 R   5.3  0.3   0:00.83 php-fpm: pool user2
12352 user2 20   0  369088  40048   2680 R   5.3  0.5   0:00.83 php-fpm: pool user2
12492 user6 20   0  345016  17020   2260 D   5.3  0.2   0:00.74 php-fpm: pool user6
12531 user2 20   0  350236  21352   2496 R   5.3  0.3   0:00.77 php-fpm: pool user2
12544 user2 20   0  354356  25056   2556 R   5.3  0.3   0:00.79 php-fpm: pool user2
12673 user7 20   0  421236  44516   3368 D   5.3  0.6   0:00.48 /usr/sbin/httpd -DFOREGROUND
12677 user8 20   0  340940  13040   3168 R   5.3  0.2   0:00.17 php-fpm: pool user8
12713 user9 20   0  347064  17836   2752 D   5.3  0.2   0:00.40 php-fpm: pool user9
  45 root      20   0       0      0      0 R   4.3  0.0 110:26.57 [kswapd0]
11675 user10 20   0  411076  37676   2616 R   4.3  0.5   0:00.59 /usr/sbin/httpd -DFOREGROUND
11800 user6 20   0  347240  17596   2220 R   4.3  0.2   0:01.02 php-fpm: pool user6
11930 apache 20   0  417204  43612   4064 S   4.3  0.5   0:00.87 /usr/sbin/httpd -DFOREGROUND
12077 user11 20   0  429928  54948   2964 R   4.3  0.7   0:00.83 /usr/sbin/httpd -DFOREGROUND
12175 user12 20   0  415200  40060   3504 R   4.3  0.5   0:00.64 /usr/sbin/httpd -DFOREGROUND
12190 user13 20   0  366644  38268   2200 S   4.3  0.5   0:00.77 php-fpm: pool user13

Connection counts:
     1 114.119.152.164
     1 138.199.36.210
     1 17.121.114.63
     1 178.85.80.125
     1 185.186.161.252
     1 185.30.177.162
     1 185.30.179.42
     1 194.110.203.45
     1 2001
     1 221.13.140.88
     1 44.203.181.79
     1 51.222.253.16
     1 54.36.148.125
     1 54.36.148.215
     1 66.249.70.47
     1 83.138.199.120
     1 89.200.205.34
     2 103.168.214.218
     2 37.97.206.65
     2 62.83.207.46
     2 94.213.40.124
     3 37.162.31.165
     3 62.194.40.235
     3 92.66.139.169
     4 126.227.228.23
     4 5.88.238.63
     5 212.121.96.163
    15 127.0.0.1
   252 13.82.63.184
   271 188.166.201.130

IP '188.166.201.130' currently has '270' connections

Connection info for '188.166.201.130':
tcp        0      0 OurIPv4:36348   188.166.201.130:443     TIME_WAIT
This IP address cannot be blocked though, as we use an API that stopped working for the duration of me blocking this IP address. So I want to know which pages are visited or how the load comes to be in the first place. Grepping both IP addresses did not return anything in any of the logs in the log viewer, but 13.82.63.184 did show up everywhere in /var/log/httpd/domains, so it's a crawler. But we don't want to have any crawler just randomly coming up to our server and be like "Here, this specific API call, let me just do that like 270 times". So we want to know how often that API gets called, where it's being called from (upon which page visit) and what the destination IP address is.
We tested tcpdump using the following command on our cellphones on mobile data, with results that were not helpful in any regard:
tcpdump -i eth0 -n dst host 37.19.221.163
 
Last edited:
We've figured that the crawler isn't just a crawler but is also trying to find exploits on the server. I think our solution would be to block any traffic that follows "GET /wp-login.php" or "POST /wp-plain.php" or whatever method. How do we create a firewall rule that, upon visiting this URL on any website on the VPS, would block said IP address?
 
I guess you've stopped eye-scanning of the page far before you could see the section #2
The newer system works in tandem with the previous, and will scan the logs for the other services (Apache, Dovecot, Exim, ProFTPd, SSHd).
Click to expand...

This another section from the same page related to your request: https://docs.directadmin.com/direct...el.html#brute-force-monitoring-for-xmlrpc-php

Another page for the same: https://docs.directadmin.com/directadmin/general-usage/securing-with-bfm.html

p.s. Come on, reading is rather useful :)
 
zEitEr said:
I guess you've stopped eye-scanning of the page far before you could see the section #2
The newer system works in tandem with the previous, and will scan the logs for the other services (Apache, Dovecot, Exim, ProFTPd, SSHd).
Click to expand...

I am not sure how to interpret this, I don't know how to read it. It does not say what to do with the web page visits/requests.

The /usr/local/directadmin/data/templates/brute_filter.list contains options:

wordpress3=ip_after=&ip_until= -&text=] "POST /&text2=/wp-login.php&text3=" 302%20&count_multiplier=4

What this does, is after all log parsing is done, the Brute Force Monitor will count how many entries were triggered for that item (e.g., wordpress3) for that given IP.
Click to expand...

Then I will need to know first what web page is visited in order to create a new filter myself. It is useful! So this is step 2, but I need to know step 1 first.
 
patrickkasie said:
It does not say what to do with the web page visits/requests.
Click to expand...

Since it's all under BFM, the only possible action is to block offending IP, unless you use "only alert" method.

patrickkasie said:
Then I will need to know first what web page is visited in order to create a new filter myself.
Click to expand...

That depends on a pattern they use. Do they always hit "/wp-login.php" before doing another 1 request? Or they might hit "/wp-login.php" and then visit 2+ pages on the same domain? How many times do they hit "/wp-login.php"? Probably it's sufficient to count only hits to "/wp-login.php"? If this is the case, the BFM can do it without custom filters.
 
We don't know. I was hoping you could tell me. All I'm seeing is the IP address in that list in my first message popping up all the time. I fixed this by changing the CT_LIMIT to 100 in CSF, and now that problem is no longer that specific IP address. It's a different IP address now, on a different website. It's now happening on the user wof, we just don't know what domain or page

Edit: the IP address does not generate any items in our logs, including /wp*
 
I can say what is going on on your server and who is attacking (if anybody is at all) only if I connect to your server and read logs.

If you want to reduce impact of brute-force attacks on your server, follow the guide from post above. You might check my settings and script here: https://github.com/poralix/directadmin-bfm-csf By running an installer you will get CSF/LFD installed and BFM enabled. That's enough for a beginning in most cases.
 
zEitEr said:
I can say what is going on on your server and who is attacking (if anybody is at all) only if I connect to your server and read logs.
Click to expand...
I will have to discuss this with my employer.
zEitEr said:
If you want to reduce impact of brute-force attacks on your server, follow the guide from post above. You might check my settings and script here: https://github.com/poralix/directadmin-bfm-csf By running an installer you will get CSF/LFD installed and BFM enabled. That's enough for a beginning in most cases.
Click to expand...
CSF/LFD has been installed and BFM (for logins and the aforementioned filters) is also active.
 
patrickkasie said:
CSF/LFD has been installed and BFM (for logins and the aforementioned filters) is also active.
Click to expand...

OK, so you have the measures active. And you will still see attacks from various IPs. That's expected. The tools work, and they do their work after they detect attacks. They won't block IPs until they attack your server.

patrickkasie said:
I will have to discuss this with my employer.
Click to expand...

p.s. Since I don't work for DirectAdmin, my services are commercial.
 
You must log in or register to reply here.
Back
Top