Steadily increasing CPU and Network load over the past year

[BBM]

I'm experiencing a steadily increasing CPU and Network-load over the past year and I think maybe even longer before.

My VPS has 2 cpu's but the load according to the VPS-hoster is on average around 90%, so 1 CPU is almost fully busy. I think that's way too much for the number of accounts (60) on the server.
A large user-account has been added late last year so that accounts for some increase, but I tend to think the average load has been increasing maybe due to more hack-attempts or less-efficient processes running at the moment.
I've been tuning the Maria-database a number of times with success I think, and some time before that I've also adjusted the apache-settings, to eliminate occassional out-of-memory-issues. Maybe I'll have to look into that again some more.

Basicly, I would like some pointers where to look for possible issues and how to decrease the load again, or make certain processes more efficient.

 

[scriptkitty]

Hi!

What do these commands show?

top
sar
sar -r
sar 3 20

Do you use Awstats or any site software that could help you analyze the traffic?

Do you use Wordfence Live Traffic of any resource-intensive processes like this? Do you actively use ClamScan? Over the day, when does it peak? During updates? During backups?

Do you have a lot of mysql processes running?

mysqladmin proc

Have you inquired your host about this and have they provided any insight into what uses the most resources on the server?

 

[ericc]

My VPS has 2 cpu's but the load according to the VPS-hoster is on average around 90%, so 1 CPU is almost fully busy. I think that's way too much for the number of accounts (60) on the server.

Wow, that is dense.

I also noticed large increase in queries recently (wp-logins, xmlrpc and just random strings) and bad bots steadily hammering sites / consuming resources / causing high loads.

 

[vlijmenlive]

Knowing TransIP at least you can be sure that the server is OK.

But with the info provided it's difficult to give you good feedback that's adjusted to your situation. Having 60 users doesn't really say anything, more important is what they host, how many visitors they have, do they use caching or not. The increase might just be explained by the fact that 1 users has a traffic increase. Even without a big increase on bandwith usage this could be one of the causes.

Could you provide us with more info, as Kitty mentioned, checking the output of 'top' could be a good starting point. Also check your services on memory usage. If you don't have enough RAM available your cpu load will also increase.

 

[BBM]

Thanks for the replies.
The top-command shows mysql is heavily used (as usually on the server), and thinking about it some more this could lead to the high load alone perhaps.

Pretty much all sites/accounts on the VPS are under my control and most of them are fairly 'stationary' WordPress-sites. Caching would help, but I wonder how much.
The large account recently added has a fairly active phpBB-forum.

In the beginning I've had some database connection errors but since I've 'tuned' the mysql-conf I think I found a good balance between the available memory (4GB) and the number of connections.
High load-warnings are sent when there's a spike in visits that sprawn apache-calls, I think (php-FPM).

I'm also worried a bit by the constant hammering of Wordpress-logins and also Exim, ssh and Proftpd.
The ssh default port is changed so I'm not to worried about that yet. I've shutdown proftpd temporary yesterday as a test.
There are some CSF custom regex rules present to try and limit the number of attacks on the services but I don't think they work all that properly at the moment.

Below are some screenshots of the results of the commands asked by Kitty;
(Showing mysql as the top-load, ranging from about 30% to 71% shown below)

 

[Zhenyapan]

Check "amerikaa" looks like his scritps causes overload, also install opcache
/usr/local/directadmin/custombuild/build update
/usr/local/directadmin/custombuild/php_extensions.conf set opcache=yes
/usr/local/directadmin/custombuild/build opcache

 

[BBM]

User "amerikaa" is a site with just very quiet phpBB-forum.
Although the forumsoftware is a bit outdated, I did notice there were only 2 users only and about 300 'guests' online and causing the load.
Tried to change and turn-off some forum-settings like Gzip-compression and some search functions but this caused an mysql-issue and severe slow down of the server.

I've now temporary suspended the account and guess what indeed, the server load is back to 'normal'.
Mysql is back to around 5-20% load.
Silly of me looking straight past something so obvious.

I will check the logs and see why that account was causing such a load.

 

[BBM]

Found out a searchbot AspiegelBot was hammering the site relentlessly.
Blocked its entire IP range and things look normal again. Rookie oversight by me.

Thanks all for the help and Zhenyapan for showing me 'the light'...