Server freezes

[provotector]

Hello. I have a fresh install of Ubuntu 22.04 on a VPS from SSD nodes. Only Direct Admin is installed. A few Wordpress sites with official addons.

The server freezes from time to time. No SSH access and all websites down. No hints of problems in the logs. No hints of problems on kdump. No hints of problems in PRTG system monitor (I'm monitoring CPU, HDD, RAM, etc. with SNMP).

Initially I only payed a DA license without support. Because this incident, I purchased a license with support.

The support told me that they don't provide support for the operating system. At this point I think the problem can be found:

a) In the fresh install of Ubuntu.
b) In Directadmin.

I don't think an LTS version of Ubuntu "out of the box" is going to have "factory" problems and I don't think it's the operating system that causes the system to crash.

I'm sure that the problem lies in Direct Admin. But the support team is no offering other response different to "the problem is in the operanting system and we don't provide support for the operating system".

What can I do? Thanks.

 

[ericosman]

Hi,

What are the specs of your server?

 

[provotector]

Hello Ericosman. Thanks for replying. The specs are:

4 vCPUs
Memory 16 GB
Disk 320 GB SSD

 

[Richard G]

Which datacenter? Like Contabo or where?
Normally on a freeze of the system there is always something in the system log.
So if nowhere anything is to be seen, one would think maybe network issue on the datacenter side?

If the system "crashes", do you need to reboot or is it coming back up by itself?
And if it comes back itself, then when it's down, can you ping it or do you get timeouts on SSH connections and the sites, because that would be pointing to network issues again.

 

[floyd]

What about sar?

 

[provotector]

Which datacenter? Like Contabo or where?
Normally on a freeze of the system there is always something in the system log.
So if nowhere anything is to be seen, one would think maybe network issue on the datacenter side?

If the system "crashes", do you need to reboot or is it coming back up by itself?
And if it comes back itself, then when it's down, can you ping it or do you get timeouts on SSH connections and the sites, because that would be pointing to network issues again.

Thanks for your reply. The datacenter location is Frankfurt.

I need to reboot via VPS panel. No response on pings.

I think that is most plausible to be a Directadmin missconfiguration the root of the problems than a network issue on a datacenter.

 

[provotector]

What about sar?

What is sar?? Thanks.

 

[Richard G]

SAR is a monitoring program.
Frankfurt, with Hetzner maybe? Maybe they can have a look.

I don't say it's impossible, but I never have seen DA doing something like that, because that would mean DA bringing down the NIC or all services and the network card always responds to a ping.

You should be able to see that in the system log. If it's a network issue, the logfile would be running until you reset the VPS.
If it would be a DA or OS issue, then a crashwould mean that for as long as the VPS is offline, there is not a single log entry anywhere and time would skip from crash time until reboot time.

Maybe try the free agent of hetrixtools to investigate if the vps is still running but just unreachable or create something via cron wich puts in a log entry every minute for the time being. If that log stops, then it's system crash. If it keeps running, it's network issue.

 

[provotector]

SAR is a monitoring program.
Frankfurt, with Hetzner maybe? Maybe they can have a look.

I don't say it's impossible, but I never have seen DA doing something like that, because that would mean DA bringing down the NIC or all services and the network card always responds to a ping.

You should be able to see that in the system log. If it's a network issue, the logfile would be running until you reset the VPS.
If it would be a DA or OS issue, then a crashwould mean that for as long as the VPS is offline, there is not a single log entry anywhere and time would skip from crash time until reboot time.

Maybe try the free agent of hetrixtools to investigate if the vps is still running but just unreachable or create something via cron wich puts in a log entry every minute for the time being. If that log stops, then it's system crash. If it keeps running, it's network issue.

I used PRTG, another monitor program.

There's no response when I ping the server under freezement status.

In the logs appear a "time jump" from freezing starts to manual reset of VPS. No network issues previously to the jump. The problem is on the OS or in DA just as you say.

The VPS is still running but unreachable, just as you say. Is a system crash.

 

[Richard G]

Indeed a system crash. Problem is that I never ever seen DA crashing a compleet system without a single trace in the logs, either system or DA logs.
So if I'm correct the system hangs itself instantly at a certain point.
I have seen that happening on pc's when memory refresh was broken in the RAM and as soon as the RAM got full, the system hang it self. That was a long time ago. And this is a VPS so that would mean a node issue, but in that case you would not be the only one.
I have no clue on whic DA setting could cause something like that.

Maybe the VPS provider can see something in the logs somewhere, or maybe somebody else has an idea.
Unless you want to try backing up the sites, installing the server newly, maybe with Alma or Debian just to try out and see if the problem persists.

@Zhenyapan ever encountered something similar?

 

[Richard G]

P.s. you do have swap space right? How much?

 

[provotector]

Indeed a system crash. Problem is that I never ever seen DA crashing a compleet system without a single trace in the logs, either system or DA logs.
So if I'm correct the system hangs itself instantly at a certain point.
I have seen that happening on pc's when memory refresh was broken in the RAM and as soon as the RAM got full, the system hang it self. That was a long time ago. And this is a VPS so that would mean a node issue, but in that case you would not be the only one.
I have no clue on whic DA setting could cause something like that.

Maybe the VPS provider can see something in the logs somewhere, or maybe somebody else has an idea.
Unless you want to try backing up the sites, installing the server newly, maybe with Alma or Debian just to try out and see if the problem persists.

@Zhenyapan ever encountered something similar?

"I never ever seen DA crashing a compleet system" I invite you to Search for "freezes" in this forum:

Centos server freeze

I've a server running centos 7.7 and directadmin. In some cases the server freezes . I tried to run dmesg | tail after the issue occurred , I see this message : dmesg | tail [ 469.120963] [] rest_init+0x77/0x80 [ 469.121044] [] start_kernel+0x450/0x471 [ 469.121126] [] ...

forum.directadmin.com

Block Freezing

Hi! Since the update of this week of Blockcracking I already have 7 server in the CPD with custombuild2.0 (spamassassin+blockcracking+easyspamfighter+pigeonhole) that all the local mails freezes them. And it displays the following error: T = spamcheck: Child process of spamcheck carried...

forum.directadmin.com

CentOS DirectAdmin freezes from on arbitrary moments

Hi. Somehow my DA installation sometimes freezes. The server is not busy at all. There are a few small sites on it. We work with two network adapters. On one adapter we allow http / ftp and on the other one we allow ssh (only reachable via our VPN). When the sites get non-responsive, I still...

forum.directadmin.com

etc...

This demonstrates 2 things.

1. I'm not the only one this happens to
2. Directadmin crashes like any other software. Is not "crashproof"

I don't understand why you find it easier for a hardware or network failure on datacenter to occur rather than a configuration failure on Directadmin.

P.D. The server don't uses swap. Haves an SSD and 16GB of RAM.

 

[ericosman]

He has his server with the company “SSD nodes”
And creating (at least) 4GB swap is important.

 

[BillyS]

Hi @provotector none of those forum posts have anything to do with DA crashing the system. One is related to CloudLinux, one has to do with with mail freezing (exim config problem) and the last has to do with a bad network interface. As others have suggested the lack of swap will lead to system instability.

The reason people suggesting looking elsewhere is because if DA had the kind of problem you're describing, there would be more people posting about it. If you're not familiar with setting up a server, you should look here. The recommendation is to have at least 2G of swap:

CloudLinux OS | Directadmin Docs

DirectAdmin Knowledge Base

docs.directadmin.com

 

[floyd]

The recommendation is to have at least 2G of swap:

I only use 512 MB of swap because really by that point the server is already slowing down because of lack of memory and high swap usage. What is really needed in that case is more ram not more swap. I have increased swap on occasion as a stop gap measure until I could get more ram installed.

 

[provotector]

Hi @provotector none of those forum posts have anything to do with DA crashing the system. One is related to CloudLinux, one has to do with with mail freezing (exim config problem) and the last has to do with a bad network interface. As others have suggested the lack of swap will lead to system instability.

The reason people suggesting looking elsewhere is because if DA had the kind of problem you're describing, there would be more people posting about it. If you're not familiar with setting up a server, you should look here. The recommendation is to have at least 2G of swap:

CloudLinux OS | Directadmin Docs

DirectAdmin Knowledge Base

docs.directadmin.com

Thanks for your reply Billys. If there is something on the server that consumes the large amount of 16 GB of RAM... How is it possible that adding just 4 GB of swap is going to solve the problem? just as floyd says, the ideal solution is to increase the amount of physical ram. The question now is how much?

I only use 512 MB of swap because really by that point the server is already slowing down because of lack of memory and high swap usage. What is really needed in that case is more ram not more swap. I have increased swap on occasion as a stop gap measure until I could get more ram installed.

Thanks for replying floyd. I'm agreed with you.

The hardest task currently required of the server is backups. So we are going to assume that the problems come from there.

How much RAM/swap can I require for crash-free operation if I am hosting 14 websites, all Wordpress or Prestashop?

It should be possible to limit the use of RAM during the backup process, even if this makes the process slower. Different backup tasks could be scheduled, each with a few websites. This behavior could be handled by the DA itself.

We really don't know how much RAM the server may need to avoid crashing. It could perfectly be 50GB. Although according to my calculations, if 14 websites take down a server with 16GB of RAM, more than 1GB of RAM is needed for each website.

Thank you! Nice thread!

 

[Richard G]

I invite you to Search for "freezes" in this forum:

In invite you to show me that these were caused by DA. Next to that, they are not complete system freezes. One most likely a hardware error, the second one only mail freeze en the third one cause by the OS as far as I could see.
However, I still never have seen it happen, I didn't say it can't happen.

I don't understand why you find it easier for a hardware or network failure on datacenter to occur rather than a configuration failure on Directadmin.

For the simple reason that this is the most likely cause.
And to answer your statement.
1.) I didn't say it was impossible, but the ones you pointed to there were none completely crashing without anything in the log
2.) I never said DA was crashproof. I only stated that I've never seen DA crashing a whole server like you are experiencing now. Ofcourse DA can crash, but I invite you to try and find something caused by DA where you didn't even could ping the server and only a reboot could fix it.
Again, not saying it's impossible, but in the time I used DA I've never seen it happen that way.

Back to the issue. Start using swap. I have 64 GB RAM and even have swap. Odd things can happen so I would advise a minimum of 4 GB swap.

It's not a lot of sites indeed, and 16 GB should be enough.
There is an ionice setting which you can use for creating backups, that might already be of help.

It's bit odd that you only want to discuss with me, but take the same advise from others... swap is an OS fix, not DA fix.

 

[BillyS]

Thanks for your reply Billys. If there is something on the server that consumes the large amount of 16 GB of RAM... How is it possible that adding just 4 GB of swap is going to solve the problem?

Swap isn't just a slow replacement for memory, it plays an important role in memory reclamation.

In defence of swap: common misconceptions

tl;dr: Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve.

chrisdown.name

How much RAM you need to host 14 sites is anyone's guess because it depends... First place to look is probably the database side and the corresponding configuration and how busy the sites are in terms of visitors. I see people tweaking database settings all the time that cause memory to grow needlessly.

 

[ericosman]

Correct me if i'm wrong, but there is also the possibility that the Node you run on is .... not the best / get's oversold too much.
Resulting in systems becoming slow, not work optimal etc.

Been there done that at Contabo...
Thats the big downside about running a VPS.