DA won't start after updating to v1.669

K

krisiskris

Verified User
Joined
Jan 2, 2019
Messages
29
So yesterday I updated DA on one of our servers from 1.668 to 1.669. Since then, we're not able to start DA at all.
Running da from the commandline results in:
Segmentation fault (core dumped)
Click to expand...
This also means any da related commands to try and rebuild or update, won't work either.
Looking into /var/log/directadmin/error.log doesn't reveal anything relevant.

We tried debugging the application, but I'm not sure what to make of the output:
(gdb) run server --debug=2000
Starting program: /usr/bin/da server --debug=2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__memchr_evex () at ../sysdeps/x86_64/multiarch/memchr-evex.S:324
324 VPCMP $0, (%rdi), %YMMMATCH, %k1
Click to expand...
Running a backtrack gives slightly more info, but I'm still uncertain what to make of it.
(gdb) bt
#0 __memchr_evex () at ../sysdeps/x86_64/multiarch/memchr-evex.S:324
#1 0x00007ffff72f312e in two_way_short_needle (needle_len=10, needle=0x7ffff7ff1411 "usage: ssh", haystack_len=<optimized out>, haystack=0x4002ed "ux-x86-64.so.2") at str-two-way.h:303
#2 __GI___memmem (needle_len=10, needle_start=<optimized out>, haystack_len=<optimized out>, haystack_start=<optimized out>) at memmem.c:73
#3 __GI___memmem (haystack_start=<optimized out>, haystack_len=<optimized out>, needle_start=0x7ffff7ff1411, needle_len=10) at memmem.c:43
#4 0x00007ffff7fedaef in ?? ()
#5 0x000000000000fd00 in ?? ()
#6 0x00007ffff7dd80e2 in _dl_map_segments (loader=<optimized out>, has_holes=<optimized out>, maplength=<optimized out>, nloadcmds=<optimized out>, loadcmds=<optimized out>, type=<optimized out>, header=0x7ffff7ffe238, fd=<optimized out>, l=0x0) at ./dl-map-segments.h:141
#7 _dl_map_object_from_fd (name=<optimized out>, origname=<optimized out>, fd=<optimized out>, fbp=0x7ffff7ffe230, realname=<optimized out>, loader=<optimized out>, l_type=<optimized out>, mode=<optimized out>, stack_endp=<optimized out>, nsid=<optimized out>) at dl-load.c:1181
#8 0x0000000000000000 in ?? ()
Click to expand...
Interestingly enough, all the sites on the server seem to be working fine. Also sending and receiving e-mails is working fine as far as I can see (tested with Roundcube to external addresses)

I ran this info through ChatGPT which is advising me to reinstall DA entirely, but I'm a bit reluctant to do that, as I don't want to lose all my configs and data.

Do you guys have any advice on that to do in this situation?

Edit: please note, there's loads of RAM available, It's not a memory related issue!
1729778183855.png

Thanks,
Kris
 
Last edited:
I tried opening a ticket but as we purchased this license via reseller TransIP, I cannot open a ticket directly. I'll try contact them, but I have a feeling they do not provide support for this type of case.
 
krisiskris said:
I tried opening a ticket but as we purchased this license via reseller TransIP, I cannot open a ticket directly. I'll try contact them, but I have a feeling they do not provide support for this type of case.
Click to expand...
Feel free to PM @fln or me directly, we’ll check it for you :)
 
[Fixed] So I managed to get TransIP to contact DA support and take a look for me. One of your experts logged into our server and reported this:
Hello,
Your server is infected:

[tempadmin@srv03 ~]$ ldd /usr/bin/da | grep httpd
/usr/sbin/libhttpd.so (0x00007f9758075000)
[tempadmin@srv03 ~]$

That file is should not be used/shown in "da" tool.

The next file is also not the default file from DA:

[tempadmin@srv03 ~]$ strings /usr/sbin/libhttpd.so | grep /
/usr/lib/apache/mod_mybchizz.so
[tempadmin@srv03 ~]$

and there are more processes/tools infected with that file on your server:

[tempadmin@srv03 ~]$ lsof /usr/sbin/libhttpd.so
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 3828686 tempadmin mem REG 253,0 11256 510897 /usr/sbin/libhttpd.so
bash 3828701 tempadmin mem REG 253,0 11256 510897 /usr/sbin/libhttpd.so
bash 3841993 tempadmin mem REG 253,0 11256 510897 /usr/sbin/libhttpd.so
lsof 3846969 tempadmin mem REG 253,0 11256 510897 /usr/sbin/libhttpd.so
lsof 3846970 tempadmin mem REG 253,0 11256 510897 /usr/sbin/libhttpd.so
[tempadmin@srv03 ~]$

You can try removing these 2 files:

/usr/sbin/libhttpd.so
/usr/lib/apache/mod_mybchizz.so

and then reboot your server to check if it helps with DA.

But we recommend to create new server with clean OS and migrate your users with sites to that new server.

Thank you.
Click to expand...
And then a second comment:
Also here is more bad files on your server:

[tempadmin@srv03 ~]$ cat /etc/ld.so.preload
/lib64/libs.so /usr/sbin/libhttpd.so
[tempadmin@srv03 ~]$
[tempadmin@srv03 ~]$ ls -la /lib64/libs.so
-rwsr-xr-x 1 root root 31048 May 22 2021 /lib64/libs.so
[tempadmin@srv03 ~]$
Click to expand...
So I went back into the server with SSH and checked his findings and also compared the results to another healthy server we have.
First I got rid of /usr/sbin/libhttpd.so and /usr/lib/apache/mod_mybchizz.so as these also doesn't exist on the healthy server. Then I rebooted the server but received a ton of errors about one of the removed files:
ERROR: ld.so: object '/usr/sbin/libhttpd.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
Click to expand...
So I opened the /etc/ld.so.preload file and removed the libhttpd.so dependency from it.

After this DA was online again, and I'm able to log in to the control panel. Checking the messages, I found this:

DirectAdmin has been updated to v1.669 with warnings​

23/10/2024, 13:39
This is an automated message notifying you that DirectAdmin has been successfully updated with warnings.
v1.668 (a506a8e) to v1.669 (522e3b0)

Warning message:
/usr/local/directadmin/scripts/update.sh:
/usr/local/directadmin/scripts/update.sh: line 46: 2793411 Segmentation fault (core dumped) ${DA_PATH}/directadmin permissions
/usr/local/directadmin/scripts/update.sh: line 111: 2793423 Segmentation fault (core dumped) ${DA_PATH}/directadmin build cron > /dev/null 2> /dev/null
/usr/local/directadmin/scripts/update.sh: line 111: 2793432 Segmentation fault (core dumped) ${DA_PATH}/directadmin build deprecation_check > /dev/null 2> /dev/null
/usr/local/directadmin/scripts/update.sh: line 119: 2793446 Segmentation fault (core dumped) ${DA_PATH}/directadmin build install

post-update (signal: segmentation fault (core dumped)):
Click to expand...

So, everything seems fine again, but I'm very uncertain as to how this happened. I highly doubt that a threat actor gained access to the server; we have SSH open on a non-default port that only has a couple of specific IP addresses whitelisted for access and logging in is only allowed via SSH-keys, no passwords.
 
Looks your server is compromised, backup your clients and setup/reinstall your server
 
I was in the loop when support staff was checking this issue. Usually forced library pre-loading is a common way to infect the system. So my initial reaction was that this should be malware. But if this is really malware, then it did a very poor job hiding itself. Maybe some very poorly written extension or some other software was trying to install itself in this way 🤷‍♂️. Do you remember installing any 3rd party software not from the distro repos?
 
No, nothing strange. Besides the default packages needed for DA to run, we've got Python and a couple of other small packages. But nothing out of the ordinary, and also all installed via yum.
This is, however, a server that was upgraded from CentOS 7 to Almalinux 8. Could that have anything to do with it?
 
You must log in or register to reply here.
Back
Top