Brute Force Monitor stopped working

C

Christophe1

Verified User
Joined
Aug 24, 2008
Messages
49
Location
Belgium
Hi all

On a recently installed, pretty standard and up-to-date directadmin server, the brute force monitor suddenly stopped working after accidentally installing fail2ban (misunderstanding with colleague).
As if the logs are no longer been read. Meanwhile, fail2ban is removed again.

The instructions on http://help.directadmin.com/item.php?id=380 were followed and originally all this worked perfect.
Manually blocking an IP from the Brute Force Monitor that was detected earlier, still works.

The dataskq is still running.

When deleting /usr/local/directadmin/data/admin/brute.conf (after backup), no new file is created.
When deleting /usr/local/directadmin/data/admin/brute.conf.lock (after backup), a new file is created.

According to directadmin.conf, the brute force monitor should be active.

Any help would be particularly appreciated...
 
FYI: /usr/local/directadmin/dataskq d200 gives:

Debug mode. Level 200
pidfile written
staring queue
Segmentation fault

Installed OS: Debian 7.8 64-bit
DirectAdmin compiled on Debian 7.0 64-bit

Any suggestion?
 
Hmm.. if it's a segfault, we'd want to run it through gdb and debug mode to find out what's going on:
Code: 
cd /usr/local/directadmin
gdb dataskq
and once in gdb, run:
Code: 
run d2000
and when it segfaults, run this to get the output
Code: 
bt full
and paste us the relevant info.

John
 
Thank you!

Starting program: /usr/local/directadmin/dataskq d2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Debug mode. Level 2000

root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Error reading ./data/admin/brute.conf: Unable to open ./data/admin/brute.conf for reading.<br>
No such file or directory<br>
. Note, it won't exist by default, so this error may be meaningless

Login Fail Match: Jan 24 06:25:24 da sshd[27980]: Failed password for invalid user root from 221.203.142.71 port 32840 ssh2
(I removed all failed logins between the first and the last)
Login Fail Match: Jan 24 06:33:22 da sshd[2080]: Failed password for invalid user radio from 222.82.212.75 port 32898 ssh2

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()


(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007ffff5f0a9ec in __pthread_initialize_minimal_internal ()
from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007ffff5f0a209 in _init () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x00007ffff74e6000 in ?? ()
No symbol table info available.
#4 0x00000000007eb4f9 in call_init ()
No symbol table info available.
#5 0x00000000007eb6bc in _dl_init ()
No symbol table info available.
#6 0x00000000007c890e in dl_open_worker ()
No symbol table info available.
#7 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
#8 0x00000000007c80dd in _dl_open ()
No symbol table info available.
#9 0x00000000007cab85 in do_dlopen ()
No symbol table info available.
#10 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
Click to expand...
 
Thanks for the output. Those are all system library calls. Can you "type <return> to continue" so we get more output?
I'd be looking for which DA call is made, which calls the system functions.

It could also be an issue relating to static vs dynamic binaries.
I could try compiling you the opposite of what you currently have, eg:
Code: 
ldd dataskq
and I'll add or remove the static compile flag based on your output.
We'll see if we need to go that far (as it's not a great solution, as it's not update-safe, in that you'd revert to the previous style of binaries)
Hopefully it's just a simple code fix :)

John
 
By comparing some files in /usr/local/directadmin/data/admin between 2 servers, I noted that some ownerships were different.

And therefore I typed:

chown diradmin:diradmin brute.conf
chown diradmin:diradmin brute_user.data
chown diradmin:diradmin brute_ip.data
chown diradmin:diradmin brute_skip.list
Click to expand...

Now I didn't get the "type <return>..."-message, but:

Starting program: /usr/local/directadmin/dataskq d2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Debug mode. Level 2000

root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Login Fail Match: Jan 24 06:48:58 da sshd[13833]: Failed password for invalid user root from 183.3.202.108 port 32875 ssh2
Login Fail Match: Jan 24 06:51:03 da sshd[15538]: Failed password for invalid user alwin from 85.69.155.61 port 37286 ssh2

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007ffff5f0a9ec in __pthread_initialize_minimal_internal () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007ffff5f0a209 in _init () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x00007ffff74e6000 in ?? ()
No symbol table info available.
#4 0x00000000007eb4f9 in call_init ()
No symbol table info available.
#5 0x00000000007eb6bc in _dl_init ()
No symbol table info available.
#6 0x00000000007c890e in dl_open_worker ()
No symbol table info available.
#7 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
#8 0x00000000007c80dd in _dl_open ()
No symbol table info available.
#9 0x00000000007cab85 in do_dlopen ()
No symbol table info available.
#10 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
#11 0x00000000007ca957 in dlerror_run ()
No symbol table info available.
#12 0x00000000007cab29 in __libc_dlopen_mode ()
No symbol table info available.
#13 0x00000000007b6a92 in __nss_lookup_function ()
No symbol table info available.
#14 0x00000000007b725b in __nss_next2 ()
No symbol table info available.
#15 0x00000000007b4a6e in getspnam_r ()
No symbol table info available.
#16 0x00000000007b489e in getspnam ()
No symbol table info available.
#17 0x0000000000523bdc in Tally::system_acount_is_suspended(Config*, char const*) ()
No symbol table info available.
#18 0x0000000000523e76 in Tally::user_is_suspended(Config*, char const*, char const*) ()
No symbol table info available.
#19 0x0000000000524396 in Tally::scan_log(Config*, char const*, int, ConfigFile&, ConfigFile&, ConfigFile&, ConfigFile&, ConfigFile&, int&) ()
No symbol table info available.
#20 0x0000000000526174 in Tally::check_brute_force_logs(Config*) ()
No symbol table info available.
#21 0x00000000005b90b6 in doQueue() ()
No symbol table info available.
#22 0x00000000005b96ea in main ()
No symbol table info available.
(gdb)
Click to expand...
 
Hmm.. that seems more like a library issue.
Looking at the code, DA does make a call to getspnam.
IF the alwin account doesn't exist, getspnam should have returned NULL, but it didn't... as I don't see
Code: 
Tally::system_account_is_suspended(alwin): unable to get user_info
in the debug output before the segfault.
This means, the process carried on in an attempt to read the returned struct pointer which is probably no good due to a faulty return.

Two options:

1) First, you could use this workaround, which doesn't solve the segfault, but prevents DA from nearing that point:
http://www.directadmin.com/features.php?id=1611

so set:
Code: 
brute_force_ignore_attempts_on_suspended=0
2) OR create a ticket at https://tickets.directadmin.com, and we can compile you a custom set of binaries, opposite to the binaries you have.. which is dependent on the ldd output mentioned in my previous reply.

John
 
The brute force monitor works again after setting:

brute_force_ignore_attempts_on_suspended=0
Click to expand...


But there is no user alwin and no users are suspended...
Problem caused by manually creating and deleting users?
 
Correct. DA does a lookup for this Username.
It doesn't exist, so the function should return NULL, but it's not returning at all, hence the library issue.
 
I'm already very pleased that there is a workaround and the brute force monitor is working again.
But is it an option to help trying to find the cause of the problem? A real update-safe solution would be great.

I removed "brute_force_ignore_attempts_on_suspended=0" from directadmin.conf and in /usr/local/directadmin/data/admin I deleted:
brute.conf
brute_ip.data
brute_log_entries.list
brute_user.data

Then, I tried to run the dataskq again. This time with segfault:

root@da:/usr/local/directadmin# gdb dataskq
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/directadmin/dataskq...done.
(gdb) run d2000
Starting program: /usr/local/directadmin/dataskq d2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Debug mode. Level 2000

root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Login Fail Match: 2016-01-26 16:03:27 login authenticator failed for (User) [209.236.124.188]: 535 Incorrect authentication data (set_id=laptop)
exim1: Unable to find ip_after from
Login Fail Match: 2016-01-26 16:03:27 login authenticator failed for (User) [209.236.124.188]: 535 Incorrect authentication data (set_id=laptop)

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb)



(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00007ffff5f0a9ec in __pthread_initialize_minimal_internal () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007ffff5f0a209 in _init () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#3 0x00007ffff74e6000 in ?? ()
No symbol table info available.
#4 0x00000000007eb4f9 in call_init ()
No symbol table info available.
#5 0x00000000007eb6bc in _dl_init ()
No symbol table info available.
#6 0x00000000007c890e in dl_open_worker ()
No symbol table info available.
#7 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
#8 0x00000000007c80dd in _dl_open ()
No symbol table info available.
#9 0x00000000007cab85 in do_dlopen ()
No symbol table info available.
#10 0x00000000007c6a76 in _dl_catch_error ()
No symbol table info available.
#11 0x00000000007ca957 in dlerror_run ()
No symbol table info available.
#12 0x00000000007cab29 in __libc_dlopen_mode ()
No symbol table info available.
#13 0x00000000007b6a92 in __nss_lookup_function ()
No symbol table info available.
#14 0x00000000007b725b in __nss_next2 ()
No symbol table info available.
#15 0x00000000007b4a6e in getspnam_r ()
No symbol table info available.
#16 0x00000000007b489e in getspnam ()
No symbol table info available.
#17 0x0000000000523bdc in Tally::system_acount_is_suspended(Config*, char const*) ()
No symbol table info available.
#18 0x0000000000523def in Tally::user_is_suspended(Config*, char const*, char const*) ()
No symbol table info available.
#19 0x0000000000524396 in Tally::scan_log(Config*, char const*, int, ConfigFile&, ConfigFile&, ConfigFile&, ConfigFile&, ConfigFile&, int&) ()
No symbol table info available.
#20 0x00000000005261c5 in Tally::check_brute_force_logs(Config*) ()
No symbol table info available.
#21 0x00000000005b90b6 in doQueue() ()
No symbol table info available.
#22 0x00000000005b96ea in main ()
No symbol table info available.
(gdb)
Click to expand...

Added "brute_force_ignore_attempts_on_suspended=0" to directadmin.conf again and ran the dataskq.
Brute force monitor is working again and I got the following:

(gdb) run d2000
Starting program: /usr/local/directadmin/dataskq d2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Debug mode. Level 2000

root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Login Fail Match: 2016-01-26 16:03:27 login authenticator failed for (User) [209.236.124.188]: 535 Incorrect authentication data (set_id=laptop)
exim1: Unable to find ip_after from
Login Fail Match: 2016-01-26 16:03:27 login authenticator failed for (User) [209.236.124.188]: 535 Incorrect authentication data (set_id=laptop)
Login Fail Match: 2016-01-26 16:11:05 login authenticator failed for ([10.15.8.10]) [66.171.17.126]: 535 Incorrect authentication data (set_id=academy)
done queue
[Inferior 1 (process 22937) exited normally]
(gdb)
Click to expand...

Files brute.conf, brute_ip.data, brute_log_entries.list and brute_user.data were automatically generated.

User "alwin" from the output this morning and users "laptop" and "academy" in the output above are no real users. They only occur in the logs.

When trying to run ldd dataskq in /usr/local/directadmin/, I got: "not a dynamic executable".
 
Create a ticket, and I'll create you some dynamic variables.
Be sure to specify the license ID in the ticket so I'll know which OS to compile them for.

John

Edit: never mind about the ticket, found your email.
 
Thank you very much!
I followed the steps to install the dynamic binaries, removed "brute_force_ignore_attempts_on_suspended=0" from directadmin.conf and ran the dataskq again:

root@da:/usr/local/directadmin# gdb dataskq
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/directadmin/dataskq...(no debugging symbols found)...done.
(gdb) run d2000
Starting program: /usr/local/directadmin/dataskq d2000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Debug mode. Level 2000

root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Login Fail Match: Jan 26 22:32:09 da dovecot[6817]: pop3-login: Disconnected (auth failed, 1 attempts in 10 secs): user=<admin@89>, method=PLAIN, rip=162.105.81.94, lip=89.106.244.57, session=<g1pdakMq3JiiaVFe>
FormCheck::isDomain: length is less than 4 (2): 89
Tally::user_is_suspended(admin@89, dovecot1): '89' is not a valid domain. Skipping exemption check.
Login Fail Match: Jan 26 22:32:28 da dovecot[6817]: pop3-login: Disconnected (auth failed, 1 attempts in 17 secs): user=<admin@89>, method=PLAIN, rip=162.105.81.94, lip=89.106.244.57, session=<sAITa0Mqf8uiaVFe>
FormCheck::isDomain: length is less than 4 (2): 89
Tally::user_is_suspended(admin@89, dovecot1): '89' is not a valid domain. Skipping exemption check.
done queue
[Inferior 1 (process 6801) exited normally]
(gdb)
Click to expand...
 
You must log in or register to reply here.
Back
Top