The service 'lfd' on server servername is currently down

B

bdj

Verified User
Joined
Feb 22, 2008
Messages
113
I'm currently getting this message every hour. It started a couple of days ago when I started adding sites. (It was empty before that)

/var/log/messages contains these lines every 2 minutes

Code: 
Oct 22 13:03:32 hossrv systemd[1]: lfd.service: start operation timed out. Terminating.
Oct 22 13:03:32 hossrv systemd[1]: lfd.service: Failed with result 'timeout'.
Oct 22 13:03:32 hossrv systemd[1]: Failed to start ConfigServer Firewall & Security - lfd.
Oct 22 13:03:37 hossrv systemd[1]: Starting ConfigServer Firewall & Security - lfd...

Clicking "ConfigServer Security & Firewall" takes 2 minutes to display

I'm not sure which log file of where to find more info about what's causing the 'timeout'

Any suggestions?



Centos 8
DA 1.63.0
CSF 14.11
 
LFD at least keeps its own log on my system, /var/log/lfd.log, see if there's anything revealing there.
 
Code: 
   0 -rw-------  1 root     root                    0 Oct 22 00:01 lfd.log

It's empty
 
What happens if you login via SSH and try to restart CSF and LFD manually, does it give any additional info?

service csf restart
service lfd restart

did you disable firewalld from Centos?
 
The thing is, when this VPS came preinstalled, at first it worked but after I started adding sites, ie using the server, these errors came.

Code: 
[root@hossrv log]# service csf restart
Redirecting to /bin/systemctl restart csf.service
------ about 2 minutes of waiting
[root@hossrv log]#
[root@hossrv log]# service lfd restart
Redirecting to /bin/systemctl restart lfd.service
Job for lfd.service failed because a timeout was exceeded.
See "systemctl status lfd.service" and "journalctl -xe" for details.
[root@hossrv log]#

Code: 
[root@hossrv log]# systemctl status firewalld
● firewalld.service
   Loaded: masked (Reason: Unit firewalld.service is masked.)
   Active: inactive (dead)


Code: 
journalctl -xe

Oct 22 16:41:36 hossrv systemd[1]: Starting ConfigServer Firewall & Security - lfd...
-- Subject: Unit lfd.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit lfd.service has begun starting up.
Oct 22 16:42:01 hossrv CROND[2284914]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 22 16:43:01 hossrv CROND[2284954]: (root) CMD (/usr/local/directadmin/dataskq)
Oct 22 16:43:06 hossrv systemd[1]: lfd.service: start operation timed out. Terminating.
Oct 22 16:43:06 hossrv systemd[1]: lfd.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- The unit lfd.service has entered the 'failed' state with result 'timeout'.
Oct 22 16:43:06 hossrv systemd[1]: Failed to start ConfigServer Firewall & Security - lfd.
-- Subject: Unit lfd.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit lfd.service has failed.
--
-- The result is failed.
 
Haven't seen you provided this one including the running status:

Code: 
systemctl status lfd

and

Code: 
systemctl status csf

and just keep looking at the log file even if it is empty:

open another terminal type this with the -F big capital letter:

tail -F /var/log/lfd.log
 
Code: 
[root@hossrv log]# systemctl status lfd
● lfd.service - ConfigServer Firewall & Security - lfd
   Loaded: loaded (/usr/lib/systemd/system/lfd.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Fri 2021-10-22 16:55:07 CEST; 21s ago
  Process: 2285410 ExecStart=/usr/sbin/lfd (code=killed, signal=TERM)
 Main PID: 457821 (code=killed, signal=KILL)

Oct 22 16:53:37 hossrv systemd[1]: Starting ConfigServer Firewall & Security - lfd...
Oct 22 16:55:07 hossrv systemd[1]: lfd.service: start operation timed out. Terminating.
Oct 22 16:55:07 hossrv systemd[1]: lfd.service: Failed with result 'timeout'.
Oct 22 16:55:07 hossrv systemd[1]: Failed to start ConfigServer Firewall & Security - lfd.


[root@hossrv log]# systemctl status csf
● csf.service - ConfigServer Firewall & Security - csf
   Loaded: loaded (/usr/lib/systemd/system/csf.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2021-10-22 16:39:40 CEST; 15min ago
  Process: 2284219 ExecStop=/usr/sbin/csf --initdown (code=killed, signal=TERM)
  Process: 2284284 ExecStart=/usr/sbin/csf --initup (code=exited, status=0/SUCCESS)
 Main PID: 2284284 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 104857)
   Memory: 0B
   CGroup: /system.slice/csf.service

Oct 22 16:39:40 hossrv csf[2284284]: iptables v1.8.4 (nf_tables):  RULE_APPEND failed (Invalid argument): rule in chain INPUT
Oct 22 16:39:40 hossrv csf[2284284]: LOGDROPIN  all opt -- in !lo out *  0.0.0.0/0  -> 0.0.0.0/0
Oct 22 16:39:40 hossrv csf[2284284]: SMTPOUTPUT  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0
Oct 22 16:39:40 hossrv csf[2284284]: csf: FASTSTART loading SMTP Block (IPv4)
Oct 22 16:39:40 hossrv csf[2284284]: csf: FASTSTART loading DNS (IPv4)
Oct 22 16:39:40 hossrv csf[2284284]: iptables v1.8.4 (nf_tables):  RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
Oct 22 16:39:40 hossrv csf[2284284]: LOCALOUTPUT  all opt -- in * out !lo  0.0.0.0/0  -> 0.0.0.0/0
Oct 22 16:39:40 hossrv csf[2284284]: iptables v1.8.4 (nf_tables):  RULE_INSERT failed (Invalid argument): rule in chain INPUT
Oct 22 16:39:40 hossrv csf[2284284]: LOCALINPUT  all opt -- in !lo out *  0.0.0.0/0  -> 0.0.0.0/0
Oct 22 16:39:40 hossrv systemd[1]: Started ConfigServer Firewall & Security - csf.
[root@hossrv log]#


I'm running the tail -F in another terminal but it's returning nothing.
 
It's the same thing... rule_insert generates failures.
I've encountered this issue last year with somebody which might have something to do with the VPS system used.

Check this page:

Error with CSF Centos 8.1 installation

Okay. Some progress I disabled those 3 blocklists mentioned in my previous message. Now, after a reboot (and starting LFD manually) LFD is running. However "ConfigServer Security & Firewall" is still awefully slow compared to a system running with CentOS 7.x. Very strange. When I click it, it...
forum.directadmin.com forum.directadmin.com

Also seen some odd thing that this can happen if there is no existing/etc/firewalld/firewalld.conf present when using Centos 8.
Check if this file is present, if not create one, even empty is good.
sudo touch /etc/firewalld/firewalld.conf
 
/etc/firewalld/firewalld.conf is present

I'm looking at /etc/sysconfig/iptables and I'm not sure what all those numbers in the [ ] sections mean. Examples only state [0:0]

Code: 
[root@hossrv firewalld]# cat /etc/sysconfig/iptables

# Generated by iptables-save v1.8.4 on Sun Oct 10 12:01:08 2021
*raw
:PREROUTING ACCEPT [4858:1141895]
:OUTPUT ACCEPT [5695:1550863]
COMMIT
# Completed on Sun Oct 10 12:01:08 2021
# Generated by iptables-save v1.8.4 on Sun Oct 10 12:01:08 2021
*mangle
:PREROUTING ACCEPT [4858:1141895]
:INPUT ACCEPT [4858:1141895]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [5695:1550863]
:POSTROUTING ACCEPT [5695:1550863]
COMMIT
# Completed on Sun Oct 10 12:01:08 2021
# Generated by iptables-save v1.8.4 on Sun Oct 10 12:01:08 2021
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun Oct 10 12:01:08 2021
# Generated by iptables-save v1.8.4 on Sun Oct 10 12:01:08 2021
*filter
:OUTPUT ACCEPT [2269:620429]
:FORWARD ACCEPT [0:0]
:INPUT ACCEPT [1729:316018]
COMMIT
# Completed on Sun Oct 10 12:01:08 2021

I've asked my hoster is they use OpenVZ
 
Update: I'm not sure why or how but /var/log/messages shows;

Code: 
Oct 22 18:12:01 hossrv systemd[1]: Starting ConfigServer Firewall & Security - lfd...
Oct 22 18:13:30 hossrv systemd[1]: lfd.service: Can't open PID file /run/lfd.pid (yet?) after start: No such file or directory
Oct 22 18:13:30 hossrv systemd[1]: Started ConfigServer Firewall & Security - lfd.

Code: 
[root@hossrv log]# systemctl status lfd
● lfd.service - ConfigServer Firewall & Security - lfd
   Loaded: loaded (/usr/lib/systemd/system/lfd.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-22 18:13:30 CEST; 3h 48min ago
  Process: 2290112 ExecStart=/usr/sbin/lfd (code=exited, status=0/SUCCESS)
 Main PID: 2290237 (lfd - sleeping)
    Tasks: 1 (limit: 104857)
   Memory: 133.4M
   CGroup: /system.slice/lfd.service
           └─2290237 lfd - sleeping

Oct 22 18:12:01 hossrvsystemd[1]: Starting ConfigServer Firewall & Security - lfd...
Oct 22 18:13:30 hossrvsystemd[1]: lfd.service: Can't open PID file /run/lfd.pid (yet?) after start: No such file or directory
Oct 22 18:13:30 hossrvsystemd[1]: Started ConfigServer Firewall & Security - lfd.
[root@hossrv log]#

I also have a /var/log/lfd.log now with data.
I'm 100% sure I didn't to anything after 17:00 CET
*confused*

It still takes a good 2 minutes to open ConfigServer Firewall & Security
 
Oke, first the etc/sysconfig/iptables file.
Exactly which distro are you using? Because imho that file looks to have content in there which should not be in there.

If I look at the date, it seems iptables is running things and configuring things and not CSF/LFD. And iptables should -not- be running as a service itself and it looks as it does now.

Remove that file.

Try this:
systemctl stop iptables systemctl disable iptables systemctl stop ip6tables systemctl disable ip6tables

Just to be sure, disable csf/lfd by using csf -x

doublecheck everything is off now by using the iptables -L command which should give this output now:
Code: 
#iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination        

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

When this is correct, start csf/lfd again via the csf -e command.

Now look if it starts.

Check if you don't have too many ip blocks already, clear the csf.deny file and limit the amount of blocks, for example 1000 for both full block and temp block, depending on the amount of memory you have, or maybe use IPSET with csf.

If the problem stays, it could have to do with OPENVZ, maybe kernel needs to be updated.
Exactly the same issue occured here:

Intermittent error: The service 'lfd' on server myserver.domain.com is currently down

Hi Yesterday afternoon, I used CustomBuild 2.0 to install the new updates. I think one of them was PHP related. I also ran yum update from the prompt. Yesterday evening and today I get intermittent errors concerning lfd:- “The service 'lfd' on server myserver.domain.com is currently down”...
forum.directadmin.com forum.directadmin.com
 
  • Like
Reactions: bdj
Allereerst; Bedankt voor je tijd

Distro : CentOS Linux release 8.4.2105


csf -x takes a long time to responds but eventually stops it

Code: 
[root@hossrv admin]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
# Warning: iptables-legacy tables present, use iptables-legacy to see them

[root@hossrv admin]# iptables-legacy -L
bash: iptables-legacy: command not found

Next:

Code: 
[root@hossrv admin]# csf -e
.
bunch of rules
.
.
ACCEPT  all opt -- in lo out *  0.0.0.0/0  -> 0.0.0.0/0
ACCEPT  all opt -- in * out lo  0.0.0.0/0  -> 0.0.0.0/0
iptables v1.8.4 (nf_tables):  RULE_APPEND failed (Invalid argument): rule in chain OUTPUT
LOGDROPOUT  all opt -- in * out !lo  0.0.0.0/0  -> 0.0.0.0/0
iptables v1.8.4 (nf_tables):  RULE_APPEND failed (Invalid argument): rule in chain INPUT
LOGDROPIN  all opt -- in !lo out *  0.0.0.0/0  -> 0.0.0.0/0
SMTPOUTPUT  all opt -- in * out *  0.0.0.0/0  -> 0.0.0.0/0
csf: FASTSTART loading SMTP Block (IPv4)
csf: FASTSTART loading DNS (IPv4)
iptables v1.8.4 (nf_tables):  RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
LOCALOUTPUT  all opt -- in * out !lo  0.0.0.0/0  -> 0.0.0.0/0
iptables v1.8.4 (nf_tables):  RULE_INSERT failed (Invalid argument): rule in chain INPUT
LOCALINPUT  all opt -- in !lo out *  0.0.0.0/0  -> 0.0.0.0/0
Job for lfd.service failed because a timeout was exceeded.
See "systemctl status lfd.service" and "journalctl -xe" for details.
● lfd.service - ConfigServer Firewall & Security - lfd
   Loaded: loaded (/usr/lib/systemd/system/lfd.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Sat 2021-10-23 16:12:50 CEST; 29ms ago
  Process: 2471863 ExecStart=/usr/sbin/lfd (code=killed, signal=TERM)
 Main PID: 2394333 (code=killed, signal=KILL)

Oct 23 16:11:20 hossrv systemd[1]: Starting ConfigServer Firewall & Security - lfd...
Oct 23 16:12:50 hossrv systemd[1]: lfd.service: start operation timed out. Terminating.
Oct 23 16:12:50 hossrv systemd[1]: lfd.service: Failed with result 'timeout'.
Oct 23 16:12:50 hossrv systemd[1]: Failed to start ConfigServer Firewall & Security - lfd.
csf and lfd have been enabled
 
Graag gedaan hoor! ;)

First of all. Did you delete the etc/sysconfig/iptables file first? Or maybe rather backup it to some other place?

As for the CSF stopping, I think these are the cause:
bdj said:
RULE_APPEND failed (Invalid argument): rule in chain OUTPUT
Click to expand...
Might be very well these will cause the timeout which in the end will stop CSF.

Problem is that I have no clue as to where they are coming from or how to fix this.
And support on the CSF forum is not always the best. But it's free so well... :)

I found a topic here which is really *exactly* the same problem, but is closed as it would be caused by csf/lfd configuration.
So I wondering which configuration error they are talking about.

Because if we check google for that error, we see it happening a whole lot, and might have to do with some conflicting issues between iptables and nftables.
I hope we can get this fixed, but at this moment I've got no clue yet.

What you could do is copy your current csf.conf to a safe place. Then download CSF from scratch and put a default csf.conf in there and see if the error still occurs, then we could be sure.
 
Oke, yesterday I decided to start from scratch.
Uninstall CSF
Stop and disable ip(6)tables and remove the etc/sysconfig/iptables file.
rotate all logs
install CSF.
Now everything is fine (apart from being flooded by emails about Excessive resource usage)

Thanks for the help.
 
bdj said:
Now everything is fine (apart from being flooded by emails about Excessive resource usage)
Click to expand...
Oke that is normal.
Either make the resource usage a bit higher, or just disable the mail send for excessive resource usage.

As for the help, you're welcome.
 
I figured that out already.
Now I'm working on these, trying to figure out what is causing this.


Code: 
Time:      Mon Oct 25 12:30:00 2021 +0200
PID:       278410 (Parent PID:134812)
Account: 6user3
Uptime:   67 seconds

Executable:
/usr/local/php74/sbin/php-fpm74

Command Line (often faked in exploits):
php-fpm: pool 6user3                                                                    

Network connections by the process (if any):
tcp: 100.200.200.31:45390 ->  46.101.254.99:80

Files open by the process (if any):
/dev/null
/dev/null
/tmp/.ZendSem.9ZErS4 (deleted)

But don't worry, this is something totally different. I'm going find a solutions to mark
 
These could be anything, could be busy sites or a script on the site which keeps the php-fpm running for that user.
You have to monitor that user's site for it.
 
You must log in or register to reply here.
Back
Top