Excessive Resource Usage - systemd-network

[mockingbird]

I'm getting these messages in the Mail Queue Administration every hour or so:

2023-02-13 15:59:29 Received from root@<snip> U=root P=local S=817 T="lfd on <snip>: Excessive resource usage: systemd-network (680 (Parent PID:680" 2023-02-13 15:59:29 <snip> R=localuser T=local_delivery defer (-29): User 0 set for local_delivery transport is on the never_users list

I did find this thread, with the same error, but for a different process. Shall I add an exclusion (if so, may I know how to please?), or is this worth investigating?

Thanks

 

[Richard G]

User 0 set for local_delivery transport is on the never_users list

This points out that there is mail for root. This could be important, maybe csf or something else is trying to notify root of the cause of the high load.

I would start taking care that you can receive root mail.

You can either edit the /etc/aliases file and change:
# root: marc
to
root: [email protected]
ofcourse change [email protected] to an email address you want to receive roots mail at.
Restart exim afterwards:
service exim restart

Or create a file /root/.forward (mind the dot) and add your email address in there.

 

[BillyS]

I just looked, I dont have systemd-network in my CSF cef.pignore. As Richard said, you should first fix the mail issue.

 

[mockingbird]

Thanks @Richard G and @BillyS

I set up the email alias and I will report back. The only thing I have aside from this fresh install is a Wordpress installation in a subdirectory (manual install of Wordpress).

 

[mockingbird]

Ok, here are the results which I've been receiving in my email:
lfd on server-<snip>.da.direct: Excessive resource usage: admin (22269 (Parent PID:22268)) Time: Tue Feb 14 02:06:10 2023 +0000 Account: admin Resource: Process Time Exceeded: 9066 > 1800 (seconds) Executable: /usr/bin/bash Command Line: -bash PID: 22269 (Parent PID:22268) Killed: No

lfd on server-<snip>.da.direct: Excessive resource usage: admin (22185 (Parent PID:22184)) Time: Tue Feb 14 02:05:10 2023 +0000 Account: admin Resource: Process Time Exceeded: 9008 > 1800 (seconds) Executable: /usr/lib/systemd/systemd Command Line: (sd-pam) PID: 22185 (Parent PID:22184) Killed: No

lfd on server-<snip>.da.direct: Excessive resource usage: admin (22184 (Parent PID:22184)) Time: Tue Feb 14 02:05:10 2023 +0000 Account: admin Resource: Process Time Exceeded: 9008 > 1800 (seconds) Executable: /usr/lib/systemd/systemd Command Line: /lib/systemd/systemd --user PID: 22184 (Parent PID:22184) Killed: No

There are more, but they're too many to list. Anyone know what's going on? This is a pretty fresh install here.

 

[BillyS]

Found a couple of posts from about a year ago, referencing CPanel. Is this you having the same issue last year?

Excessive resource usage Executable: /usr/lib/systemd/systemd Command Line: (sd-pam)

Hello, I am aware that CSF reports these via email, but I have never seen this with one of my websites . usually its /usr/bin/curl or /opt/cpanel/ea-php7.4/root/usr/sbin/php-fpm or /opt/cpanel/ea-php8/root/usr/sbin/php-fpm but never systemd or command line (sd-pam) Any reason you can think of...

forums.cpanel.net

 

[Richard G]

lfd on server-<snip>.da.direct:

You're still using an invalid hostname. It should be an FQDN hostname of yourself, and not an xxxx.da.direct hostname.
Not sure if this could contribute to the issue, but seems things are still not fine.
Always first setup hostname and domains correctly, then install scripts.

 

[mockingbird]

Found a couple of posts from about a year ago, referencing CPanel. Is this you having the same issue last year?

Excessive resource usage Executable: /usr/lib/systemd/systemd Command Line: (sd-pam)

Hello, I am aware that CSF reports these via email, but I have never seen this with one of my websites . usually its /usr/bin/curl or /opt/cpanel/ea-php7.4/root/usr/sbin/php-fpm or /opt/cpanel/ea-php8/root/usr/sbin/php-fpm but never systemd or command line (sd-pam) Any reason you can think of...

forums.cpanel.net

No sir, not me.

I wonder if it's related to my VPS... But it shouldn't be. It's quad core, 4GB ram, fast storage... Should be able to handle the workload.

I have many emails this morning lol from CSF as before... I wonder what could be causing this.

 

[Richard G]

We wrote at the same time. I just replied to you with a tip above your post.

 

[mockingbird]

You're still using an invalid hostname. It should be an FQDN hostname of yourself, and not an xxxx.da.direct hostname.
Not sure if this could contribute to the issue, but seems things are still not fine.
Always first setup hostname and domains correctly, then install scripts.

Ok, thanks, I'll fix that and report back.

 

[mockingbird]

We wrote at the same time. I just replied to you with a tip above your post.

Ok, two interesting things to note:

1) The mail queue administration is now consistently empty after changing the server over to my FQDN, however, the emails still come through (the last one complained about systemd.networkd)

2) My alias seems to be ignored now, because although I have it set up in 'etc/aliases' to "root: admin@mydomain", the emails are going to "root@mydomain" and end up in my catch-all. Besides that, which is beyond the scope of this thread and might just be the result of some minor misconfiguration somewhere, I wonder how many other people would receive these emails from CSF/LFD if they had a catch-all set up on their account.

Thanks

 

[Richard G]

My alias seems to be ignored now, because although I have it set up in 'etc/aliases' to "root: admin@mydomain", the emails are going to "root@mydomain"

Yes, well that is a bit odd. I don't know your distro but in some distro's (not Centos) the .forward is needed to get it correctly.

1.) Good to hear the queue keeps empty now. We don't have a ignore for systemd.networkd in csf.pignore and it should not be there either.
So the question should be as to why the systemd.networkd is causing this long time.
Is something hanging, is there some leak scriptor something. I'm not that good in systemd stuff, maybe somebody else has a clue, maybe @BillyS or @Hostmavi have a clue.

2.) I presume you did restart exim afterwards.
Receiving those mails is not depending on having a catch-all or not. I get all CSF e-mails to the alias I set up.
I also presume you made a typo in the masking of the domain and you set up the alias to send to [email protected] (so with the tld) and not just admin@mydomain.

 

[mockingbird]

Yes, well that is a bit odd. I don't know your distro but in some distro's (not Centos) the .forward is needed to get it correctly.

1.) Good to hear the queue keeps empty now. We don't have a ignore for systemd.networkd in csf.pignore and it should not be there either.
So the question should be as to why the systemd.networkd is causing this long time.
Is something hanging, is there some leak scriptor something. I'm not that good in systemd stuff, maybe somebody else has a clue, maybe @BillyS or @Hostmavi have a clue.

Ah, then I should elaborate on my system specifications:

OS: Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-60-generic x86_64)

admin@ubuntu:~/Download$ ./vpsb-lx64-250 Version 2.5.0, (c) 2018+ jsg (->lowendtalk.com) Machine: amd64, Arch.: x86_64, Model: Intel Core Processor (Haswell, no TSX, IBRS) OS, version: Linux 5.15.0, Mem.: 3.851 GB CPU - Cores: 4, Family/Model/Stepping: 6/60/1 Cache: 32K/32K L1d/L1i, 2M L2, ? L3 Std. Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 cflsh mmx fxsr sse sse2 ss htt sse3 pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline aes xsave osxsave avx f16c rdrnd hypervisor Ext. Flags: fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid syscall nx pdpe1gb rdtscp lm lahf_lm lzcnt

2.) I presume you did restart exim afterwards.
Receiving those mails is not depending on having a catch-all or not. I get all CSF e-mails to the alias I set up.
I also presume you made a typo in the masking of the domain and you set up the alias to send to [email protected] (so with the tld) and not just admin@mydomain.

Yessir, I restarted it. Also, correct, I did put the ".com" there despite just typing "mydomain" on the forum as the mask.

I don't think root@mydomain has an entry on the mail server (there is no "root" in the 'Email Accounts' menu under 'User'), so maybe that's why it's getting sent to my catch-all.

Yes, certainly, I'd like to know if anyone else can chime in on this... The only other thing I have set up different from stock is that I created a swapfile on the server of 6GB and have that configured in /etc/fstab for startup, since my VPS' Ubuntu install does not set up a swap partition by default.

Thanks

 

[Richard G]

admin@ubuntu:~/Download$ ./vpsb-lx64-250

Seems your hostname is still not like it should be. Or did you name it ubuntu.domain.com?

so maybe that's why it's getting sent to my catch-all.

That is correct, however I wonder why it's then send to [email protected] instead of [email protected]. But I have seen this happening on Debian (and related) systems. You could therefore try to remove or comment the mail address from the /etc/aliases file (and restart Exim) and only use the /root/.forward file with your [email protected] email address in there and see if that fixes the issue.

@zEitEr you are also good in resources needed and systemd stuff, do you have any clue about the systemd.networkd keep giving these notices?

 

[mockingbird]

Seems your hostname is still not like it should be. Or did you name it ubuntu.domain.com?

Oh, yea, that's fine, it's just a mask. The pertinent line in my /etc/hosts reads as follows:

x.x.x.x ubuntu.com ubuntu

That is correct, however I wonder why it's then send to [email protected] instead of [email protected]. But I have seen this happening on Debian (and related) systems. You could therefore try to remove or comment the mail address from the /etc/aliases file (and restart Exim) and only use the /root/.forward file with your [email protected] email address in there and see if that fixes the issue.

@zEitEr you are also good in resources needed and systemd stuff, do you have any clue about the systemd.networkd keep giving these notices?

Yea, I removed it since it was no longer doing anything after changing my FQDN.

I also just deactivated my file-type swapfile... Upon rebooting after removing the entry in fstab, I saw a 4GB swap... I checked where it was coming from with the "swapon -s" command, and I now see a 4GB swap partition on /dev/sda2 - so I wonder what's up with that since it is not something I did.

But swap usage is always nil or almost nil - still, I'll see if that changed anything.

 

[Richard G]

The pertinent line in my /etc/hosts reads as follows:

x.x.x.x ubuntu.com ubuntu

Well LoL, that's also wrong, because ubuntu.com is a domain name. In the hostfile it should be a FQDN hostname. So as long as your masking your domainname with ubuntu.com then it should be:
x.x.x.x server.ubuntu.com ubuntu
Or whatever your hostname is. Using the incorrect name there could also cause weird issues, so I would certainly take care the correct FQDN hostname (not domain name) is in there and reboot the VPS.
Also check /etc/HOSTNAME for a correct name, just to be sure. Maybe it may be empty (I always use the correct name in there) but it may not be wrong.

As for the swap I don't know. I have a swap, almost never used, but seems DA must have a swap, Zeiter is beter into quota and swap stuff. I always make things mostly default as possible and almost never have issues with these things.

 

[mockingbird]

Well LoL, that's also wrong, because ubuntu.com is a domain name. In the hostfile it should be a FQDN hostname. So as long as your masking your domainname with ubuntu.com then it should be:
x.x.x.x server.ubuntu.com ubuntu
<snip>

I've gone ahead and made the following change to /etc/csf/csf.conf:

PTC_USERTIME = "1800"

to

PTC_USERTIME = "0"

I then issued "sudo csf-r" and "sudo service lfd restart".

Since all the emails had to do with this setting in particular, and they were finishing 1 minute ahead of the threshold, I don't think there's any real issue here. Alternatively, I could have also set the limit higher, to something like "1900".

As for the hosts, my domain isn't ubuntu.com or a subdomain of ubuntu.com. I am simply using ubuntu.com as a mask for my real server name (and I altered the bash prompt to read "admin@ubuntu:~" only in my post here on the forum, not on my actual server) .

Thanks everyone for your help.

 

[Richard G]

PTC_USERTIME = "1800"

You mean PT_USERTIME? I've got that at PT_USERTIME = "3000" and no mails.
If you only got mails of these, then you could set it to 0 too.

I am simply using ubuntu.com as a mask for my real server nam

Yep I understood that, but normally people mask the names in a correct way so those who are helping can see that there is no problem, which is why I wondered about the ubuntu.com instead of server.ubuntu.com (so masked), just wanted to be sure.

Glad you figured it out with CSF.

 

[mockingbird]

You mean PT_USERTIME? I've got that at PT_USERTIME = "3000" and no mails.
If you only got mails of these, then you could set it to 0 too.

Ah yes, indeed it is PT_USERTIME and not PTC_USERTIME...

Ok, yours is at 3000 you say by default? Thanks for that. Perhaps it would be prudent for the developers to take a note of that. The default of "1800" for Ubuntu seems insufficient. Thanks again

...oh, and not a single email since the change.

 

[Richard G]

Ok, yours is at 3000 you say by default?

No I think the default is 1800 since I got that on the other servers, but on one server I probably have raised this for the same reason as you to 3000 at a certain point.
I will put it back to 1800 and see what happens.