Currently running Almalinux 8.5 and DA 1.63.1. I've noticed the last few weeks a few of my syslog files, (etc/log/secure, etc/log/pureFTP, etc/log/messages) are not receiving recorded log entries. Mail and Exim aren't either but I have those services turned off. What's so strange is the syslog appears to write into other files (Apache, DA, etc) - just not these. At either rate - thought I'd shoot this over to the collective and try to find an answer:
Thus far I've
I'm also noticing LFD log errors to the effect of "*SYSLOG CHECK* Failed to detect check line [sfmeKdH36QAOcPxwumqYJYq15F] sent to SYSLOG"
I've seen a few solutions to the same issue from years past, but I've tried those options and came up empty handed.
UPDATE: Dec 2, 2021
Did a lot of trial and error and, for whatever reason, rsyslog just refuses to grab data from the journal. Did just about everything the RSYSLOG docs recommend short of just doing a fresh OS install. Removed rsyslog and installed syslog-ng. Had a lot of work at the onset because it wasn't referencing imjournal, but I'm actually getting log entries now. We'll see how long the good times last, but I'll post an update if something changes.
I'm really irked I couldn't figure it out though, so if anyone has ideas what caused this in the first place feel free to post. It's vexed me
.
Update Dec 3, 2021 - The Solution
Still had issues....then I realized that maybe the problem has nothing to do with the reader but rather the system journal itself. Low and behold, I ran
And...got a lot of errors back. Somehow the journal file itself got corrupted! I imagine what was probably happening in the background is when rsyslog utilized the imjournal process to find its place in the log, the journal was corupted in such a way that it couldn't figure out how to continue. That's also probably why, for context, everything stopped on the same day, with a large gap, then stopped again. I let my journal files get pretty large, so it would be a while before everything is a-ok. The solution:
As root or using sudo:
Restart the systemd.journald process and syslog-ng (or rsyslog) and...tada.
Sometimes the simplest solution fixes the problem. On the positive side, I actually like syslog-ng a bit better, and it's a more mature logger anyway, so I'll keep it
Thus far I've
- Restarted the RSYSlog service and journal service. Confirmed both are running:
-
- All log files exist and set RW for root / root
- Attempted a manual write to the log files as the root user using logger and posted notice did not record.
#### MODULES ####
module(load="imuxsock" # provides support for local system logging (e.g. via logger command)
SysSock.Use="off") # Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
module(load="imjournal" # provides access to the systemd journal
StateFile="imjournal.state") # File to store the position in the journal
#module(load="imklog") # reads kernel messages (the same are read from journald)
#module(load="immark") # provides --MARK-- message capability
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")
# Use default timestamp format
module(load="builtinmfile" Template="RSYSLOG_TraditionalFileFormat")
# Include all config files in /etc/rsyslog.d/
include(file="/etc/rsyslog.d/*.conf" mode="optional")
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
I'm also noticing LFD log errors to the effect of "*SYSLOG CHECK* Failed to detect check line [sfmeKdH36QAOcPxwumqYJYq15F] sent to SYSLOG"
I've seen a few solutions to the same issue from years past, but I've tried those options and came up empty handed.
UPDATE: Dec 2, 2021
Did a lot of trial and error and, for whatever reason, rsyslog just refuses to grab data from the journal. Did just about everything the RSYSLOG docs recommend short of just doing a fresh OS install. Removed rsyslog and installed syslog-ng. Had a lot of work at the onset because it wasn't referencing imjournal, but I'm actually getting log entries now. We'll see how long the good times last, but I'll post an update if something changes.
I'm really irked I couldn't figure it out though, so if anyone has ideas what caused this in the first place feel free to post. It's vexed me
.Update Dec 3, 2021 - The Solution
Still had issues....then I realized that maybe the problem has nothing to do with the reader but rather the system journal itself. Low and behold, I ran
And...got a lot of errors back. Somehow the journal file itself got corrupted! I imagine what was probably happening in the background is when rsyslog utilized the imjournal process to find its place in the log, the journal was corupted in such a way that it couldn't figure out how to continue. That's also probably why, for context, everything stopped on the same day, with a large gap, then stopped again. I let my journal files get pretty large, so it would be a while before everything is a-ok. The solution:
As root or using sudo:
Then to clear out the old log entirely
Restart the systemd.journald process and syslog-ng (or rsyslog) and...tada.
Sometimes the simplest solution fixes the problem. On the positive side, I actually like syslog-ng a bit better, and it's a more mature logger anyway, so I'll keep it
Last edited:
.