System Messages Cleaunup Script v1.1

soulshepard

soulshepard

Verified User
Joined
Feb 7, 2008
Messages
123
System Messages Cleanup Script v1.2.2

06-07-2014: added the patch suggestions for bigEsmurf, the mention bug has been fixed
12-01-2013: added installer script for version v1.2.1, this script will install, and add the crontab & logrotate config, and it will retain the config settings made in the scipts, in a sense implace upgrading is prepared.
12-01-2013: Update to v1.2.1 : the script can now handle the file increments of 000,001, 00X files that can exist in the ticket folders.
UPDATE: the script will now archive your admin and all users ticket messages in one big logfile. and you can clean up the users messages also, and you can put in a skiplist for users you do not want to clean

Hello all,

I like the system messages that directadmin produces, only it can grow very big. especially with messages from csf firewall, and backups and to note this can take up serious inode space if you leave it accumulating over time.. and deleting them is a pain and takes time. i know i have seen people talking about this and i tought there were some cleanup scripts allready place?
In anycase i created a new one.. this one will archive all messages for the admin or all users in a logfile and keep your server messages clean with a user tunable retention of XX days. hopefully its of some use for someone.

the code might look big but i do some fancy bash looping with extra checks to stop and or prevent possible harm. to note: its on your own risc. in the script i place extra checks and config options. like output to a log or screen or a debug level and the most important switch is the testmode .. if its set to 1 and this is the default it will not delete or update the ticket list or anything else.

The main function of this scripts is to parse the /usr/local/directadmin/data/(users/user or admin)/tickets.list and check if the entries here have a corresponding folder with 00X.conf and 00X.msg files in there.
if not it will be removed from the ticket list. and when you set the keep days to 31 it will search for tickets olders then 31 days and remove the 00X.conf and 00X.msg files and the ticket gets removed from the ticket list.
but before it will remove anything it will add all lines of the old tickets into a history logfile. so you can search in one logfile of all the system messages of the past. in combination with the logrotator it can keep a 1 year or longer of history logfiles. so you can use it in a cron or use it one time to clean up quickly.. with or without a log.

it was a nice puzzle and i think i will expand this with more options to filter out backups and other type of messages to get those reported to a monitoring solution.

The only case you will see old files in your ticket folder is if the ticket list entries are gone and the files were not deleted. perhaps in these rare cases i will add this to the script. so be sure to check also your ticket folder after you run the script to clean up even more old stale files that are not in any ticket list.. this does take up huge amounts of inodes ;)

the current version is the v1.2.2 thanks to bigEsmurf for some nice patchwork

download: View attachment installdasysv1.2.2.txt to /usr/local/directadmin/scripts/custom and rename it to installcleandasys.sh (installer)
md5sum : 68e7f87b9f20ed376b7d22930994f4e9

download: View attachment cleandasysmsgv1.2.2.txt to /usr/local/directadmin/scripts/custom and rename it to cleandasysmsg.sh (manual download)
md5sum : 5d22007acfa45900cc6eb2464948c07d


The installer script: installcleandasys.sh
This will download the scriptfile from the da forum, check m5dums, configure the cron & logrotate , and you can enable the script from the installer, but other settings, need to be changed manually. The script will retain config settings for each time you run the installer, also you crontab if you dare to point it to an central one. but default we use a seperate one.
Code: 
wget -O /usr/local/directadmin/scripts/custom/installcleandasys.sh http://forum.directadmin.com/attachment.php?attachmentid=1785&d=1399493265
sh /usr/local/directadmin/scripts/custom/installcleandasys.sh disable
sh /usr/local/directadmin/scripts/custom/installcleandasys.sh enable
use the optione enable to turnoff the testmode and or use disable to enable the testmode
note: when run intially installed with enable it will have debug logging turned on.

to run:
Code: 
sh /usr/local/directadmin/scripts/custom/cleandasysmsg.sh

other options configurable inside the script.
Code: 
options:
daticketlist=/usr/local/directadmin/data
ticketlistfilename=tickets.list
daticketfolder=/usr/local/directadmin/data/tickets
dauserdatafolder=/usr/local/directadmin/data/users

keepamountofdays=10 # the amount of days you want to keep.
keepmessagehistoryinlog=1 # archive the messages to a logfile.

processallusermessages=1 # process all users in directadmin to clean the messages or it will default to admin.
skipusers="xxxxusers1kjdkjnsd1 xxxusers2kljadladj"  # skip theses users from the autocleaning

logfile=/var/log/directadmin/sysmsgcleanup.log
messageshistorylogfile=/var/log/directadmin/sysmsghistory.log

# debug parameters
testmode=0
debugoutput=0   # show the skipped or valid messages in the logs that are found and other debug messages.
outputtolog=1   # log all actions to the logfile.
outputtocrt=1   # show all action on the screen.

Tested platform list: (please reply your platform)
Code: 
CentOS release 5.8 (Final)
CentOS release 6.5 (Final)

installer example scriptoutput:
Code: 
[root@serverx somefolder]# sh installcleandasys.sh enable
Cleandasysmsg 1.2.1 script installer
checking installed script settings in : /usr/local/directadmin/scripts/custom/cleandasysmsg.sh
retaining:testmode=0
retaining:debugoutput=1
checking if the curl binary exists
fetching http://forum.directadmin.com/attachment.php?attachmentid=1697&d=1389495684 as /usr/local/directadmin/scripts/custom/cleandasysmsg121.sh
installed md5:742cf573626bf8e3e17083089ba32373 /usr/local/directadmin/scripts/custom/cleandasysmsg121.sh, with expected md5:742cf573626bf8e3e17083089ba32373
makeing a backup of the old cronfile /etc/cron.d/cleandasysmsg_cron to /root/ins/cleandasysmsg_cron.1389570946.backup
preparing new cronfile in /root/ins/cleandasysmsg_cron.1389570946:..
reverting backup to origional file because existing cron entry has been found for script : cleandasysmsg.sh
file /etc/logrotate.d/cleandasysmsg_logrotate allready exists, skipping
option forced : setting testmode:0
patching the script /usr/local/directadmin/scripts/custom/cleandasysmsg121.sh with testmode=0
patching the script /usr/local/directadmin/scripts/custom/cleandasysmsg121.sh with debugoutput=1
done.


Manual install:
Code: 
wget -O /usr/local/directadmin/scripts/custom/cleandasysmsg.sh http://forum.directadmin.com/attachment.php?attachmentid=1783&d=1399492942
sh /usr/local/directadmin/scripts/custom/cleandasysmsg.sh
note: default runs the script in testmode. please allways do one run in testmode if you install in on a new platform / server

manual install cron:
add this line to a new cron for each week on saturday for log cleanup
file:/etc/cron.d/cleandasysmsg_cron
Code: 
0 7 * * 6  root sh  /usr/local/directadmin/scripts/custom/cleandasysmsg.sh > /dev/null 2>&1

manual install logrotate:
To lograte the archived logfile you could add a logrotate to run each month and to keep 12 months. so you keep a year of system messages for all users in a log file/archive
and to note for big ervers this file for me never comes above 200Mb. and then compress it will be greatly reduced. it is good to have the logfiles archieved it could also be a evedence keeper.
create the file:/etc/logrotate.d/cleandasysmsg_logrotate
Code: 
/var/log/directadmin/sysmsgcleanup.log /var/log/directadmin/sysmsghistory.log {
        monthly
        missingok
        start 0
        rotate 12
        compress
        create 0644 diradmin diradmin
}

The script code:
Code: 
#!/bin/bash

# version 1.2.2 of the directadmin message system purge or auto cleanup
# by [email protected]
#
# change the script settings and set the testmode to 0 to do ticket removal and file/folder removal of ticket data
# use at your own risk. allways test and double check results!
#
# set the amount of days you want to keep in the message system
# and set the correct paths for the ticket folder or list if they are changed from the default.
#
# to allow the logfile to grow please add it to the logrotate configuration /etc/logrotate.d/directadmin or your own custom
# also with the message history file it can grow to a few MB for a lot of messages. and rotating this each X can be setup manually.
#
# to note, when you process the user messages also then you need to have the script run with the correct rights
#
# to use the skipuser list use skipuser="user1 user2" with a space delimited
#
# 12-01-2014: changed script to also detect 000,001 and increment files under the same ticket folder.
#             looping and log messages changed.
# 12-01-2013: added installer script for version v1.2.1, this script will install, and add the crontab & logrotate config, and it will retain the config settings made in the scipts, in a sense implace upgrading is prepared.
# 06-07-2014: added the patch suggestions for bigEsmurf, the mention bug has been fixed
#


##### script settings

daticketlist=/usr/local/directadmin/data
ticketlistfilename=tickets.list
daticketfolder=/usr/local/directadmin/data/tickets
dauserdatafolder=/usr/local/directadmin/data/users

keepamountofdays=10 # the amount of days you want to keep.
keepmessagehistoryinlog=1 # archive the messages to a logfile.

processallusermessages=1 # process all users in directadmin to clean the messages or it will default to admin.
skipusers="xxxxusers1kjdkjnsd1 xxxusers2kljadladj"  # skip theses users from the autocleaning

logfile=/var/log/directadmin/sysmsgcleanup.log
messageshistorylogfile=/var/log/directadmin/sysmsghistory.log

# debug parameters
testmode=0
debugoutput=1   # show the skipped or valid messages in the logs that are found and other debug messages.
outputtolog=1   # log all actions to the logfile.
outputtocrt=1   # show all action on the screen.

############################################################################################################ no config needed below


# global vars

# for numeric regular expression
regnumcheck='^[0-9]+$'


# logical checks

if [[ ! -d $daticketlist ]]; then
        echo unable to find directadmin data folder for user/tickets.lists quiting..
        exit 2
fi

if [[ ! -d $daticketfolder ]]; then
        echo unable to find the directadmin ticket folder quiting..
        exit 2
fi

if [[ ! -d $dauserdatafolder ]]; then
        echo unable to find the directadmin user data folder quiting..
        exit 2
fi

regnumcheck='^[0-9]+$'
if  [[ ! $keepamountofdays =~ $regnumcheck ]] ; then
        echo please enter a valid numeric amount of days you want to keep as system messages
        exit 2
fi

if  [[ ! $testmode =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the test mode on or off with the values 0 or 1
        exit 2
fi

if  [[ ! $debugoutput =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the displayskips on or off with the values 0 or 1
        exit 2
fi

if  [[ ! $outputtolog =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the ioutputtolog on or off with the values 0 or 1
        exit 2
fi

if  [[ ! $outputtocrt =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the ioutputtocrt on or off with the values 0 or 1
        exit 2
fi

if  [[ ! $keepmessagehistoryinlog =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the keepmessagehistoryinlog on or off with the values 0 or 1
        exit 2
fi

if  [[ ! $processallusermessages =~ $regnumcheck ]] ; then
        echo please enter a valid numeric number to turn the processallusermessages on or off with the values 0 or 1
        exit 2
fi

if [[ $outputtolog -eq 1 ]] ; then
        if [[ ! -f $logfile ]] ; then
        touch $logfile
                if [[ ! -f $logfile ]] ; then
                echo error cannot create log file in $logfile.
                exit 2
                fi
        fi
fi

if [[ $keepmessagehistoryinlog -eq 1 ]] ; then
        if [[ ! -f $messageshistorylogfile ]] ; then
        touch $messageshistorylogfile
                if [[ ! -f $messageshistorylogfile ]] ; then
                echo error cannot create the message history logfile in $messageshistorylogfile.
                exit 2
                fi
        fi
fi


############################################################################################################### functions

filedatediff() {
    d1=$(date +%s)  # current date in timestamp
    d2=$(stat -c%Y $daticketfolder/$folderSysmsg/$filenameSysmsg/$1) # timestamp of file
#    echo $(( (d1 - d2) / 86400 )) # calculate days difference between timestamps
 echo $(( (d1 - d2 ) / (60*60*24) ))
}

folderdatediff() {
    d1=$(date +%s)  # current date in timestamp
    d2=$(stat -c%Y $daticketfolder/$folderSysmsg/$filenameSysmsg) # timestamp of file
#    echo $(( (d1 - d2) / 86400 )) # calculate days difference between timestamps
 echo $(( (d1 - d2 ) / (60*60*24) ))
}


datelog() {
date +%d-%m-%Y-%H:%M:%S:%N
}

datetimestamptoarchivedate() {
date -d @$1 +'%d-%m-%Y-%H:%M:%S'
}

msglog() {
dateline=$(datelog)

if [[ $outputtocrt -eq 1 ]] ; then
echo $dateline - $1
fi

if [[ $outputtolog -eq 1 ]] ; then
echo $dateline - $1 >> $logfile
fi

}

archivemsglog() {

dateline=$(datelog)

if [[ $outputtocrt -eq 1 ]] ; then
echo $dateline - $1
fi

if [[ $outputtolog -eq 1 ]] ; then
echo $dateline - $1 >> $logfile
fi

if [[ $keepmessagehistoryinlog -eq 1 ]] ; then
echo $dateline - $1 >> $messageshistorylogfile
fi

}

archivemsglogpreservedate() {

dateline=$(datelog)

if [[ $keepmessagehistoryinlog -eq 1 ]] ; then

        if [[ -f $daticketfolder/$folderSysmsg/$filenameSysmsg/$1 ]] ; then

                datelinefile=`stat -c%Y $daticketfolder/$folderSysmsg/$filenameSysmsg/$1`

                archivemsglog "archiving ticket: $ticketlistname, user: $loopedusername, file: $daticketfolder/$folderSysmsg/$filenameSysmsg/$1, date: $(datetimestamptoarchivedate "$datelinefile")"

                cat $daticketfolder/$folderSysmsg/$filenameSysmsg/$1 | while read line || [ -n "$line" ];
                do
                        echo "$dateline - $loopedusername: $(datetimestamptoarchivedate "$datelinefile") : $line" >> $messageshistorylogfile

                        if [[ $outputtocrt -eq 1 ]] ; then
                                if [[ $debugoutput -eq 1 ]] ; then
                                        echo "$dateline - $loopedusername: $(datetimestamptoarchivedate "$datelinefile") : $line"
                                fi
                        fi
                done
        fi
fi

}


############################################################################################################### main

msglog "starting directadmin system messages cleanup with parameters:"
msglog "directadmin ticketlist: $daticketlist"
msglog "directadmin ticketfolder: $daticketfolder"
msglog "logfile: $logfile"
msglog "messageshistorylogfile: $messageshistorylogfile"
msglog "keeping $keepamountofdays days of system messages"
msglog "testmode:$testmode, debugoutput:$debugoutput, outputtolog:$outputtolog, outputtocrt:$outputtocrt, keepmessagehistoryinlog:$keepmessagehistoryinlog"

# get the userlist and remove the skiplist  (to get the new arruserlist)

arrdausers=( $(find -H $dauserdatafolder -maxdepth 1 -mindepth 1 -type d -printf "%f\n" | sort ) )  # get the directadmin user, with find its safer then an ls.. so you only list types and things you need


arrdaskiplist=( $(echo $skipusers) )  # get the list of users you want to skip

        # remove the skiplist from the userlist
        for i in "${arrdausers[@]}"; do
             skip=
                     for j in "${arrdaskiplist[@]}"; do
                         [[ $i == $j ]] && { skip=1; break; }
                     done
                     [[ -n $skip ]] || arruserlist+=("$i")
        done

        # loopdebug if needed
        if [[ $debugoutput -eq 1 ]] ; then
                for fulllistentry in "${arrdausers[@]}"
                do
                        msglog "fulllist:$fulllistentry"
                done
        fi
        if [[ $debugoutput -eq 1 ]] ; then
                for skiplistentry in "${arrdaskiplist[@]}"
                do
                        msglog "skiplist:$skiplistentry"
                done
        fi

        if [[ $debugoutput -eq 1 ]] ; then
                for skipuser in "${arruserlist[@]}"
                do
                        msglog "moddedlist:$skipuser"
                done
        fi


# implode the array to a string
printf -v moddeduserlist "%s\n" "${arruserlist[@]}"

# and if we have disabled the all users to process then we simple only process admin and destroy our previous loop work ;)
if [[ ! $processallusermessages -eq 1 ]] ; then
moddeduserlist="admin"
fi

        if [[ $debugoutput -eq 1 ]] ; then
                for skipuser in "$(echo $moddeduserlist)"
                do
                        msglog "processing:$skipuser"
                done
        fi
# walk all users or just admin
for processdauser in $(echo $moddeduserlist)
do
loopedusername=$processdauser

msglog "processing tickets for user: $processdauser"

checkadminlcasevar=$(echo $processdauser | tr '[:upper:]' '[:lower:]')

if [[ ! "$checkadminlcasevar" == "admin" ]] ; then
processdauser="users/$processdauser"
fi

# ofcourse if the ticketlist file actually exists then we can continue.
if [[ -f $daticketlist/$processdauser/$ticketlistfilename ]] ; then

msglog "selecting ticketfile: $daticketlist/$processdauser/$ticketlistfilename"

 # process the messages
 for listsysmsg in $(cat $daticketlist/$processdauser/$ticketlistfilename)
        do
                #pointer to meassure acctions and display message
                actionpointer=0
                # fetch the ticket list format
                folderSysmsg=${listsysmsg:0:6}  # where 6 is the fixed length
                filenameSysmsg=$(echo $listsysmsg | sed 's/^0*//' | cut -d "=" -f 1)
                filenameSysmsg=$(printf "%03d\n" "$filenameSysmsg") # suggested by bigEsmurf http://forum.directadmin.com/showthread.php?t=48021
                ticketlistname=$(echo $listsysmsg | cut -d "=" -f 1)
                        #echo $folderSysmsg/$filenameSysmsg
                if [[ $debugoutput -eq 1 ]] ; then
                msglog "checking: $daticketfolder/$folderSysmsg/$filenameSysmsg"
                fi

                if [[ -d $daticketfolder/$folderSysmsg/$filenameSysmsg ]] ; then

                               msglog "checking file dates for ticket $ticketlistname in folder: $daticketfolder/$folderSysmsg/$filenameSysmsg"

                                # need to check for all the numeric .conf and msg. files we cannot assume only 000. files exist so we need to loop to make sure.
                                # begin of patch bigEsmurf http://forum.directadmin.com/showthread.php?t=48021
                                TICKETFILES=$(find -H $daticketfolder/$folderSysmsg/$filenameSysmsg -type f \( -name "*.conf" -o -name "*.msg" \) -printf "%P\n" | sort)
                                if [ ! -z "$TICKETFILES" ]
                                then
                                        actionpointer=0
                                else
                                        msglog "no files found in $daticketfolder/$folderSysmsg/$filenameSysmsg, entry will be deleted"
                                        actionpointer=1
                                fi
                                for listmsgconffiles in $TICKETFILES
                                # end of patch bigEsmurf http://forum.directadmin.com/showthread.php?t=48021
                                 do
                                        if [[ -f $daticketfolder/$folderSysmsg/$filenameSysmsg/$listmsgconffiles ]] ; then
                                                getSysmsgdaysold=$(filedatediff $listmsgconffiles)
                                                        # check if the result is a number, then continue.
                                                        if  [[ $getSysmsgdaysold =~ $regnumcheck ]] ; then

                                                                # get the older and the new seperated.
                                                                if [[ $getSysmsgdaysold -gt $keepamountofdays ]] ; then

                                                                                # only do in action stations
                                                                                if [[ $testmode -eq 0 ]] ; then

                                                                                        # declare removal
                                                                                        msglog "processing file $listmsgconffiles in folder : $daticketfolder/$folderSysmsg/$filenameSysmsg, The file is $getSysmsgdaysold days old, keeping only $keepamountofdays"

                                                                                        if [[ $keepmessagehistoryinlog -eq 1 ]] ; then
                                                                                                        archivemsglogpreservedate "$listmsgconffiles"
                                                                                        fi
                                                                                        # delete the older files
                                                                                        actionpointer=1
                                                                                        msglog "deleting file:$daticketfolder/$folderSysmsg/$filenameSysmsg/$listmsgconffiles"
                                                                                        rm $daticketfolder/$folderSysmsg/$filenameSysmsg/$listmsgconffiles  > /dev/null 2>&1

                                                                                          if [[ -f "$daticketfolder/$folderSysmsg/$filenameSysmsg/$listmsgconffiles" ]] ; then
                                                                                                msglog "ERROR: file $daticketfolder/$folderSysmsg/$filenameSysmsg/$listmsgconffiles still exists, perhaps its readonly?"
                                                                                                exit 2
                                                                                          fi

                                                                                else
                                                                                        msglog "TESTMODE:file $listmsgconffiles in folder : $daticketfolder/$folderSysmsg/$filenameSysmsg, The file is $getSysmsgdaysold days old, keeping only $keepamountofdays"
                                                                                        msglog "testmode is on will not perform deletes or updates"
                                                                                fi
                                                                        #done


                                                                else
                                                                        if [[ $debugoutput -eq 1 ]] ; then
                                                                        msglog "skipping: $daticketfolder/$folderSysmsg/$filenameSysmsg its $getSysmsgdaysold days old and we keep $keepamountofdays"
                                                                        fi
                                                                fi # exit check file older


                                                        fi # exit check if result is numeric of date

                                        fi # exit from check listmsgconffiles

                                    done  # exit from the findlistmsgconffiles


                        if [[ $testmode -eq 0 ]] ; then

                          if [[ actionpointer -eq 1 ]] ; then
                                msglog "deleting folder:$daticketfolder/$folderSysmsg/$filenameSysmsg"
                                rmdir $daticketfolder/$folderSysmsg/$filenameSysmsg > /dev/null 2>&1

                                        if [[ -d "$daticketfolder/$folderSysmsg/$filenameSysmsg" ]] ; then
                                                msglog "ERROR: folder $daticketfolder/$folderSysmsg/$filenameSysmsg still exists, check why the ${listmsgconffiles%.conf}.conf and ${listmsgconffiles%.conf}.msg or possible other files are not deleted!. perhaps manual created files with dates and or symlinks present."
                                                msglog " dont forget to remove the ticketlist entry manual when completed with sed -i '/'$ticketlistname'/d' $daticketlist/$processdauser/$ticketlistfilename"
                                                exit 2
                                        else
                                               # continue with removal of the ticketlist entry
                                               # updating the ticket list with removing the line
                                               msglog "removing ticketlist entry : $ticketlistname"
                                               sed -i '/'$ticketlistname'/d' $daticketlist/$processdauser/$ticketlistfilename
                                        fi
                           fi

                                if [[ $debugoutput -eq 1 ]] ; then
                                 msglog "no ticket files found older then $keepamountofdays days in folder $daticketfolder/$folderSysmsg/$filenameSysmsg"
                                fi
                        fi
                # here we have an entry in the ticket list that does not exist on the file system so we remove it
                else
                        msglog "cannot find : $daticketfolder/$folderSysmsg/$filenameSysmsg, removing $ticketlistname from ticketlist"
                        if [[ $testmode -eq 0 ]] ; then
                                sed -i '/'$ticketlistname'/d' $daticketlist/$processdauser/$ticketlistfilename
                        else
                                 msglog "testmode is on will not perform deletes or updates"
                        fi
                fi # exit from  check folder existance


        #cleanup
        folderSysmsg=
        filenameSysmsg=
        msgsubject=
        getSysmsgdaysold=
        msgsubject=
        done
 msglog "completed user: $loopedusername"
else
msglog "$daticketlist/$processdauser/$ticketlistfilename does not exists, skipping"
fi
done

example log output:

logfile:/var/log/directadmin/sysmsgcleanup.log
Code: 
12-01-2014-03:50:45:774205000 - starting directadmin system messages cleanup with parameters:
12-01-2014-03:50:45:777490000 - directadmin ticketlist: /usr/local/directadmin/data
12-01-2014-03:50:45:780470000 - directadmin ticketfolder: /usr/local/directadmin/data/tickets
12-01-2014-03:50:45:783664000 - logfile: /var/log/directadmin/sysmsgcleanup.log
12-01-2014-03:50:45:786485000 - messageshistorylogfile: /var/log/directadmin/sysmsghistory.log
12-01-2014-03:50:45:789392000 - keeping 5 days of system messages
12-01-2014-03:50:45:792183000 - testmode:0, debugoutput:0, outputtolog:1, outputtocrt:1, keepmessagehistoryinlog:1
12-01-2014-03:50:45:804700000 - processing tickets for user: admin
12-01-2014-03:50:45:810674000 - selecting ticketfile: /usr/local/directadmin/data/admin/tickets.list
12-01-2014-03:50:45:822330000 - checking file dates for ticket 000001283 in folder: /usr/local/directadmin/data/tickets/000001/1283
12-01-2014-03:50:45:834769000 - processing file 000.conf in folder : /usr/local/directadmin/data/tickets/000001/1283, The file is 9 days old, keeping only 5
12-01-2014-03:50:45:844434000 - archiving ticket: 000001283, user: admin, file: /usr/local/directadmin/data/tickets/000001/1283/000.conf, date: 02-01-2014-06:51:20
12-01-2014-03:50:45:897064000 - deleting file:/usr/local/directadmin/data/tickets/000001/1283/000.conf
12-01-2014-03:50:45:907629000 - processing file 000.msg in folder : /usr/local/directadmin/data/tickets/000001/1283, The file is 9 days old, keeping only 5
12-01-2014-03:50:45:918117000 - archiving ticket: 000001283, user: admin, file: /usr/local/directadmin/data/tickets/000001/1283/000.msg, date: 02-01-2014-06:51:20
12-01-2014-03:50:45:970839000 - deleting file:/usr/local/directadmin/data/tickets/000001/1283/000.msg
12-01-2014-03:50:45:975638000 - deleting folder:/usr/local/directadmin/data/tickets/000001/1283
12-01-2014-03:50:45:980059000 - removing ticketlist entry : 000001283
...
30-12-2013-01:59:12:207901889 - processing tickets for user: webuser111
30-12-2013-01:59:12:210515468 - selecting ticketfile: /usr/local/directadmin/data/users/webuser111/tickets.list
30-12-2013-01:59:12:215976284 - checking: /usr/local/directadmin/data/tickets/000000/549
30-12-2013-01:59:12:223530843 - archiving ticket: 000000549, user: webuser111, file: /usr/local/directadmin/data/tickets/000000/549/000.conf, date: 08-12-2011-22:17:13
30-12-2013-01:59:12:247005985 - archiving ticket: 000000549, user: webuser111, file: /usr/local/directadmin/data/tickets/000000/549/000.msg, date: 08-12-2011-22:17:13
30-12-2013-01:59:12:251928819 - REMOVING: /usr/local/directadmin/data/tickets/000000/549 its 752 days old, keeping only 10
30-12-2013-01:59:12:253290486 - removing ticketlist entry : 000000549
30-12-2013-01:59:12:260761838 - checking: /usr/local/directadmin/data/tickets/000002/2419
30-12-2013-01:59:12:268720602 - archiving ticket: 000002419, user: webuser111, file: /usr/local/directadmin/data/tickets/000002/2419/000.conf, date: 19-02-2013-02:20:27
30-12-2013-01:59:12:292109348 - archiving ticket: 000002419, user: webuser111, file: /usr/local/directadmin/data/tickets/000002/2419/000.msg, date: 30-12-2013-01:51:19
30-12-2013-01:59:12:297065068 - REMOVING: /usr/local/directadmin/data/tickets/000002/2419 its 313 days old, keeping only 10
30-12-2013-01:59:12:298446413 - removing ticketlist entry : 000002419
30-12-2013-01:59:12:302536129 - completed user: webuser111

logfile:/var/log/directadmin/sysmsghistory.log
Code: 
30-12-2013-01:59:10:693671510 - archiving ticket: 000005475, user: admin, file: /usr/local/directadmin/data/tickets/000005/5475/000.conf, date: 19-12-2013-01:05:01
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : from=diradmin
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : name=Message System
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : priority=30
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : status=open
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : subject=Brute-Force Attack detected in service log on User(s) root
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : type=message
30-12-2013-01:59:10:693671510 - admin: 19-12-2013-01:05:01 : user=multiple
30-12-2013-01:59:10:719140153 - archiving ticket: 000005475, user: admin, file: /usr/local/directadmin/data/tickets/000005/5475/000.msg, date: 19-12-2013-01:05:01
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 : A brute force attack has been detected in one of your service logs.
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 :
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 : User root has 15827 failed login attempts: dovecot1=135&exim2=13&proftpd1=23&sshd4=15656
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 :
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 : Check 'Admin Level -> Brute Force Monitor' for more information
30-12-2013-01:59:10:719140153 - admin: 19-12-2013-01:05:01 : http://help.directadmin.com/item.php?id=404
...
30-12-2013-01:59:12:223530843 - archiving ticket: 000000549, user: webuser111, file: /usr/local/directadmin/data/tickets/000000/549/000.conf, date: 08-12-2011-22:17:13
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : from=diradmin
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : name=Message System
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : priority=30
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : status=open
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : subject=Your backups are now ready
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : type=message
30-12-2013-01:59:12:223530843 - webuser111: 08-12-2011-22:17:13 : user=multiple
30-12-2013-01:59:12:247005985 - archiving ticket: 000000549, user: webuser111, file: /usr/local/directadmin/data/tickets/000000/549/000.msg, date: 08-12-2011-22:17:13
30-12-2013-01:59:12:247005985 - webuser111: 08-12-2011-22:17:13 : Backup created
 
Last edited:
i have found a server that had 002 and 003.msg files .. i will update the script to detect these and parse these in a loop .. in versie 1.2.1
 
the script has been updated to version 1.2.1 it will now also process the incremental 000 and 001,002 etc files that can exist in a ticket folder..
the log messages has been changed and the looping a little bit. the origional post has been updated with a new donwload link and md5sum

and an installer has been added to speed thigns up.
 
Last edited:
Greate script, thanx! I had to change a few things though, messages below 1000 were not getting deleted. i.e. ticket 000000000 resulted in the right folderSysmsg, but with a finenameSysmsg of "". I added:
Code: 
filenameSysmsg=$(printf "%03d\n" "$filenameSysmsg")

That way "" will become 000, 1 will become 001, etc... 1234 will stay 1234 :)

Another thing I had a problem with was that the folders didn't have any messages. So I added:
Code: 
TICKETFILES=$(find -H $daticketfolder/$folderSysmsg/$filenameSysmsg -type f \( -name "*.conf" -o -name "*.msg" \) -printf "%P\n" | sort)
if [ ! -z "$TICKETFILES" ]
then
	actionpointer=0
else
	msglog "no files found in $daticketfolder/$folderSysmsg/$filenameSysmsg, entry will be deleted"
	actionpointer=1
fi
for listmsgconffiles in $TICKETFILES

Everything works great now!
 
Could you paste the entire script with those edits? Or, soulshepard could you edit your first post with new version including this fix?

Regards
 
Hello All

@bigEsmurf
Thanks for the bugfix!

with the line
Code: 
filenameSysmsg=$(printf "%03d\n" "$filenameSysmsg")

you mean actually as extra definition under the existing?
like:
Code: 
                # fetch the ticket list format
                folderSysmsg=${listsysmsg:0:6}  # where 6 is the fixed length
                filenameSysmsg=$(echo $listsysmsg | sed 's/^0*//' | cut -d "=" -f 1)
                filenameSysmsg=$(printf "%03d\n" "$filenameSysmsg") # added by bigeSmurf
                ticketlistname=$(echo $listsysmsg | cut -d "=" -f 1)

@SelleRone

yes i will update the main post as soon as i get a reply from bigEsmurf to confirm and test the changes.
 
Last edited:
Please be sure to post and let us know when the script update is ready; I've got thousands of messages on one server and I'm anxiously waiting :).

Jeff
 
nobaloney said:
Please be sure to post and let us know when the script update is ready; I've got thousands of messages on one server and I'm anxiously waiting :).

Jeff
Click to expand...

its ready jeff.. the patch works great. i updated the installer and the code ontop. but the fastes way is to use the installer..it will do the manual steps as descript above

please test the script i am really happy with the outcome.. but do keep in mind it can run for a while if you have years of message build up.. but when its complete and in the cron it will keep the amount of messages to the retention of days configured in the script.

i will update the script later on to make it with a seperate config file.

when you run the script in screen manually then also check out with tail the following logs

# for the script log actions
tail -f /var/log/directadmin/sysmsgcleanup.log
# the logfile that will hold the archive of all messages of the server in one logfile. it is really usefull with searching for the deleted message history.
tail -f /var/log/directadmin/sysmsgcleanup.log


Soul
 
Last edited:
@soulshepard:

I'm finally ready to test. I want to make sure I understand everything in your last post:

If I just do a new install following instructions in the first post can I just run it, or need I do anything else?

Thanks.

Jeff
 
nobaloney said:
@soulshepard:

I'm finally ready to test. I want to make sure I understand everything in your last post:

If I just do a new install following instructions in the first post can I just run it, or need I do anything else?

Thanks.

Jeff
Click to expand...

yes the first post is updated (including the source script and installer, so you can follow the manual install or the installer)

when you run the installer below it will install the script cleandasysmsg.sh in : /usr/local/directadmin/scripts/custom/

Code: 
wget -O /usr/local/directadmin/scripts/custom/installcleandasys.sh http://forum.directadmin.com/attachment.php?attachmentid=1785&d=1399493265
sh /usr/local/directadmin/scripts/custom/installcleandasys.sh disable
sh /usr/local/directadmin/scripts/custom/installcleandasys.sh enable

ater that you can run :

sh /usr/local/directadmin/scripts/custom/cleandasysmsg.sh
 
Thanks. I got busy again but I've got a server I need to try it on soon.

Jeff
 
i got it installed on a few servers again.. and its working very nice to keep your system messages clean. and to keep a searchable archive of logfiles..
i think i need to convert this to a plugin.. and will try to find the time to research how..

ps: i also noticed there are sometimes 000.conf.lock files i guess this must be rare..
 
Last edited:
I've just coded several tools in the Message System to:
1) Delete Messages from the tickets.list if you're done with them. Can be based on subject, or data (before or after, in days)

2) Automatically have DA clear out tickets and/or messages from disk (data/tickets/0000*/*) if they're older than a certain time.
tickets.list files will be cleaned of the missing ticket numbers once they're read in, so no need to keep the tickets.list clean from stale data.
Different days can be used for tickets vs messages, and the date for ticket comparison will be the last reply in the ticket thread (so a thread started a long time ago is not deleted, if there is a new message)

https://www.directadmin.com/features.php?id=1696

It's in the pre-release system, if anyone wants to try it out.

John
 
Thx John, thats a really good addition i can acrive this script, altho this script does one thing that could be included in what you made.

- placing all messages contents into a logfile and rotate it with logrotate and zip the old archives + the option to tell how long you would want to keep the logfile
- i use this as an way to search for the old messages, i know i can copy the messages to email, a workarround, but if the mail does not work then i lost that data.

thanks for creating the feature into directadmin
 
You must log in or register to reply here.
Back
Top