multi-server log parsing

donco

Verified User
Joined
Feb 5, 2004
Messages
45
We have a multi-server customer who wishes to see their site usage for all there servers in one place.

What is the best way to implement this?

The working theory is to copy/cat all the server log files into their primary server log files (sort if necessary), and allow the usual DirectAdmin/webalizer/et al processes to take it from there.

However, before I do this, I need to know the best place to tie into "the system". When do I have to have the data pulled across? Before which process kicks off?

Is it possible to configure the system to parse in multiple files, rather than just one?

Thanks,
-Don
 
Hello,

If all logs are placed (and possibly sorted by date) into one log file, it could work...

The process that does webalizer is the dataskq, and it is triggered when:
echo "action=tally&value=all" >> /usr/local/directadmin/data/task.queue
is run. I don't think that the different domain names would show up.. but if it's the same domain on multiple servers via round robin dns, then it would probably work fine.

The cron is run from /etc/cron.d/directadmin_cron, which I believe runs at 12:10am daily. You can change it as needed.

If you need to parse from multiple log files, then your best bet would be to just implement the entire thing seperate from DA.

John
 
I don't think webalizer can parse logs out of order; I think you WILL have to sort by date.

Jeff
 
As far as I can see, I need to be ready for these two events:
  1. DirectAdmin bandwidth accounting @ 00:10 with *.bytes files
  2. webalizer web log parsing @ 04:02 with *.log files (and *.error.log?)
    [/list=1]

    In other words, I need the *.bytes files for each domain to be consolidated (no sort necessary) and ready to go for DirectAdmin by 00:10. Whereas webalizer requires all the *.log files (and *.error.log?) for each domain to be consolidated and sorted by 04:02.

    Does that sound about right?

    Am I missing anything?

    Is there any reason not to have webalizer kick off at 00:10 along with the DirectAdmin accounting process?

    Thanks, as always, for your time and attention in this matter,
    -Don
 
Actually, both 1 and 2 are part of the "tally" which start at 00:10. The domains are processed one at a time for each user. So you'll want to make sure everything you're doing is done by then.

John
 
So the domain specific files must not be tied into the cron.daily webalizer or logrotate, as they kick off (along with all of cron.daily) at 04:02...?
 
Oh.. if you have a webalizer00 file in cron.daily, you can delete that. It's just the cron that comes with webalizer.. we don't use it for the domains. Logrotate will only affect the error_log, access_log, suexec_log ..etc... correct, it doesn't touch the domain specific logs.

John
 
DirectAdmin Support said:
Oh.. if you have a webalizer00 file in cron.daily, you can delete that.
Do you mean 00webalizer?
we don't use it for the domains.
Where do you start webalizer, John?
Logrotate will only affect the error_log, access_log, suexec_log ..etc... correct, it doesn't touch the domain specific logs.
However, I presume it can be easily made to. If I set up logrotate to affect domain specific logs, will it interfere with any other process?

Jeff
 
Hello,

yes 00webalizer :)

Webalizer is run in the tally. It's executed a large tree, going through each Reseller, then to their users, then to each domain. Each part of the tree returns with all totals for all items. When the last tree (Reseller) returns, the admin.usage file is written. So webalizer will start on one domain when the tally starts, and will end when the last domain is being tallied.. so through the whole tally webalizer jobs are being run. If you rotate the domain specific logs.. it's not a huge deal. However, if you swap out logs so DA can't see them, there may be a whole in the webalizer stats.

John
 
Back
Top