Webalizer Statistics

S

sracicot

New member
Joined
Oct 12, 2005
Messages
1
Location
New Hampshire
Hello,


I am looking for the ability to customize the webalizer.conf file that is used to generate the stats for each of the website.

Any body know how I can go about doing this?

Basically I would like the option of adding HideSite, Group Sites, Group Referrers, etc. commands that make the reports a little cleaner and nicer.


Thanks
Steve
 
sracicot said:
...Group Sites, Group Referrers, etc....
Click to expand...

By that, do you mean to combine multiple virtual sites into one large stats report showing merged 'group site' stats in one report?

If so, that is exactly what I have been asking about for a long time with nothing at all accomplished, even though I was willing to pay for a custom script to do it. Can anyone here do this?
 
I believe that DA manages Webalizer with individual domain log files. I don't believe you can group domains that are in separate log files being managed separately.

Perhaps I'm wrong.

Jeff
 
the GroupDomains feature refers to
domains in one site's hit stats,
and not to the domains you're
hosting on the server, indeed.

add

GroupDomains 2
HideAllSites

to your webalizer.conf

re-tally, and have a look.

maybe a bit of a jumble though
 
Last edited:
squirrelhost said:
the GroupDomains feature refers to
domains in one site's hit stats,
and not to the domains you're
hosting on the server,....
Click to expand...

Don't understand what you are referring to as far as so called Group Domains go, but do understand what Jeff said, which is the big issue I have.

There must be 2 terminologies involved defining whatever Group Domains are meant to mean. Must be a communications problem.
 
I've discussed this with my partner.

What you want done isn't easy, because DA uses separate log files for the domains.

To enable this it would be necessary to, after Webalizer runs, append all the logfiles together to make one monolithic file, and then run Webalizer again on that monolithic file, to create a separate set of stats.

It would be quite a bit of work for us, because we don't know Webalizer well enough to "just do it". We may be willing to tackle it on an hourly basis.

Perhaps you can find someone else with more familiarity.

Jeff
 
sounds like you do want all 'domains' on
the server to be aggregated re stats
somehow.

I only know and understand the
webalizer configuration, and how
GroupDomains is defined and works there.

dealing with log files isn't particularly
difficult, I may have a look this weekend,
maybe others will too. you'd probably
want something small and clean, rather
than a monolothic pile of poo, as some
scripting tends to resemble.
 
it's getting rather late here in good old London,
so before I have my horlicks and retire,
I'll tell you how I would solve this little problem.

First, I'll define what problem I'd solve...

suppose I'm a reseller, with a few
user domains. I'd like to have all stats for the
domains aggregated, as if webalizer
processed all those site's stats together,
and see results in one big stats page.

rather than do too much work, I'd let
apache and webalizer do it for me as much
as possible,

each httpd.conf specifies where the access
log file is. But apache can write the same
data to as many log files as you specify. so I'd:

1. customise the httpd.conf I use to
write all the site's data to one extra
'special' log file. I'd use the same
combined logformat. each vhost (or
'domain' or 'site') can actually write to
this one log file all at once, as apache allows.

2. I've now got one big log file somewhere,
and would like this to be processed, but
where is it? how to do it?

3. i'd create a 'special' subdomain, as the
reseller, maybe called 'globalstats' or
whatever.

the access log file for this, is our one big
combined log file, with all our sites' data,
and I could also add the reseller's data in
there too - in fact it would be if i'd only
used the same custom httpd.conf template
for all.

4. voila! ( I think ).

maybe when I get up tomorrow morning
someone will have done it! if not
i'll have a go myself.

If this was for all the domains on the
server, not just per-reseller, then I'd
create a 'dummy' domain by admin,
and use the log file for this as the
one written to by all domains on the
server - admin would then access the
stats for this site from his 'user' level
just like any other domain set up
by the 'admin reseller' account.

could use the same 'fake' domain name
for this on all servers, allstats.com, or
whatever.
 
Last edited:
Whether or not it's an appropriate exclamation remains to be seen, but you've thought it out well, and I'd completely forgotten about having multiple log files. Great idea!

Care to give it a try?

Jeff
 
trying it now - i'll wait till tomorrow to check
any results.

I've set up a domain (see below), as this
looked to be easier:

there are a few gotchas?:-

I don't want hits to this domain
(in fact, I'll just be using the
'server IP' to log in and check stats, so
maybe any gotcha re stats for the dummy
site go away.

With a DA server, in general there's no
need for a 'main domain' hence I didn't
want to use a subdomain.

I could set up
stats.<hostname> and have this as a
'domain' on each server, but as this
does resolve, what to do about hits
I don't want - maybe block with .htaccess
and ignore the 403 entries in the
stats anyway ?

This is what I went with, so if you're
company uses bloggs.com, using my
current convention, if this was my
first server in Germany, this server would
be de1.bloggs.com (you could use
datacenter location, suitably abbreviated
in this), so I'd have

stats.de1.bloggs.com

which should have the .htaccess in public_html
and where I'm hoping all the logs will
be written.

I added a line to each httpd.conf, and just
so i could go with the flow for the moment,
that for the domain above has two entries !
one created as usual, and the other I added.
(duplicates any hits, but all are 403:forbidden).

After adding this it would be possible to
run a script, which updated all httpd.conf
files, adding a log entry in all, followed
by putting custom httpd.confs in the
custom dir (automates for future domains).

so if you wanted all this, maybe click a button and

1. get hostname, and set up domain
stats.<hostname>

2. munge all existing httpd.confs

3. put custom httpd.conf into /custom/ dir.

4 have a cup of tea ?

I'll wait and see..
 
just checked on server, as as
expected each hit to the websites on there
is also appended onto the
new 'site' logs.

will let webalizer run, and will see later what
custom webalizer configurations can help make
the best sense of the data.
 
works fine, here's a simple step by steup guide.

1. create a new account on your server,
username anything you like, domain
anything you like, let's call it test.com

2. custom virtual host conf:

copy

/usr/local/directadmin/data/templates/virtual_host.conf

to

/usr/local/directadmin/data/templates/custom/virtual_host.conf

Edit this one and add the line:

CustomLog /var/log/httpd/domains/test.com.log combined

after the line:

CustomLog /var/log/httpd/domains/|DOMAIN|.log combined

3. re-write:

echo "action=rewrite&value=httpd" >> /usr/local/directadmin/data/task.queue

(I believe that'll re-create all the domains' httpd.conf files)

4. restart apache

/usr/local/etc/rc.d/httpd restart

(for freebsd that is)


wait till webalizer updates, or re-tally everything
now:

echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

now log into DA as the new user,
and check domain test.com's stats - the log files for
all domains will now be adding to this
domain's log files, and the stats will now
be an aggregation of all site's stats on
the server.

(Note - you can add another CustomLog entry
for the ssl vhost config, and for the subdomain
version, in case you miss some here)
 
Hi, Will this work for multiple virtual sites? Any limit on how many?

Unfortunately, do not know how to implement it. Can you do it for me? I would be happy to pay you a reasonable fee for your time.

Please let me know. Thanks!

squirrelhost said:
works fine, here's a simple step by steup guide.

1. create a new account on your server,
username anything you like, domain
anything you like, let's call it test.com

2. custom virtual host conf:

copy

/usr/local/directadmin/data/templates/virtual_host.conf

to

/usr/local/directadmin/data/templates/custom/virtual_host.conf

Edit this one and add the line:

CustomLog /var/log/httpd/domains/test.com.log combined

after the line:

CustomLog /var/log/httpd/domains/|DOMAIN|.log combined

3. re-write:

echo "action=rewrite&value=httpd" >> /usr/local/directadmin/data/task.queue

(I believe that'll re-create all the domains' httpd.conf files)

4. restart apache

/usr/local/etc/rc.d/httpd restart

(for freebsd that is)


wait till webalizer updates, or re-tally everything
now:

echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

now log into DA as the new user,
and check domain test.com's stats - the log files for
all domains will now be adding to this
domain's log files, and the stats will now
be an aggregation of all site's stats on
the server.

(Note - you can add another CustomLog entry
for the ssl vhost config, and for the subdomain
version, in case you miss some here)
Click to expand...
 
works for every domain on the server,
if you create/edit the one conf file in
the custom directory.

if you have root access to your server,
i'll write a little script to do it, will reply soon.

everything has a limit, but all we're doing
here is produce webstats using a large
log file. it's your own responsibility to
ensure your server can cope with what
you throw at it.
 
tested this on freebsd. worked ok.

if you save this as blah.pl on your server,
and if you've set up a 'dummy' domain
whose stats you want to be the aggregated
stats for all domains on the server,
i.e. test.com, say, then simply do

perl blah.pl test.com


just follow the instruction. here's the script:

#!/usr/bin/perl

# - writes apache access log data for all domains on server to specified domain's logfile
# - creates: /usr/local/directadmin/data/templates/custom/virtual_host(2).conf
# - adds new CustomLog line to this file
# - re-writes all domain httpd.conf files
# - logging will start after apache is restarted

my ($domain) = $ARGV[0];
die "usage: ./$0 <domain>\n" unless $ARGV[0];

$logfile = "/var/log/httpd/domains/${domain}.log";
$logline = "\tCustomLog /var/log/httpd/domains/${domain}.log combined";

if (!(-e $logfile)) {
print "log file for domain ${domain} not found - check if domain exists on server\n";
exit;
}

print "which apache version - 1.3 or 2.0, enter '1' for apache 1.3, or enter '2' for apache 2.0 ";
$option = <STDIN>;
chomp $option;
if ($option eq "1") {
$in = "/usr/local/directadmin/data/templates/virtual_host.conf";
$out = "/usr/local/directadmin/data/templates/custom/virtual_host.conf";
}
elsif ($option eq "2") {
$in = "/usr/local/directadmin/data/templates/virtual_host2.conf";
$out = "/usr/local/directadmin/data/templates/custom/virtual_host2.conf";
}
else { print "unknown option selected for apache version..\n"; exit; }

if ((-e $out)) {
print "custom httpd.conf already exists - continue? (y/n) ";
$option = <STDIN>;
chomp $option;
if ($option ne "y") {
print "exiting\n";
exit;
}
}

open(INFO, $in);
@lines = <INFO>;
close(INFO);

open(INFO, ">$out");
foreach $line (@lines) {
if ($line =~ ErrorLog) {
print INFO "${logline}\n";
}
print INFO $line;
}
close(INFO);

print "re-write all httpd.conf files? (y/n) ";
$option = <STDIN>;
chomp $option;
if ($option eq "y") { system(" echo \"action=rewrite&value=httpd\" >> /usr/local/directadmin/data/task.queue "); }

exit;
 
Last edited:
'fraid I only use BSD, but all this does
is copy the default file(s), and
edit then new custom version(s).

if the paths are the same, then
I doubt it'd matter.

backup your current file(s),
run the script, see what happens.

in the worst possible scenario,
you've a new file(s) you don't
want, so delete it.
 
Code: 
which apache version - 1.3 or 2.0, enter '1' for apache 1.3, or enter '2' for apache 2.0 1
custom httpd.conf already exists - continue? (y/n) y
re-write all httpd.conf files? (y/n) y

nothing happen
 
You must log in or register to reply here.
Back
Top