HowTo: Custom webalizer wrapper for DA v0.0.1

markus

Verified User
Joined
Nov 7, 2004
Messages
134
Location
Girona, Catalunya
HowTo: Custom webalizer wrapper for DA v1.0.0

NOTE: Please, consider this a beta version. :rolleyes:

DOWNLOAD:
http://www.phpmix.com/files/directadmin/webalizer

Code:
# ======================================================================
# Custom webalizer wrapper for DA - version 1.0.0 (2005-01-11)
#
# Upgrades and further information
# --------------------------------
#
# Please, check out the DirectAdmin (DA) How-To Guides forum:
# [url]http://www.directadmin.com/forum/forumdisplay.php?forumid=40[/url]
#
# History: See change log at the end of these notes.
#
#
# Short Description
# -----------------
#
# This script replaces the webalizer executable called by DA.
# By doing so, it allows you to:
#
# a) Use your own webalizer.conf, even using specific settings
#    per domain (or even per subdomain), if you wish.
#
# b) Enable/Disable webalizer stats globally or per domain. This
#    might be useful, for instance, if you're using some other Apache
#    log analizer like AWStats, etc. for some or all your domains.
#
# In short, this script allows you to call the 'original' webalizer
# ONLY for the domains you want, the way you exactly want. :-)
#
#
# Compatibility
# -------------
#
# As of now (see version and date on top), this script supports:
#
# - DirectAdmin 1.23.3
# - Webalizer 2.01
# - Red Hat Linux Enterprise 3
#
# Please, confirm if you can make it run on other systems. Thanks
#
# ======================================================================
# 1. README
# ======================================================================
#
# 1.1.	How current DirectAdmin allows you to customize Webalizer?
# ----------------------------------------------------------------
#
# Quoted from here:
# [url]http://help.directadmin.com/item.php?id=27[/url]
#
# DirectAdmin uses the default webalizer settings by not specifying a
# webalizer.conf file at all. If you want to adjust the settings for
# webalizer, you can create your own basic webalizer.conf file for DA
# to use. This file must be domain independant because this one file
# will be used for all domains without any modifications to it.
#
# This file should be stored here:
# /usr/local/directadmin/data/template/custom/webalizer.conf
#
#
# 1.2. When DirectAdmin calls Webalizer?
# --------------------------------------
#
# DirectAdmin calls webalizer as part of the tally process which is
# triggered by a daily cron task.
#
# You can manually trigger this procedure by executing the following
# command:
#
# echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
#
# Please, read this for further information:
# [url]http://help.directadmin.com/item.php?id=48[/url]
#
#
# 1.3. How DirectAdmin calls Webalizer?
# -------------------------------------
#
# DirectAdmin uses the root user to call webalizer for each domain and
# subdomain as follows:
#
#   webalizer -p -n HOSTNAME -o OUTPUT_DIR INPUT_LOG
#
# where:
#
#   HOSTNAME    - Might either be domain.tld or subdomain.domain.tld
#   OUTPUT_DIR  - This is the directory where the webalizer results
#                 will be stored (HTML and PNG files, etc.)
#   INPUT_LOG   - This is the input log file that will be processed.
#
# examples (*):
#
#   webalizer -p -n example.com \
#             -o /home/user1/domains/example.com/stats \
#             /var/log/httpd/domains/example.com.log
#
#   webalizer -p -n subdom.example.com \
#             -o /home/user1/domains/example.com/stats/subdom \
#             /var/log/httpd/domains/example.com.subdom.log
#
# (*) Note the \ symbol means the command continues in the next line ;-)
#
#
# 1.4. DirectAdmin limitations regarding Webalizer?
# -------------------------------------------------
#
# a) DA does not allow you to use domain specific tokens in the
#    webalizer.conf custom template (see 1.1 above).
#
# b) DA runs the webalizer for all domains and subdomains defined in
#    the system. But, what if you want to use a different Apache log
#    analizer like AWStats, etc.?
#
#
# 1.5. What can we do?
# --------------------
#
# Well, meanwhile DA does not directly support more features related to
# webalizer... we can replace the webalizer executable with a little
# script, so that we can parse the command line and then:
#
# a) Decide if we really want to run webalizer for this domain. This is
#    useful if, instead, you want to run another Apache log analizer.
#
# b) Use a custom webalizer.conf with the ability to use hostname
#    related parameters.
#
# c) Use the same webalizer.conf for all domains or use different .conf
#    files per domain (or even per subdomain). This also allows your own
#    users to create the webalizer.conf file the way they wish.
#
#
# ======================================================================
# 2. INSTALLATION
# ======================================================================
#
# This installation procedure assumes you're logged in as root !!!
#
# 2.1.	Create the file we will use to specify the domains we want to
#		allow or deny using webalizer.
#
#		cd /etc/virtual
#		touch webalizer_domains
#		chmod 660 webalizer_domains
#
#    You should manually edit this file to add or remove domains.
#
#    The syntax of the webalizer_domains file is pretty simple. You can
#    specify a domain in each line. Blank lines are ignored. You can
#    even use comments with the symbol #.
#
#    You can customize the 'meaning' of the webalizer_domains file using
#    the WEBALIZER_DEFAULT parameter in the custom webalizer script.
#
#    WEBALIZER_DEFAULT="allow" means allow webalizer by default. So, you
#    just use the webalizer_domains file to specify the domains that you
#    want to deny from using webalizer.
#
#    WEBALIZER_DEFAULT="deny" means deny webalizer by default. So, you
#    just use the webalizer_domains file to specify the domains that you
#    want to allow to use webalizer.
#
#
# 2.2.	Replace the webalizer executable with this script.
#
#		cd /usr/bin
#		mv webalizer webalizer-orig
#		wget [url]http://files.phpmix.com/directadmin/webalizer[/url]
#		chmod 750 webalizer
#		chown webalizer:root webalizer
#
#
# 2.3.	Customize the custom webalizer script (this script).
#
#	* Please, check out the 'Configuration Section'.
#	* Specially the WEBALIZER_DEFAULT parameter.
#
#
# 2.4.	Install the logrotate script for the new webalizer logs.
#
#		cd /etc/logrotate.d
#		wget -O webalizer [url]http://files.phpmix.com/directadmin/webalizer.logrotate[/url]
#
#
# 2.5.	Customize your webalizer.conf files. Any of the following
#		files is optional. It will be used only if it exists.
#
#	The GLOBAL configuration file should be created here:
#		/usr/local/directadmin/data/templates/webalizer.conf
#
#	DOMAIN configuration files should be created here:
#		/home/$USER/domains/$DOMAIN/stats/webalizer.conf
#
#	SUBDOMAIN configuration files should be created here:
#		/home/$USER/domains/$DOMAIN/stats/$SUBDOMAIN/webalizer.conf
#
#	Note, you should replace $USER $DOMAIN and $SUBDOMAIN with valid
#	values for your installation.
#
#	In those .conf files you can use any of the following tokens:
#		|HOSTNAME|
#		|DOMAIN|
#		|USER|
#
#
# And you're done!
#
# Please, check the chapter 1.3 in README section on how to execute
# webalizer to test the installation.
#
# Remember to check the webalizer_log if something seems to fail.
#
# ----------------------------------------------------------------------
#
# Copyright (c) 2005 Markus, phpmix.com
#
# Note that me nor phpmix.com have any affiliation with DirectAdmin.
#
# This script is released under the GNU GPL v2. You can find a copy of
# the license here: [url]http://www.gnu.org/copyleft/gpl.html[/url]
#
# THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, WHETHER ORAL OR WRITTEN, EXPRESS OR IMPLIED.
#
# ======================================================================
# CHANGELOG:
#
# version 1.0.0 (2005-01-11)
# - Added support for subdomains.
# - Added more and better documentation.
# - Added more information messages to the log (troubleshooting).
#
# version 0.0.1 (2005-01-08)
# - First attempt.
# - It does not support subdomains (yet).
# - It needs better documentation.
# - The idea sounds promising, though. ;-)
#
# ======================================================================
 
Last edited:
This is an example of global webalizer.conf, you should place it here:
/usr/local/directadmin/data/templates/webalizer.conf

Code:
GMTTime		yes
VisitTimeout	1800
DNSCache	/var/www/usage/dns_cache.db
DNSChildren	10

PageType	htm*
PageType	cgi
PageType	txt
PageType	*html
PageType	php*
PageType	asp*
PageType	pl

HideURL		*.gif
HideURL		*.GIF
HideURL		*.jpg
HideURL		*.JPG
HideURL		*.png
HideURL		*.PNG
HideURL		*.ra

TopAgents	30
AllAgents	yes
TopCountries	30
TopReferrers	30
AllReferrers	yes
TopSites	30
TopKSites	20
AllSites	yes
TopURLs		30
TopKURLs	20
AllURLs		yes
TopEntry	30
TopExit		30
TopSearch	20
AllSearchStr	yes
TopUsers	0
AllUsers	no

IgnoreSite	*|DOMAIN|
IgnoreSite	localhost*
IgnoreReferrer	*|DOMAIN|/*
HideReferrer	Direct Request

As you'll see, you can use the token |DOMAIN| in this configuration file. Our webalizer script will replace it with the current domain. This is useful, for instance, to create webalizer rules to ignore self referers, etc.

Comments are welcome. :)
 
To test this script, you can trigger DA to run webalizer for all domains with the following command:

Code:
echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue
 
I tried these modifications but when I execute the command for it to run. It does not seem to be updating. How can I troubleshoot this problem? Thanks.
 
Also,
/etc/virtual/webalizer_domains

does not exist by default, this must be created.
 
rldev said:
I tried these modifications but when I execute the command for it to run. It does not seem to be updating. How can I troubleshoot this problem? Thanks.
Please, look at /var/log/httpd/webalizer_log (or where your WEBALIZER_LOG parameter points to). All webalizer output should be redirected this this log, plus some messages generated by the script itself to easilly identify each executionor whether a domain has been skipped, etc.

You could also test this script calling it from the command line like this:

Code:
/usr/bin/webalizer -p \                       (press enter)
   -n |DOMAIN| \                              (press enter)
   -o /home/|USER|/domains/|DOMAIN|/stats \   (press enter)
   /var/log/httpd/domains/|DOMAIN|.log

This is how DA calls webalizer. Just replace the tokens (|DOMAIN| and |USER|) with somthing else.


rldev said:
Also,
/etc/virtual/webalizer_domains

does not exist by default, this must be created.
If this file doesn't exists, the script behaves (as if it was empty) according to the value of the WEBALIZER_DEFAULT variable.



I have updated the first post on this topic with some notes. Please, keep in mind this is still Beta. I just figured it doesn't work correctly with subdomains.


More soon... :)
 
Last edited:
I get the following error when running webalizer -p

/usr/bin/webalizer: line 112: syntax error near unexpected token `)'
/usr/bin/webalizer: line 112: `below)'

Also there is no log being created.

log location is:

WEBALIZER_LOG="/var/log/httpd/webalizer_log"


I think part of the problem is copying the new webalizer script from the forum. I copied and pasted the script twice and easch time when I run webalizer -p it gives a different line error. Perhaps you can attach this script in a txt file?


After fixing the errors in the webalizer files. I get the following error:

/usr/bin/webalizer: line 83: /etc/virtual/domainowners: Permission denied
/usr/bin/webalizer: line 114: please,: command not found
/usr/bin/webalizer: line 151: logs: command not found
 
Last edited:
I have replaced the code with a link to the file. See top topic. Thanks for taking the time to test this thingy... :)

As per the "permission denied" error:

You could create the file following these steps (as root):

Code:
touch /etc/virtual/webalizer_domains
chown mail:mail /etc/virtual/webalizer_domains
chmod 644 /etc/virtual/webalizer_domains

Syntax is just like /etc/virtual/domains. That is a single domain per line.
 
The problem is:

/usr/bin/webalizer: line 83: /etc/virtual/domainowners: Permission denied

which is referenced in your script.

-rw-r--r-- 1 mail mail 43 Jan 10 10:49 domainowners
 
What do you mean by running the script as root? I am using root. So if I am root shouldn't not matter who owns the files for the purposes of running webalizer -p ?
 
The problem seems to be the permissions on /etc/virtual/domainowners which is set to 644 by default. If you change the permissions to say 750 the script will start but give the following errors:

/etc/virtual/domainowners: line 1: lsmith.com:: command not found
/etc/virtual/domainowners: line 2: rocksolidnet.com:: command not found
 
Last edited:
I don't know but something seem up with the file you posted. I recpoied it from the link you provided and I now get the following errors:

[root@tp2 bin]# webalizer
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options:
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
-m[fr] val
-W compat --compat
-W copyleft --copyleft
-W copyright --copyright
-W dump-variables[=file] --dump-variables[=file]
-W gen-po --gen-po
-W help --help
-W lint[=fatal] --lint[=fatal]
-W lint-old --lint-old
-W non-decimal-data --non-decimal-data
-W profile[=file] --profile[=file]
-W posix --posix
-W re-interval --re-interval
-W source=program-text --source=program-text
-W traditional --traditional
-W usage --usage
-W version --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
/usr/bin/webalizer: line 2: {print$1}: command not found
 
Yup! ...I believe the error is the command you're using. Sorry if I didn't make it clear. I've rewritten the message where it is posted.

Here it is again:
Code:
/usr/bin/webalizer -p \                       (press enter)
   -n |DOMAIN| \                              (press enter)
   -o /home/|USER|/domains/|DOMAIN|/stats \   (press enter)
   /var/log/httpd/domains/|DOMAIN|.log

The three lines are part of the same command. The symbol \ means the command continues in the next line.

Alternatively you could enter:
Code:
/usr/bin/webalizer -p -n |DOMAIN| -o /home/|USER|/domains/|DOMAIN|/stats /var/log/httpd/domains/|DOMAIN|.log
Do you see what I mean?
 
Yes I see what you mean but the errors are similar:

[root@tp2 bin]# /usr/bin/webalizer -p -n |DOMAIN| -o /home/|USER|/domains/|DOMAIN|/stats /v
ar/log/httpd/domains/|DOMAIN|.log
bash: DOMAIN: command not found
bash: -o: command not found
bash: USER: command not found
bash: /domains/: No such file or directory
bash: DOMAIN: command not found
bash: /stats: No such file or directory
bash: DOMAIN: command not found
bash: .log: command not found
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options:
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
-m[fr] val
-W compat --compat
-W copyleft --copyleft
-W copyright --copyright
-W dump-variables[=file] --dump-variables[=file]
-W gen-po --gen-po
-W help --help
-W lint[=fatal] --lint[=fatal]
-W lint-old --lint-old
-W non-decimal-data --non-decimal-data
-W profile[=file] --profile[=file]
-W posix --posix
-W re-interval --re-interval
-W source=program-text --source=program-text
-W traditional --traditional
-W usage --usage
-W version --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
/usr/bin/webalizer: line 2: {print$1}: command not found

domainowners still gives a permission denied so the script can not attatin the proper values to process.
 
Replace the tokens |DOMAIN| and |USER| with proper values.

For instance:
/usr/bin/webalizer -p -n example.com
-o /home/sample/domains/example.com/stats
/var/log/httpd/domains/example.com.log
All should be entered in the same command line. I just splitted the line to avoid breaking the post format.

The "permission denied" error seems like it is trying to write to the file. Using the proper command line should fix it. I believe.
 
Thanks for the help. Same problems. I'll let a few others test this.
 
rldev said:
/usr/bin/webalizer: line 42: /etc/virtual/domainowners: Permission denied
line 42? That one should be a comment. This must be an error releted to the copy/paste operation when I first posted the code (on the first message) instead of a link to the file. :confused:


Anyway, thanks again for taking the time to test it. :)
 
Hi all!

I have been rewriting some things in this script and updated the file and top post on this topic. I've changed the version number to 1.0.0

Changes:
- Added support for subdomains.
- Added more and better documentation.
- Added more information messages to the log (troubleshooting).

I should say I started to write bash scripts a couple of months ago or so. So, perhaps there are things that could be done much better.

I tried to create a script that should not eat much resources, though.

Please, let me know if you can see anything that could improved, suggestions, etc.

Thanks

PS: If it works, just enjoy! :D

PS: I've tried to change to topic title as well. But it didn't worked. :eek:
 
markus said:
PS: I've tried to change to topic title as well. But it didn't worked. :eek:
What would you like it changed to?

I'll see what I can do.

Jeff
 
Back
Top