MailGraph HOWTO

BIG! Howto for MailGraph: http://www.stat.ee.ethz.ch/mailgraph.cgi

There are issues at the moment with mailgraph and parsing the exim logfiles. I'll be on a vacation for a couple of weeks. After it i'll try to resolve the issues. If you'll follow the howto completly, the issues with "Useless use of string.... " are resolved. The only left issue is parsing the exim logs


First we need to have to fix 2 dependecies: RRDTool (if you already have it skip it) and File::Tail (which requires Time::HiRes):

* CPAN modules:

- perl -MCPAN -e shell (First time running, keep pressing enter or make your selection until "cpan>")
cpan> install File::Spec
cpan> install Time::HiRes (Could give errors ignore)
cpan> install File::Tail (Could give errors ignore)
cpan> install Date::Calc (Could give errors ignore)
quit

* RRDTool needs Freetype2 (if you don't have it follow instructions below)
- cd /usr/tmp
- wget http://surfnet.dl.sourceforge.net/sourceforge/freetype/freetype-2.3.7.tar.gz
- tar zxvf freetype-2.3.7.tar.gz
- cd freetype-x-x-x
- ./configure
- make
- make install

* RRDTool also needs LibART (if you don't have it follow instructions below)

- cd /usr/tmp
- wget ftp://ftp.gnome.org/pub/GNOME/sources/libart_lgpl/2.3/libart_lgpl-2.3.20.tar.gz
- tar zxvf libart_lgpl-*.tar.gz
- cd libart_lgpl-x-x-x
- ./configure
- make
- make install

* Now we need RRDTool :


- cd /usr/tmp
- wget ftp://ftp.bit.nl/mirror/rrdtool/rrdtool.tar.gz
- tar zxvf rrdtool.tar.gz
- cd rrdtool-x.xx.x
* Ok first determine where freetype and libart is located (if it is in /usr/include/freetype2 do nothing) and run following command:
- export CPPFLAGS="-I[PATH FREETYPE2] -I [PATH LIBART]"
In my case: - export CPPFLAGS="-I/usr/local/include/freetype2 -I/usr/local/include/libart-2.0"


- ./configure (Laugh when notice appears on the end )
- make (Some nice errors could flow over you're screen)
- make install

* I never liked dirs with version numbers on my system so fix the rrdtool dir:

- mv /usr/local/rrdtool-x-x-xx /usr/local/rrdtool

* A lot nicer!
* Download the MailGraph package:

- cd /usr/tmp
- wget http://mailgraph.schweikert.ch/pub/mailgraph-1.14.tar.gz
- tar zxvf mailgraph-exim.tar.gz
- cd mailgraph

* Edit mailgraph.cgi and mailgraph.pl
* Add after comments and after first "use" statements: use lib qw( /usr/local/rrdtool/lib/perl);

# locate librrd.so.2
# ln -s /usr/local/rrdtool/lib/librrd.so.2 /usr/lib/librrd.so.2

Where you'll put in the correct location.

Do the same for libart_lgpl_2.so.2

# locate libart_lgpl_2.so.2
# ln -s /usr/local/lib/libart_lgpl_2.so.2 /usr/lib/libart_lgpl_2.so.2

* Fire it up from shell:

- ./mailgraph.cgi

* Still getting Time/HiRes Errors ?? Manual installation of perl package:

- cd /usr/tmp
- wget http://search.cpan.org/CPAN/authors/id/J/JH/JHI/Time-HiRes-1.65.tar.gz
- tar zxvf Time-HiRes-1.65.tar.gz
- cd Time-HiRes-1.65
- perl Makefile.pl
- export LC_ALL=C
- make
- make test
- make install

* Now all should be fine
* Move it to it's original destination

- mv /usr/tmp/mailgraph /usr/local/mailgraph

* edit the mailgraph-init
* CHANGE: MAILGRAPH_PL=/usr/local/mailgraph/mailgraph.pl (Like that)
* CHANGE: MAILLOG=[you log location normaly /var/log/exim/mainlog]
* CHANGE: RRD_DIR=/var/lib/mailgraph
* Change whatever you like to be different, but the RRD_DIR is hardcoded in the cgi (so you will have to change that too)
*
* edit mailgraph.cgi and change the temp_dir to ex. /tmp


- mkdir /var/lib/mailgraph

* Fire it up:

- sh mailgraph-init start
- ps aux | grep mailgraph (should show up perl -w /usr/local/mailgraph....)

* Ok it now monitors you're maillog for changes and writes the changes to /var/lib what's set in the init script
* Let's set it up on one of you domains:

- cp /usr/local/mailgraph/mailgraph.cgi /home/[user]/domains/[domain]/public_html/cgi-bin/mailgraph.cgi

* Go and test it http://domain/cgi-bin/mailgraph.cgi
* No images ?? (check if you did everything as described and check you're apache error_log)


OK and to enable VIRUS to be calculated with it, change you're deny message to begin with MALWARE

Let me know if it worked

Edited above mailgraph-init changing section. you should use /var/log/exim/mainlog and add -e as parameter.

Do you have an example of this working in a production environment? I'd like to see the results returned (like if it records SPAM messages flagged by spamassassin, etc)

--Josh

jstrzalko said:

Do you have an example of this working in a production environment? I'd like to see the results returned (like if it records SPAM messages flagged by spamassassin, etc)

--Josh

Click to expand...

Hi Josh,

A working URL:

http://www.fusion-ict.nl/cgi-bin/mailgraph.cgi

Spam isn't flagged normally, as these aren't written in the log files of exim.

so I guess the question is... how did you get yours to log SPAM. You using Postfix instead?

Thanks,
--Josh

Hehehe very easy I use blacklists Everything blocked by blacklists is shown as spam

Not really nice, but it will give a nice graph

DO you mind emailing me your exim.conf file (or specifically where your blacklists are and where they go) I'd like to incorporate that into my settings.

[email protected]

Thanks,
--Josh

jstrzalko said:

DO you mind emailing me your exim.conf file (or specifically where your blacklists are and where they go) I'd like to incorporate that into my settings.

[email protected]

Thanks,
--Josh

Click to expand...

Blacklists are already integrated in DA's exim.conf

You only need to add the domains that should use blacklisting in the /etc/virtual/rbl_domains (or something).

After that there are some changes you'll have to make to the scripts of mailgraph

fusionictnl said:

You only need to add the domains that should use blacklisting in the /etc/virtual/rbl_domains (or something).

Click to expand...

The filename is /etc/virtual/use_rbl_domains.

After that there are some changes you'll have to make to the scripts of mailgraph

Click to expand...

Can you document those changes for us?

Jeff

Ok I will do the basic changes to have them count. So you don't have to recode the mailgraph.pl:

You'll have to change the deny messages of you're filters (virus/blacklists etc.)

For blacklist mine are:

deny message = Sorry but you're blacklisted at $dnslist_domain

changed it to:

deny message = SPAM Sorry but you're blacklisted at $dnslist_domain


For virusses:

deny message = This message contains a virus or other malware ($malware_name)

Changed to:

deny message = MALWARE This message contains a virus or other malware ($malware_name)

These entry's are arround line 500.

It's very important to have the strings at the beginning as it scans for that in the exim logs.

For all other ones the strings as followed:

BOUNCED - Bounced messages
REJECTED - Rejected messages
SPAM - Spam messages
MALWARE - Virus messages
MIME_ERROR - Incorrect Mime messages
BAD_ATTACHMENT - Incorrect attachments

We'll this should be enough to get some realistic stats.. Except for spamassassin tagged messages.

Probably a mistake of mine, but when I follow the howto and I try

/usr/local/mailgraph$ sh mailgraph-init start

Click to expand...

It says :

Starting mail statistics grapher: mailgraph
Can't locate RRDs.pm in @INC (@INC contains: /usr/lib/perl5/5.8.1/i386-linux-thread-multi /usr/lib/perl5/5.8.1 /usr/lib/perl5/site_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.1 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.1/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.1 /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.0 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.1/i386-linux-thread-multi /usr/lib/perl5/5.8.1 .) at /usr/local/mailgraph/mailgraph.pl line 218.
BEGIN failed--compilation aborted at /usr/local/mailgraph/mailgraph.pl line 218.

Click to expand...

Any ideas ?

Best regards,
Olivier Baum
http://www.basm.be/

Can someone explain how they got MailGraph to tail the multiple log files that Exim uses? I have mine tailing the "mainlog" file, there fore it doesn't get the bounced or spam messages, as they are logged in the "rejected" file.

The default exim.conf file I built for DA saves all reject information both in mainlog and in rejectlog.

Jeff

os2lover said:

Probably a mistake of mine, but when I follow the howto and I try


It says :



Any ideas ?

Best regards,
Olivier Baum
http://www.basm.be/

Click to expand...

You probaly missed this step:

* Edit mailgraph.cgi and mailgraph.pl
* Add this after the comments: use lib qw( /usr/local/rrdtool/lib/perl ../lib/perl );
* Fire it up from shell:

cwallace said:

Can someone explain how they got MailGraph to tail the multiple log files that Exim uses? I have mine tailing the "mainlog" file, there fore it doesn't get the bounced or spam messages, as they are logged in the "rejected" file.

Click to expand...

They are both as jlasman says logged in mainlog and rejectlog.

fusionictnl said:

You probaly missed this step:

* Edit mailgraph.cgi and mailgraph.pl
* Add this after the comments: use lib qw( /usr/local/rrdtool/lib/perl ../lib/perl );
* Fire it up from shell:

Click to expand...

Ok you're right, I missed the mailgraph.pl
I then followed the rest of the HOWTO
But then I linked it to :

ln -s /usr/local/mailgraph/mailgraph.cgi /home/basm/domains/basm.be/public_html/basm.be/cgi-bin/mailgraph.cgi
and now I get an error 500 ? (http://basm.be/cgi-bin/mailgraph.cgi)

Sorry to be such a pain in the ass.

I get the same error when trying to view the graphs.

Install went ok and I get no errors when run from command line:-
./mailgraph.cgi
or
sh mailgraph-init start

I have checked my /tmp directory and I have no images in there?
the apache error from my logs is this:-

BTW Do the images get updated on the fly on every page load, or do they get made every few minutes or so regardless of if the cgi is run.

Many thanks
Stephen

xgeek said:

I get the same error when trying to view the graphs.

Install went ok and I get no errors when run from command line:-
./mailgraph.cgi
or
sh mailgraph-init start

I have checked my /tmp directory and I have no images in there?
the apache error from my logs is this:-

Code:

[Sat Jan  1 14:20:46 2005] [error] [client xx.xx.xxx.xxx]
 Premature end of script headers: /home/admin/domains/[censored]/public_html/cgi-bin/mailgraph.cgi

BTW Do the images get updated on the fly on every page load, or do they get made every few minutes or so regardless of if the cgi is run.

Many thanks
Stephen

Click to expand...

/var/lib/mailgraph holds 2 rdd files, wich are created by the process "mailgraph-init start" wich are created on change of you're maillogs (realtime almost)

The stats are generated on the fly through rdd cgi file. These hold a rdd database and could be a little bit off the realtime. But still represent a perfect result.

The 500 error is a standard cgi problem and could be off a right problem. Check you're apache logs and suexec logs as they will probably show you where to look.

If you're sure that when executing the CGI file works.

hmm the only error I get are:-

and the apache one in the post above.

I have other cgi scripts that run okay so I am not sure what else to do.

Cheers

Well in the apache error log I also get :

[Sat Jan 1 15:07:28 2005] [error] [client 82.35.241.144] Premature end of script headers: /home/basm/domains/basm.be/public_html/cgi-bin/mailgraph.cgi

What could be causing this ?