DirectAdmin bandwidth calculation says more compared to webalizer

[streamservice]

Hello,

After detecting some strange bandwidth reports in DirectAdmin I started to look closer to what was reported by DirectAdmin (compared to webalizer).

DirectAdmin reported (on a system consuming below 20mbit/s, source the switch) that multiple users used over 80TB in 1 day. By simple comparing what network connection would be required to do over 200TB/day on a server I decided that DirectAdmin couldn't be correct in this situation.

After resetting the counters I started to test it on my own site to generate traffic (opening the site and pressing F5).
Webalizer reports: 1426KBytes
DirectAdmin reports: 79.0MB for Apache

For this test I disabled http://[ip]/~[username]

Apache version: 2.4.3
Debian version: 6
DirectAdmin version: 1.42.1

From httpd.conf:

LogFormat "%O \"%r\"" homedir

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%O %I" bytes
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

This is the last problem I can find to our current DirectAdmin setup. What could be the solution to this problem?

 

[streamservice]

Just created a user account with the same name and the same domain on another server and loaded a copy of the logs there. It reports below 2MB bandwidth usage for Apache.

 

[DirectAdmin Support]

Hello,

Hmm... hard to say.

The apache logs that DA uses is at:
/var/log/httpd/domains/domain.com.bytes

Note that DA logs both inbound and outbound traffic, including headers.
Webalizer only includes outbound data, and (I believe) only the data.. not the headers (I could be wrong with the headers bit)

In any case, ensure you're using the most recent version of DA, and type:

cd /usr/local/directadmin/custombuild
./build update
./build rewrite_confs

to ensure you've got a recent copy of the httpd.conf setup, and a the most recent copy of the DA parser.

Lastly, check the bandwidth breakdown:
Admin Level -> Show All Users -> username -> Bandwidth [Details]

which is taken from the files:
/usr/local/directadmin/data/users/username/bandwidth.tally
/usr/local/directadmin/data/users/username/bandwidth.tally.cache (easier to read)

John

 

[streamservice]

Hello John,

After checking the /var/log/httpd/domains/domain.com.bytes files I discovered the problem and why it did only happen on the more busy sites. In our setup we use DirectAdmin in a way it wasn't invented to be used (with a, currently small, cluster). We have multiple web servers and when 2 or more requests did come in in a short period of time it resulted in lines in the .bytes file like the lines below:
23123 3467687787686 657
123 35345
1231 4634654654 67565

Now I added a few additional scripts to first create the .bytes and .log files per server and just before the tally combine them (for the .bytes file this is easy, for the other files it is a little bit more difficult).

On request I can create some documentation regarding our setup and put them online here. That way HA hosting with DirectAdmin will become possible. On irc.freenode.net ##directadmin questions regarding our setup can be asked.