Bandwidth problems

N

netsolid

Verified User
Joined
Feb 8, 2010
Messages
35
One VPS, a centos 5.5 with DA1.37 gives incorrect bandwidth calculations.

When i check the totals, the total from the eth card is 82Gb, while the current usage is 110Gb.
One account has according to his account bandwidth used 29,2Gb this month with apache is 29.1Gb. When we check the awstats the usage is almost 5Gb wich is normal for that account.

When i check the usage of the VPS the bandwith is 115Gb.

So i have few questions. The first on is why is the bandwidth for this account so high in DA. How does DA calculate the account trafic and how does AWstats calculate the trafic. There is just a little bit FTP trafic on this account. Where can i check where this usage is coming from.

The second one is why is there a difference between the bandwidth of the eth card and the current usage. Its almost 30Gb.

I hope someone can help me out.
 
@netsolid:

How are you getting the figures from your eth card? Are you also figuring all logical eth ports, for example eth0:1, etc.?

Jeff
 
Hi jeff,

Those figures are coming from DA and from the VPS panel. But theres something very weird going on. Our customar is now user 10Gb a day according to DA, to AWstats and webalizer the normale bandwith of 200Mb a day.

We cant figure out where DA is calculating this amount from. The logs are all showing less trafic than DA.
 
I don't have an answer for you; perhaps someone else will respond.

Have you contacted DirectAdmin Support?

Jeff
 
Hello,

Check the "Details" for the bandwidth when viewing the User, eg:

Admin Level -> Show All Users -> username -> Bandwidth: Details

and let us know which service/aspect is using up the data.


Note, if you've got an old exim.pl/exim.conf file, then it's possible that emails stuck in your spool queue are being billed to the User for each attempt. The newer exim.pl file will log the message ID and DA will remove duplicates. (will only work for messages sent *after* the new exim.pl is installed....)
http://help.directadmin.com/item.php?id=51


In any case, check:
/usr/local/directadmin/data/users/username/bandwidth.tally

and bandwidth.tally.cache (same as the "Details" button) to see what shows up there. FYI, any entries with just numbers and no details will be emails. The exim.pl will include more info (include the message ID) with each email delivery.


Also keep in mind that the webalizer/awstats bandwidth usage only logs apache.
DA logs much more than just apache (but if the value is higher than your device, something might be off)

John
 
Here are the details, like i noticed its all apache.

Code: 
Date Apache Email Ftp DirectAdmin Other Total Email Deliveries 
2011 02 01 11.34 GB 45.9 KB 0.00 KB 6.75 MB 0.00 KB 11.35 GB 39 
2011 02 02 378.6 MB 4.74 KB 0.00 KB 328 KB 0.00 KB 379.0 MB 4 

total 11.71 GB 50.7 KB 0.00 KB 7.07 MB 0.00 KB 11.72 GB 43

Code: 
The tally.cache:
2011 02 01=da=7080241&email=47046&email_count=39&email_count_incoming=0&email_count_outgoing=39&ftp=0&http=12175040539&http_ul=84891077&other=0&total=12182167826
2011 02 02=da=335431&email=4850&email_count=4&email_count_incoming=0&email_count_outgoing=4&ftp=0&http=397039638&http_ul=2152339&other=0&total=397379919
simpletotal=12579547745
total=da=7415672&email=51896&email_count=43&email_count_incoming=0&email_count_outgoing=43&ftp=0&http=12572080177&http_ul=87043416&other=0&total=12579547745

If you would like to see the bandwidth.tally i can mail it to you.
 
Hello,

That narrowed it down greatly.
We can see that the usage is pointed at apache.

DA counts both the domain usage, but also counts the /~username access.
Check /var/log/httpd/homedir.log for any excessive entries for the given User (the homedir.log is rotated nightly, so is not a historical log)

The /~username access is not counted by awstats/webalizer, but is counted by DA.


It's also possible that the /var/log/httpd/domains/domain.com.bytes files were very large and not tallies/rotated for some time. If this happens, then the first tally on it would include the entire bytes log, which may span more than 1 day (which is why it's important to ensure things are working smoothly). Webalizer/awstats would show normal spans since the dates are in each log entry, however the bytes log doesn't have a date/time. If we had "details" for the previous month, we would expect to see holes for the days that didn't get computed (if this is in fact affecting you.. this is just a guess)

John
 
The homedir.log shows nothing special and the bytes file is also small. Is there somethings else we can check.
 
Hello,

The homedir.log would be for "todays" usage, so likely anything of relevance is probably gone, as far as logs go. From this point, I would say to monitor the usage and see if it remains in the 300 MB to 1G range (whatever is normal). If this range continues and only the 1st day of the month is off, then my guess would be my previous one, where the domain.com.bytes was leftover from a previous time (but cannot confirm this).

In any case, if you want to manually lower that value, edit the bandwidth.tally file and change the "http_bw_total" for the first "type=tally" line. Set it to a rough guess as to what the bandwidth *should* be, and when the next full tally runs, the usage will go down to the value you've set (plus other days, etc).

From then, you'd really only be able to keep monitoring everything to see if the issue happens again.

John
 
Hi John,

This doesnt accur on the first of the month, last month is was random on different days. I checked and there was no pattern i could detect. This all started last december. When i check awstats it shows normal bandwidth but when i check the googlebot bandwidth of this month in awstats is almost 11Gb.

How can we log the exact usage in apache without DA rotating the logs. We want to know what is using all this the bandwidth.
 
ahhhhhh now i noticed how awstats is calculating the bandwidth. And i missed a row.

This is what awstats is showing:

Code: 
Unique visitors Number of visits Pages Hits Bandwidth Viewed traffic 
* 420  534(1.27 visits/visitor) 2375 (4.44 Pages/Visit) 25151(47.09 Hits/Visit) 430.81 MB (826.11 KB/Visit) 
Not viewed traffic * 37701 38955 11.23 GB

I missed the not viewed trafic :rolleyes:

The next question is why google is visiting that much. But that no DA issue.
 
Just my two cents. I've never liked and trusted bandwidth calculating based on parsing logs (not going to hurt someone's feelings). I do not know how actual is it now, but the last time I checked, I noticed one ugly thing, that Awstats shown too great numbers.

When somebody downloads a big file in several streams, Awstats shows as if that file was downloaded not once, but twice or even more. Thus bandwidth gets too high in Awstats.

I'm not sure if it is the same with Directadmin.
 
Hello,

Apache if the %b option is used with apache bytes logging, each trigger of a download will log the entire size of the file, even if the file isn't downloaded all the way. Download accelerators that break a file into 5 parts (started at different spots) will trigger 5x the bandwidth actually used, if the %b option is used. We used to use this a long time ago.

Now, we use the %O option, which logs the actual traffic from the pipe, and not the size of the file. (logging is done post-transfer). Check /etc/httpd/conf/httpd.conf, and you should have something like:
Code: 
LogFormat "%O %I %t" bytes
The %t is fairly new, so you might not have it. DA logs the uploaded apache traffic as well now (1.37.0), if %t is set.

John
 
We've been experiencing same issue starting this month on one of our servers.

Random accounts will have extraondinary bandwidth jumps (accounts that use 3-4gb a month jump to 30gb overnight). We know this is NOT real traffic as our RTG graphs show no spikes in traffic to account for this...it would be obvious if a sudden 200-300gb daily jump in traffic were to happen.

I'v discovered instances of multiple dataskq's going back days in teh process list which MAY be what's happening..but not 100% sure.

In the mean time, anyone have a quick and dirty method of disabling the suspend at limit flag on all accounts globally?
 
Hello,

Click the "Details" next to the bandwidth usage for the given User to see a breakdown of daily usage, and on which day is happening. Once what and when is know, the logs can be cross-references for that day/time to see if the specific cause can be found.
In the mean time, anyone have a quick and dirty method of disabling the suspend at limit flag on all accounts globally?
Click to expand...
Yes:
Code: 
cd /usr/local/directadmin/data/users
perl -pi -e 's/suspend_at_limit=ON/suspend_at_limit=OFF/' */user.conf
John
 
John, been there done that. Don't forget, I've been managing DA for almost 7 years now :). This is NOT any specific day..seen it 5/2, 5/3, 5/4, 5/5...5/10.

Different set of users each day, never same ones twice. Random sites within their accounts. THis is a DEFINITE issue with dataskq getting stuck and looping...only logical cause I can find.

Thanks for the perl patch.
 
If you spot the dataskq sitting for too long, type:
Code: 
killall -USR1 dataskq
to tell it to dump it's current processing location to the errortaskq.log.

John
 
You must log in or register to reply here.
Back
Top