Webalizer not compiling stats for one domain.

[Jeff_Infopop]

I'm having an issue with Webalizer on one of our machines. It seems that it has decided to update the stats for all of the domains on the server accept for one.

I've run this:

echo 'action=tally&value=all' >> /usr/local/directadmin/data/task.queue

and the process completes, but still no stats are compiled for the one domain.

I can manually run webalizer like so:

/usr/bin/webalizer -p -n domain.com -o /home/user/domains/domain.com/stats /var/log/httpd/domains/domain.com.log

and it updates the stats.

Is there some switch I have to throw somewhere for the cron to process for this domain? Or am I going to have to setup my own cron for this single domain's webalizer?

 

[DirectAdmin Support]

Hello,

Try:

echo 'action=tally&value=[b]username[/b]&type=user' >> /usr/local/directadmin/data/task.queue

To see if the task queue just isn't making it that far.

John

 

[Jeff_Infopop]

Ok, that worked, so I looked for the cut off point. It looks like this is happening for the last 19 domains on this server.

I ran the tally all task again and then checked the directadmin errortaskq.log and this is all I found.

2004:03:23-11:16:00: DaTaskQ is already running. Delete the file /var/run/dataskq.pid to reset

Is there another log file somewhere that's going to tell me why or where it is choking up?

 

[WilcoOnline]

Strange i got the some problem

/var/run/ it doesn't have a dataskq.pid file but the error log say it's so

 

[DirectAdmin Support]

Hello,

the "DaTaskQ is already running" error is just because the dataskq runs once per minute. If the tally take more than one minute (which it almost always does) then the dataskq program that is run in the 2nd minute will check for the pid file and quit so that we don't get 2 instances running. There is only a problem if the pid file exists and *no* instances of the dataskq are running. In which case, just delete the pid file. (the dataskq will automatically delete the pid file if it's more than a few hours old .. i'd have to check the exact time, but it does delete it)

John

 

[Jeff_Infopop]

Yeah, I figured it was something like that. That isn't the reason the cron stops working before it finishes though.

Do you have any ideas where I could start looking?

 

[DirectAdmin Support]

Hello,

If you are up to it, you could try this:

cd /usr/local/directadmin

type "date" to get the date of the server.. wait until it's just past second "0" so that the dataskq won't run what you do.

Quickly, paste:

echo 'action=tally&value=username&type=user' >> /usr/local/directadmin/data/task.queue

then run:

./dataskq d

and check for any strange output. You need to run that before the time pases to the next minute, or else the cron'd dataskq will run the action=tally command before you.

Also, have a look for any weird usage total in case (/usr/local/directadmin/data/users/username/user.usage)

and check for the date last modified for the files in the users directory in case the taskq is quitting prematurely, then we can tell what it did last. (/usr/local/directadmin/data/users/username/* and ./domains/*)

John

 

[Icheb]

I have the same problem but with > 10 users...

So i tried a total retally, and than started debug mode:

[root@horus directadmin]# ./dataskq d
In debug mode
root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Warning: Truncating oversized request field
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Segmentation fault

Don't think that's supposed to happen...
Also, the errorlog generates a nice:
2004:03:25-00:10:59: DaTaskQ is already running. Delete the file /var/run/dataskq.pid to reset
entry

This is been happening since the 17th of this month.

edit:
To help debug, i've ran all users manually with the described method, no crashes, just a few warnings...
So it must be somewhere in DA's other tally modules. (Am i right when i say not all traffic things are updated, example, i've a user called x, in 'show all users' this user has used 1.1 GB, when i go to the user details it's 5.9 GB...

Possible bug in DA ?

 

[DirectAdmin Support]

Hello,

Yes, that look like a bug, it should never segfault, so if it does, it means it's wandering out of it's memory space. We just need to find out why. If you have gdb (debugger program).. you can do the same thing but run "gdb dataskq", then "run d" . It should run a bit, then will give you a segfault message again. You can type "bt full" to get a full backtrace. We'll need to figure out where the segfault is occuring, so if you want us to hunt it down for you, just send us your IP and root password to [email protected], or use https://www.directadmin.com/clients/safesubmit.php.

I've added the bug to the versions system, but don't have any info to describe the bug yet.

Thanks,

John

 

[Icheb]

Alright, I'll start testing and preparing to submit immedately...

edit:
Not sure where to put this information;

[root@horus directadmin]# /root/retally.sh
[root@horus directadmin]# gdb dataskq
GNU gdb Red Hat Linux (5.2.1-4)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
(gdb) run d
Starting program: /usr/local/directadmin/dataskq d
In debug mode
root priv set: uid:0 gid:0 euid:0 egid:0
pidfile written
staring queue
Warning: Truncating oversized request field
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record
Error: Skipping oversized log record

Program received signal SIGSEGV, Segmentation fault.
0x081a4ec3 in free ()
(gdb) bt full
#0  0x081a4ec3 in free ()
No symbol table info available.
#1  0x08183123 in operator delete(void*) (ptr=0x40002ff8)
    at ../../../../libstdc++-v3/libsupc++/del_op.cc:39
No locals.
#2  0x0818314f in operator delete[](void*) (ptr=0x40002ff8)
    at ../../../../libstdc++-v3/libsupc++/del_opv.cc:36
No locals.
#3  0x080705fd in delString(char*&) ()
No symbol table info available.
#4  0x08070625 in delString(char*&, char*&) ()
No symbol table info available.
#5  0x080b6b84 in Tally::tallyReseller(int) ()
No symbol table info available.
#6  0x080ba268 in Tally::tallyHo(ConfigFile*, int) ()
No symbol table info available.
#7  0x080b5789 in Tally::tallyAll(int) ()
No symbol table info available.
#8  0x080ba25c in Tally::tallyHo(ConfigFile*, int) ()
No symbol table info available.
#9  0x080e4100 in executeCommand(char const*) ()
No symbol table info available.
#10 0x080e5633 in processQueue(char const*) ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#11 0x080e5739 in doQueue() ()
No symbol table info available.
#12 0x080e5b23 in main ()
No symbol table info available.
#13 0x08188abe in __libc_start_main ()
No symbol table info available.
(gdb)

If DA support still needs access, please let me know in a reaction, i'll submit it than immedately.

 

[DirectAdmin Support]

Thanks for the quick response. I'll start tracking that immediately. If I need more info, I'll let you know. If we need more details, we might put a debug binary on there, which is essentially the same thing, but gives far more accurate information. (line numbers etc)

John

 

[Icheb]

Ok, the faster this is solved, the better...
Starting to get a total chaos here, the admin panel doesn't update values, so i can't see who exactly is suspended, users start complaining why there only is a tally every 2 days etc (manual tally, user by user seems to work ok)

 

[DirectAdmin Support]

Hello,

If you could send your root login info to https://www.directadmin.com/clients/safesubmit.php, including the reseller name that doesn't get updated, I'll try and track it down based on the values that are being used.

John

 

[Icheb]

Done

 

[DirectAdmin Support]

Thanks for the feedback. Bug has been found and fixed. Workaround can be found here:

http://www.directadmin.com/features.php?id=351

John

 

[Icheb]

Thanks for the feedback. Bug has been found and fixed. Workaround can be found here:

http://www.directadmin.com/features.php?id=351

John

Thank you for the quick response

 

[Turbo M]

i am having this same problem and actually just had DA installed a week ago from NAC.net

how is this fixed? I didn't really understand that link.

 

[klaver]

dataskq segmentation fault

Got a similar error, different output:


jupiter:/usr/local/directadmin# gdb dataskq

GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

(gdb) run d
Starting program: /usr/local/directadmin/dataskq d
In debug mode
Debug set to level 10
root priv set: uid:0 gid:0 euid:0 egid:0

Program received signal SIGSEGV, Segmentation fault.
0x08085641 in errorLog ()
Current language: auto; currently asm
(gdb) bt full
#0 0x08085641 in errorLog ()
No symbol table info available.
#1 0x0814d478 in main ()
No symbol table info available.
(gdb)

 

[klaver]

and some strace output:


jupiter:/usr/local/directadmin# strace ./dataskq d
execve("./dataskq", ["./dataskq", "d"], [/* 20 vars */]) = 0
uname({sys="Linux", node="jupiter.grafix-is.com", ...}) = 0
brk(0) = 0x83d8000
brk(0x83f9000) = 0x83f9000
socket(PF_FILE, SOCK_STREAM, 0) = 4
connect(4, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
close(4) = 0
open("/etc/nsswitch.conf", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=465, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
read(4, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 465
read(4, "", 4096) = 0
close(4) = 0
munmap(0x40000000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=22572, ...}) = 0
old_mmap(NULL, 22572, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40000000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libnss_compat.so.2", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000\22\0"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0644, st_size=28616, ...}) = 0
old_mmap(NULL, 31628, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40006000
old_mmap(0x4000d000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x6000) = 0x4000d000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libnsl.so.1", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 <\0\000"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0644, st_size=73304, ...}) = 0
old_mmap(NULL, 80544, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x4000e000
old_mmap(0x4001f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x11000) = 0x4001f000
old_mmap(0x40020000, 6816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40020000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`Z\1\000"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0755, st_size=1254660, ...}) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0x83d9ba0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
old_mmap(NULL, 1264972, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40022000
old_mmap(0x4014c000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x129000) = 0x4014c000
old_mmap(0x40155000, 7500, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40155000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/ld-linux.so.2", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\f\0\000"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0755, st_size=88904, ...}) = 0
old_mmap(NULL, 88464, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40157000
old_mmap(0x4016c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x15000) = 0x4016c000
close(4) = 0
munmap(0x40000000, 22572) = 0
brk(0) = 0x83f9000
brk(0x841a000) = 0x841a000
brk(0) = 0x841a000
open("/etc/nsswitch.conf", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=465, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
read(4, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 465
read(4, "", 4096) = 0
close(4) = 0
munmap(0x40000000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=22572, ...}) = 0
old_mmap(NULL, 22572, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40000000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libnss_nis.so.2", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\34\0"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0644, st_size=33440, ...}) = 0
old_mmap(NULL, 36620, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x4016d000
old_mmap(0x40175000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x7000) = 0x40175000
close(4) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libnss_files.so.2", O_RDONLY) = 4
read(4, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\35"..., 512) = 512
fstat64(4, {st_mode=S_IFREG|0644, st_size=34748, ...}) = 0
old_mmap(NULL, 38044, PROT_READ|PROT_EXEC, MAP_PRIVATE, 4, 0) = 0x40176000
old_mmap(0x4017f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 4, 0x8000) = 0x4017f000
close(4) = 0
munmap(0x40000000, 22572) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40000000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40000000, 8217) = 0
close(4) = 0
setresuid32(-1, 0, -1) = 0
setresgid32(-1, 0, -1) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40000000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40000000, 8217) = 0
close(4) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40000000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40000000, 8217) = 0
close(4) = 0
setresgid32(-1, 106, -1) = 0
setresuid32(-1, 106, -1) = 0
stat64("/usr/local/directadmin/conf/directadmin.conf", 0xbffff9b0) = -1 EACCES (Permission denied)
open("/usr/local/directadmin/conf/directadmin.conf", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)
setresuid32(-1, 0, -1) = 0
setresgid32(-1, 0, -1) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40000000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40000000, 8217) = 0
close(4) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40000000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40000000, 8217) = 0
close(4) = 0
setresgid32(-1, 65534, -1) = 0
setresuid32(-1, 65534, -1) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
write(1, "In debug mode\n", 14In debug mode
) = 14
write(1, "Debug set to level 10\n", 22Debug set to level 10
) = 22
setresgid32(-1, 0, -1) = 0
setresuid32(-1, 0, -1) = 0
getegid32() = 0
geteuid32() = 0
getgid32() = 0
getuid32() = 0
write(1, "root priv set: uid:0 gid:0 euid:"..., 41root priv set: uid:0 gid:0 euid:0 egid:0
) = 41
geteuid32() = 0
getegid32() = 0
setresuid32(-1, 0, -1) = 0
setresgid32(-1, 0, -1) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40001000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40001000, 8217) = 0
close(4) = 0
open("/etc/passwd", O_RDONLY) = 4
fcntl64(4, F_GETFD) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
_llseek(4, 0, [0], SEEK_CUR) = 0
fstat64(4, {st_mode=S_IFREG|0644, st_size=8217, ...}) = 0
mmap2(NULL, 8217, PROT_READ, MAP_SHARED, 4, 0) = 0x40001000
_llseek(4, 8217, [8217], SEEK_SET) = 0
munmap(0x40001000, 8217) = 0
close(4) = 0
setresgid32(-1, 106, -1) = 0
setresuid32(-1, 106, -1) = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++



jupiter:/usr/local/directadmin# ls -al

total 11752
drwxr-xr-x 7 diradmin diradmin 4096 2006-11-15 12:18 .
drwxrwsr-x 15 root staff 4096 2006-04-03 02:10 ..
drwx------ 2 diradmin diradmin 4096 2006-11-15 12:27 conf
drwx------ 6 diradmin diradmin 4096 2006-10-24 16:22 customapache
-rwx------ 1 diradmin diradmin 154946 2006-11-11 01:00 da-popb4smtp
drwx--x--x 8 diradmin diradmin 4096 2006-11-17 11:23 data
-rwx------ 1 diradmin diradmin 3886007 2006-11-11 01:00 dataskq
-rwx------ 1 diradmin diradmin 3872974 2006-11-11 01:00 directadmin
drwx------ 2 diradmin diradmin 4096 2005-06-01 15:37 log
-rwx------ 1 diradmin diradmin 13190 2006-11-11 01:00 logger
drwx--x--x 4 diradmin diradmin 4096 2006-11-15 12:37 scripts
-rw------- 1 diradmin diradmin 4052936 2006-09-25 16:30 update.tar.gz



jupiter:/usr/local/directadmin# ls -al conf

total 24
drwx------ 2 diradmin diradmin 4096 2006-11-15 12:27 .
drwxr-xr-x 7 diradmin diradmin 4096 2006-11-15 12:18 ..
-rw------- 1 diradmin diradmin 1577 2006-11-15 12:27 directadmin.conf
-rw------- 1 diradmin diradmin 1298 2006-11-15 12:35 license.key
-rw------- 1 diradmin diradmin 1298 2005-06-01 15:22 license.old
-rw------- 1 diradmin diradmin 30 2005-06-01 15:25 mysql.conf

 

[DirectAdmin Support]

Hello,

Login to your DirectAdmin Clients section... and check the OS that's set in your license. We usually see segfaults when the binaries for a different OS are being used.

John