High disk usage, why?

[patrik]

I'm having some problem with one of our servers which reports very high disk usage. I'm using the tool gstat, check output:

dT: 0.510  flag_I 500000us  sizeof 240  i -1
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    9     73     47    501  150.0     25    965  105.8   98.8| ad4
    9     73     47    501  150.0     25    965  105.9   98.8| ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0| ad6
    9     73     47    501  150.0     25    965  105.9   98.8| ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0| ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0| ad4s1c
    0      0      0      0    0.0      0      0    0.0    0.0| ad6s1


dT: 0.510  flag_I 500000us  sizeof 240  i -1
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
   14     86     53   3869   78.0     33    329   56.5  103.7| ad4
   14     86     53   3869   78.0     33    329   56.5  103.7| ad4s1
    0      0      0      0    0.0      0      0    0.0    0.0| ad6
   14     86     53   3869   78.0     33    329   56.5  103.7| ad4s1a
    0      0      0      0    0.0      0      0    0.0    0.0| ad4s1b
    0      0      0      0    0.0      0      0    0.0    0.0| ad4s1c
    0      0      0      0    0.0      0      0    0.0    0.0| ad6s1

501kBps + 965kBps = 1466kBps = 11728kBitps = 11.4MBitps
11.4 mbit per second shouldn't be a problem for SATA disk, right? I think this is really strange and don't really know how to troubleshoot. top doesn't report that any process is using very much CPU or mem.
Here's an example of top -m io (ordered by total):

  PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
73383 mysql         210    660     50     23      0     73   6.23% mysqld
74967 root           31     22      0     26      0     26   2.22% imapd
73638 apache         16     12      7      0      7     14   1.20% httpd
73664 apache         18     50      8      0      2     10   0.85% httpd
74406 apache         14     10      4      0      4      8   0.68% httpd

Anything strange here? Mysqld has the highest percent almost all the time but sometimes clamd or imapd takes the first place. I can't see anything wrong here, do you?
I have tried to stop apache, exim, clamd, mysqld, vm-pop3d, bacula-client, but with no luck. All of a sudden the disk usage drops again and everything seems normal.
I need some help here, thanks.

 

[lacrimos]

edited!edited!edited!edited!edited!edited!edited!edited!edited!

 

[patrik]

With gstat as I wrote. gstat comes with newer versions of FreeBSD.