imap, very high load and slow imap

Today our main server had a bit of a problem, one user has an mailbox of 58 MB, and uses Imap.
I've already received complaints about how slow it is, but when he tried to retrieve his mail today, the server load jumped up to somewhere around 78.89, after trying to kill services and solve it without an reboot, the load went up to 144.00.
Please, is there some way to solve these issues ?

No longer supporting Imap is a valid solution, but I don't think my customers will like that. But I don't like the idea every time this user fetches the new messages the load jumps up and the server isn't really responding anymore.
It's a freakin' PIV 2.4 GHz, it should be able to handle Imap connections...

Can someone please help me with a way to solve this ?

While doing a test on the server when the begin load was 0.02, it jumped up to about 2.30 when the user tried to fetch his mail using outlook express...

This has to do with the awful combination of the mbox format, imap and the uw-imap daemon.

wdv said:

This has to do with the awful combination of the mbox format, imap and the uw-imap daemon.

Click to expand...

That's indeed what I guessed after doing a search on the DA forums, but am I really the only one having problems with this ?
For the time being I've asked the customer to stop checking mail while we're at peak hours (normal server load at peak hours for that server is 0.89, but mysql starts committing suicide if imapd starts processing that mailbox together with more than 12 query's/sec, in combination with a busy apache server).

I don't think you are the only one having these problems. It's just "impossible" to switch now to maildir + another imap daemon like courier.

Why is it impossible? Some have claimed to have done it. Unfortunately they have not made the information public. It's not impossible, but it won't be easy though. This is a very real problem that is going to become more and more of a problem in the future. I have plenty of clients that solely use imap and have mailboxes bigger then 58MB.

UWIMAP and MBOX really hinder what is an otherwise great product. Since email is one of the most important aspects of web hosting, I believe DA should offer us a alternative solution. Mbox can be converted to Maildir. Easy, no but I think this is more important then adding another OS.

I know if I told my clients we had to change the mail format to greatly improve performance, most would have no problem with it. I rather get yelled at for a week then to get yelled out for the duration I offer DA

I urge anyone experiencing the same problems and wishes such a change to email DA. I can't think of anything that would improve DA more(maybe DNS clustering).

rldev said:

Why is it impossible? Some have claimed to have done it. Unfortunately they have not made the information public. It's not impossible, but it won't be easy though. This is a very real problem that is going to become more and more of a problem in the future. I have plenty of clients that solely use imap and have mailboxes bigger then 58MB.

UWIMAP and MBOX really hinder what is an otherwise great product. Since email is one of the most important aspects of web hosting, I believe DA should offer us a alternative solution. Mbox can be converted to Maildir. Easy, no but I think this is more important then adding another OS.

I know if I told my clients we had to change the mail format to greatly improve performance, most would have no problem with it. I rather get yelled at for a week then to get yelled out for the duration I offer DA

I urge anyone experiencing the same problems and wishes such a change to email DA. I can't think of anything that would improve DA more(maybe DNS clustering).

Click to expand...

I totally agree with you (Oh btw, I've got DNS clustering running here, it's rather easy, but a bit dangerous without the precautions that aren't written yet).

My next question would be; why aren't other imap daemons able to read the format DA uses ?
I do seem to remember Maildir uses different files for each mail instead of one giant file, but shouldn't these daemons be able to handle it ?
(Why aren't there any temp files or something that say 'E-mail x begins on line y and ends on line z', that way only lines which doens't have that are needed to be read on an incremental update in the mail client).


At the moment I'm thinking of asking the customer in question to login useing POP3, download everything from his IN folder and dele it after he's done, so that the imap is empty. That should give some more speed to it, although I hate to offer it as solution.

Maildir is a must. Just patching something up to read mbox is not a proper solution.

Well the good news is that DA does agree that a change needs to inevitably be made. Don't expect it soon though, but it is on the todo list.

I know Jeff has been kind enough to put up a test server for someone. Perhaps someone will be able to assist with this in order to help the DA team out and get the ball rolling.

I wish i had time to dev for this.
But i, for the time being, don't even have enough time to finish programming projects we're developping for DA server admins/DA servers.

I know the DA dev team is kinda busy and would/should welcome all help, so now we have to wait for someone that actually has the time to do something about it...

I'm digging into this... courier imap and maildir is kind of a must for me.

If there's a test server I can play on that would definitely help. Can someone point me at info about how quotas for mail is currently calculated?

Jeff Lasman who you will always see on this board had a test server up. You may want to ask him.

ballyn said:

I'm digging into this... courier imap and maildir is kind of a must for me.

If there's a test server I can play on that would definitely help. Can someone point me at info about how quotas for mail is currently calculated?

Click to expand...

Quota is set on the mailbox size and is set via the "quota" parameter in the exim.conf. The lastest exim.conf only seems to have a quota on virtual mailboxes.

I am looking back into the courier support. It took a few minutes of looking at it, to remember where I left off months ago. Hopefully I can have something to do some testing with in the near future.

ballyn said:

If there's a test server I can play on that would definitely help. Can someone point me at info about how quotas for mail is currently calculated?

Click to expand...

If anyone wants to work on getting DA to work with Maildir, I'd be happy to donate the use of a test server, and one static IP#.

I'll try to get DA to donate a test license.

Let me know.

Jeff

I've put together a quick howto on using courier-imap...

http://www.directadmin.com/forum/showthread.php?s=&threadid=6968

Here is what I have so far if someone wants to do some testing/cleanup. You would just replace the preauthcustom.c in the courier-authlib and it should just work. This version uses /home/${domainowner}/virtual/${domain}/${virtualuser}/Maildir as the Maildir. There may be some bugs and I'm sure there are, but it seems to authentiacate correctly using authtest but I did not test it fully.

I see a few things in this version that need to be fixed due to the scope of some variables (like pw). Also the version of courier-authlib tha I used was 0.53 if that makes a difference.