Quota exceeded and dovecot

kristian

Verified User
Joined
Nov 4, 2005
Messages
436
Location
Norway
Hi,

I'm not sure if this problem applies to installations without dovecot, but as I've only seen it with, I'm posting here.

When a customer has used up all his diskspace according to quota-settings, I get various error-messages in my log:

Feb 19 10:54:06 web1 dovecot[4232]: POP3([email protected]): write_full(/home/user/imap/domain.tld/popaccount/Maildir/dovecot-uidlist.lock) failed: Disk
quota exceeded

Feb 19 10:54:06 web1 dovecot[4232]: POP3([email protected]): Couldn't init INBOX: Internal error occurred. Refer to server log for more informat
ion. [2007-02-19 11:54:06]

Also seeing this alot:

Feb 19 09:50:08 web1 dovecot[4232]: POP3([email protected]): Couldn't init INBOX: Unknown error

And:

Feb 19 11:45:50 web1 dovecot[4232]: POP3([email protected]): broken sync positions

Feb 19 11:45:50 web1 dovecot[4232]: POP3([email protected]): Fixed index file /home/user/imap/domain.tld/popaccount/Maildir/dovecot.index: log file syn
c pos 0,0 -> 2, 24

The customer has huge problems POPing their mail due to this, is this working as intended? I see the various .lock and .index-files are owned by the user, so I assume they are counted towards the quota.

My dovecot-version is rc22.

Anyone have any ideas on how to ease the pain?

Thanks,
 
We saw this problem when we moved from the old mbox system to the new Dovecot Maildir system.

We traced it to the fact that our CentOS installs were by default implementing quotas on the /home partition but not on the /var partition. Since Dovecot mail has moved from /var to /home, and now email is triggering hard quota limits.

We resolved it with a combination of client education and larger quotas.
 
I have seen this kind of behavior on my existing Ensim system where an account would have issues if it goes over the alloted quota. I believe this to happen because quotas are enabled for the /home directory. Since this same thing happens on my Ensim server, I might have a few things you can try if you are still having issues.

One thing that has worked for me in the past is to increase the quota for each mail account and educate the user about the use of e-mail. If an account does go over it's quota and they can't POP their account, try having them access SquirrelMail, which uses IMAP to access the account, and have them delete some junk mail that might be in their inbox. If they get an error when they try to login or delete any messages, which most times will happen, temporarily increase the e-mail accounts quota and have them connect again. When they are finished downloading their mail, put the quota back the way it was.

The big thing is educate the user and tell them that they need to check their mail more often if they know they are getting a lot of mail.

I'm not sure if DirectAdmin with Dovecot handles this the same way or if this will work for you, but this is something that you can try if you still experience issues.

Hope this helps. Thanks.
 
We resolved it with a combination of client education and larger quotas.hen we found thi

Larger quotas is just pushing the problem further away, it doesn't remove it, and if we're to increase quotas temporarily while users can clean up their mail, it will involve extra work which I'm not too sure is the right way to go.

As for education - yes. I basically agree, but customers expect things to work, and when things break, they blame us. I don't feel it's a good thing to blame them back for having poor mail-habits or greedy with the hosting-package they choose.. :) And then there's the issue with a domain with maybe hundreds of email-accounts - one full pop-account can mess things up for everyone else, and I don't want to spend time educating 100 people.. And what about vacation? People have vacations all the time, and when they return, they'll have a problem (and possibly lost mail too, see next paragraph).

I think there is a need for a technical solution to this problem. Personally, I would like to see an option to exclude mail from quotas entirely, to prevent blocking new mails when quotas are exceeded. Blocking mails seems like a "cheap" solution to me.

At the very least, it should be possible to have the dovecot control-files (.index and .lock and the like) to be excluded from the quotas (owned by the system instead of the user, perhaps?).

I don't know how much, if anything, is tweaked by the DA-team before implementing dovecot, but is this something that can be done?

(Btw, your last line(s) were chopped it seems.)
 
So what you want is to have quotas but to ignore them? While you can do that, I still think the answer is education. People need to know that their limits are just that, limits.

It's important to realize that the only quotas that affect email delivery are those imposed by your OS distribution. You can turn off quotas for your /home partition if you want, and then DA will report on quotas nightly but email will still work.

You ask if the control files can be excluded from the quotas? Dovecot writes the those files as the user. So they have to have some combination of ownership and permissions that's (a) secure, and (b) works with Dovecot.

I've deleted the extraneous stuff from the end of my post, thanks :) .

Jeff
 
So what you want is to have quotas but to ignore them? While you can do that, I still think the answer is education. People need to know that their limits are just that, limits.

I would agree with you normally, but the problem in this case is that the customer has no control over the incoming mails to his domain.

If the customer suddenly receives alot of mail, or a few huge mails, and his email-service stops working, no education will help his problem. He will call us, and we have to:

1) Remove some mail manually.
2) Increase his quota, let him clean up, and set the quota back down.

I'm actually surprised you don't see the problem with this. If a quota exceeded simply meant the customer wouldn't receive new mails, that would be ok, but when the customer can't even fetch the mail he already have, I'm getting worried.
 
I see the problem. I think if you need to resolve it then you probably want to turn off quotas at the OS level. As I already pointed out. But then you'll have the problem; your clients will be able to use enough resources to bring your server to its knees.

Years ago, when I first started in the hosting business, I used to leave quotas off, and check usage every night and bill based on usage. A lot of people wouldn't pay for the excess usage, and I found one client with runaway email (for example caused by a combination of of a catchall account and a nasty spammer using multiple nonexistent return addresses at the domain on our server to send their spam) could shut down the entire server. I decided I'd rather have one domain shut down than the whole server.

Jeff
 
Ideally, when a user is over quota:

New incoming mail should be rejected with an error message such as "over quota" or "mailbox full".

The user should be able to read, delete, and purge mail without problems.

One way to implement this is to have a small C program that acts as the mail delivery agent, and implements mailbox size limits independent of the filesystem. If the filesystem quotas are quite a bit more generous than the mailbox size limits, this should solve the problem.

Rahul
 
I came across this which might be of some use for others as well. I haven't tried implementing it yet, but it would at least solve the errors people get when POPping their mail I assume.

http://wiki.dovecot.org/Quota/FS

Specifically the settings for :INDEX= and :CONTROL=.

Also, since this thread started, many of the bugs causing the errors should have been fixed (rc29 is latest at the time of writing).
 
Forgot to mention - if you decide to change the location of your index- and control-files, you most likely want to copy/move the existing -uidlist and -keywords files to the new location. If you don't, they will be recreated, but with new IDs, so clients will download them all over again as new mails.

The .index-files will also be automatically be recreated at the new location, but this isn't a problem unless you have very many users, as the recreation will take some resources.
 
Hello,

Thanks for the info kristian.
The CONTROL= solution seems like a good one, assuming the rc29 doesn't fix the problems). However, since our 1 standard dovecot.conf can't predict how an Admin's partitions and quotas are setup, we can't do this by default (eg, many systems have everythign on one large / partition).

For the time being we'll rely on the fixes with new dovecot releases to address this issue. If you are an admin who needs a solution, the the CONTROL= option can be setup and used on a per-server basis.

John
 
Softquota and quota-check

Hi guys,

I was wondering whether just informing the users that they are over their quota isn't good enough? My solution below is far from elegant, but with a few brains like on this thread it must be easy to make an airtight script.

On an Ensim-server I had a look at (problem raised it's head there too), I just wrote a script that checks the userquota's at night and sends the user (and/or the admin) an email when over the softlimit. Warnquota had trouble with uid's so I had to extract the offending users based on a grace-period flag.

One modification this does require, is that softquota has to be switched on for users (default 'off' for Ensim and probably for DA too). If you set grace period to 7 days, for example, the user could receive 6 emails (one every day) until his/her account is blocked. Seems like a fair warning to me...

My script is below, which is a quick hack but does the job there. It sends an email to the admin, but can be modified to send to the user. As long as the mail and other files are in the home dir, it should suffice.

Code:
#!/bin/bash
#
#  quota-check_script_harro.sh
#  Harro 2007
#
#  This script has been made to check user quota and when a user is over his/her quota,
#  to send an email to the admin.  This scripts has been made because Ensim seems to
#  cause problems with looking up UID's of virtual users ("warnquota: Can't get name for uid"a)
#

#  Adjust variables here:

TRIGGER_WORD=days
[email protected]
MESSAGE_BODY=`repquota -au | grep $TRIGGER_WORD`

#  check quota to see whether anyone is over the limit (trigger is the appearance of a
#  grace-period indicator (normally "days", but configurable)

echo TRIGGER_WORD = $TRIGGER_WORD
echo ADMIN_EMAIL = $ADMIN_EMAIL
echo MESSAGE_BODY = $MESSAGE_BODY
echo


if [ "repquota -au | grep $TRIGGER_WORD" ]
then
echo Found user over quota!  Sending mail to $ADMIN_EMAIL
/usr/sbin/sendmail -t <<fff
From: $ADMIN_EMAIL
To: $ADMIN_EMAIL
Subject: User over quota

$MESSAGE_BODY
fff

fi

# CRONTAB entry:
# 30 1 * * * /root/quota-check_script_harro.sh
 
Hi,

I haven't really looked much into how quotas work, as I'm personally not a huge fan of such limits. I do see the need though. :)

From repquota -au on my system running DA, it seems there is a grace-period of 7 days:

*** Report for user quotas on device /dev/sda9
Block grace time: 7days; Inode grace time: 7days

It goes on to show numbers for all users, such as:

Block limits File limits
User used soft hard grace used soft hard grace
----------------------------------------------------------------------
kristianr -- 147840 512000 512000 126 0 0

Maybe not easy to see here, but the 'grace'-columns are empty, I guess that means 1) user isn't over quota and all is well, or 2) grace isn't activated on the system and/or user.

I also does not know how grace-periods work - would we have to have different values for the soft and hard-quotas for that to do any good?
(As in, soft-limit is 100MB, hard-limit is 150MB, grace-period is 7 days. You're then allowed to use more than 100MB up to 150MB for 7 days before things go south.)

I would like to see a more elegant handling of disk-full problems in DA too, at least when it comes to mail. Many customers feels the service is poor if we reject mails to them, specially when they're small companies. And no, it doesn't matter how much you educate them, they'll still feel the same.


As for dovecot fixing the problem themselves, I think I read that if the disk was full, the indexes, uidlists etc would only be kept in memory, with the consequences that might lead to.
 
Back
Top