[How-To] Exim + Dovecot + Sieve + Roundcube + ManageSieve

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
* * * WARNING! Custombuild 2.0 now supports Sieve/pigeonhole * * *
* * * Don't follow this guide with CustomBuild 2.0 on your server. * * *


Sieve (mail filtering language)

From Wikipedia said:
Sieve is a programming language that can be used to create filters for email. It owes its creation to the CMU Cyrus Project, creators of Cyrus IMAP server.

Used software and its versions

DirectAdmin: 1.40.3
Exim: 4.77
dovecot: 2.1.3
RoundCube webmail: 0.7.2
dovecot-2.1-pigeonhole-0.3.0



1. Configuring Dovecot
Go to http://pigeonhole.dovecot.org/download.html and find a proper version of pigeonhole for your dovecot version.

Pigeonhole is the name of the project that adds support for the Sieve language ·(RFC 5228) and the ManageSieve protocol (RFC 5804) to the Dovecot Secure IMAP Server. In the literal sense, a pigeonhole is a a hole or recess inside a dovecot for pigeons to nest in. It is, however, also the name for one of a series of small, open compartments in a cabinet used for filing or sorting mail. As a verb, it describes the act of putting an item into one of those pigeonholes. The name `Pigeonhole' therefore well describes an important part of the functionality that this project adds to Dovecot: sorting and filing e-mail messages.

So let's get and install it:
Code:
[B]cd /usr/local/src
wget [URL]http://www.rename-it.nl/dovecot/2.1/dovecot-2.1-pigeonhole-0.3.0.tar.gz[/URL]
tar -zxvf dovecot-2.1-pigeonhole-0.3.0.tar.gz
cd dovecot-2.1-pigeonhole-0.3.0
./configure --prefix=/usr
make && make install[/B]
Now we ready to update dovecot settings, but firstly make sure you've got a backup copy of its configs:
Code:
[B]cd /etc/
cp dovecot.conf dovecot.conf-bak[/B]
Then open /etc/dovecot.conf for editing (with nano, vi, mcedit, etc), and add the following lines to the right bottom of the file:
Code:
[B]service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  service_count = 1
  process_min_avail = 1
  vsz_limit = 64M
}
service managesieve {
  process_limit = 10
}
protocol sieve {
    managesieve_max_line_length = 65536
    managesieve_implementation_string = dovecot
    log_path = /var/log/dovecot-sieve-errors.log
    info_log_path = /var/log/dovecot-sieve.log
}
plugin {
    sieve = ~/.dovecot.sieve
    sieve_global_path = /var/lib/dovecot/sieve/default.sieve
    sieve_dir = ~/sieve
    sieve_global_dir = /var/lib/dovecot/sieve/global/
}
protocol lda {
    mail_plugins = $mail_plugins sieve
    postmaster_address = postmaster@domain.com
    hostname = server.domain.com
    auth_socket_path = /var/run/dovecot/auth-master
    log_path = /var/log/dovecot-lda-errors.log
    info_log_path = /var/log/dovecot-lda.log
}
protocol lmtp {
    mail_plugins = $mail_plugins sieve
    log_path = /var/log/dovecot-lmtp-errors.log
    info_log_path = /var/log/dovecot-lmtp.log
}[/B]

After that, find there a line
Code:
[B]protocols = imap pop3[/B]

and add lmtp sieve to its end, so it would look like this:
Code:
[B]protocols = imap pop3 lmtp sieve[/B]

Then find auth section:
Code:
[B]service auth {
  user = root
}[/B]

and make it to look something like this:
Code:
[B]service auth {
  user = root
  unix_listener auth-master {
    user = mail
    group = mail
    mode = 0664
  }
}[/B]

Now we should create some logs (it's up to you to setup rotation of these logs):
Code:
[B]touch /var/log/dovecot-debug.log /var/log/dovecot-lda.log /var/log/dovecot-lmtp.log /var/log/dovecot-sieve.log /var/log/dovecot-lda-errors.log /var/log/dovecot-lmtp-errors.log /var/log/dovecot-sieve-errors.log[/B]
Code:
[B]chown mail:mail /var/log/dovecot-debug.log /var/log/dovecot-lda.log /var/log/dovecot-lmtp.log /var/log/dovecot-sieve.log /var/log/dovecot-lda-errors.log /var/log/dovecot-lmtp-errors.log /var/log/dovecot-sieve-errors.log[/B]
Code:
[B]chmod 660 /var/log/dovecot-debug.log /var/log/dovecot-lda.log /var/log/dovecot-lmtp.log /var/log/dovecot-sieve.log /var/log/dovecot-lda-errors.log /var/log/dovecot-lmtp-errors.log /var/log/dovecot-sieve-errors.log[/B]

This is How to create default empty sieve rule (Thanks to user defomaz)
Code:
 # mkdir /var/lib/dovecot/sieve/
 # touch /var/lib/dovecot/sieve/default.sieve
 # sievec /var/lib/dovecot/sieve/default.sieve
 # chown mail:mail /var/lib/dovecot/sieve/* -R
We've finished with Dovecot, let's restart it and test it. With the following commands you should see something like that comes bellow (if you see something completely different, then re-check your steps):
Code:
[B]# ps aux | grep -v grep | grep managesieve-login
dovecot  20372  0.0  0.1  44608  2988 ?        S    16:53   0:00 dovecot/managesieve-login[/B]

and this
Code:
[B]# netstat -an | grep LISTEN | grep :4190
tcp        0      0 0.0.0.0:4190                0.0.0.0:*                   LISTEN[/B]


2. Configuring Roundcube for using managesieve plugin

Code:
[B]cd /var/www/html/roundcube/plugins/managesieve/
cp config.inc.php.dist config.inc.php[/B]

Open /var/www/html/roundcube/plugins/managesieve/config.inc.php find lines
Code:
[B]// managesieve server port
$rcmail_config['managesieve_port'] = 2000;[/B]

update it look like this (change port from 2000 to 4190)
Code:
[B]// managesieve server port
$rcmail_config['managesieve_port'] = 4190;[/B]

The Pigeonhole ManageSieve service now binds to TCP port 4190 by default due to the IANA port assignment for the ManageSieve service

Then open /var/www/html/roundcube/config/main.inc.php
And find line
Code:
[B]$rcmail_config['plugins'] = array('password');[/B]
Replace it with (or add ,'managesieve')
Code:
[B]$rcmail_config['plugins'] = array('password','managesieve');[/B]

Now it's the right time to make sure RoundCube can connect to Sieve service, so open Roundcube in your browser
http://domain.com/roundcube/-> Personal Settings -> Filters -> Add filter

If you do not see any error message there, then it's OK and RounCube seems to be able to connect to Sieve service of Dovecot. So let's create a simple filter to make sure the things work ok, and later we will use this rule for a full check of our settings:



Note, you might loose these changes (made in /var/www/html/roundcube/config/main.inc.php and /var/www/html/roundcube/plugins/managesieve/config.inc.php) on Roundcube upgrading with custombuild script. So it's up to you to protect it from overwriting (see link to "Using custom configs Roundcube" at the bottom of this How-To for details). Also you might need to re-build pigeonhole after every dovecot upgrading or reinstalling


2. Configuring Exim for using Dovecot Deliver (LDA)

Code:
[B]cd /etc
cp exim.conf exim.conf-bak[/B]

Open /etc/exim.conf fir editing, find the lines
Code:
[B]address_reply:
  driver = autoreply[/B]

add after them
Code:
[B]dovecot_virtual_delivery:
   driver = pipe
   command = /usr/libexec/dovecot/deliver -d $local_part@$domain -f $sender_address -a $original_local_part@$original_domain
   message_prefix =
   message_suffix =
   delivery_date_add
   envelope_to_add
   return_path_add
   log_output
   user = "${lookup{$domain}lsearch* {/etc/virtual/domainowners}{$value}}"
   group = mail
   temp_errors = 64 : 69 : 70: 71 : 72 : 73 : 74 : 75 : 78[/B]

Find the line
Code:
[B]transport = virtual_localdelivery[/B]

Replace it with
Code:
[B]#transport = virtual_localdelivery
transport = dovecot_virtual_delivery[/B]

Restart Exim and send an email in order to check the filters!
Code:
[B]# cat /var/log/dovecot-lda.log
Apr 08 17:45:55 lda(po4ta@domain.ru): Info: sieve: msgid=<66523BD3B58443CC93FAEED102FCE7E2@VDS54.RU>: stored mail into mailbox 'INBOX.@vds54-ru'[/B]

That's it!

Some usefull links:



Need help?!

Do you want our assistance to install and configure Exim/Dovecot/Sieve/Managesieve/Roundcube on your server?
CONTACT US FOR DETAILS!

We're ready to help you!

---

Permanent link of this How-To http://plugins-da.net/info/directadmin-exim-dovecot-sieve-roundcube


UPD1: To switch to LMTP from LDA one need to apply some changes. Please read this post http://forum.directadmin.com/showthread.php?t=43332&page=3&p=236255#post236255

UPD2: Custombuild 2.0 now supports Sieve and you don't need to follow this guide in case you are using CB 2.0.

UPD3: With implementing of Sieve into Exim/Dovecot you'll get Mailbox Quotas not working, it would mean, that if you set a limit for a Mailbox to 10Mb (for example) or whatever you set, you'll be still able to overcome the limit.
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Alex, can you post somewhere, or point us to, some reasons to use Sieve and the Dovecot LDA? It would be nie to have everything in one place under this thread.

Also, will updating Dovecot break Sieve? I ask because your instructions say to get the proper version of Sieve for your version of Dovecot.

Thanks for your work.

Jeff
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
Jeff,

I've been searching for something like this for some time. I did not know anything about Sieve before, and now I'm happy I can set rules for sorting incoming emails. For some reasons I don't want my email program to do it, I'd prefer my mail server to sort emails and put them as I wish (for each group of messages I create a separate folder and then sort them).

If you upgrade or reinstall Dovecot it seems you need to re-build pigeonhole, without that dovecot might fail to start. I did not studied this moment yet. So I can't say for now, why it happens.

You can find Pigeonhole v0.3.0 for Dovecot v2.1 and Pigeonhole v0.2.6 for Dovecot v2.0 (old stable) here http://pigeonhole.dovecot.org/download.html (you can find this link in How-To above as well), so I guess within one branch you should not worry much about Pigeonhole version.

Please, feel free to ask more questions if you have any.
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
By the way, if to speak about alternatives, default filters (which comes with exim and directadmin) can block emails according to rules you can set via directadmin interface (matching words, size), but there is no opportunity to move emails to a separate folder with their help. So if I want to have all news letters in a separate folder, and emails from Bob in an another one, I'll need to move them manually or set a rule in my PC email program. That is what I did before... but I work with one email box from different places, and it was a pain to sync the rules between different versions of one email programs on different desktops.

And now with the help of Sieve and Roundcube I can spent some time to write rules, and see my emails sorted without my direct controlling.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
I may be beginning to understand Sieve. Does sieve allow virtual mailbox users to set up filters which run on the imap mail store on the server so both desktop and webmail clients will see email filtered substantially the same as desktop clients can do now, but with all clients seeing the same filtering?

If so, then it sounds interesting.

If not, then what am I missing?

Thanks.

Jeff
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
Yes Jeff, that seems you've got the idea. Note filtering is actual if you use IMAP to connect to the mail server, but not POP3.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Now I'm getting even more interested... personally, since I'm switching from POP to IMAP for my own mail management, so I can manage from anywhere I happen to be.

But do we really want to encourage users to use IMAP? If all our users start saving all their email on our servers two things happen:

1) our storage needs, and therefore our storage costs, go up.

2) we'll find our users blaming us if they ever lose email.

Should this be part of a premium email offering, then, and not part of all our email support?

Jeff
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
Should this be part of a premium email offering, then, and not part of all our email support?
Jeff, with all my respect to you and your contribution into Directadmin community, I can't answer your question, as answering it comes beyond the scope of this How-To. Note, directadmin can be used on one-project servers, not only for shared hosting. So if anyone would need Sieve here is the How-To. I can't make decisions about who and why would ever need it, and why not. There were some questions on these forums without full replies; there was a request for implementing such a feature, and at last there was my own interest and needs... so that's why you and others can now read and use the How-To. And I'm glad and happy to share it.

I know and I'm agree, that using IMAP might increase a server load, if you force all customers to use it, and it might make you to buy extra storages to store all those emails on a server. I guess if you need, there is probably a possibility to disable IMAP for all accounts, and sell access via IMAP as an optional service. So it's up to you to decide whether you are going to use this on your servers or not.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
Jeff, with all my respect to you and your contribution into Directadmin community, I can't answer your question, as answering it comes beyond the scope of this How-To.
You're right of course. I usually like to keep all discussion together if it makes sense to keep it together, but I agree, the discussion of whether or not to implement something probably doesn't belong in the How-To.

I absolutely appreciate your work and will probably use this in my own email, but as for my clients, it may become part of a premium email offering we're working on.

Jeff
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
Now I'm getting even more interested... personally, since I'm switching from POP to IMAP for my own mail management, so I can manage from anywhere I happen to be.

But do we really want to encourage users to use IMAP? If all our users start saving all their email on our servers two things happen:

1) our storage needs, and therefore our storage costs, go up.

2) we'll find our users blaming us if they ever lose email.

Should this be part of a premium email offering, then, and not part of all our email support?

Jeff
POP3 is a protocol of the past. People are more and more mobile and expect to have instant access to their data from anywhere.
1) Then I think you should price your plans accordingly, bust storage is cheap these days.
2) You're right. You need a solid DR plan in place and you could sell data recovery support if they lose the data by their own mistake. You also bear some extra responsibilities because of the linked privacy and security issues.
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
The great thing about sieve is that customers no longer need to log into the control panel to manage their email filters or vacation message. It just doesn't make sense to do it that way.

Sieve can also be used at delivery time to file spam into the correct folders while updating some flags, instead of having to rely on Exim filters.

And a few things regarding the howto.
This is not needed as those are already the default values:
managesieve_max_line_length = 65536
managesieve_implementation_string = dovecot
inet_listener sieve {
port = 4190
}
service_count = 1

Logs probably have to be created before they can be used and global sieve scripts should be tested first by compiling them.

Why did you choose to run lmtp as root instead of lda as mail?
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
2) You're right. You need a solid DR plan in place and you could sell data recovery support if they lose the data by their own mistake. You also bear some extra responsibilities because of the linked privacy and security issues.
The big issue as I see it is that to guarantee no lost email, all email needs to be stored in multiple locations all the time, and not just by nightly backups. Have you, or anyone else using DirectAdmin, approached this? How?

While we can probably do this (external storage of all email, in either SAN or NAS) I'd think it beyond what most small hosting providers using DirectAdmin would want to do.

Is it time to open up a new subforum under the E-Mail forum, to discuss this?

Jeff
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
The big issue as I see it is that to guarantee no lost email, all email needs to be stored in multiple locations all the time, and not just by nightly backups.
That's if you want to offer a very small RPO, which would be overkill for most small businesses using small hosting providers, but should be doable with the right replication hardware/software. How and where to back up your data depends on the SLA you want to offer though. We find it easier to do hourly incremental backups :).
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
I'm not sure what RPO means :).

Hourly still seems like quite a bit of server impact; would you be so kind as to share your method for incrementally backing up? File by file? Or by tarring and rsync?

Though... I'm not sure how to put a positive spin on we save all your email except for that delivered in the past hour.

Perhaps offer the service and then in the fine print only guarantee best efforts on storage; offering guaranteed storage at a higher price as a special service.

Jeff
 

zEitEr

Super Moderator
Joined
Apr 11, 2005
Messages
13,880
Location
GMT +7.00
Why did you choose to run lmtp as root instead of lda as mail?
Is that question to me? If so, I can't say for sure why exactly I did it so (I did not write notes for myself when worked on this project). I can say only that I've met a lack of documentation on the subject. And I tried to put separated parts of configs together and make it work. And as soon as I made it work I haven't tried yet to improve any part of it. So if you've got it working your way with lda as mail, that's good.
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
@nobaloney - simply put, your RPO is how much data you're willing to lose, expressed in time from the incident. You need a multi-core server, so that the impact on making backups is as low as possible.
We're using duply on top of duplicity because it uses rsync, encrypts the data with GPG and supports many different backends, including S3.
If you're on FreeBSD, you could send ZFS deltas to a remote server and have a mirror of your clients' data, but that's only worth doing if you offer High Availability.

The spin could be that you care about your client's data and that you guarantee that you would only loose 1, 6 or 12 hours worth of data. Duply is flexible enough that you could have different configs for different SLAs.

@zEitEr - Yes, it was for you :). OK about the Howto. When we went for sieve 2 years ago, lmtp wasn't available and then with 2.0 it was unreliable. It's only recently that it's become usable, but I don't see any advantages of switching, except for the fact that it's a daemon.
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
simply put, your RPO is how much data you're willing to lose, expressed in time from the incident.
I'm not sure what RPO means. We simply don't want to give our clients the false impression that it's safe to leave mail on the server and that we guarantee we'll always be able to restore all of it. When clients choose to use IMAP and not POP we're put in that position. Some of our clients use POP and some don't; we leave it up to them.
You need a multi-core server, so that the impact on making backups is as low as possible[./quote]
Our modern servers are all multi-core or even multi-processor/multi-core, and some are multi-core/multi-processor/multi-threaded. Some older servers are still in service. If we decide to offer this level of service we'll of course use latest technology; probably go so far as to use NAS, if we can configure it so we can add drives in real-time.
We're using duply on top of duplicity because it uses rsync, encrypts the data with GPG and supports many different backends, including S3.
Looks interesting. How do you decide which directories to back up? Do you use the directory lists in the DirectAdmin System backup as a guide? Or did you do your own research? Do you have a restore tool or do you restore manually? If manually, do you have a prepared guide so you can do both systemwide and individual restores as quickly as possible? As for me, I prefer the DirectAdmin Admin Backup/Transfer facility; it works well and we find it easy to do selective backups. While automatic restores restore the entire per-user backup, we can manually extract down to the per file level if required. It works for us but the disadvantage of course is that it's time consuming.
If you're on FreeBSD, you could send ZFS deltas to a remote server and have a mirror of your clients' data, but that's only worth doing if you offer High Availability.
We're not.
The spin could be that you care about your client's data and that you guarantee that you would only loose 1, 6 or 12 hours worth of data. Duply is flexible enough that you could have different configs for different SLAs.
Interesting thought. I'm wondering how to put a positive spin on that, though. Currently we recommend users retrieve email often. I do that as well. Can IMAP be set up to duplicate everything locally (I've only recently started using it myself and this may be an option)?

Jeff
 

interfasys

Verified User
Joined
Oct 31, 2003
Messages
2,099
Location
Switzerland
RPO = Recovery Point Objective.

We simply don't want to give our clients the false impression that it's safe to leave mail on the server
What about their user created content or DB files then? How is this any different?

How do you decide which directories to back up?
We're talking about user data here, so /home + databases should cover it. It depends on your SLA.

Do you have a restore tool or do you restore manually
From the command line. It's very flexible.

If manually, do you have a prepared guide so you can do both systemwide and individual restores as quickly as possible?
It's essential when doing Disaster Recovery.

Can IMAP be set up to duplicate everything locally
I'm not sure... It should be possible to send the data to multiple delivery servers via lmtp.


I like the DA tool for system wide backups because it's easy to restore DA specific data, but rsync is more efficient for certain tasks.
 
Top