Backup all incoming mail using shadow_transport in Exim

kristian

kristian

Verified User
Joined
Nov 4, 2005
Messages
440
Location
Norway
Sometimes a user deletes a mail from their mailbox before it has been backed up over night, leaving no way to recover it. In order to be able to recover those mails, the use of shadow_transport in Exim can be used. This will essentially deliver a copy of an incoming mail to a separate backup Maildir directory. This directory can/should be purged of old files/mails in a cronjob to ensure it doesn't fill up. So how can we set this up?

The begin transports section in /etc/exim.conf file contains a line that includes /etc/exim.transports.pre.conf if it exists. This is where we can place our new transport, for example something like this:

Code: 
shadow_copy_incoming_email:
  driver = appendfile
  create_directory
  directory_mode = 0770
  maildir_format
  mode = 0660
  # This will expand to: /var/local/mail-backup/YYYY-MM-DD/mydomain.com/myuser/Maildir/
  directory = /var/local/mail-backup/${substr_0_10:$tod_log}/$domain_data/${lookup{$local_part}lsearch,ret=key{/etc/virtual/$domain_data/passwd}{$value}}/Maildir/
  delivery_date_add
  envelope_to_add
  return_path_add
  group = mail
  user = mail

In addition to this, we need to add the shadow_transport = shadow_copy_incoming_email directive to the existing transport, where we want to have this additional backup delivery. This is where I am a little stuck.

The goal is the only backup mail that is delivered to an actual mailbox (hence the lookup against passwd, and not aliases for example). It looks like, on my test server, the only transport used for this is dovecot_lmtp_udp, but I might be wrong. This particular transport also doesn't include any additional config files, which means I would need to modify /etc/exim.conf directly, which would then be overwritten.

So to sum up, my questions are:
1) Which transports would shadow_transport need to be added to, to cover all local deliveries of virtual mail accounts?
2) Can we have a way of adding shadow_transport to those transports, without having to modify /etc/exim.conf directly?
 
If you edit exim.conf you’re going to lose the edits on exim update. Personally I only update exim with a custom script that executes the update and reapplies my customizations to exim.conf, but no solution is perfect because whether you prefer my way or something more brutal like making exim.conf immutable, if an update changes how you need to apply the customization or it changes key config syntax, either exim or your custom config is broken without manual intervention.

I’d consider a different approach. Put a sieve filter on the email account to copy all inbound mail to another address. Deleting old email is still an issue, but regardless of which solution you use for copying the data you’re looking at the same work for automating a task to clear it. Since you’re using maildir, suppose you could just add to the user’s crontab a find command to delete old files in the maildir of the address you redirected to: https://www.howtogeek.com/howto/ubuntu/delete-files-older-than-x-days-on-linux/
 
mxroute said:
If you edit exim.conf you’re going to lose the edits on exim update.
Click to expand...
Yes, that's the trouble. I see that for the (seemingly unused) transport called virtual_localdelivery, there is a line at the end of it that includes /etc/exim/virtual_localdelivery.conf.post if it exists. If similar .include_if_exists directives were added to the other transports, then that would allow us to customize them further without losing changes on updates.


mxroute said:
I’d consider a different approach. Put a sieve filter on the email account to copy all inbound mail to another address.
Click to expand...
I'm not 100% up to speed on the possibilities with sieve, but we do have it enabled. Is it possible to create a global sieve filter that will become active for any and all mail accounts, and that can't be removed or disabled by the user? If the sieve filter only can deliver mail to another address, then we would need to create an "archive" account for each existing account, and make sure we created those for any future created mail accounts as well,

Generally I think it makes more sense to perform this kind of shadow copy in Exim though, but not sure I can argue very well for it.
 
kristian said:
Yes, that's the trouble. I see that for the (seemingly unused) transport called virtual_localdelivery, there is a line at the end of it that includes /etc/exim/virtual_localdelivery.conf.post if it exists. If similar .include_if_exists directives were added to the other transports, then that would allow us to customize them further without losing changes on updates.



I'm not 100% up to speed on the possibilities with sieve, but we do have it enabled. Is it possible to create a global sieve filter that will become active for any and all mail accounts, and that can't be removed or disabled by the user? If the sieve filter only can deliver mail to another address, then we would need to create an "archive" account for each existing account, and make sure we created those for any future created mail accounts as well,

Generally I think it makes more sense to perform this kind of shadow copy in Exim though, but not sure I can argue very well for it.
Click to expand...

Hmm yeah I see what you mean. You want to do it by default for everyone, makes sense. I don't think hooks will cover it and most likely DA won't be altered to do it short term (though it could make a good feature request). I'm going to adjust my recommendation to doing what I do. I use custombuild on all of my servers and rather than running a rebuild all to upgrade all services, I run individual scripts to update services and add customizations back to them. For example, when I update exim on my fleet, I run this now:

github.com

da_server_updates/exim/update_exim.sh at master · mxroute/da_server_updates

Scripts for updating our DirectAdmin servers. Contribute to mxroute/da_server_updates development by creating an account on GitHub.
github.com github.com

And then just using your preferred method of running something across the fleet. I use pdsh but you might prefer salt, etc.

If you can just get down the scripting logic to always add the necessary config to exim.conf, copy my system there and you may be able to go above and beyond without the manual workload each time.
 
mxroute said:
Hmm yeah I see what you mean. You want to do it by default for everyone, makes sense. I don't think hooks will cover it and most likely DA won't be altered to do it short term (though it could make a good feature request). I'm going to adjust my recommendation to doing what I do. I use custombuild on all of my servers and rather than running a rebuild all to upgrade all services, I run individual scripts to update services and add customizations back to them.
Click to expand...
I use ansible currently to update/manage the servers, so I could do something similar, and it would probably work well. I try to avoid deviating too much from The Normal Way of how such self contained build systems like CustomBuild works though. Special cases tends to pile up, and become hard to manage after a while. :)

I added a feature request for it at https://feedback.directadmin.com/b/...lude-if-exists-to-more-transports-in-eximconf. It's a pretty simple change, since it won't actually affect any existing setups. Hopefully it'll be picked up.

Thanks for your input!
 
Hey,

I have just found that topic od forum. It is what I am looking for - to archive all incoming mails to separate catalog for backup. I have added:

shadow_copy_incoming_email directive as Kristian wrote, and also

shadow_transport = shadow_copy_incoming_email

for local_delivery

but nothing happens. I created /var/local/mail-backup/ and give access 777 just for tests but nothing appears there.

Do you have any idea?
 
Anyone has idea how to make it passible to archive all incoming mails in exim?

I have just found information about forwarding all emails from server to separate mail but it is not what is good if we have a lot of customers.
 
@xeryph
what's wrong with just regular account backups (maybe backups with email only). Or you can rsync user home-s with pattern, or something else.
 
You must log in or register to reply here.
Back
Top