[Request] Remote Email Server

Do you want to see Remote Email server in future DirectAdmin versions?

  • Yes

    Votes: 30 90.9%
  • No

    Votes: 0 0.0%
  • Not sure

    Votes: 3 9.1%

  • Total voters
    33

Maniak

Verified User
Joined
Aug 25, 2004
Messages
220
Location
Switzerland
Hello,

We are a long-term partner of DirectAdmin and manage about 100 DirectAdmin servers for various customers to date and about 12 of them are internal servers.

Over the time, we have come to a certain level of operational excellence and have scripted and customized these servers, so we can manage most aspects smoothly. Out of all customers, there is however one concern that keeps coming back to the table and it's email messages. We probably have 80 to 90% of the requests related to emails in one way or another, because email remains a complex matter regardless of the system used and the sack/protect game with spammers, harvesters and script kiddies is always ongoing.

The issues we notice are:

1. Incoming spam (about 95% of the email we receive)
2. Outbound spam (most likely hacked CMS and weak passwords sending thousands of messages)
3. Ports 25 being blocked by ISPs
4. Quota of mailbox reached
5. Problematic deliverability due to bad IP reputation (usually because of the above)
6. Forwarders sending messages which contains spams and which was not properly filtered at incoming time
7. Various problems

Basically, we have dealt with most of these problems with various workaround, including:

1. Use advanced spam filter
2. Use advanced spam filter (outbound)
3. Document and help customers switch to 587
4. Resize mailboxes upon complains
5. Configure temporary smarthost with floating IPs and use point 2 of above
6. Use point 1 above
7. Sale our support :)

So, for most of the above we have designed and successfully found solutions to workaround, however, maintaining email servers is complex and time consuming and hence we would like to request, just like it's possible with MySQL (http://help.directadmin.com/item.php?id=140) to setup "remote email server" only and connect DirectAdmin together with this "remote email server" only. This way, we could centralize email and it would help operate the following:

- Filter inbound messages more easily
- Backup all messages more easily
- Filter outbound messages more easily
- Sign outbound messages more easily
- Maintain one consolidated DNS with a single SPF, MX and RDNS
- Manage only one main Dovecot/Exim/Roundcube instance
- Keep a "local" Exim on each DA server for PHP, CGI scripts and system alerts
- Allow only certain IP to send inbound message if not authentificated hence forcing inbound filters check for non authentificated users
- Use and focus our energy for more important tasks
- Enventually but not necessarily leverage this to develop a standard API to switch/drop-in a different mail server as the idea of a single server could be use to also replace Exim by any other mail system which can do the same, because Exim with Roundcube or Outlook remains a feature-weak replacement for enterprise messaging which would be a big plus for DirectAdmin customers and Hosters in general

We hope other people out there agree with us and will support this request.

Gregory
 
Last edited:

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
I like the idea, and if I ever have a spare year or two of time I'll look into it :D.

Some of the issues of course revolve around keeping track of all users in real-time, but nothing that couldn't be figured out.

But as I said, likely a time-consuming project.

Count me interested.

Jeff
 

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
8,905
I'll abstain from voting, but would be in favor :)
Martynas and I have already been in discussion about it... and the first thing we'd add would be a very basic API implementation for Email Account management..
If account "fred" is created on domain.com on box A, then DA would match is on domain.com on box B.
(and deletion, password changing, renaming)

But that's step 1, and would be all it would do at first.
Once it's done, we'd see how best to move forward.

As Jeff mentioned, very big project, and the logistics of it would be somewhat "gross", hence the small steps for now.

John
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,788
Location
A Coruña, Spain
I'm interested too, this would also be a step forward for a clustering structure.

For the system i had implemented (for tests at now) it had to "share" many folders to have same email account on multiple servers.

An API implementation and a "slave" (as for example DirectSlave does for DNS) would be just awesome.

Regards
 

nobaloney

NoBaloney Internet Svcs - In Memoriam †
Joined
Jun 16, 2003
Messages
26,119
Location
California
So just what are we discussing here? Bacup MX or true separation? I'm guessing true separation, but I believe the code for Backup MX would be a great start.

But if we're doing true separation I'd rather not see mailbox structure on both servers. Because it could create confusion when debugging, searching spam, etc.

Anyone remember Igor Seletsky? Now known for Cloud Linux, but his earlier product, H-Sphere, approached separation by requiring separate servers which could be installed within the same OS on the same physical server, or on separate servers. Interesting approach.

As my understanding goes, I do NOT mean VPS or virtual servers. By the word [i[server[/i] I mean the daemon and necessary files encapsulated in a relatively self-contained structure, which could be installed on separate boxes. And then all that was required in the base API (used by the control panel) was to point to the location of the server.

I don't think this structure could easily be replicated in DirectAdmin; H-Sphere, after all, was designed for this from the ground up.

Interesting project :).

Jeff
 

Maniak

Verified User
Joined
Aug 25, 2004
Messages
220
Location
Switzerland
Hi Jeff,

So just what are we discussing here? Bacup MX or true separation? I'm guessing true separation, but I believe the code for Backup MX would be a great start.
My original request: true separation.

But if we're doing true separation I'd rather not see mailbox structure on both servers. Because it could create confusion when debugging, searching spam, etc.
I agree with you. One server hosting the emails for a given domain, period.

I don't think this structure could easily be replicated in DirectAdmin; H-Sphere, after all, was designed for this from the ground up.
But adding a "remote email server" would add a lot of value to DirectAdmin for sure.

Interesting project :).
Agreed here again! :)
 

Maniak

Verified User
Joined
Aug 25, 2004
Messages
220
Location
Switzerland
Hi John,

Martynas and I have already been in discussion about it... and the first thing we'd add would be a very basic API implementation for Email Account management..
It is a good start.

If account "fred" is created on domain.com on box A, then DA would match is on domain.com on box B.
(and deletion, password changing, renaming)
I personally would add "a unique marker" (or key) which is unique server-wide, but which can be influenced (if allowed so) by user in DA itself if necessary an match both the domain and the "key" in case of duplicate entry. Hence in case of conflict, it is unlikely that the user can "violate" that key on the remote host and we always secure the email from "unallowed" access. You can even do security based on domain, key and IP.

As Jeff mentioned, very big project, and the logistics of it would be somewhat "gross", hence the small steps for now.
Indeed, but you capitalize on your product once again with a great feature. Thanks for being open to the request by the way.
 

Petertjuh360

Verified User
Joined
Nov 7, 2010
Messages
350
I have the same request and problems as the topic starter. This will be a nice and large improvement for DirectAdmin. I already was searching for a solution but I could not manage a way to find this out.
 

mmx

Verified User
Joined
May 8, 2005
Messages
112
Location
Montreal, QC, Canada
I've always wanted this type of infrastructure, and thought about implementing it with shared storage (a combination of NFS+ZFS and/or CARP+HAST thrown in somewhere) and configuration files pushed across a pool of servers (separate Exim, Dovecot, SpamAssassin, and ClamAV instances). While I haven't dived into Exim and Dovecot documentation yet, I do believe it's possible to set something up without breaking DA. One of my ideas was to use a LDAP server or a SQL database to handle authentication across a pool of hosts, and have DirectAdmin interface/synchronize user ACLs with said auth server whenever an account is added, modified or removed. If I am not mistaken, some of this functionality can be added through scripts that are fired off every time a function in the control panel has been executed (need to check DirectAdmin API). The individual Exim/Dovecot servers will have to be setup manually and maintained individually, outside the scope of DirectAdmin+CustomBuild, of course.

Dovecot pages to read:Director and Replication. Some guy blogged about his clustered setup.

This idea is on my list of todos / stuff to play with, after I finish up PortsBuild. :)
 

ben29

Verified User
Joined
Jul 20, 2006
Messages
473
Location
isreal
that will be amazing.
but i think according to security reasons it should be very secure connection
 

Nobby

Verified User
Joined
Jan 6, 2015
Messages
5
This in invaluable.
Separating mail from web has many advantages.
creating a mysql/mariadb single database that all DA boxes talk to, centralises mail, allows for easy expansion using multiple front ends to, say NFS backend, like emc or netapp devices, easy to expand as demand requires, each front end can have its own replicated mysql db copy to lessen the load talking to localhost by not stressing the master server that all DA boxes talk to for add/del etc of users.
this is in use with broadband retail unit, using dovecote mysql and postfix, since postfix and dovecote are designed to work together, with special hook, shows good cooperation between weitse and timo.

second major advantage is anti spam/virus testing which is very resource intensive.

best advantage is, if web server or mail server is down, the customers not completely isolated off of the internet
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,788
Location
A Coruña, Spain
I'd implemented a clustering for DA, is still not finished (doenst have a easy web interface to mange them but was working fine.

It was using GlusterFS and was clustering all services, so, all services (httpd, mail, etc...) are on all servers.

Is not the same you're talking about, services are not separated for each server, but it does work so far.

Regards
 

Nobby

Verified User
Joined
Jan 6, 2015
Messages
5
This in invaluable.
Separating mail from web has many advantages.
creating a mysql/mariadb single database that all DA boxes talk to, centralises mail, allows for easy expansion using multiple front ends to, say NFS backend, like emc or netapp devices, easy to expand as demand requires, each front end can have its own replicated mysql db copy to lessen the load talking to localhost by not stressing the master server that all DA boxes talk to for add/del etc of users.
this is in use with broadband retail unit, using dovecote mysql and postfix, since postfix and dovecote are designed to work together, with special hook, shows good cooperation between weitse and timo.

second major advantage is anti spam/virus testing which is very resource intensive.

best advantage is, if web server or mail server is down, the customers not completely isolated off of the internet
Time was scarce yesterday, so I forgot to add..

This should be a simple matter , all the DA boxes need to do is talk to 1, as in "any" mysql server, the actual mail servers would not even need DA on it, sure it means admins need to know what they are doing, but its very very simple to get that postfix/mysql/dovecot combination going for virtual hosts

All DA needs is to know the master database location, name, DB-user, pass, and tables to access, or, of course DA could decide what structure they want and include a schema we install into mysql, that way DA does not need to configure mail servers or pop3 or whatever, admins do it, all DA does is add/delete domains, users/aliases in a database.


yes, it means a tad more work like maybe 15 mins more by admins and things are running perfect, and those who have no idea about how to do it, would be far better off sticking with how things are now.


PS: Who ever is responsible for these dumb random things, you need to take into account not everyone knows your countries slang, please replace with something thats internationally understandable.
 

r3chn3r

Verified User
Joined
Jan 13, 2013
Messages
103
This would make me a happy camper indeed. I would love to see this feature implemented. :)

I have a dev box at home where my ISP blocks ports. I still haven't figured out how to get the email working despite trying various things I've found documentation for. Just doesn't want to work with mailgun I suppose.
 
Last edited:

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
8,905
Feature added, available for beta testing now:
https://www.directadmin.com/features.php?id=1745

As promised, it has very limited functionality, so please from refraining when you notice it doesn't create the DA account/domain if it's not on the other server :)
It does require you to set things up in a matching server-pair before you start using it.

For added control I've added allow/deny files that you can add either User accounts or Domain names.
When testing, I do recommend you add your testing domain to the allow file, so other domains/accounts are not affected.

Binaries are now available in the pre-release section:
http://help.directadmin.com/item.php?id=408

Good luck, happy testing!

John
 

SeLLeRoNe

Super Moderator
Joined
Oct 9, 2004
Messages
6,788
Location
A Coruña, Spain
Very very nice feature!

As far as i understand, this will not sync email data, just accounts right? Any hint for email data? Rsync?

Thanks

Best regards
 

DirectAdmin Support

Administrator
Staff member
Joined
Feb 27, 2003
Messages
8,905
Thanks! and yes it's very basic/simple.. it just mimics account creation/deletion/change/suspend/unsuspend.
Correct, rsync will be your best friend for data syncing.

John
 
Top