Table of Contents
Self-hosting mail tips
From: https://www.reddit.com/r/linuxadmin/comments/8x7ll3/selfhosting_mail/
It's not “easy” but it's probably not as hard as a lot of people make it out to be. I've hosted my own mail servers for years and if you tick off all the boxes on your setup you should be okay.
When you say you've got a clean IP, have you run it through an RBL checker or three? If you're doing both IPv4 and IPv6, be sure to check both.
So here's what you'll need:
I recommend running your MX on a dedicated host. Don't use the box for any other services, including client mail services (IMAP/POP). Have the MX deliver mail to your mailbox server, and have the mailbox server smarthost mail to the MX. EDIT: this effectively means running two MTAs, but the one on the mailbox host will have a very basic config. It doesn't need to talk to the rest of the world, just your MX host.
You'll need a static IP (sounds like you've got this already) and you'll need forward and reverse DNS records that match the mail host name (the name the SMTP server reports to foreign SMTP servers). You'll want a TLS certificate for that name as well. You can use Let's Encrypt for this, so it needn't cost you anything. You'll want to add SPF, DKIM, and DMARC records to your mail-enabled DNS zones.
You'll want to sign your outgoing mail with DKIM headers. HOWEVER, for those keys to be trusted by receiving systems the zone you are storing the keys in must be DNSSEC enabled. Otherwise, most foreign systems will regard the public keys as untrusted. If your current DNS solution supports DNSSEC, you're in luck. If not, you might want to look at a different service.
You will need a service on your MX to perform DKIM tasks, either opendkim or the DKIM module for amavis.
You don't say which MTA you're planning to use, but if you use postfix review the documentation for connection control, and anti-UCM. It's sort of a fine line between being too restrictive and blocking legitimate email and so open that you let a lot of garbage in. You'll quickly find out that there are a lot of 'legit' mail servers out there that are improperly configured. Especially if they run Exchange. I would also recommend you use the postscreen module and check incoming connections against Spamhaus or some other RBL. It really cuts down on the garbage.
In addition to the MTA-level controls you'll also likely want to run amavis for anti-spam filtering with spamassassin, antivirus with clamav, and if you do that you might as well use the DKIM module in amavis instead of running a separate standalone service. It will just save you one extra hop in your mail flow (amavis runs on the MTA, usually bound to localhost on port 10025. postfix forwards the mail to amavis for scanning or signing and then gets it back to deliver by receiving it on another localhost bound port, e.g. 10024).
You'll want to take the normal security steps. Key authentication for ssh, run a MAC (AppArmor or SELinux), host-based firewall, fail2ban, etc. Obviously, setup a schedule for patching and backups.
You don't mention how you're going to deliver to your mailbox server - direct or ETRN. If the latter, be sure to restrict the ETRN request sources.
And… that's about it. Good luck with the project!