Reliable, Deliverable, Self-Hosted Email
I have been on an ongoing quest to free myself from cloud services for years now. During this time, I have hosted my personal email (
@bloomqu.ist) on a
Google Apps G Suite Google Workspace account, which, while convenient, also means that my personal emails are at the whims of one of the world’s most privacy-hostile companies.
Google’s famous slogan.
Obviously, this cannot stand forever. I wanted to self-host my email, but I know that self-hosting email is fraught with issues. The problems I wanted to avoid are as follows:
- Open-source email software is complicated to set up if you were not a sysadmin in the 90’s.
- Popular email providers like Google Mail frequently block emails from residential/public cloud IP addresses for anti-spam reasons.
- If your self-hosted mail server goes down, you can potentially miss out on important email.
I cannot use an email service that does not have reliable sending and receiving. Happily, each of these problems has a solution:
- The Mailu project bundles antispam, POP3, IMAP, SMTP, webmail, administrative interface, etc. into a set of Docker containers that can be managed with Docker Compose or your container tool of choice. This gives us a good, well-architected base setup to configure further.
- Use a trusted SMTP outgoing relay to send email. Yes, this is not self-hosting, but you cannot “self-host” an outgoing email anyways, it is ultimately leaving your network one way or another.
- Use a backup email server to receive email when your server is down. Set up MX records with a lower priority than your self-hosted mail server to have them act as a fallback. You can either self-host this backup or use a public service. Then, when the primary server is back up, use IMAP to mirror messages from the backup automatically.
In fact, Mailu is so easy to use, that we can configure (2) and (3) out of the box. Here is a step-by-step guide to configuring Mailu with an outbound SMTP gateway and backup MX server:
Part 1: Install Mailu
Follow the Mailu setup guide published on their website. Make sure to enable the
fetchmail service -
fetchmail will pull email from our backup mail server.
Set up your DNS records appropriately. For most use cases, this means pointing an A record from your mail hostname (in my case,
mail.chary.us) to your mail server’s public IP, and configuring MX records on the domains that will be receiving email:
~ dig A mail.chary.us mail.chary.us. 300 IN A 126.96.36.199 ~ dig MX chary.us chary.us. 300 IN MX 1 mail.chary.us. ~ dig MX bloomqu.ist bloomqu.ist. 300 IN MX 1 mail.chary.us.
We do not need to set up SPF/DKIM DNS records at this time, since we will be using an outgoing SMTP relay to send email, not the mail server itself.
Part 2: Set up an outgoing SMTP relay
To prevent other email servers from discarding our emails because they’re from an IP with no reputation for mail sending, we can use an outgoing SMTP relay instead of sending email ourselves.
There are many options available for this, with Mailgun, Mandrill by Mailchimp and SendGrid by Twilio being the three heavy-weights of the industry. I chose to go with SendGrid, because their free plan offers sending 100 emails/day forever, which is far more emails than I can see myself sending.
Once you have an account with your SMTP relay provider, you will want to add the domain(s) that you will be sending from to your account. In SendGrid, this is called “Sender Authentication”. As part of this, you will configure DNS CNAME records for DKIM and SPF:
Confused about how SPF can work like this, even though it’s being set on
yourdomain.com? SPF is used to validate the
Return-Path header, not
From, and SendGrid uses
em0000.yourdomain.com as the
Return-Path. 💥. Read more on StackOverflow.
Now that the domains are configured, obtain the credentials for your provider’s SMTP gateway. In SendGrid, this is under the “API Keys” settings panel. Your API key is your password.
mailu.env file, update the following environment variables:
# Change to your SMTP relay host - port 587 will automatically use SSL RELAYHOST=smtp.sendgrid.net:587 # Change to your username, on SendGrid this is the literal string "apikey" RELAYUSER=apikey # Change to your password, on SendGrid this is the API key you obtained RELAYPASSWORD=your-api-key-goes-here
If you have already started Mailu, run
docker-compose down; docker-compose up -d to reload the env file.
You should now be able to send an outgoing email (for example, via the Mailu webmail) and by inspecting the
Received header at the recipient end, see that it was relayed through SendGrid:
// more headers Received: from xtranbvx.outbound-mail.sendgrid.net (xtranbvx.outbound-mail.sendgrid.net. [188.8.131.52]) // more headers
If this does not work, you can tail the logs using
docker-compose logs -f --tail=0 and re-send the email to see what the error was.
Part 3: Set up the backup email server
If, heaven forbid, your power goes out, your server gets knocked off the shelf, or the fiber line to your house is cut, you will not want to miss out on incoming e-mails. Although some mail transfer agents (MTAs) will retry delivery if the recipient SMTP server is unavailable, you really really don’t want to be relying on “maybe the sender will retry” when it comes to important email.
Luckily, this is pretty easy to fix, by the nature of how MX (Mail EXchanger) DNS records work. Each record has a priority, with the lowest priority records being considered first. Records with equal priority are “randomly” chosen by the sending MTA. For example, with the following MX records, mail would always be delivered to
mail.foo.net is down, in which case it will go to
10 < 20):
foo.net. 300 IN MX 10 mail.foo.net. foo.net. 300 IN MX 20 backup-mail.foo.net.
You can either set up your own mail server for this purpose, or use a public Internet service as the backup.
If you decide to self-host a backup mail server, make sure the infrastructure is separate from your primary mail server. There is some discussion on the Mailu repo about setting up a backup MX server, and if you DuckDuckGo “backup MX server”, you can find plenty of information about setting up a
postfix server in this manner.
I went with the non-self-hosted option - since I already have GSuite set up to receive emails for all my domains, I can use it as the “backup” MX. Here is what my MX records look like:
➜ ~ dig MX bloomqu.ist bloomqu.ist. 300 IN MX 1 mail.chary.us. bloomqu.ist. 300 IN MX 3 aspmx.l.google.com. bloomqu.ist. 300 IN MX 5 alt1.aspmx.l.google.com. bloomqu.ist. 300 IN MX 5 alt2.aspmx.l.google.com. bloomqu.ist. 300 IN MX 10 aspmx2.googlemail.com. bloomqu.ist. 300 IN MX 10 aspmx3.googlemail.com.
As a result, MTAs will first attempt to deliver to
mail.chary.us. If my server is down, delivery will fall back to Google Mail.
There are also paid services that offer to do specifically this - a cursory search found this affordable offering.
Once your backup MX records are in place, the last step is to configure Mailu to automatically retrieve email from your backup MX using
fetchmail. This is available via the web administration interface. Navigate to
/admin and click on “Fetched accounts”. From here, you can configure automatic fetching via IMAP or POP3. My setup for Google Apps looks like this:
Note that Mailu is configured to only pull unread email via “Fetched accounts”.
fetchmail will run every 60 seconds to pull in email.
Part 4: Next Steps
Now you are sending and receiving email in a safe, reliable way, while still maintaining control of your data at rest. 🎉🎉🎉!
Life is happy!
After setting everything up, you may wish to take some additional steps to enhance and secure your setup:
- Set up a catch-all address for your domains using
Aliasesin the Mailu admin panel.
- Establish a backup strategy - as with anything self-hosted, the data is now your ultimate responsibility. Nobody will be around to help you recover lost emails when your server’s hard drive inevitably crashes, so establish a strategy now.
- Increase your mailbox quota - by default, Mailu creates all accounts with a 1GB mail quota, which is pretty small.
- Disable any services you don’t need running 24/7 - this could include the admin panel and webmail.
- Test your email setup by sending and receiving emails from your friends. This will help catch any errors with your setup before they manifest into truly embarrassing email problems.