Self-Hosted Business Email with Docker Mailserver

When I started using my own business domain, I also wanted the email addresses to match it. Instead of using a personal email address for every conversation, I wanted addresses such as name@company.com, info@company.com, sales@company.com and support@company.com.

At first, this looks like a simple task: connect a domain to a mail server and create a few mailboxes. In practice, there is much more involved. A working business email setup depends on DNS, MX records, SPF, DKIM, DMARC, TLS, mailbox storage and outgoing mail delivery all being configured correctly.

I decided to work with a Docker-based mail environment because I already use Docker and Linux for several of my self-hosted applications.

The Setup

At a high level, my mail infrastructure follows this flow:

Business Domain
      ↓
DNS / MX
      ↓
Docker Mailserver
      ↓
Mailboxes
      ↓
Roundcube Webmail
      ↓
SMTP / Outgoing Delivery

Each component has a different responsibility. The DNS records tell other mail systems where messages for the domain should go. Docker Mailserver handles the actual mailboxes and mail protocols, while Roundcube provides a browser interface for reading and sending email.

Domain and DNS

This is where most of the actual work is, and where most mistakes happen — not in the mail server software itself.

• A mail subdomain, like mail.yourdomain.com, pointed at the server with an A (or AAAA) record.
• An MX record on the domain, telling other mail servers where to deliver messages for @yourdomain.com.
• An SPF TXT record, listing which servers are allowed to send mail for the domain.
• A DKIM TXT record, publishing the public key used to verify signed outgoing mail.
• A DMARC TXT record, telling receiving servers what to do with mail that fails SPF or DKIM, and where to send reports about it.

Each of these is a DNS record, not a mail server setting — which means propagation delay is a real factor. A record that looks correct can still fail checks for a while simply because it hasn't propagated everywhere yet.

Docker Mailserver

I run the mail server in Docker rather than installing Postfix, Dovecot, and the rest of a traditional mail stack directly on the host. The appeal is the same reason I run most of my self-hosted infrastructure in Docker: the services are isolated from each other, the configuration lives in version-controlled files instead of scattered system config, and rebuilding or migrating the setup means redeploying containers rather than reconstructing a server by hand.

That said, Docker solves the deployment problem, not the mail delivery problem. Getting the container running is the easy part. DNS, authentication, and deliverability still need to be configured correctly regardless of how the mail server itself is packaged.

Creating business email accounts

Once the server and DNS are working, adding a mailbox is mostly bookkeeping: pick an address, set a password, and it's live. Role-based addresses turned out to matter more than I expected — info@, sales@, and support@ each map to a different kind of message, so mail can be organized (or handed to a different person, or auto-responded to differently) by which address it arrived on, instead of everything landing in one inbox and being manually sorted.

I'm not going to publish the actual passwords, DKIM private key, or server details here — those stay in the server configuration, not in a blog post.

Roundcube webmail

Roundcube gives the mailboxes a normal browser-based inbox — login, inbox, sent folder, drafts, composing and replying, all through a web interface rather than needing a desktop mail client configured with IMAP/SMTP settings. For checking mail from a phone or a machine that isn't already set up with an email client, that's the practical benefit: open a browser, log in, done.

SPF, in plain terms

SPF (Sender Policy Framework) is a DNS record that lists which mail servers are allowed to send email claiming to be from your domain. When another mail server receives a message from yourdomain.com, it can check the SPF record and see whether the sending server is actually on the approved list.

It's worth being precise about what SPF does and doesn't do: it's one signal among several, not a complete anti-spoofing solution on its own. A message can still fail SPF and be delivered anyway, or pass SPF and still be judged spam for other reasons. It's meaningfully useful, but it's not the whole picture — which is exactly why DKIM and DMARC exist alongside it.

DKIM, in plain terms

DKIM (DomainKeys Identified Mail) works differently: the mail server cryptographically signs each outgoing message with a private key, and the corresponding public key is published in DNS. A receiving server can verify that signature — confirming the message really was sent by a server authorized for the domain, and that it wasn't altered in transit.

The private key never leaves the server. Only the public key goes into DNS, which is exactly how it's supposed to work.

DMARC, in plain terms

DMARC builds on both of the above. It checks whether SPF and DKIM results align with the domain in the message's visible "From" address, and lets the domain owner publish a policy for what receiving servers should do with mail that fails — plus an address to receive aggregate reports about that mail.

The detail that matters in practice: DMARC policy should be rolled out gradually. Starting at p=none (monitor and report, but don't reject anything) lets you see what the reports actually say before moving to p=quarantine and eventually p=reject. Jumping straight to a strict policy risks legitimate mail — including your own, if something in the setup isn't quite aligned yet — getting rejected outright.

SMTP relay

Sending mail directly from a self-hosted server works, but a freshly self-hosted IP has no sending history, and plenty of residential and cloud IP ranges are treated with suspicion by large mail providers regardless of how correctly SPF, DKIM, and DMARC are configured. Many hosting providers also block outbound port 25 entirely.

Routing outgoing mail through a reputable SMTP relay sidesteps both problems — it sends through infrastructure with an established reputation instead of a brand-new IP. It's not a guarantee that every message lands in the inbox rather than spam; deliverability depends on more than the sending infrastructure. But it removes one of the more common points of failure.

Testing the setup

Before trusting it, I worked through each piece separately rather than assuming that "it looks configured" meant "it works":

• DNS resolution for the mail subdomain and MX record
• Sending a test message out, and confirming it arrives
• Sending a message in, and confirming it's received
• Checking the SPF, DKIM, and DMARC results on a received test message (most major providers show these in the message headers)
• Checking whether test messages land in the inbox or the spam folder
• Replying from the receiving side, to confirm the round trip works
• Logging into Roundcube itself, not just verifying the server is running

Common problems

A rough list of what actually went wrong, versus what I expected to go wrong:

• Can receive but can't send. Usually outbound port 25 blocked by the hosting provider, or missing/misconfigured relay credentials.
• Can send but can't receive. Usually the MX record pointing at the wrong host, or a firewall blocking inbound mail ports.
• Mail landing in spam. Often SPF or DKIM not actually passing (worth checking the headers directly rather than assuming), or a DMARC policy that's stricter than the current setup can pass.
• SPF failure. Frequently a missed sending source — a relay or a script sending mail that isn't listed in the SPF record.
• DKIM failure. A selector mismatch between what's published in DNS and what the mail server is actually signing with.
• DMARC alignment issues. SPF and DKIM can each pass individually while still failing DMARC alignment, if the domains involved don't line up the way DMARC expects.
• DNS changes "not working." Usually just propagation delay — checking again after time has passed resolves more of these than any configuration change does.
• Docker container not running. The most mundane cause of "email stopped working" — worth checking before debugging DNS at all.
• Wrong mail hostname. The hostname the server identifies itself with during SMTP conversations should match what's actually in DNS, or some receiving servers get suspicious.

Security

A few things I treat as non-negotiable, though none of them make the setup "completely secure" — security here is about reducing risk, not eliminating it:

• Strong, unique passwords for every mailbox, not reused from elsewhere
• TLS for both the mail protocols and the webmail interface
• Keeping the mail server image and Roundcube updated
• Exposing only the ports that actually need to be public
• Restricting administrative access separately from mailbox access
• Watching the mail logs for repeated failed login attempts
• Rate limiting or blocking after repeated auth failures, where supported

Backups

Email is business data — invoices, client conversations, records — not a disposable cache, so it gets backed up like anything else that matters:

• Mailbox data itself
• The mail server's configuration
• DKIM-related configuration, stored securely rather than in a public repository
• The Docker volumes the whole setup depends on
• A written note of which DNS records exist and why, since "what did I actually configure six months ago" is a real question during a rebuild

What I learned

Setting up email taught me that running a mail server isn't really about starting a Docker container — that part takes minutes. DNS, authentication, sending reputation, TLS, and deliverability all have to work together, and each one can quietly break the others. It's a good example of infrastructure where the visible piece (a mailbox you can log into) is the smallest part of the actual work.

It also connects directly to the rest of what I do: the same Docker, Linux, and Cloudflare DNS work behind this mail server is what runs my self-hosted projects and the Laravel applications I deploy — DNS, containers, and server configuration aren't separate skills from web development, they're the part of it that keeps everything else actually reachable.

FAQ

Related expertise

• Self-hosting on a Raspberry Pi with Docker
• Docker, Linux and Cloudflare DNS skills
• Self-hosted infrastructure project

If you're setting up custom-domain email, DNS records, or a Docker-based mail server and want a second opinion on the configuration, you can contact me with your current setup and what isn't working. I'm also currently open to Full Stack Developer and Software Engineer opportunities in Dubai and across the UAE.