Your Own Mail Server on a VPS: Installing Postfix and Dovecot from Scratch

Running your own mail server makes sense when you need full control over email delivery on a company domain, want to understand how mail works at the protocol level, or simply do not want to depend on a third-party mail service. It is not a five-minute one-off job, though: besides installing Postfix and Dovecot you will have to watch your IP reputation, renew TLS certificates on time, and work out why a message did not arrive. This article covers only the technical part: installation, basic configuration, and a working Postfix (receives and sends mail over SMTP) and Dovecot (serves it to users over IMAP and POP3) setup on a VPS running Ubuntu 24.04 LTS.

Installing Postfix and Dovecot

Everything installs with one command: Postfix itself and three Dovecot modules, for IMAP, POP3 and LMTP (the protocol Postfix will use to hand mail over to Dovecot, explained below).

sudo apt update
sudo apt install postfix dovecot-imapd dovecot-pop3d dovecot-lmtpd

Ubuntu 24.04 ships Postfix 3.8.6 and Dovecot 1:2.3.21+dfsg1, which is enough for a new installation; you do not need third-party repositories. Postfix also provides /usr/sbin/sendmail, the classic command scripts and programs use to send mail. A tool for reading and sending mail from the terminal (the mail / mailx commands) is not included. If you need it for tests, install it separately: sudo apt install mailutils.

What the Postfix installer asks

During installation a text debconf dialog appears with two or three questions:

• General type of mail configuration. Choose Internet Site: the server will receive and send mail directly over SMTP instead of relaying everything through a smarthost.
• System mail name. This question appears only if the installer could not determine the name automatically.
• Other destinations to accept mail for. The list of domains for which the server accepts mail as local. By default it contains $myhostname, the mail name, the short host name, localhost.localdomain and localhost. This list becomes the value of the mydestination parameter in the config.

Basic Postfix configuration: main.cf

Right after installation the configuration already works, although it is minimal. You can view it in clean form (only the parameters that are set, without comments) with postconf -n. The real output is longer (around 25 lines in alphabetical order); here are only the lines that matter at this step:

smtpd_tls_security_level = may
smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key
myhostname = <short system name, NOT an FQDN>
myorigin = /etc/mailname
mydestination = $myhostname, <mailname>, <hostname>, localhost.localdomain, localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
inet_interfaces = all
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination

The important lines:

• smtpd_tls_security_level = may: STARTTLS is already on and offered to everyone who connects. The connection is encrypted from the start, even before any manual TLS setup. Whether the client trusts the certificate is a separate question.
• smtpd_tls_cert_file and smtpd_tls_key_file point to the self-signed ssl-cert-snakeoil certificate that Ubuntu generates at installation. Encryption works, but mail clients will show an untrusted certificate warning until you connect Let’s Encrypt (section below).
• myhostname is taken from the system host name. If it is not set as an FQDN (common on a fresh VPS whose hostname looks like a short service name), Postfix leaves it as is. On a production server, check the name and set it explicitly if needed: sudo postconf -e "myhostname = mail.example.com", then restart Postfix. This is the name the server announces in the SMTP greeting (HELO/EHLO), and many receiving servers pay attention to it when scoring spam reputation.
• myorigin = /etc/mailname is the domain added to the sender address for locally sent mail when none is given. The value comes from the contents of /etc/mailname.
• mydestination lists the domains whose mail is considered local and delivered locally. The official Postfix documentation (postfix.org/BASIC_CONFIGURATION_README.html) warns that removing $myhostname or localhost.$mydomain from this list can cause mail delivery loops, so change this parameter with care.
• mynetworks lists networks allowed to relay mail through this server without authentication. By default it is only localhost, which is correct. Extend it only for a specific need.
• smtpd_relay_restrictions decides who can use the server as a relay at all. The default already protects against turning the server into an open relay for spam. That is why, later, we add SASL for authenticated sending instead of weakening this parameter.

If you later need several domains with separate mailboxes (virtual addresses rather than system Linux users), Postfix has a separate mechanism, virtual mailbox domains, described at postfix.org/VIRTUAL_README.html. It uses its own parameters (virtual_mailbox_domains, virtual_mailbox_maps and so on) and is officially incompatible with listing the same domain in mydestination: the two mechanisms are mutually exclusive. This article follows the simpler path for a start, with ordinary Linux system users, so we do not go into virtual domains here.

Configuring Dovecot: switching to Maildir

Before any changes Dovecot stores mail in the mbox format, which you can see in the output of doveconf -n: mail_location = mbox:~/mail:INBOX=/var/mail/%u. mbox is one big file per mailbox with messages appended one after another. Most modern guides recommend Maildir, for good reason:

Parameter	mbox (default)	Maildir
Storage	one file per mailbox	a separate file per message
Concurrent access	requires file locking	no locking, safer with simultaneous access
Recovery after a failure	corruption can affect the whole mailbox	corruption affects a single message

Change the format in /etc/dovecot/conf.d/10-mail.conf:

sudo nano /etc/dovecot/conf.d/10-mail.conf

Find the mail_location line and replace its value with:

mail_location = maildir:~/Maildir

You do not need to configure authentication separately: out of the box Dovecot works through PAM and Linux system users (passdb { driver = pam }, userdb { driver = passwd }). In this scheme a mailbox for a new person is simply a Linux user: sudo useradd -m mailuser and sudo passwd mailuser. You do not need a separate account database. Dovecot’s TLS certificate is by default a symlink to the same self-signed snakeoil certificate Postfix uses, so IMAP and POP3 sessions are encrypted right after installation as well.

Connecting Postfix and Dovecot through LMTP

LMTP (Local Mail Transfer Protocol) is the final delivery protocol by which Postfix hands an accepted message to Dovecot, which then places it in the right user’s Maildir. This is the official, recommended way to connect them (see doc.dovecot.org).

First enable LMTP reception on the Dovecot side. Open /etc/dovecot/conf.d/10-master.conf and, inside the service lmtp { } section, add a unix socket:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
}

Now tell Postfix to use this socket for delivery. Set it in /etc/postfix/main.cf with postconf:

sudo postconf -e "mailbox_transport = lmtp:unix:private/dovecot-lmtp"

Restart both services so the changes take effect:

sudo systemctl restart dovecot postfix

The most common mistake in this setup: auth_username_format

At this step most beginners have everything technically right, yet mail still does not reach the mailbox. This is the one genuinely tricky detail in the whole installation, so it deserves its own section.

By default Dovecot uses auth_username_format = %Lu. It makes Dovecot look the user up by the full login Postfix sends, including the domain, for example testmail@example.com. The problem is that no such Linux system user exists. There is a user testmail, without a domain in the name. As a result Postfix accepts the message from outside without complaint, but at the LMTP handover to Dovecot the message bounces back to the sender with this error:

550 5.1.1 User doesn't exist

Without fixing this parameter, the Postfix and Dovecot setup described here will not work: messages will keep bouncing with this error. The fix is one line in /etc/dovecot/conf.d/10-auth.conf:

auth_username_format = %n

%n strips the domain and keeps only the name before the “@”. Dovecot now looks for testmail instead of testmail@example.com and finds the right Linux user. Restart again after the change:

sudo systemctl restart dovecot

SASL: authentication for sending mail

So far the server can only receive incoming mail. To send mail through the same server from an ordinary mail client (with a login and password), you need SASL, and the easiest way is through the same Dovecot, which already knows how to verify system users.

In /etc/dovecot/conf.d/10-master.conf, inside the service auth { } section, add a socket for Postfix:

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Tell Postfix to use this socket:

sudo postconf -e "smtpd_sasl_type = dovecot"
sudo postconf -e "smtpd_sasl_path = private/auth"
sudo postconf -e "smtpd_sasl_auth_enable = yes"

And in /etc/dovecot/conf.d/10-auth.conf set the allowed authentication mechanisms:

auth_mechanisms = plain login

The login mechanism is not there for show: it is needed for compatibility with older mail clients such as Outlook and Windows Mail that do not support plain plain. Restart both services after the changes.

Port 587: accepting mail from clients with authentication

Port 25 is meant for server-to-server mail transfer, not for sending from personal devices. Many providers and mail clients now require a separate port 587 (submission) with mandatory authentication for outgoing mail. The stock /etc/postfix/master.cf contains a submission block, but it is commented out by default.

Do not uncomment the stock lines one by one. In the stock file this block has different options, and one of them is an empty smtpd_relay_restrictions=. If you uncomment the block as is, without fixing exactly that line, you end up with an open relay on port 587: the server would start forwarding mail anywhere without a real check. Replace the whole block with this one:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

The key line is smtpd_relay_restrictions=permit_sasl_authenticated,reject: it is non-empty and requires authentication for any relaying through this port. To verify the relay is closed, connect to port 587 without authenticating and try to send a message to a foreign domain. The server should refuse (554 5.7.1 Recipient address rejected: Access denied) instead of accepting it.

If you also need port 465 with implicit TLS (for older clients that cannot do STARTTLS), in the current master.cf it is called submissions, with an “s” at the end. This is the modern name defined in RFC 8314, and older guides may call it by the outdated term “smtps”. You configure it the same way as the submission block, with TLS and SASL options.

After changing the configuration, restart Postfix:

sudo systemctl restart postfix

Firewall: opening the required ports in UFW

Even if the services already listen on all interfaces, mail will be unreachable from outside while the ports are closed by the firewall. Conveniently, the Postfix and Dovecot packages register ready-made profiles with their ports in UFW. You can list them with:

sudo ufw app list

Among others you will see Postfix, Postfix Submission, Postfix SMTPS, Dovecot IMAP, Dovecot Secure IMAP, Dovecot POP3 and Dovecot Secure POP3. It is simpler to open them by name than to remember port numbers:

sudo ufw allow Postfix
sudo ufw allow 'Postfix Submission'
sudo ufw allow 'Dovecot Secure IMAP'
sudo ufw allow 'Dovecot Secure POP3'

The unencrypted versions, plain IMAP and POP3 without TLS (ports 143 and 110), are deliberately not opened here: today’s mail clients all support TLS, and an unencrypted login means a password in clear text. If UFW has never been enabled on the server, allow SSH before running ufw enable (sudo ufw allow OpenSSH, or your non-standard port), otherwise you risk losing access to the server along with the mail setup.

One more thing to check in advance: outbound port 25 is not always open by default at VPS providers. Many block outgoing traffic on it so the server cannot be used to send spam. This is not a universal rule; it depends on the specific plan and provider. You can check it yourself:

nc -zv -w5 smtp.gmail.com 25

If netcat is not installed, you can get the same result with bash alone:

timeout 5 bash -c "echo > /dev/tcp/smtp.gmail.com/25" && echo open || echo closed

TLS certificate: from self-signed to Let’s Encrypt

A self-signed certificate encrypts traffic, but mail clients will complain about an untrusted certificate. Before issuing a real certificate, the domain mail.example.com must point to this server’s IP with an A record: certbot verifies domain ownership with an HTTP request to port 80.

There are two paths depending on whether a web server is already running on the machine:

• If nginx is already running, install the plugin and get a certificate with automatic configuration: sudo apt install certbot python3-certbot-nginx, then sudo certbot --nginx.
• If there is no web server and port 80 is free, you can issue the certificate in standalone mode: sudo certbot certonly --standalone. If port 80 is taken by another web server, standalone mode will not work. Stop that server temporarily while the certificate is issued.

After issuing the certificate, put its paths into Postfix:

sudo postconf -e "smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem"
sudo postconf -e "smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem"

And into Dovecot, in /etc/dovecot/conf.d/10-ssl.conf. The syntax matters here: the file path must be preceded by a < character, which means “read the contents of the file”:

ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

sudo systemctl restart postfix dovecot

On Ubuntu 24.04, automatic renewal does not go through cron as in older guides but through the systemd timer certbot.timer, which is already enabled after the certbot package is installed. You do not need to start anything separately. (A file /etc/cron.d/certbot is also installed by the package, but it disables itself when it sees systemd on the system, so it is not an error if you notice it there.)

The one thing worth adding is a hook so the daemons pick up the new certificate after EVERY renewal without a full restart. Important: running certbot renew --deploy-hook ... by hand right after issuing will not save anything. It will only run when a real renewal comes due (usually 30 days before expiry), and a one-off command will print “No renewals were attempted” and that is the end of it, the hook is stored nowhere. The hook must be given when the certificate is issued, so certbot writes it into the renewal config and the timer runs it every time:

sudo certbot --nginx --deploy-hook "systemctl reload postfix dovecot"

(For the certonly --standalone variant, add the same flag to the same command.) You can check that the hook was saved by looking at /etc/letsencrypt/renewal/mail.example.com.conf: a line renew_hook = systemctl reload postfix dovecot should appear there.

Checking that everything works

First, the status of the services:

systemctl status postfix
systemctl status dovecot

For Postfix the status will most likely show “active (exited)”, which is normal behavior and not an error. The postfix.service unit is Type=oneshot with ExecStart=/bin/true: effectively a stub that exists only for systemd integration. The real processes (master and its children) are started and supervised by a separate internal instance unit. You can see it with systemctl status 'postfix@*', where you will find the familiar “active (running)”. If you decide that “exited” on the main unit means “not working” and start restarting the service in circles, nothing will change, because the real problem, if there is one, is not there. Dovecot is simpler: it has the usual “active (running)” status.

Next, check that the ports are really listening:

ss -tlnp

After full setup the list should include 25 and 587 (Postfix), as well as 110, 143, 993 and 995 (Dovecot: POP3, IMAP and their secure versions).

For a sending test, swaks is convenient:

sudo apt install swaks
swaks --to user@example.com --server 127.0.0.1

If everything is configured correctly, the message should appear in ~/Maildir/new/ as a file named like <timestamp>.M<uniq>P<pid>.<hostname>,S=<size>,W=<size>. Keep the log open at the same time:

tail -f /var/log/mail.log

On successful delivery the chain in the log looks like this: postfix/smtpd accepts the connection, postfix/local determines the recipient, then comes a line about the handover to transport=lmtp, and finally Dovecot’s own record that the message was saved to INBOX. If the chain breaks at the LMTP step with an error about a nonexistent user, go back to the auth_username_format section above. It is almost always that.

What next

The server is configured and technically ready to receive and send mail, but without DNS records external servers will simply not find it. You will need an MX record pointing to this server (with priorities if there are several servers) and an SPF record allowing it to send mail on behalf of your domain.

After that, set up DKIM (a digital signature for messages) and DMARC (a policy that tells receiving servers what to do with messages that fail SPF or DKIM). Without them, mail from a new domain will very likely end up in spam, especially at large providers. And there is a separate topic that takes more than one day: the reputation of a new IP address. A fresh server simply has none, and building it requires gradually increasing sending volume and monitoring blacklists.