Install your own mail server with Modoboa on Debian

Would you rather keep your email on your own server under your own domain instead of leaving it with a free webmail provider? Modoboa is one of the most convenient ways to do that. The installer it ships with sets up Postfix, Dovecot, a database, a spam filter, webmail and an administration interface in a single run.

This guide covers the installation on Debian 12 and Debian 11. Debian 10 is no longer supported and should not be used for a new mail server.

Requirements

• Your own VPS or root server with a fresh Debian installation. We recommend using a small server exclusively for mail (not mandatory, but sensible), because the hostname and the reverse DNS entry get adapted to the mail domain.
• Root access over SSH.
• Your own domain where you can set the DNS records yourself.
• A static IPv4 address for which you can create a reverse DNS entry.
• The mail ports have to be reachable: 25 (SMTP), 587 and 465 (sending by your users), 143 and 993 (IMAP) plus 80 and 443 for webmail and for issuing the certificate.

Important: run the installation on a freshly installed system. An already running web server or an existing database will regularly make the installer abort.

Preparing the system

First bring the package list and all installed packages up to date:

apt update && apt upgrade -y

Then install the packages that the installer expects:

apt-get install sudo git python3 mariadb-server -y

Finish the database setup and answer the questions of the wizard. Choose a strong password for the database administrator:

sudo mysql_secure_installation

Setting the hostname

The server needs a fully qualified hostname that matches the mail domain you will use later. Without this step the installation fails:

hostnamectl set-hostname mail.<DOMAIN>.<TLD>

Ideally you create the A record for exactly this name in DNS right now. The installer requests a Let's Encrypt certificate later on, and that only works if the name already points to your server IP.

Downloading the Modoboa installer

cd /tmp
git clone https://github.com/modoboa/modoboa-installer
cd modoboa-installer

Creating and adjusting the configuration file

First let the installer create only the configuration file and stop after that:

python3 ./run.py --stop-after-configfile-check <DOMAIN>.<TLD>

Open the generated file:

nano installer.cfg

Adjust the following sections. This gives the server a free SSL certificate from Let's Encrypt and connects it to the local database:

[certificate]
generate = true
type = letsencrypt

[letsencrypt]
email = admin@<DOMAIN>.<TLD>

[database]
engine = mysql
host = 127.0.0.1
install = true

Use an email address that you actually read. Let's Encrypt sends notifications there if a renewal does not go through.

Starting the installation

python3 ./run.py mail.<DOMAIN>.<TLD>

The run takes a few minutes. Use the time to set the DNS records.

Setting the DNS records

A record and MX record

The A record points to your server IP, and the MX record tells the world which host accepts mail for your domain. The priority "10" is the usual value when there is only one mail server:

1st record:
Subdomain: mail
Type: A (IPv4)
Target: <Server-IP>
The result should look like this: mail.<DOMAIN>.<TLD> IN A <Server-IP>

2nd record:
(Subdomain: @)
Type: MX
Target: mail.<DOMAIN>.<TLD>
The result should look like this: @ IN MX 10 mail.<DOMAIN>.<TLD>

SPF, DKIM and DMARC

Without these three records your mail will very likely end up in the spam folder of your recipients. Large providers now require them explicitly.

SPF defines which servers are allowed to send for your domain. As a TXT record on the main domain:

Type: TXT
Name: @
Value: v=spf1 mx -all

DKIM signs your outgoing mail cryptographically. Modoboa can generate the keys per domain on its own. After the installation you find the public key in the administration interface, in the settings of the respective domain. Enter that value as a TXT record in your DNS.

DMARC tells the recipients how to handle mail that fails SPF or DKIM. Start with a monitoring setting and tighten it later:

Type: TXT
Name: _dmarc
Value: v=DMARC1; p=none; rua=mailto:postmaster@<DOMAIN>.<TLD>

Reverse DNS (PTR)

The PTR record resolves your IP address back to the hostname. Many mail servers flatly reject connections without a matching PTR record. With us you set it yourself in the customer panel:

"My Services" -> "select server" -> "click the Reverse DNS button" -> mail.<DOMAIN>.<TLD>

What matters is that the PTR record and the hostname match exactly.

First steps after the installation

If the installation went through, reboot the server once:

reboot

After that you reach the administration interface at https://mail.<DOMAIN>.<TLD>. Log in with the administrator account that the installer created, and change the password as the very first step. Then create your domain and the mailboxes you need.

Next, test in both directions: send a message from your new server to an external mailbox, and one from outside to your new address. The quickest way to verify that SPF, DKIM, DMARC and PTR really take effect is one of the freely available mail testers on the web.

Common problems

• The installation aborts immediately: the hostname is not a fully qualified domain name. Check it with hostname -f.
• The certificate is not issued: the A record for mail.yourdomain does not point to the server, or port 80 is blocked. Let's Encrypt validates the domain over port 80.
• Mail ends up in spam: usually the PTR record is missing, or SPF and DKIM are not set. Check all four records before you run the server in production.
• Outgoing mail is rejected: the IP address may be on a block list. Most operators offer a way to look this up and to request removal.
• The installer reports a database error: MariaDB has not been finished with mysql_secure_installation, or the database section in installer.cfg does not fit.
• Nothing works after a reboot: check the services involved with systemctl status postfix dovecot nginx.

A mail server of your own needs more care than other services: updates, backups of the mailboxes and an eye on deliverability. If you take that on, you get full control over your communication in return.

Author: XEfnexX (pseudonym)