Linux: Setup Mail Server

Table of Contents#

1. Prerequisites
2. Choosing Mail Server Components
3. Setting Up the Core Server
   • 3.1 Install Postfix (MTA)
   • 3.2 Install Dovecot (MDA/IMAP/POP3)
4. DNS Configuration for Deliverability
   • 4.1 MX Records
   • 4.2 A & PTR Records
5. Security Hardening
   • 5.1 SSL/TLS with Let’s Encrypt
   • 5.2 SPF, DKIM, and DMARC
   • 5.3 Firewall Configuration
6. Webmail Interface (Roundcube)
7. Testing the Mail Server
8. Troubleshooting Common Issues
9. References

Prerequisites#

Before diving in, ensure you have the following:

• Linux Server: A VPS or dedicated server running Ubuntu 22.04 LTS (recommended for stability). Other distros (e.g., CentOS) work, but commands may vary.
• Static IP Address: ISPs often block port 25 for dynamic IPs, so a static IP is required.
• Domain Name: A registered domain (e.g., example.com) to use for email addresses (e.g., [email protected]).
• Root Access: SSH access with sudo privileges to configure system settings.
• Basic Linux Skills: Familiarity with the terminal, editing config files (e.g., nano/vim), and package managers (apt).

Choosing Mail Server Components#

A mail server relies on several components working together. Here’s a breakdown of key roles and recommended tools:

Setting Up the Core Server#

3.1 Install Postfix (MTA)#

Postfix is a robust, open-source MTA. Install and configure it as follows:

Step 1: Update System Packages#

sudo apt update && sudo apt upgrade -y  

Step 2: Install Postfix#

sudo apt install postfix -y  

During installation, a prompt will appear. Select Internet Site and enter your domain (e.g., example.com) as the "System mail name."

Step 3: Configure Postfix#

Edit the main configuration file:

sudo nano /etc/postfix/main.cf  

Update the following settings (replace example.com with your domain):

# Basic settings  
myhostname = mail.example.com  # FQDN of your mail server  
mydomain = example.com         # Your domain  
myorigin = $mydomain           # Emails appear to come from @example.com  
inet_interfaces = all          # Listen for connections on all interfaces  
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain  
 
# Network restrictions (prevent open relay)  
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128  
home_mailbox = Maildir/        # Store emails in ~/Maildir/ (instead of mbox)  
mailbox_command =              # Leave empty (Dovecot will handle delivery)  
 
# Enable submission port (587) for authenticated users  
smtpd_sasl_type = dovecot  
smtpd_sasl_path = private/auth  
smtpd_sasl_auth_enable = yes  
smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination  

Save and exit (Ctrl+O, Ctrl+X in nano).

Step 4: Restart Postfix#

sudo systemctl restart postfix  
sudo systemctl enable postfix  # Start on boot  

3.2 Install Dovecot (MDA/IMAP/POP3)#

Dovecot handles delivering emails to user inboxes and allows clients to fetch emails via IMAP/POP3.

Step 1: Install Dovecot#

sudo apt install dovecot-core dovecot-imapd dovecot-pop3d -y  

Step 2: Configure Dovecot#

Edit the main config file:

sudo nano /etc/dovecot/dovecot.conf  

Add/uncomment:

listen = *                  # Listen on all interfaces  
protocols = imap pop3       # Enable IMAP and POP3  

Step 3: Configure Mail Location#

Tell Dovecot where to store emails:

sudo nano /etc/dovecot/conf.d/10-mail.conf  

Set:

mail_location = maildir:~/Maildir  # Match Postfix's home_mailbox setting  

Step 4: Enable Authentication#

Dovecot uses PAM (Pluggable Authentication Modules) to authenticate system users. Edit:

sudo nano /etc/dovecot/conf.d/10-auth.conf  

Set:

disable_plaintext_auth = yes  # Require encrypted connections for auth  
auth_mechanisms = plain login  

Step 5: Configure SSL (Temporary)#

We’ll use Let’s Encrypt later, but for now, enable self-signed SSL to test:

sudo nano /etc/dovecot/conf.d/10-ssl.conf  

Set:

ssl = required  
ssl_cert = </etc/ssl/certs/dovecot.pem  
ssl_key = </etc/ssl/private/dovecot.pem  

Step 6: Restart Dovecot#

sudo systemctl restart dovecot  
sudo systemctl enable dovecot  

Step 7: Create a Test User#

Emails are tied to Linux system users. Create a test user:

sudo adduser john  # Follow prompts to set a password  

Dovecot will automatically create the ~/Maildir directory when John receives his first email.

DNS Configuration for Deliverability#

Emails won’t be delivered (or will be marked as spam) without proper DNS records. Configure these via your domain registrar (e.g., Namecheap) or DNS host (e.g., Cloudflare).

4.1 MX Records#

MX (Mail Exchange) records tell other servers where to send emails for your domain.

Add an MX record:

• Host: @ (or leave blank)
• Value: mail.example.com (FQDN of your mail server)
• Priority: 10 (lower numbers = higher priority; use 10 for a single server).

4.2 A & PTR Records#

• A Record: Maps mail.example.com to your server’s static IP.

  • Host: mail
  • Value: 192.168.1.100 (replace with your server IP).

• PTR Record (Reverse DNS): Maps your server’s IP back to mail.example.com.

  • Contact your VPS provider to set this (e.g., DigitalOcean allows PTR edits in the control panel).

Security Hardening#

5.1 SSL/TLS with Let’s Encrypt#

Encrypt emails in transit with SSL/TLS using Let’s Encrypt (free certificates).

Step 1: Install Certbot#

sudo apt install certbot python3-certbot-apache -y  # Apache is required for certbot  

Step 2: Generate SSL Certificate#

sudo certbot certonly --standalone -d mail.example.com  

Certificates will be saved to /etc/letsencrypt/live/mail.example.com/.

Step 3: Configure Postfix to Use SSL#

Edit /etc/postfix/main.cf:

# SSL settings  
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem  
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem  
smtpd_tls_security_level = may  
smtpd_tls_protocols = !SSLv2, !SSLv3  
smtp_tls_security_level = may  

Restart Postfix:

sudo systemctl restart postfix  

Step 4: Update Dovecot SSL#

Edit /etc/dovecot/conf.d/10-ssl.conf:

ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem  
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem  

Restart Dovecot:

sudo systemctl restart dovecot  

5.2 SPF, DKIM, and DMARC#

These records prevent email spoofing and improve deliverability.

SPF (Sender Policy Framework)#

SPF specifies which IPs are allowed to send emails for your domain. Add a TXT record:

• Host: @
• Value: v=spf1 ip4:192.168.1.100 -all (replace 192.168.1.100 with your server IP).
  • ip4:192.168.1.100: Allows this IP to send emails.
  • -all: Rejects emails from unlisted IPs.

DKIM (DomainKeys Identified Mail)#

DKIM signs emails with a private key; receivers verify using a public DNS record.

Step 1: Install OpenDKIM

sudo apt install opendkim opendkim-tools -y  

Step 2: Configure OpenDKIM

Edit /etc/opendkim.conf:

Domain                  example.com  
KeyFile                 /etc/opendkim/keys/example.com/mail.private  
Selector                mail  
SOCKET                  inet:8891@localhost  

Step 3: Create Key Directory

sudo mkdir -p /etc/opendkim/keys/example.com  
sudo opendkim-genkey -D /etc/opendkim/keys/example.com/ -d example.com -s mail  

This generates mail.private (private key) and mail.txt (public key).

Step 4: Set Permissions

sudo chown -R opendkim:opendkim /etc/opendkim  
sudo chmod 600 /etc/opendkim/keys/example.com/mail.private  

Step 5: Configure Postfix to Sign Emails

Edit /etc/postfix/main.cf:

milter_default_action = accept  
milter_protocol = 6  
smtpd_milters = inet:localhost:8891  
non_smtpd_milters = inet:localhost:8891  

Step 6: Add DKIM DNS Record

View the public key:

sudo cat /etc/opendkim/keys/example.com/mail.txt  

Add a TXT record with:

• Host: mail._domainkey
• Value: The content of mail.txt (e.g., v=DKIM1; k=rsa; p=...).

DMARC (Domain-based Message Authentication, Reporting, and Conformance)#

DMARC tells receivers how to handle emails that fail SPF/DKIM checks. Add a TXT record:

• Host: _dmarc
• Value: v=DMARC1; p=quarantine; sp=quarantine; adkim=s; aspf=s; fo=1; rua=mailto:[email protected]
  • p=quarantine: Quarantine (mark as spam) failed emails.
  • rua: Send daily reports to [email protected].

5.3 Firewall Configuration#

Allow essential ports using UFW (Uncomplicated Firewall):

sudo ufw allow 25/tcp      # SMTP (incoming/outgoing)  
sudo ufw allow 587/tcp     # Submission (authenticated SMTP)  
sudo ufw allow 465/tcp     # SMTPS (SSL)  
sudo ufw allow 143/tcp     # IMAP  
sudo ufw allow 993/tcp     # IMAPS (SSL)  
sudo ufw allow 110/tcp     # POP3  
sudo ufw allow 995/tcp     # POP3S (SSL)  
sudo ufw allow ssh         # Keep SSH access!  
sudo ufw enable  

Webmail Interface (Roundcube)#

Roundcube is a popular webmail client for accessing emails via a browser.

Step 1: Install Dependencies#

sudo apt install apache2 mysql-server php libapache2-mod-php php-mysql php-curl php-gd php-imagick php-intl php-json php-mbstring php-xml php-zip -y  

Step 2: Create a MySQL Database#

Log into MySQL:

sudo mysql -u root -p  

Run these commands (replace password with a strong password):

CREATE DATABASE roundcube;  
CREATE USER 'roundcube'@'localhost' IDENTIFIED BY 'password';  
GRANT ALL PRIVILEGES ON roundcube.* TO 'roundcube'@'localhost';  
FLUSH PRIVILEGES;  
EXIT;  

Step 3: Install Roundcube#

Download the latest Roundcube:

wget https://github.com/roundcube/roundcubemail/releases/download/1.6.3/roundcubemail-1.6.3-complete.tar.gz  
tar -xf roundcubemail-1.6.3-complete.tar.gz  
sudo mv roundcubemail-1.6.3 /var/www/roundcube  
sudo chown -R www-data:www-data /var/www/roundcube  

Step 4: Configure Apache#

Create a virtual host file:

sudo nano /etc/apache2/sites-available/roundcube.conf  

Add:

<VirtualHost *:80>  
    ServerName mail.example.com  
    DocumentRoot /var/www/roundcube  
 
    <Directory /var/www/roundcube>  
        Options FollowSymLinks  
        AllowOverride All  
        Require all granted  
    </Directory>  
 
    ErrorLog ${APACHE_LOG_DIR}/roundcube.error.log  
    CustomLog ${APACHE_LOG_DIR}/roundcube.access.log combined  
</VirtualHost>  

Enable the site and restart Apache:

sudo a2ensite roundcube.conf  
sudo a2enmod rewrite  
sudo systemctl restart apache2  

Step 5: Complete Roundcube Setup#

Access http://mail.example.com/installer in a browser. Follow the setup wizard:

• Database: MySQL, username roundcube, password password, database roundcube.
• IMAP Settings: localhost, port 993, SSL.
• SMTP Settings: localhost, port 587, STARTTLS.

Delete the installer directory after setup:

sudo rm -rf /var/www/roundcube/installer  

Testing the Mail Server#

Step 1: Send a Test Email#

Log into Roundcube via https://mail.example.com (use Let’s Encrypt to enable HTTPS for Apache). Compose an email to a personal account (e.g., Gmail) and send it.

Step 2: Verify Deliverability#

• Check if the email arrives in the inbox (not spam).
• Use MX Toolbox to validate DNS records and SPF/DKIM/DMARC.
• Check Postfix logs for errors:

  sudo tail -f /var/log/mail.log  

Troubleshooting Common Issues#

• Emails not sending: Check /var/log/mail.log for errors (e.g., DNS issues, port 25 blocked by ISP).
• Emails marked as spam: Ensure SPF/DKIM/DMARC records are correct (use MX Toolbox).
• Roundcube login failed: Verify Dovecot is running (sudo systemctl status dovecot) and IMAP ports are open.

References#

• Postfix Official Documentation
• Dovecot Wiki
• Let’s Encrypt Certbot Guide
• OpenDKIM Setup Guide
• Roundcube Documentation
• MX Toolbox (DNS and mail server testing)

By following these steps, you’ve built a secure, self-hosted mail server on Linux. Regularly update packages (sudo apt update && sudo apt upgrade) and monitor logs to keep it running smoothly! 🚀