4.52

OpenSMTPD with Dovecot, virtual users and domains, SQLite3, RSpamd antispam

Image
Configure OpenSMTPD mail server with RSpamd antispam, integrate with Dovecot IMAP4 storage, share a virtual accounts / authentication system of users / domains based on SQLite3 between OpenSMTPD and Dovecot.

How to use OpenSMTPD (with RSpamd), sharing virtual accounts and authentication with Dovecot IMAP4 - using an SQL database as storage provider for all accounts

This article presents a working scenario tested under several production servers. It is intended for small and medium scale. For large scale instead of SQLite3 database you can adapt this article and use a database server such as PostgreSQL or MySQL.


The article covers the setup for OpenBSD, Debian Linux, Ubuntu Linux operating systems.
All the below setup and operations must use the root system account.

Required packages for this setup:


(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script
Install required packages - OpenBSD
OpenBSD comes by default with OpenSMTPD mail server installed in the base system. Install just the SQLite3, OpenSMTPD extras, RSpamd, Dovecot packages using a shell:
$ pkg_add sqlite3
$ pkg_add opensmtpd-extras
$ pkg_add rspamd
$ pkg_add dovecot
(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script
Install required packages - Debian or Ubuntu Linux (in this example only ; for other Linux distros see the specific distro documentation how to install packages)
For Debian Linux, Ubuntu Linux install the SQLite3, OpenSMTPD, OpenSMTPD extras, RSpamd, Dovecot (imapd, lmtpd, sqlite) packages using a shell:
$ apt install sqlite3
$ apt install opensmtpd
$ apt install opensmtpd-extras
$ apt install rspamd
$ apt install dovecot-imapd
$ apt install dovecot-lmtpd
$ apt install dovecot-sqlite

For compatibility with all email clients this setup will use the plain and the login authentication mechanisms available in Dovecot IMAP4 server.
For security reasons, as the plain or the login authentication are not very secure it will enforce the use of TLS1.2 as required in both: Dovecot and OpenSMTPD servers for all clients.

(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script

Setup of the TLS certificates for both servers: OpenSMTPD and Dovecot IMAP4

There are several options to have TLS certificates (RSA or EC): get them from a certificate authority or use self-generated certificates. To self-generate the certificates (RSA), use the below example by running the following commands in a shell:
$ cd ~
$ openssl req -new -x509 -nodes -newkey rsa:4096 -keyout mail.key -out mail.crt -days 365
$ mkdir /etc/ssl/mail-servers
$ mv mail.key /etc/ssl/mail-servers/
$ mv mail.crt /etc/ssl/mail-servers/

(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration OpenSMTPD Server opensmtpd conf

OpenSMTPD base config, including the config to use RSpamD filter and the SQLite3 authentication extras

Edit the OpenSMTPD config file smtpd.conf to add or adapt the following lines in config as below. If using certificates from a certificate authority uncomment the line containing the `ca.crt` to enable the verification of the certificate authority too. If using EC certificates uncomment also the line which ends with `dhe auto`. The example below will use a dummy host name such as `my-email.domain`. This will setup also the required config for OpenSMTPD to authenticate using an SQLite3 database (will be setup below, later). Add the RSpamd filter to be used by OpenSMTPD. Setup the OpenSMTPD delivery system and auth to use shared authentication with Dovecot via the SQLite3 database accounts.
OpenBSD OpenSMTPD config: /etc/mail/smtpd.conf
Debian Linux / Ubuntu Linux OpenSMTPD config: /etc/smtpd.conf
### OpenSMTPD config file: smtpd.conf

## enable TLS high only ciphers
smtp ciphers HIGH

## setup of TLS certificates
#ca  my-email.domain  cert "/etc/ssl/mail-servers/ca.crt"
pki  my-email.domain  key  "/etc/ssl/mail-servers/mail.key"
pki  my-email.domain  cert "/etc/ssl/mail-servers/mail.crt"
#pki my-email.domain  dhe  auto

## OpenSMTPD Auth against SQlite3
table credentials  sqlite:/etc/vmail-auth/opensmtpd-auth-sqlite.conf
table vdomains     sqlite:/etc/vmail-auth/opensmtpd-auth-sqlite.conf
table vusers       sqlite:/etc/vmail-auth/opensmtpd-auth-sqlite.conf
table vuserinf     sqlite:/etc/vmail-auth/opensmtpd-auth-sqlite.conf

# RSpam filter for OpenSMTPD ; will use port 25 standard and port 587 for authenticate virtual users for SMTP send messages
filter "rspamd" proc-exec "filter-rspamd"
listen on lo0    port 25                                              filter "rspamd"
listen on egress port 25  tls         pki unix.uxm                    filter "rspamd"
listen on lo     port 587 tls-require pki unix.uxm auth <credentials> filter "rspamd"
listen on egress port 587 tls-require pki unix.uxm auth <credentials> filter "rspamd"

# Deliver the email messages directly to Dovecot IMAP4 via Dovecot LMTP (use LMTP IP:PORT not socket, which may require aditional setup of privileges ... !!!)
#action "deliver-virtual" lmtp "/var/dovecot/lmtp" rcpt-to virtual <vusers>
action "deliver-virtual" lmtp "127.0.0.1:10024" rcpt-to virtual <vusers> userbase <vuserinf>
match from any for domain <vdomains> action "deliver-virtual"

# Deliver local system messages for the existing real system accounts
action "deliver-relay" relay
match      from local for any action "deliver-relay"
match auth from any   for any action "deliver-relay"


(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Dovecot IMAP4 Server dovecot conf

Dovecot base config, including the config to use SQLite3 authentication extras

Edit the Dovecot config file dovecot.conf to add or adapt the following lines in config as below. If using certificates from a certificate authority replace the `mail.crt` with `full-chain.crt`. The `full-chain.crt` can be created by running in a shell the following command: cat mail.crt ca.crt > full-chain.crt. It must use the user/group vmail/vmail with UID=5200 and GID=5200 and the plain and login as auth mechanisms with using TLS as security provider. Using other auth mechanisms in Dovecot such as cram-md5 or others needs a slighly different approach in the way that passwords are stored in SQLite3 database and may not be compatible with all versions and all clients. You can combine this configuration to use the mail-crypt encrypted storage for Dovecot.
### Dovecot config file: dovecot.conf

ssl = required
ssl_min_protocol = TLSv1.2
ssl_cipher_list = HIGH
ssl_key = </etc/ssl/mail-servers/mail.key
## without CA (simple certificate)
ssl_cert = </etc/ssl/mail-servers/mail.crt
## with CA (full chain certificate)
#ssl_cert = </etc/ssl/mail-servers/full-chain.crt

disable_plaintext_auth = no
auth_mechanisms = plain login

## If something does not work after all the setup is completed uncomment the following two lines and look at the messages in Dovecot's logs to have a clue what is wrong ...
#auth_debug = yes
#auth_debug_passwords = yes

### vmail: uid=5200 ; gid=5200 ###
mail_uid = vmail
mail_gid = vmail
mail_location = maildir:/var/vmail/%u
mail_plugins = quota

# use the SQLite auth (will be setup below)
passdb {
    driver = sql
    args = /etc/vmail-auth/dovecot-auth-sqlite.conf.ext
}
userdb {
    driver = sql
    args = /etc/vmail-auth/dovecot-auth-sqlite.conf.ext
}

# lmtp is required because OpenSMPTD will use it to deliver incoming email messages to Dovecot directly
protocols = lmtp imap

base_dir = /var/dovecot/

postmaster_address = postmaster@my-email.domain
mail_max_userip_connections = 10

service lmtp {
    inet_listener lmtp {
      address = 127.0.0.1
      port = 10024
    }
    unix_listener lmtp {
      group = vmail
      user = vmail
      mode = 0660
    }
}

plugin {
    quota = maildir:User quota
    quota_rule = *:storage=50MB
    quota_warning = storage=95%% quota-warning 95 %u
    quota_exceeded_message = Fail: User Quota Exceeded
}

# IMAP folders definition

namespace inbox {

    inbox = yes
    separator = .

    mailbox "Drafts" {
    auto = subscribe
    special_use = \Drafts
    }

    mailbox "Sent" {
    auto = subscribe
    special_use = \Sent
    }
    mailbox "Sent Mail" {
    auto = no
    special_use = \Sent
    }

    mailbox "Trash" {
    auto = subscribe
    special_use = \Trash
    }
    mailbox "TRASH" {
    auto = no
    special_use = \Trash
    }

    mailbox "Junk" {
    auto = subscribe
    special_use = \Junk
    }
    mailbox "Spam" {
    auto = no
    special_use = \Junk
    }

    mailbox "Archive" {
    auto = create
    special_use = \Archive
    }
    mailbox "Archives" {
    auto = no
    special_use = \Archive
    }

    mailbox "Notes" {
    auto = create
    }

}


(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script

The vmail user and group required for the shared storage of email messages for both OpenSMTPD and Dovecot

Create the vmail user and group with the exact UID=5200 and GID=5200, using a shell as below:
$ mkdir /var/vmail
$ groupadd -g 5200 vmail
$ useradd -g vmail -u 5200 -s /sbin/nologin -d /var/vmail -m vmail
$ chown /var/vmail vmail:vmail

(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration SQL: SQLite3 sql script

The SQLite3 database required as shared authentication for email accounts for both OpenSMTPD and Dovecot

Create the SQLite3 Database dump file. The database structure should be stored in a file named /tmp/vmail-db-sqlite.dump.sql. The database structure contains the SQL schema and data (sample with 2 domains, 3 accounts and 1 alias), can be seen below:
--- SQlite dump file: vmail-db-sqlite.dump.sql

BEGIN TRANSACTION;

---
--- Schema
---


---- Table: vdomains (Virtual Domains)


CREATE TABLE vdomains (
    'id' INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    'domain' VARCHAR(63) NOT NULL
);

CREATE UNIQUE INDEX 'vdomains_uidx_id' ON `vdomains` (`id` ASC);
CREATE UNIQUE INDEX 'vdomains_idx_domain' ON `vdomains` (`domain` ASC);


--- Table: vmail (Virtual Email Addresses)


CREATE TABLE 'vmail' (
    'id' INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    'account' VARCHAR(96) NOT NULL,
    'pass' VARCHAR(255) NOT NULL DEFAULT '',
    'quota_mb' INTEGER NOT NULL DEFAULT 100,
    'quota_msg' INTEGER NOT NULL DEFAULT 1000
);

CREATE UNIQUE INDEX 'vmail_uidx_id' ON `vmail` (`id` ASC);
CREATE UNIQUE INDEX 'vmail_uidx_account' ON `vmail` (`account` ASC);
CREATE INDEX 'vmail_idx_pass' ON `vmail` (`pass`);


--- Table: valias (Virtual Email Aliases)


CREATE TABLE valias (
    'id' INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    'recipient' VARCHAR(96) NOT NULL,
    'destination' VARCHAR(96) NOT NULL,
    'virtualid' INTEGER NOT NULL DEFAULT 0
);

CREATE UNIQUE INDEX 'valias_uidx_id' ON `valias` (`id` ASC);
CREATE INDEX 'valias_idx_recipient' ON `valias` (`recipient` ASC);
CREATE INDEX 'valias_idx_destination' ON `valias` (`destination` ASC);
CREATE INDEX 'valias_idx_virtualid' ON `valias` (`virtualid` ASC);


--- Table: _info (MetaInfo)


CREATE TABLE '_info' (
    'key' VARCHAR(100) PRIMARY KEY NOT NULL,
    'value' TEXT NOT NULL
);

CREATE UNIQUE INDEX '_info_uidx_key' ON `_info` (`key` ASC);


---
--- Data (edit this data to follow your particular needs) ; This is only a sample !!!
---

--- create 2 sample email domains
INSERT INTO `vdomains` (`id`,`domain`) VALUES ('1','my-email.domain');
INSERT INTO `vdomains` (`id`,`domain`) VALUES ('2','my-email.domain2');

--- create 3 sample email accounts
--- the password for all 3 accounts below is just a sample: 'sample-password'
--- the encrypted password password has been created using the dovecot admin utility as (run in a shell):
--- doveadm pw -s BLF-CRYPT
--- all three accounts below have a sample quota of max 100MB and max 7500 messages
INSERT INTO `vmail` (`id`,`account`,`pass`,`quota_mb`,`quota_msg`) VALUES ('1','account1@my-email.domain','{BLF-CRYPT}$2y$05$S0d1Dif4B.F2.Rhw6R7.JuYvsoDwv6CmkHcrZ8JbP19eC.NyXmxjC','100','7500');
INSERT INTO `vmail` (`id`,`account`,`pass`,`quota_mb`,`quota_msg`) VALUES ('2','account2@my-email.domain','{BLF-CRYPT}$2y$05$W2OnqAK82wpZCs4E6/G21OJnxXVVb99no/EMqDrL8ZCVEox6ifi/y','100','7500');
INSERT INTO `vmail` (`id`,`account`,`pass`,`quota_mb`,`quota_msg`) VALUES ('3','account@my-email.domain2','{BLF-CRYPT}$2y$05$6d20muKiYPjZeUjvFvQYUODeR6C4YbAEmVO.xGVfZcYgd5hjdBJZm','100','7500');

--- create an email alias as postmaster for the domain my-email.domain
INSERT INTO `valias` (`id`,`recipient`,`destination`,`virtualid`) VALUES ('1','postmaster@my-email.domain','account2@my-email.domain','1');
--- create an email alias as postmaster for the domain my-email.domain2
INSERT INTO `valias` (`id`,`recipient`,`destination`,`virtualid`) VALUES ('2','postmaster@my-email.domain2','account@my-email.domain2','3');

---

INSERT INTO `_info` (`key`,`value`) VALUES ('about','Virtual Mail System for OpenSMTPD + Dovecot');
INSERT INTO `_info` (`key`,`value`) VALUES ('copyright','(c) 2022 w3soft.org');
INSERT INTO `_info` (`key`,`value`) VALUES ('license','BSD');

---

COMMIT;

--- #END dump


(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script

Create the SQLite3 Database file

The SQLite3 database should be created as /etc/vmail-auth/vmail-auth.sqlite3 using the SQLite3 Database dump file from above, previous stored as /tmp/vmail-db-sqlite.dump.sql. This DB file will be used as the shared Authentication for of the email virtual domains, accounts and aliases for both Dovecot and OpenSMTPD. Type in a shell the following commands:
$ mkdir /etc/vmail-auth
$ cd /etc/vmail-auth
$ sqlite3 vmail-auth.sqlite3 < /tmp/vmail-db-sqlite.dump.sql

Create or update a domain

Use the table vdomains, and INSERT or UPDATE accordingly also the email accounts in table vmail and the aliases (if any) in the valias table. Look at the sample SQL data from above to have a hint how.

Create or update an alias

Use the table valias and look at the sample SQL data from above for a hint. The table valias structure is like this: id is the unique ID of the alias ; the field recipient is the alias email address that must not exist in the real accounts table vmail ; the fields destination and virtualid must be in sync and have a value as destination = vmail.account and virtualid = vmail.id from a specific record that already exists in the table vmail. Cross domain aliases are allowed too by this setup.

Create or update an account

Use the table vmail. For INSERT a new unique id must be issued. For UPDATE never update the unique ID field id in this table or other tables, the id field have to remain as it was created first time. UPDATE is allowed for any of the following fields: account as email address, pass as password, quota_mb as quota in megabytes (use 0 for no quota), quota_msg as the quota for the maximum number of messages to this account (use 0 for no quota).

(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Unix / Linux Shell: Bash, Ksh shell script

Create or update a password for an email account

The passwords are encrypted using BLF-CRYPT and stored in the SQlite3 database in the vmail table, column pass. To create a new password, run the following command in a shell, enter the password and get the output of the command and update it in the SQlite3 DB by example:
{BLF-CRYPT}$2y$05$W2OnqAK82wpZCs4E6/G21OJnxXVVb99no/EMqDrL8ZCVEox6ifi/y as password.
The encrypted output for the same password will be different most of the time because the password utility includes a password salt automatically for this type of password.
$ doveadm pw -s BLF-CRYPT
(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration SQL: SQLite3 sql script

Update the password in the SQLite3 DB

Let's say the above command output was: {BLF-CRYPT}$2y$05$W2OnqAK82wpZCs4E6/G21OJnxXVVb99no/EMqDrL8ZCVEox6ifi/y.
Connect to SQLite3 database using sqlite3 vmail-auth.sqlite3 and you will get a prompt like sqlite>. Then enter the following SQL statement as below, over the SQLite3 prompt to update the password for the account with id=1 by example. If the account has a different id adapt the SQL statement below. Adapt the SQL statement from below with the real password provided by the password utility from above. Write the following statement (don't forget the trailing ; to close the statement, then hit ENTER.
UPDATE vmail SET pass = '{BLF-CRYPT}$2y$05$W2OnqAK82wpZCs4E6/G21OJnxXVVb99no/EMqDrL8ZCVEox6ifi/y' WHERE id = 1;

After this quit from SQlite3 by typing the .quit command and hit ENTER (ex: sqlite> .quit).

(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Dovecot IMAP4 Server dovecot conf

OpenSMTPD specific config to use the shared storage with Dovecot and authenticate against the shared SQLite3 database

Create the OpenSMTPD auth file as /etc/vmail-auth/opensmtpd-auth-sqlite.conf with the following content for using SQLite3 database shared auth. Notice, it uses the exact UID=5200 and GID=5200 as the vmail user/group created above. Failing to set the exact UID/GID as defined with the vmail above will fail sharing the email messages between OpenSMTPD and Dovecot in a proper way.

### OpenSMTPD config, auth SQLite3: opensmtpd-auth-sqlite.conf
## user/group: vmail/vmail as UID=5200 ; GID=5200

dbpath /etc/vmail-auth/vmail-auth.sqlite3

# Domain lookup query ; rows = 1 ; fields = 1 (domain VARCHAR)
query_domain SELECT domain FROM vdomains WHERE ((domain = ?) AND (domain != '')) LIMIT 1 OFFSET 0;

# Alias lookup query ; rows >= 0 ; fields = 1 (email VARCHAR)
query_alias SELECT 'vmail_' || virtualid AS alias FROM valias WHERE ((recipient = ?) AND (recipient != '')) LIMIT 250 OFFSET 0;

# UserInfo query ; rows = 1 ; fields = 3 (uid INTEGER, gid INTEGER, maildir VARCHAR, password VARCHAR)
query_userinfo SELECT 5200 AS uid, 5200 AS gid, 'var/vmail/deliver-virtual:lmtp/vmail_' || id AS maildir FROM vmail WHERE (('vmail_' || id = ?) AND (account != '') AND (pass != '')) LIMIT 1 OFFSET 0;

# Credentials lookup query ; rows = 1 ; fields = 2 (email VARCHAR, password VARCHAR)
query_credentials SELECT account AS user, pass AS password FROM vmail WHERE ((account = ?) AND (account != '') AND (pass != '')) LIMIT 1 OFFSET 0;


(c) 2022 w3soft.org, license: BSD learning resources for software development and operating systems administration configuration Dovecot IMAP4 Server dovecot conf

Dovecot specific config to use the shared storage with OpenSMTPD and authenticate against the shared SQLite3 database

Create the Dovecot auth file as /etc/vmail-auth/dovecot-auth-sqlite.conf.ext with the following content for using SQLite3 database shared auth. Notice, it uses the exact UID=5200 and GID=5200 as the vmail user/group created above. Failing to set the exact UID/GID as defined with the vmail above will fail sharing the email messages between OpenSMTPD and Dovecot in a proper way.

### Dovecot config, auth SQLite3: dovecot-auth-sqlite.conf.ext
## user/group: vmail/vmail as UID=5200 ; GID=5200

driver = sqlite
connect = /etc/vmail-auth/vmail-auth.sqlite3

# used by IMAP to check authentication on login
default_pass_scheme = BLF-CRYPT
password_query = SELECT account AS user, '{BLF-CRYPT}' || pass AS password FROM vmail WHERE ((account = '%u') AND (account != '') AND (pass != '')) LIMIT 1 OFFSET 0

# used by LMTP to know where to deliver (the OpenSMTPD will have to choose the real user for aliases ...)
user_query = SELECT '/var/vmail/' || vmail.account || '/' AS home, 'maildir:/var/vmail/' || vmail.account || '/' AS mail, 5200 AS uid, 5200 AS gid, '*:storage=' || vmail.quota_mb || 'MB:messages=' || vmail.quota_msg AS quota_rule FROM vmail WHERE ((account = (SELECT valias.destination FROM valias WHERE (valias.recipient = '%u') LIMIT 1 OFFSET 0)) AND (account != '') AND (pass != '')) LIMIT 1 OFFSET 0


Final considerations: Important

DISCLAIMER for this article
  • the method in this article was tested to work with OpenSMTPD 6.7, RSpamD 3.2 and Dovecot 2.3 ; it may not work as expected with other versions
  • the w3soft.org is not responsible for any data loss or other issues (including security) regarding the usage of OpenSMTPD, RSPamD, Dovecot and SQLite explained in this article
  • if something is not working after this setup is completed check the following:
    • the /var/vmail path is owned recursively by vmail user/group: chown -R vmail:vmail /var/vmail
    • the following ports have to be opened in the firewall: OpenSMTPD 25 and 587 ; Dovecot 143 and 993
    • the TLS certificates are readable by both servers: OpenSMTPD and Dovecot
    • the SQLite3 database is readable by both servers: OpenSMTPD and Dovecot
    • Dovecot's LMTP is running and listening on 127.0.0.1 at port 10024 so OpenSMTPD to be able to deliver incoming messages directly to Dovecot
    • check if Dovecot's authentication works correctly by using an email IMAP4 client to connect to Dovecot on port 143 TLS or 993 SSL using an email account (username is the same as the email address) and a password defined in SQLite3 table vmail
    • check if OpenSMTPD's authentication works correctly by using an email SMTP client to connect to OpenSMTPD on port 25 STARTTLS or 585 SSL using an email account (username is the same as the email address) and a password defined in SQLite3 table vmail
    • check if the incoming messages are working ; check if you can send a message
    • try to debug the Dovecot's authentication by uncommenting the auth_debug_passwords or auth_debug in dovecot.conf
    • check OpenSMTPD configuration by running in a shell smtpd -n
    • check Dovecot configuration by running in a shell doveconf 1>/dev/null && echo $?
      Adapting this example to the real world:
    • use TLS certificates provided by a real Certificate Authority
    • use real domains, real email accounts and real passwords not the sample ones as provided by this example
    • double check your security, firewall and privileges before setting up a live real email server with OpenSMTPD / Dovecot / RSpamD with an SQLite3 auth backend and shared storage as in this example
    • sometimes changing data in the SQLite3 database such as domains or accounts needs to restart the both OpenSMTPD and Dovecot services.