Postfix, Dovecot and OpenLDAP on Fedora 21
by Sebastien Mirolo on Mon, 30 Mar 2015
So far we had run the mail server with static hash files. As we add
contributors and machines in the mix, it is about time to introduce
a central account service (i.e. OpenLDAP). Since Rackspace and Amazon
deprecated their first generation instances, re-building the mail
infrastructure from scratch on a new system seemed appropriate - plus,
it is fun!
Following is a diagram of how we planned to hook-up the authentication for accessing inboxes and sending e-mails.
The idea is to configure PAM to authenticate users through LDAP, then have Dovecot (IMAP/POP) to rely on PAM authentication and Postfix (SMTP) rely on Dovecot pass through authentication.
Most setup deal with multiple e-mail addresses through alias maps and forwards
but here, we wanted to have physically separate inboxes for each email
address and a single user owning multiple addresses, as in:
dn: uid=johnd,ou=people,dc=mydomain,dc=com objectClass: inetorgperson cn: John Doe mail: johnd@mydomain.com mail: info@mydomain.com
This last requirement turned out in a puzzle of incompatible pieces to fit together. We did. Here is the story of that journey.
Setting up OpenLDAP
The first step was to bring-up OpenLDAP. OpenLDAP is notoriously complex to setup because the error messages are cryptic and often unrelated to the root cause of the issue.
We will install both the OpenLDAP servers, and clients - there is a high probability we will need to run some command lines to probe and test the setup.
$ yum install openldap-servers openldap-clients
By default slapd uses syslog to send information and log error messages. Since we are using the syslog-ng variant, we will add a file as such:
$ vi /etc/syslog-ng/conf.d/slapd.conf
destination d_ldap { file("/var/log/slapd.log"); };
filter f_ldap { program(slapd); };
log { source(s_sys); filter(f_ldap); destination(d_ldap); };
$ sudo systemctl restart syslog-ng
We are testing the setup on Fedora 21 here so we need to come back a bit on how OpenLDAP is compiled on Fedora systems. We will want to use ldaps:// (LDAP over TLS) which means we will have to configure keys and certificates. That is where things get serious. No matter how we configure the various certificate databases, we end-up with an error messages that read:
TLS: error: connect - force handshake failure: errno 0 - moznss error -5938 TLS: can't connect: TLS error -5938:Encountered end of file. # or TLS: cannot open certdb '/etc/openldap', error -8018:Unknown PKCS #11 error.
By default on Fedora, OpenLDAP is linked with the moznss implementation of TLS. You can check by running ldd
$ ldd /usr/sbin/slapd | grep nss
libnss3.so => /lib64/libnss3.so (0x00007f621ed2a000)
libnssutil3.so => /lib64/libnssutil3.so (0x00007f621d636000)
You might try to get moznss certificate database to work. If so, here and here are good starting points.
Fortunately, there are two other common open source implementation of TLS: openssl and gnutls. Now OpenSSH, Dovecot and Postifx all are linked with OpenSSL by default on Fedora. So you can see the use of MozNSS in OpenLDAP as either a security feature (a different code base) or as a security risk (a different code base).
[UPDATE: From OpenLDAP-and-MozNSS-Compatibility-Layer: " OpenLDAP in Fedora had been compiled with NSS (MozNSS, i.e. Mozilla Network Security Services) as a crypto library for several years. This effort was driven only by Fedora downstream and its derivatives. However, this implementation had not been optimal and for that it was later decided to move back to OpenSSL which is the preferred crypto library within OpenLDAP upstream community. "]
We just went ahead and recreated RPM packages that are linked against OpenSSL.
# 1. Download the source package and prerequisites
$ sudo yum install yum-utils rpmdevtools cracklib-devel cyrus-sasl-devel \
groff krb5-devel libtool libtool-ltdl-devel nss-devel openssl-devel \
tcp_wrappers-devel unixODBC-devel perl-ExtUtils-Embed
$ rpmdev-setuptree
$ yumdownloader --source openldap
$ rpm -i openldap-2.4.40-2.fc21.src.rpm
# 2. Update the spec file and rebuild the packages
$ diff -u prev ~/rpmbuild/SPECS/openldap.spec
- --with-tls=moznss \
+ --with-tls=openssl \
$ rpmbuild -ba ~/rpmbuild/SPECS/openldap.spec
# 3. Install the RPMs
$ rpm -i ~/rpmbuild/RPMS/x86_64/openldap-2.4.40-2.fc21.x86_64.rpm --force
$ rpm -i ~/rpmbuild/RPMS/x86_64/openldap-servers-2.4.40-2.fc21.x86_64.rpm --force
$ rpm -i ~/rpmbuild/RPMS/x86_64/openldap-clients-2.4.40-2.fc21.x86_64.rpm --force
# 4. Check sldapd is linked against the expected libraries
$ ldd /usr/sbin/slapd | grep ssl
libssl.so.10 => /lib64/libssl.so.10 (0x0000xxxxxxxxx000)
Sure enough the slapd service will not start right of the box. It can't open the bind ldapi:// to a socket. SELinux won't let it do so - easily fixed.
$ sudo sh -c 'grep slapd /var/log/audit/audit.log | audit2why -M slapd'
type=AVC msg=audit(1422216270.205:722): avc: denied { unlink } for pid=1520\
comm="slapd" name="ldapi" dev="tmpfs" ino=15555 \
scontext=system_u:system_r:slapd_t:s0 \
tcontext=unconfined_u:object_r:var_run_t:s0 tclass=sock_file permissive=0
$ sudo sh -c 'grep slapd /var/log/audit/audit.log | audit2allow -M slapd'
$ semodule -i slapd.pp
$ sudo rm slapd.pp slapd.te
At first we can configure OpenLDAP to point to the exact same key and cert files we are using for Dovecot and Postfix. That would work but it is not ideal when we are looking to extend the use of OpenLDAP to support single sign-on.
Since there are no reasons we would want the OpenLDAP server to be available on a public DNS, nor any reason unknown parties should talk to the server, we will use a self-signed certificate.
# Create private key and Certificate Signing Request (CSR)
$ sudo openssl req -new -sha256 -newkey rsa:2048 -nodes \
-keyout /etc/pki/tls/private/ldap.key \
-out /etc/pki/tls/certs/ldap.csr
Common Name (eg, your name or your server's hostname) []: Use private IP address here
# Self-signing
$ openssl x509 -req -days 365 -in /etc/pki/tls/certs/ldap.csr \
-signkey /etc/pki/tls/private/ldap.key \
-out /etc/pki/tls/certs/ldap.crt
There is not much we can do without a root user, or DN in LDAP terminology. Hence we run slappasswd to get a password hash we will copy/paste into the configuration files.
$ slappasswd
New password:
Re-enter new password:
{SSHA}password_hash
OpenLDAP is versatile enough that it can update configuration files through regular LDAP command line client utilities. Figuring out how to that at this point is just more complex than need be so we just edit the config files in place, update the CRC values and restart the slapd daemon.
$ diff -u prev /etc/openldap/ldap.conf
-TLS_CACERTDIR /etc/openldap/cacerts
+TLS_CACERT /etc/pki/tls/certs/ldap.crt
+TLS_REQCERT demand
$ diff -u prev /etc/openldap/slapd.d/cn=config/olcDatabase={0}config.ldif
+olcRootPW: {SSHA}password_hash
$ diff -u prev /etc/openldap/slapd.d/cn=config/olcDatabase={2}hdb.ldif
-olcSuffix: dc=my-domain,dc=com
-olcRootDN: cn=Manager,dc=my-domain,dc=com
+olcSuffix: dc=mydomain,dc=com
+olcRootDN: cn=Manager,dc=mydomain,dc=com
+olcRootPW: {SSHA}password_hash
+olcAccess: {0}to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
+olcAccess: {0}to attrs=userPassword by self write by dn.base="cn=Manager,dc=mydomain,dc=com" write by anonymous auth by * none
+olcAccess: {1}to * by dn.base="cn=Manager,dc=mydomain,dc=com" write by self write by * read"
$ diff -u /etc/openldap/slapd.d/cn=config.ldif
-olcTLSCACertificatePath: /etc/openldap/certs
-olcTLSCertificateFile: "OpenLDAP Server"
-olcTLSCertificateKeyFile: /etc/openldap/certs/password
+olcTLSCACertificatePath: /etc/pki/tls/certs
+olcTLSCertificateFile: /etc/pki/tls/certs/ldap.crt
+olcTLSCertificateKeyFile: /etc/pki/tls/private/ldap.key
We want to get rid of the warnings in the log file so we update the CRC32 after modifying the config files in place. To recompute the CRC32 we use the following command:
$ yum install perl-String-CRC32
$ tail -n +3 /etc/openldap/slapd.d/config-file \
| perl -e 'use String::CRC32; printf("%x\n", crc32(\*STDIN));'
It is time to start the OpenLDAP daemon and install standard schemas to describe users and POSIX accounts.
$ systemctl enable slapd $ systemctl start slapd # RFC 4524 $ ldapadd -x -H ldap:/// -f /etc/openldap/schema/cosine.ldif -D "cn=config" -W # still necessary though might be deprecated ... $ ldapadd -x -H ldap:/// -f /etc/openldap/schema/nis.ldif -D "cn=config" -W # RFC 2798 $ ldapadd -x -H ldap:/// -f /etc/openldap/schema/inetorgperson.ldif \ -D "cn=config" -W
Let's now add our first user to the directory. In the process we will add required top domain and groups. A good read at this point is Specifying Directory Entries Using LDIF
$ cat people.ldif
# Without creating a top level organization we might end-up with errors like:
# ldap_add: Insufficient access (50)
# additional info: no write access to parent
dn: dc=mydomain,dc=com
objectclass: top
objectclass: domain
dc: mydomain
description: Example inc.
dn: ou=people,dc=mydomain,dc=com
objectclass: top
objectclass: organizationalUnit
ou: people
description: People with credentials on mydomain infrastructure
dn: ou=groups,dc=mydomain,dc=com
objectClass: top
objectClass: organizationalUnit
ou: groups
description: Unix groups on mydomain infrastructure
dn: uid=johnd,ou=people,dc=mydomain,dc=com
objectClass: inetorgperson
objectClass: posixaccount
sn: Doe
cn: John Doe
uid: johnd
uidNumber: 1000
gidNumber: 1000
homedirectory: /home/johnd
shell: /bin/bash
mail: johnd@mydomain.com
mail: info@mydomain.com
dn: cn=wheel,ou=groups,dc=mydomain,dc=com
cn: wheel
objectClass: top
objectClass: posixGroup
gidNumber: 10
memberUid: johnd
dn: cn=dev,ou=groups,dc=mydomain,dc=com
cn: dev
objectClass: top
objectClass: posixGroup
gidNumber: 1000
memberUid: johnd
$ ldapadd -x -H ldap:/// -f people.ldif \
-D "cn=Manager,dc=mydomain,dc=com" -W
# Let's check we can access the newly created user 4 different way:
$ ldapsearch -xLLL -b "dc=mydomain,dc=com" uid=johnd
$ ldapsearch -xLLL -D uid=johnd,ou=people,dc=mydomain,dc=com
$ ldapsearch -xLLL -b dc=mydomain,dc=com \
-D uid=johnd,ou=people,dc=mydomain,dc=com
$ ldapwhoami -x -D uid=johnd,ou=people,dc=mydomain,dc=com -W
On a side note, to export the ldap directory tree into ldif format (very useful on system upgrades), we can use slapcat as such:
$ slapcat -v -l people.ldif
Though before uploading the ldif file back, you will need to prune a bunch of generated fields out of the dump. Failing to do so, you end up with Constraint violation errors.
ldap_add: Constraint violation (19)
additional info: structuralObjectClass: no user modification allowed
Until now we used the ldap:// port. It is time to turn on TLS and test connection through the secure ldaps:// port. After we replaced moznss by openssl, most errors are straightforward to fix. They are either related to the SELinux configuration or to the domain name associated to the EC2 instance.
$ service slapd restart # SELinux TLS: could not use key file `/etc/pki/tls/private/ldap.key'. TLS: error:0200500D:system library:fopen:Permission denied bss_file.c:393
There seems to be a bug in OpenLDAP because it will require the private key to be accessible by the user passed on the slapd command line (ex: -u lapd).
$ sudo chmod 750 /etc/pki/tls/private $ sudo chmod 640 /etc/pki/tls/private/ldap.key $ sudo chgrp -R ldap /etc/pki/tls/private
$ ldapsearch -d -1 -xLLL -H ldaps:// -b "dc=mydomain,dc=com" uid=johnd
# domainName
TLS: hostname (ip-aa-bb-cc-dd.us-east-2.compute.internal) does not match \
common name in certificate (*.mydomain.com).
TLS: can't connect: TLS: hostname does not match CN in peer certificate.
This error will appear because we used the Private IP address in the Certificate Signing Request, not the private domain name. We thus tweak the command as in:
$ ldapsearch -d -1 -xLLL -H ldaps://PrivateIP/ -b "dc=mydomain,dc=com" uid=johnd
It doesn't hurt to associate the public hostname with the loopback either so requests for mail.mydomain.com initiated locally do not wander around.
$ diff -u prev /etc/hosts -127.0.0.1 localhost +127.0.0.1 mail.mydomain.com localhost
Setting up Postfix
$ sudo yum install postfix $ sudo systemctl enable postfix $ sudo systemctl start postfix
Postfix takes a while to get used to but once you know your way around, is pretty straightforward to bend to your exact will.
There are plenty of posts on setting up postfix. We will skip here through some of the details and focus on creating virtual mailboxes. All mailboxes are own by a vmail user account. We are only interested to use OpenLDAP for virtual_mailbox_maps here.
$ sudo adduser -u 5000 -M -s /sbin/nologin vmail $ sudo mkdir -p /var/db/mail/vhosts/mydomain.com $ sudo chown vmail:vmail /var/db/mail/vhosts/mydomain.com $ diff -u prev /etc/postfix/main.cf +virtual_mailbox_domains = mydomain.com +virtual_mailbox_base = /var/db/mail/vhosts +virtual_mailbox_maps = ldap:/etc/postfix/ldap-vmailbox +virtual_minimum_uid = 100 +# vmail:vmail equivalent to 5000:5000 +virtual_uid_maps = static:5000 +virtual_gid_maps = static:5000 +virtual_alias_maps = hash:/etc/postfix/virtual $ diff -u prev /etc/postfix/virtual postmaster@mydomain.com postmaster @mydomain.com info@mydomain.com $ sudo postmap /etc/postfix/virtual $ diff -u prev /etc/postfix/ldap-vmailbox server_host = localhost version = 3 search_base = ou=people, dc=mydomain, dc=com query_filter = mail=%s result_attribute = mail result_format = %d/%u
At this point we test we can retrieve maildrop directories from LDAP.
$ postmap -q johnd@mydomain.com ldap:/etc/postfix/ldap-vmailbox mydomain.com/johnd/, mydomain.com/info/
We get both maildrops. That is a problem though it is behaving as documented in LDAP_TABLE (5). It seems impossible to use the actual key that was used for the lookup as the result of the query. We will have to rely on hash files. Unfortunately in postconf (5) it is stated:
Note 1: for security reasons, the virtual(8) delivery agent disallows regular expression substitution of $1 etc. in regular expression lookup tables, because that would open a security hole.
It means we must add each mapping explicitly in the vmailbox hash file. (My personal opinion is that there were alternative ways of insuring security outside simply and purely disabling regular expressions...)
$ diff -u prev /etc/postfix/main.cf +virtual_mailbox_maps = hash:/etc/postfix/vmailbox $ diff -u prev /etc/postfix/vmailbox johnd@mydomain.com mydomain.com/johnd/ info@mydomain.com mydomain.com/info/ $ sudo postmap /etc/postfix/vmailbox
Moving on to testing with the local sendmail.
"Unable to locate mail" is an error that baffled me for quite some time until I finally realize spamc was not using the postfix sendmail as I was expected but ssmtp sendmail instead.
$ tail /var/log/maillog
sSMTP[3357]: Unable to locate mail
sSMTP[3357]: Cannot open mail:25
postfix/pipe[3356]: B1D1B4035B: to=<johnd@mydomain.com>,\
relay=spamassassin, delay=0.76, delays=0.6/0.01/0/0.15, dsn=5.3.0,\
status=bounced (Command died with status 1: "/usr/bin/spamc".\
Command output: sendmail: Cannot open mail:25 )
That is fixed by changes the default mta alternative.
$ ls -la /etc/alternatives/mta
/etc/alternatives/mta -> /usr/sbin/sendmail.ssmtp
$ alternatives --display mta
$ alternatives --config mta
# or
$ alternatives --set mta /usr/sbin/sendmail.postfix
$ ls -la /etc/alternatives/mta
/etc/alternatives/mta -> /usr/sbin/sendmail.postfix
Another warning we found annoying and wanted to fix is an untrusted TLS connection to gmail.
$ tail /var/log/maillog
postfix/smtp[3788]: Untrusted TLS connection established \
to gmail-smtp-in.l.google.com[74.125.20.27]:25:\
TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
First we test we have a correct CA bundle to connect to gmail.
$ openssl s_client -showcerts -CAfile /etc/pki/tls/certs/ca-bundle.crt \
-starttls smtp -connect gmail-smtp-in.l.google.com:25
Then we modify our postfix configuration to use this CA bundle file. We also force connection to gmail to be verified.
$ diff -u prev /etc/postfix/main.cf +smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt +smtpd_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt $ vi /etc/postfix/tls_policy gmail.com verify .gmail.com verify $ sudo postmap /etc/postfix/tls_policy
Adding a SPF record
At this point we are able to send and receive mail. To prevent SPAM filters to miscategorize mail coming from our server, we add a SPF record to the DNS record.
We allow A records, domain's MXes and amazon SES (since the webapp relies on SES) to send mail for the domain, prohibit all others.
# TXT Records (Optional Text Fields) "v=spf1 a mx include:amazonses.com -all"
OK. Time to access our e-mails from an external client.
Setting up Dovecot
$ yum install dovecot libtool-ltdl $ systemctl enable dovecot $ systemctl start dovecot
Dovecot spits out very informative error messages so it is straightforward to walk through what needs to be fixed after the initial configuration. Here again SELinux is often the culprit. On modern Linux distributions, it is a good habit to always check in /var/log/audit/audit.log when something looks odd.
$ grep dovecot /var/log/audit/audit.log | audit2why -M dovecot # Dovecot and SELinux $ sudo semanage fcontext -a -t dovecot_spool_t "/var/db/mail/vhosts(/.*)?" $ sudo restorecon -R -v /var/db/mail/vhosts
Here we hit a wall to configure authentication through PAM while allowing multiple e-mail inboxes per users.
There is three ways to connect to a LDAP directory from PAM, the old way and two new ways. Unfortunately after hours of fiddling with each of them, they fall short of delivering the expected result. To make it quick:
- pam_ldap has a pam_login_attribute for retrieving a directory entry. It is though a deprecated package on Fedora and must be compiled from source. it has been written off as leaking memory and buggy beyond repair.
- nss-pam-ldapd is most likely the first choice you will find when searching for "pam ldap", though Fedora prefers their sssd implementation as a default. For us the issue is the lack of pam_login_attribute option. Even with pam_regex we couldn't bypass that limitation (because of a 3-way interaction with Dovecot).
- sssd is heavy on local caching (its selling point) but is either buggy by design or by accident. It will not support to index its cache by the key you use for retrieving a directory entry. Instead it will insists on using the first e-mail address associated with an entry.
Dovecot allows to authenticate directly against a LDAP directory, so finally,
that is the configuration we implemented.
$ diff -u prev /etc/dovecot/conf.d/10-auth.conf
-!include auth-system.conf.ext
-#!include auth-ldap.conf.ext
+#!include auth-system.conf.ext
+!include auth-ldap.conf.ext
$ diff -u prev /etc/dovecot/conf.d/10-mail.conf
-#mail_location =
+mail_location = maildir:/var/db/mail/vhosts/%d/%n
$ diff -u prev /etc/dovecot/conf.d/10-master.conf
# Postfix smtp-auth
- #unix_listener /var/spool/postfix/private/auth {
- # mode = 0666
- #}
+ unix_listener /var/spool/postfix/private/auth {
+ mode = 0666
+ user = postfix
+ group = postfix
+ }
$ diff -u prev /etc/dovecot/conf.d/10-ssl.conf
-ssl_cert = mydomain.com.crt
+ssl_key = mydomain.com.key
$ diff -u /etc/dovecot/conf.d/auth-ldap.conf.ext
-userdb {
- driver = ldap
- args = /etc/dovecot/dovecot-ldap.conf.ext
-}
+#userdb {
+# driver = ldap
+# args = /etc/dovecot/dovecot-ldap.conf.ext
+#}
-#userdb {
-# driver = static
-# args = uid=vmail gid=vmail home=/var/vmail/%u
-#}
+userdb {
+ driver = static
+ args = uid=vmail gid=vmail home=/var/db/mail/vhosts/%d/%n
+}
$ vi /etc/dovecot/dovecot-ldap.conf.ext
# Both slapd and dovecot are running on the same machine. We rely on
# on the clear text ldap:// port and localhost here in case the domain
# name for the machine is reset incorrectly somewhere in a reboot sequence.
uris = ldap://localhost/
ldap_version = 3
tls_ca_cert_file = /etc/pki/tls/certs/ldap.crt
base = dc=mydomain,dc=com
auth_bind = yes
auth_bind_userdn = uid=%n,ou=people,dc=mydomain,dc=com
user_filter = (&(objectClass=posixAccount)(mail=%u))
We are not believers in STARTTLS, so we disable Dovecot listening on clear text POP and IMAP ports.
$ diff -u prev /etc/dovecot/conf.d/10-master.conf
service imap-login {
inet_listener imap {
- #port = 143
+ port = 0
}
service pop3-login {
inet_listener pop3 {
- #port = 110
+ port = 0
}
After configuring all of the email stack, we try to send and receive e-mails from the OSX Mail.app client. Here weird connections to Dovecot IMAP or POP can often be solved by a process of "quit and restart Mail.app".
Conclusion
As usual with software, a lot of time went into figuring out the politics, community and changing landscape behind various pieces of the puzzle. I hope this article has been helpful to you.
Side Notes
Newest approach to PAM-LDAP authentication will pass through an intermediate generic secure authentication daemon, either sssd or nslcd.
SSSD
The log output that made me believe sssd is using a different key for retrieving and LDAP entry and indexing its cache.
[sssd[be[default]]] [sdap_get_generic_ext_step] (0x0400): \
calling ldap_search_ext with [(&(mail=info@mydomain.com\
(objectclass=posixAccount)(&(uidNumber=*)(!(uidNumber=0))))\
[dc=mydomain,dc=com].
[sssd[be[default]]] [sdap_save_user] (0x0400): Save user
[sssd[be[default]]] [sdap_save_user] (0x0400): Processing user johnd
[sssd[be[default]]] [sdap_save_user] (0x0400): Adding user principal\
[johnd@mydomain.com] to attributes of [johnd].
[sssd[be[default]]] [sysdb_get_real_name] (0x0040): Cannot find user\
[info@mydomain.com] in cache
[sssd[be[default]]] [sysdb_search_by_name] (0x0400): No such entry
Some posts on sssd worth reading:
Switching from SSSD to NSS-PAM
SSSD is the default authentication daemon on Fedora 21. Though I don't know any reasons to switch to nslcd at this point, the instructions to do so are pretty straightforward.
$ sudo yum install nss-pam-ldapd $ diff -u prev /etc/nslcd.conf -base dc=mydomain,dc=com +base dc=mydomain,dc=com $ sudo systemctl enable nslcd $ sudo systemctl start nslcd $ diff -u prev /etc/nsswitch.conf -passwd: files sss -shadow: files sss -group: files sss +passwd: files ldap +shadow: files ldap +group: files ldap
Modifying LDAP entries
Here is an example on how to use ldapmodify to update an LDAP entry.
$ cat john.ldif
dn: uid=john,ou=people,dc=mydomain,dc=com
changetype: Modify
add: loginshell
loginshell: /bin/sh
$ ldapmodify -x -H ldap:/// -f johnd.ldif \
-D "cn=Manager,dc=mydomain,dc=com" -W
$ ldapsearch -xLLL -b "dc=mydomain,dc=com" uid=johnd
by Sebastien Mirolo on Mon, 30 Mar 2015
