A central directory service is a common fragment of Enterprise IT infrastructures. Frequently companies organize their complete user management through a directory service, giving them the comfort of SSO. This makes it a requirement for services shared by corporate users to seamlessly integrate with the authentication service. The integration of a directory service – may it be an OpenLDAP, Apache Directory Server, or Active Directory – is one of the most common cornerstones of a Hadoop installation.
In up coming posts I am going to highlight some of the necessary steps for a dependable integration of Hadoop in today’s secure enterprise infrastructures including a demonstration of Apache Argus. As a preliminary step we are going to revisit some basic principals in this post that comprises a secure PKI, and a central OpenLDAP directory service. The knowledge of this is going to be presumed going forward. In this post CentOS is used as the operation system.
Creating a CA and Signing the Certificate
Before we begin with the installation of an LDAP directory service we are going to pay attention to the setup of a PKI infrastructure, or at least at some of the core elements of such. A PKI consists of a handful of components that need to be understood well. In addition often there is no clear naming convention to the files, and formats involved that it can be quite confusing.
The components a PKI consist of are (*):
- Certificate Authority (CA)
- Registration Authority
- Central Directory
- Certificate Management System
- Certificate Policy
Common naming convention according to X.509 are (*):
- .pem – (Privacy-enhanced Electronic Mail) Base64 encoded DER certificate, enclosed between “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–“
- .cer, .crt, .der – usually in binary DER form, but Base64-encoded certificates are common too (see .pem above)
- .p7b, .p7c – PKCS#7 SignedData structure without data, just certificate(s) or CRL(s)
- .p12 – PKCS#12, may contain certificate(s) (public) and private keys (password protected)
- .pfx – PFX, predecessor of PKCS#12 (usually contains data in PKCS#12 format, e.g., with PFX files generated in IIS)
We will want our directory service to encrypt the communication between our clients in a trusted manner. For this purpose we are going to create a CA that we will further use to sign our used certificates. Client and services will have the same CA and will therefor be able to verify each others requests.
Creating a CA
Let’s begin by creating a Certificate Authority (CA) using OpenSSL. In CentOS you will find the openssl.cnf under /etc/pki/tls/ which we will use. Some minor changes are required in order to be usufuil for our purposes. But before we do any changes it is wise to make a copy of the file.
$ cp /etc/pki/tls/openssl.cnf $ vi /etc/pki/tls/openssl.cnf
The CA will be created under the default path /etc/pki/CA so you should make sure you have this setup correctly under the section [CA_default] in the OpenSSL configuration file. The CA certificate is going to be stored under /etc/pki/CA/certs while the key will reside under /etc/pki/CA/private. The database used by the CA to store information about signed certificates will be stored in the file index.txt directly in the folder /etc/pki/CA. In addition to that a serial number is needed. Create these files first:
$ touch index.txt $ echo '01' > serial.txt
Then make sure you have the following in your OpenSSL configuration file (/etc/tls/openssl.cnf):
# OpenSSL example configuration file.
# This is mostly being used for generation of certificate requests.
....
[ ca ]
default_ca = CA_default # The default ca section
[ CA_default ]
dir = /etc/pki/CA # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
crl_dir = $dir/crl # Where the issued crl are kept
database = $dir/index.txt # database index file.
#unique_subject = no # Set to 'no' to allow creation of
# several ctificates with same subject.
new_certs_dir = $dir/newcerts # default place for new certs.
certificate = $certs/ca.crt # The CA certificate
serial = $dir/serial.txt # The current serial number
crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
crl = $dir/crl.pem # The current CRL
private_key = $dir/private/ca.key # The private key
RANDFILE = $dir/private/.rand # private random number file
x509_extensions = usr_cert # The extentions to add to the cert
# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
# Extension copying option: use with caution.
copy_extensions = copy
# Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs
# so this is commented out by default to leave a V1 CRL.
# crlnumber must also be commented out to leave a V1 CRL.
# crl_extensions = crl_ext
default_days = 365 # how long to certify for
default_crl_days= 30 # how long before next CRL
default_md = default # use public key default MD
preserve = no # keep passed DN ordering
The above configuration is the way we want our default CA based on the OpenSSL configuration to be. As we will also make use of signing_policy and signing_req , you should make sure your openssl.cnf also contains the below configuration sections:
# For the CA policy [ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional [ signing_policy ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional [ signing_req ] subjectKeyIdentifier=hash authorityKeyIdentifier=keyid,issuer
From here we go ahead and create our CA using OpenSSL. We do this by creating a SSL request (req) using -x509 . Omitting -x509 would create us a normal PKI request, leaving it create us here a CA valid for 10 years. If you don’t want to secure your private key with a password you can add -nodes to the here provided command. Having no password for the key is less secure, but in a testing environment frees you from providing a password each time you sign a certificate.
$ cd /etc/pki/CA $ openssl req -config /etc/pki/tls/openssl.cnf -new -x509 -extensions v3_ca -keyout private/ca.key -out certs/ca.crt -days 3650
The private key also needs also be secured by limiting the access rights. We can achieve this by:
$ chmod 0400 private/ca.key
To view the the content of your newly create CA you can try this:
$ openssl x509 -in certs/ca.crt -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 10452765967770567559 (0x910faf443d41c387)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=DE, ST=Bavaria, L=Munich, O=Hortonworks Inc., OU=Security CA, CN=knox.cluster/emailAddress=ca@localhost
Validity
Not Before: Sep 11 09:22:17 2014 GMT
Not After : Sep 8 09:22:17 2024 GMT
Subject: C=DE, ST=Bavaria, L=Munich, O=Hortonworks Inc., OU=Security CA, CN=knox.cluster/emailAddress=ca@localhost
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:d4:63:a4:b9:bb:60:44:b2:21:c8:dc:a5:75:d9:
b9:85:22:00:c4:1b:4d:66:e7:9d:76:91:fa:b7:1e:
c3:b5:04:9a:f2:79:5e:6b:d0:0e:73:59:ab:c0:0e:
3a:53:5a:66:4f:dc:b0:75:51:0a:54:99:d9:b2:5a:
20:12:84:24:db:c1:bf:a2:bd:d9:e6:46:b9:c5:95:
e0:a1:24:03:81:77:2e:a0:97:64:a2:62:a3:c0:2e:
58:9c:27:f7:0b:ab:23:69:97:24:88:da:d9:6c:a2:
98:42:75:39:f3:2e:85:db:b0:50:cd:2a:d8:cf:bf:
ca:7e:55:ea:c4:13:36:1e:8b:23:b2:91:d1:44:e8:
59:78:3b:4b:e0:db:74:4e:86:4b:f0:df:50:2a:8a:
c3:23:53:c7:fa:6d:50:56:4a:83:17:d6:67:90:a7:
dd:06:c9:05:c3:cd:67:90:18:c2:ef:6f:21:fc:f7:
7f:22:94:52:0a:6d:69:97:22:b7:7f:30:6e:90:32:
66:dd:e7:24:8f:0c:d0:3a:d2:a5:a9:83:b2:11:28:
78:4d:98:4d:ad:be:33:87:97:3b:d6:0b:74:f9:41:
2c:bb:8c:a1:6c:ff:98:e1:c7:7c:2b:eb:13:52:15:
1f:48:6b:15:dc:b9:de:99:c8:a1:47:61:d0:a2:a3:
83:7d
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
47:A7:FB:D4:23:D9:C9:2E:F5:61:93:35:FE:E3:5B:F8:F0:33:79:38
X509v3 Authority Key Identifier:
keyid:47:A7:FB:D4:23:D9:C9:2E:F5:61:93:35:FE:E3:5B:F8:F0:33:79:38
X509v3 Basic Constraints:
CA:TRUE
Signature Algorithm: sha1WithRSAEncryption
5e:04:d3:0f:2d:63:d5:c0:7e:5b:c5:45:66:1a:82:05:e9:ab:
27:f8:d1:d4:84:56:fb:bf:3f:3d:f0:78:0e:e0:91:88:92:81:
3b:73:e6:e7:e1:2b:74:da:ca:81:60:b6:9f:68:d2:6d:71:e5:
cb:53:16:64:61:93:b3:0b:56:5c:ab:71:3c:69:e4:40:c9:f7:
66:37:de:20:32:71:a8:02:b2:6b:91:96:93:a6:9d:75:c0:fb:
ad:ea:06:f6:6a:8f:81:c4:be:fc:43:9c:d3:91:42:46:60:7c:
d6:0e:81:31:8c:37:81:82:e0:2f:e8:3d:25:20:fb:b6:1f:44:
83:a0:41:ad:fa:32:80:08:1c:e6:e5:30:c7:5f:4d:a3:8c:e4:
c8:ea:f2:de:e7:69:d2:d5:4f:ea:bd:38:a4:77:a3:13:3f:6f:
e4:c4:7a:90:e6:2c:32:23:44:25:43:87:7a:ba:cc:c0:d3:f8:
46:4f:f2:db:31:f1:3e:af:5e:3d:f1:90:b5:e0:41:9b:81:9e:
e6:ca:fa:60:a6:58:7f:bd:6b:7a:85:f8:22:15:ca:c2:7d:01:
10:8c:e3:ba:9c:4a:ad:d6:57:af:09:ef:e5:61:24:07:8e:e4:
9b:f7:48:db:8e:dc:eb:ae:03:90:43:05:27:05:42:e5:77:41:
15:dc:99:65
We can now proceed with the creation of our server certificate which we will sign using the prior created CA. Creating a server certificate we issue almost the same openssl request command as before but omitting the -x509 flag:
$ openssl req -config /etc/pki/tls/openssl.cnf -newkey rsa:2048 -sha256 -nodes -out knox_cert.csr -outform PEM -keyout knox_key.pem # signing the request $ openssl ca -config /etc/pki/tls/openssl.cnf -policy signing_policy -extensions signing_req -out knox_cert.pem -infiles knox_cert.csr
In the prior step we also already went ahead and signed the fresh certificate with our own CA. Check it out with this command:
$ openssl x509 -in knox_cert.pem -text -noout Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C=DE, ST=Bavaria, L=Munich, O=Hortonworks Inc., OU=Security CA, CN=knox.cluster/emailAddress=ca@localhost Validity Not Before: Sep 11 18:28:53 2014 GMT Not After : Sep 11 18:28:53 2015 GMT Subject: C=DE, ST=Bavaria, O=Hortonworks Inc., OU=Hadoop CA, CN=knox.cluster/emailAddress=knox@localhost Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a8:a0:57:68:70:af:49:14:7a:21:23:99:50:4f: dd:6e:58:a4:29:a7:14:08:da:72:d5:19:9c:7b:17: 5a:ca:ac:a6:70:4b:cb:50:0e:f4:9c:5a:d0:4f:c2: 99:b7:06:3a:9c:e5:62:1f:5b:5c:0b:99:ad:79:77: bc:42:f2:46:ab:75:d9:1a:4f:96:62:ea:ca:3c:4c: cc:ee:56:9f:64:0e:4e:f5:0f:e6:75:83:8e:e3:be: 86:21:6d:c6:d6:07:a0:eb:37:f2:da:fb:d7:1e:e8: 40:64:c8:80:3a:b1:95:dd:b1:de:a4:f1:4f:bd:51: a7:0e:11:e5:f0:f6:83:e8:0b:43:2b:39:7b:93:31: 99:c8:59:24:b2:bd:e7:d4:f1:2f:56:30:73:04:da: c3:f8:22:0f:74:47:3a:01:ae:b8:d5:23:9e:b3:83: b4:cf:8e:f5:c8:dd:41:53:61:bb:8b:ec:29:14:8f: d0:10:b5:3f:a5:c0:41:a5:e9:ef:07:41:4d:ef:c5: 3d:4c:ab:ad:0e:31:18:0f:43:cb:e2:a5:23:18:a8: 71:50:b4:9e:48:35:8e:53:62:fe:4e:97:38:9d:56: 4e:87:9a:d7:f3:d7:93:7f:ee:63:37:0a:dd:0f:b3: e2:1f:67:d4:18:31:54:0f:f3:68:e6:67:7b:61:5c: 7e:91 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 40:09:52:3E:C0:72:8F:58:6A:5D:ED:7E:9D:60:12:82:38:BC:1B:4B X509v3 Authority Key Identifier: keyid:47:A7:FB:D4:23:D9:C9:2E:F5:61:93:35:FE:E3:5B:F8:F0:33:79:38 X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Non Repudiation, Key Encipherment Signature Algorithm: sha1WithRSAEncryption 60:a1:50:b4:78:23:a4:de:f6:b9:8e:07:17:26:4f:c4:cc:c6: 5f:b7:60:a4:58:cf:0d:42:d7:f5:07:ef:d2:ab:75:1b:19:37: 47:d4:96:8a:03:33:72:4d:13:e7:8a:6d:5b:f5:79:02:60:bc: b2:a8:7f:cd:ab:c1:f7:bf:52:42:03:e3:1c:fd:46:bf:c2:7a: 1e:02:d7:a9:7d:8a:a3:d3:15:53:0c:85:2e:43:46:09:e9:bc: 02:52:15:d6:2d:fe:81:97:29:85:6b:6d:de:e6:db:29:2b:6d: 48:15:fd:fe:3d:9d:d6:99:b3:66:cb:45:1a:1d:29:d4:23:b3: d2:1b:b4:33:4c:4e:6c:bd:57:c5:a8:c4:d4:96:6d:2f:78:79: 2e:8b:69:0d:ea:81:f5:f2:94:a5:62:a8:e0:7c:9e:09:2e:94: 7c:b5:ca:d8:ba:6a:a2:bc:aa:10:c3:70:af:d8:d0:11:8b:a4: ea:d0:ed:ba:02:7c:3f:02:74:99:8c:21:43:16:ef:8d:51:4c: b0:1b:c8:c6:44:6c:7b:37:28:62:c0:be:2a:65:da:3f:4f:53: ca:89:3d:68:8f:e1:2e:a1:96:a7:ac:c1:ca:cc:d7:3d:21:ef: 8f:fb:44:39:87:52:2f:e4:80:32:0f:8a:a3:66:70:e8:94:88: 4e:1c:cc:13
Our freshly minted certificate valid unitl 11th of September 2015. Let go ahead and create now our directory service and secure it with this certificate.
Installing OpenLDAP with CA Signed Certificate
Installing OpenLDAP on CentOS is quite straightforward and can be achieved in a few simple steps. Below in the section “Further Readings” you should be able to find some more resources if you are experience some issues following the here provided procedure.
Let’s first install OpenLDAP and configure it using slapd.conf according to our needs:
$ yum install openldap-servers openldap-clients $ mv slapd.d slapd.d.origin # making a backup # preparing the database $ cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG $ chown -R ldap:ldap /var/lib/ldap
For administrative operations or as a binding user that is allowed to search the directory (usually any user is allowed to search) we are going to create a Manager or root account. This account needs a password we store it hashed within in the slapd.conf configuration file. To create the password hash you can use the slappasswd command:
$ slappasswd
New password:
Re-enter new password:
{SSHA}jNxchGtQCR2OJlHB9Rnmn0r9qHbli81t
Copy the complete {SSHA}…. part into your slapd.conf configuration at the appropriate place. You can use the here provide configuration by replacing the rootpw with your password.
You can go ahead and use the below configuration which also applies password policies. The policies are provided in the next section. Both the slapd.conf and the policies were taken from here. Go there for further details.
#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include /etc/openldap/schema/core.schema
include /etc/openldap/schema/cosine.schema
include /etc/openldap/schema/inetorgperson.schema
include /etc/openldap/schema/nis.schema
# Added for policy
include /etc/openldap/schema/ppolicy.schema
# Allow LDAPv2 client connections. This is NOT the default.
allow bind_v2
pidfile /var/run/openldap/slapd.pid
argsfile /var/run/openldap/slapd.args
moduleload ppolicy.la
# The next three lines allow use of TLS for encrypting connections using a
# dummy test certificate which you can generate by changing to
# /etc/pki/tls/certs, running "make slapd.pem", and fixing permissions on
# slapd.pem so that the ldap user or group can read it. Your client software
# may balk at self-signed certificates, however.
# TLSCACertificateFile
# TLSCertificateFile
# TLSCertificateKeyFile
database bdb
suffix "dc=mycorp,dc=net"
rootdn "cn=root,dc=mycorp,dc=net"
rootpw {SSHA}oHEyp39sSpIDrZD3zaC1qiybbI97GRrZ
# PPolicy Configuration
overlay ppolicy
ppolicy_default "cn=default,ou=policies,dc=mycorp,dc=net"
ppolicy_use_lockout
ppolicy_hash_cleartext
# The database directory MUST exist prior to running slapd AND
# should only be accessible by the slapd and slap tools.
# Mode 700 recommended.
directory /var/lib/ldap
# Indices to maintain for this database
index objectClass eq,pres
index ou,cn,mail,surname,givenname eq,pres,sub
index uidNumber,gidNumber,loginShell eq,pres
index uid,memberUid eq,pres,sub
index nisMapName,nisMapEntry eq,pres,sub
For password policy place this into a file named ppolicy.ldif in the /etc/openldap directory:
dn: ou = policies,dc=mycorp,dc=net objectClass: organizationalUnit objectClass: top ou: policies # default, policies, example.com dn: cn=default,ou=policies,dc=mycorp,dc=net objectClass: top objectClass: pwdPolicy objectClass: person cn: default sn: dummy value pwdAttribute: userPassword pwdMaxAge: 7516800 pwdExpireWarning: 14482463 pwdMinLength: 2 pwdMaxFailure: 10 pwdLockout: TRUE pwdLockoutDuration: 60 pwdMustChange: FALSE pwdAllowUserChange: FALSE pwdSafeModify: FALSE
Testing our configuration prior to starting the service:
$ slaptest -f /etc/openldap/slapd.conf (db errors can be ignored, because we've not started it yet)) $ service slapd start $ chkconfig slapd on
Before we apply our certificate to the server let’s first test our setup by adding entries to the database including a test user. The entries are stored in a file named users.ldif that should contain something similar to this:
version: 1 dn: dc=mycorp,dc=net dc: mycorp objectClass: top objectClass: domain # Please replace with site specific values dn: dc=hadoop,dc=mycorp,dc=net objectclass: organization objectclass: dcObject o: Hadoop dc: hadoop # Entry for a sample people container # Please replace with site specific values dn: ou=people,dc=hadoop,dc=mycorp,dc=net objectclass:top objectclass:organizationalUnit ou: people # Entry for a sample end user # Please replace with site specific values dn: uid=guest,ou=people,dc=hadoop,dc=mycorp,dc=net objectclass:top objectclass:person objectclass:organizationalPerson objectclass:inetOrgPerson cn: Guest sn: User uid: guest userPassword:guest-password
We now add this to our directory and search for our guest user. If you want to provide your password at command prompt instead of writing it out to your shell replace every -w your_rootpw with -W.
$ ldapadd -x -D "cn=root,dc=mycorp,dc=net" -w your_rootpw -f users.ldif $ ldapsearch -H ldap://knox.cluster -D "cn=root,dc=mycorp,dc=net" -w horton '(uid=guest)' # user search with debug info $ ldapsearch -H ldap://knox.cluster -D "cn=root,dc=mycorp,dc=net" -w horton -d7 '(uid=guest)' # search all with debug $ ldapsearch -H ldap://knox.cluster -D "cn=root,dc=mycorp,dc=net" -w horton -d7
We are now almost done. What is left to do is secure the service via SSL. For this we need to provide our created signed certificate to the server configuration in addition to the CA certificate itself. Also the client would need the CA certificate. We do this by adding the following to ldap.conf and slapd.conf under /etc/openldap:
TLSCACertificateFile /etc/openldap/ca.crt TLSCertificateFile /etc/openldap/knox_cert.pem TLSCertificateKeyFile /etc/openldap/knox_key.pem
Please make sure that you’ve also copied the files provided here into the appropriate place.
BASE dc=mycorp,dc=net URI ldap://knox.cluster TLS_CACERT /etc/pki/CA/certs/ca.crt
Now restart the service and test the configuration by StartTLS:
$ service slapd restart $ ldapsearch -H ldap://knox.cluster -D "cn=root,dc=mycorp,dc=net" -w your_rootpw -ZZ -d7
This should work without any issues. We now have our service secured by our own signed certificate over StartTLS. The difference between SSL and StartTLS can be seen as the folllows:
- ldap:// + StartTLS should be directed to a normal LDAP port (normally 389), not the ldaps:// port.
- ldaps:// should be directed to an LDAPS port (normally 636), not the LDAP port.
By default OpenLDAP after installation will not run on port 636 providing no SSL connectivity over ldaps:// . You can change this by adjusting /etc/sysconfig/ldap by changing the option SLAPD_LDAPS from no to yes.
# Run slapd with -h "... ldaps:/// ..." # yes/no, default: no SLAPD_LDAPS=yes
We can test this like with another search over ldaps:// this time:
$ ldapsearch -H ldaps://knox.cluster:636 -D "cn=root,dc=mycorp,dc=net" -w horton -d7 # testing SSL using openssl openssl s_client -connect know.cluster:389 -CApath /etc/pki/CA/certs/
Further Readings
- How do you sign OpenSSL Certificate Signing Requests with your Certification Authority?
- OpenLDAP Software 2.4 Administrator’s Guide
- Installing OpenLDAP on Redhat / CentOS 6.3
- Chapter 6 OpenLDAP password policy overlay
- Kerberos (Amazon)
- LDAP System Administration (Amazon)
- Implementing SSL / TLS Using Cryptography and PKI (Amazon)
henning.kropponline.de 
OpenLDAP Setup with CA Signed Certificate on CentOS http://t.co/igiQR5vCZj
LikeLike