At some point you might require to connect your dashboard, data ingestion service or similar to a secured and kerberized HDP cluster. Most Java based webcontainers do support Kerberos for both client and server side communication. Kerberos does require very thoughtful configuration but rewards it’s users with an almost completely transparent authentication implementation that simply works. Steps described in this post should enable you to connect your application with a secured HDP cluster. For further support read the links listed at the end of this writing. A sample project is provided on github for hands-on exercises.
Setup the Environment
You can build a quick test environment using a custom vagrant environment, download the latest HDP Sandbox, or use a Docker image (very basic setup). In all cases this article helps you in kerberizing your environment with a local installed KDC. Once kerberized you will also have to install Tomcat of course. This uses CentOS 7 with Tomcat 7 with config files being located under /etc/tomcat and webapps under /var/lib/tomcat/webapps:
$ yum install -y tomcat $ ll /etc/tomcat/ insgesamt 208 drwxrwxr-x 3 root tomcat 22 4. Feb 18:00 Catalina -rw-rw-r-- 1 tomcat tomcat 12257 12. Mai 2015 catalina.policy -rw-rw-r-- 1 tomcat tomcat 6294 12. Mai 2015 catalina.properties -rw-rw-r-- 1 tomcat tomcat 1394 12. Mai 2015 context.xml -rw-rw-r-- 1 tomcat tomcat 547 12. Mai 2015 log4j.properties -rw-rw-r-- 1 tomcat tomcat 3288 12. Mai 2015 logging.properties -rw-rw-r-- 1 tomcat tomcat 6536 12. Mai 2015 server.xml -rw-rw-r-- 1 tomcat tomcat 1568 12. Mai 2015 tomcat.conf -rw-rw---- 1 tomcat tomcat 1998 12. Mai 2015 tomcat-users.xml -rw-rw-r-- 1 tomcat tomcat 163385 12. Mai 2015 web.xml $ systemctl start tomcat $ systemctl status tomcat tomcat.service - Apache Tomcat Web Application Container Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled) Active: active (running) since Do 2016-02-04 21:38:13 UTC; 7s ago Process: 1774 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS) Main PID: 1819 (java) CGroup: /system.slice/tomcat.service └─1819 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dc... Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.startup.Catalina load Feb 04 21:38:14 one.hdp server[1819]: INFO: Initialization processed in 372 ms Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.core.StandardService startInternal Feb 04 21:38:14 one.hdp server[1819]: INFO: Starting service Catalina Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.core.StandardEngine startInternal Feb 04 21:38:14 one.hdp server[1819]: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54 Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.startup.HostConfig deployWAR Feb 04 21:38:14 one.hdp server[1819]: INFO: Deploying web application archive /var/lib/tomcat/webapps/hdp-web.war Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.loader.WebappClassLoader validateJarFile Feb 04 21:38:14 one.hdp server[1819]: INFO: validateJarFile(/usr/share/tomcat/webapps/hdp-web/WEB-INF/lib/jsp-api-2.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offendin...pression.class Hint: Some lines were ellipsized, use -l to show in full. $ ll /var/lib/tomcat/webapps/ insgesamt 42452 drwxr-xr-x 4 tomcat tomcat 51 4. Feb 21:35 hdp-web -rw-r--r-- 1 root root 43468411 4. Feb 21:35 hdp-web.war
Since Ambari already runs on port 8080 I changed the port to 8099 for this demo.
A Sample WebApp (WebHDFS)
As a sample web application connecting to the cluster we are going to use a basic HDFS implemention that let’s it’s users define a path to browse the file system. This is the code we need for that:
public class WebHdfsServlet extends HttpServlet implements Servlet {
public WebHdfsServlet(){};
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String path = request.getParameter("path");
if (path == null) path = "/";
Configuration conf = new Configuration();
conf.set("fs.defaultFS","webhdfs://one.hdp:50070");
FileSystem fs = FileSystem.get(conf);
FileStatus[] fsStatus = fs.listStatus(new Path(path));
PrintWriter out = response.getWriter();
out.println("<html>");
out.println("<head><title>hdfs dfs -ls /</title></head>");
out.println("<body><h1>/</h1><ul>");
for(int i = 0; i < fsStatus.length; i++){
out.println("<li>" + fsStatus[i].getPath().toString() + "</li>");
}
out.println("</ul></body>");
out.println("</html>");
out.close();
}
}
The web.xml contains the following resource description:
<web-app>
<display-name>HDP Web Sample</display-name>
<servlet>
<servlet-name>WebHdfsServlet</servlet-name>
<servlet-class>hdp.webapp.WebHdfsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>WebHdfsServlet</servlet-name>
<url-pattern>/webhdfs</url-pattern>
</servlet-mapping>
</web-app>
Once deployed and with a none secured cluster the result of opening http://one.hdp:8099/hdp-web/webhdfs?path=/user in your browser the result should look like this:
The result will look very different once the cluster is secured using Kerberos:
Configuring Tomcat for Kerberos
In a next step to make the web application work with the secured cluster we will have to configure Tomcat for Kerberos authentication. If configured correctly this would be enough to make our web application work again. There is no need to make any changes on the code. It is worth mentioning that in this scenario of course the actions performed on the cluster will always be executed by the Tomcat user. Of course this is only sufficient for simple scenarios, but in general a proxy user setup would be advised. Please see my post about a secure HDFS client for that.
Creating a Tomcat Principal
Here Tomcat will to the authentication with the cluster transparently to the applications being deployed on the webserver. As, if you followed the steps in mentioned early to kerberize the cluster, the KDC does not have any credentials stored for a tomcat user, we first need to create a tomcat principal in the KDC. Additionally as we obviously can not type in the password every time the service wants to authenticate, we need to download a so called keytab, which is basically an encrypted file containing the password of the user. The file needs to be properly secured using stand measures like POSIX access rights. The steps required are as following:
$ kadmin.local -q "addprinc -randkey tomcat/one.hdp@MYCORP.NET" Authenticating as principal root/admin@MYCORP.NET with password. WARNING: no policy specified for tomcat/one.hdp@MYCORP.NET; defaulting to no policy Principal "tomcat/one.hdp@MYCORP.NET" created. $ kadmin.local -q "xst -k /etc/tomcat/tomcat.keytab tomcat/one.hdp@MYCORP.NET" Authenticating as principal root/admin@MYCORP.NET with password. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/tomcat/tomcat.keytab. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/tomcat/tomcat.keytab. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/tomcat/tomcat.keytab. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/tomcat/tomcat.keytab. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/tomcat/tomcat.keytab. Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/tomcat/tomcat.keytab. $ chown tomcat. /etc/tomcat/tomcat.keytab $ ll /etc/tomcat/tomcat.keytab -rw------- 1 tomcat tomcat 394 6. Feb 21:24 /etc/tomcat/tomcat.keytab $ klist -kt /etc/tomcat/tomcat.keytab Keytab name: FILE:/etc/tomcat/tomcat.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET 2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
With the steps above we create the tomcat principal in the KDC with a random generated password. Next we download the keytab and place it under /etc/tomcat in that way, that only the tomcat user is able to read it.
All is left now is to configure Tomcat to use the created principal to authenticate requests against the secured cluster.
Configure JAAS and GSS-API
The most common approach to this is to use Java’s Authentication and Authorization Service (JAAS) which is a pluggable authentication mechanism just like PAM. This does not require the import of additional packages as JAAS is build in since JDK 1.4. In order to use JAAS for authentication in requires a login configuration file describing the type of method being used and it’s parameters applicable. For the Tomcat service we will create the jaas.conf file under /etc/tomcat/jaas.conf enabling GSS-API for Kerberos.
GSS-API was designed as a common service being able to access different security services. The GSS-API/Kerberos subsystem allows a Java application to authenticate to Kerberos once, and then use the acquired security credentials to access a whole array of services securely, including directory services.
com.sun.security.jgss.krb5.initiate {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
principal="tomcat/one.hdp@MYCORP.NET"
useKeyTab=true
keyTab="/etc/tomcat/tomcat.keytab"
storeKey=true;
};
Will have to make this configuration known to the Tomcat services, so it is aware of the security context to use. Adding the following Java options to the runtime creation of the service is enough to accomplish a kerberized security context. File /etc/tomcat/tomcat.conf :
... # You can pass some parameters to java here if you wish to JAVA_OPTS="-Djava.security.auth.login.config=/etc/tomcat/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false" ...
Restarting tomcat will make the sample application work with the secured cluster.
There are a few ways to validate the authentication is working and Tomcat is configured correctly. For once you need to check if Tomcat was started with the right options. So please check by for example using the status command of systemctl:
$ systemctl status tomcat
tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled)
Active: active (running) since Sa 2016-02-06 21:57:02 UTC; 3s ago
Process: 13784 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
Main PID: 13829 (java)
CGroup: /system.slice/tomcat.service
└─13829 /usr/lib/jvm/jre/bin/java -Djava.security.auth.login.config=/etc/tomcat/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -classpath ...
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.startup.Catalina load
Feb 06 21:57:02 one.hdp server[13829]: INFO: Initialization processed in 380 ms
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.core.StandardService startInternal
Feb 06 21:57:02 one.hdp server[13829]: INFO: Starting service Catalina
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.core.StandardEngine startInternal
Feb 06 21:57:02 one.hdp server[13829]: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.startup.HostConfig deployWAR
Feb 06 21:57:02 one.hdp server[13829]: INFO: Deploying web application archive /var/lib/tomcat/webapps/hdp-web.war
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
Feb 06 21:57:02 one.hdp server[13829]: INFO: validateJarFile(/usr/share/tomcat/webapps/hdp-web/WEB-INF/lib/jsp-api-2.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offendi...pression.class
Hint: Some lines were ellipsized, use -l to show in full.
Authentication attempts are logged at the KDC, so if everything works as expected you would see the tomcat user appear in the logs:
# tail /var/log/krb5kdc.log
Feb 06 21:56:07 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:51 one.hdp krb5kdc[8109](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:57:41 one.hdp krb5kdc[8109](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454795861, etypes {rep=18 tkt=18 ses=18}, tomcat/one.hdp@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:57:41 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454795861, etypes {rep=18 tkt=18 ses=18}, tomcat/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Further Readings
- hdp-web-sample (Git)
- Tomcat Kerberos Howto
- Java Authentication and Authorization Service (JAAS)
- Configuring Kerberos for Glassfish and Tomcat
- Hadoop in Secure Mode
- Configuring Tomcat for Kerberos and Impersonation
- GSS-API/Kerberos v5 Authentication
- Generic Security Services Application Program Interface (GSS-API)
henning.kropponline.de 
One thought on “Connecting Tomcat to a Kerberized HDP Cluster”