Connecting Tomcat to a Kerberized HDP Cluster

At some point you might require to connect your dashboard, data ingestion service or similar to a secured and kerberized HDP cluster. Most Java based webcontainers do support Kerberos for both client and server side communication. Kerberos does require very thoughtful configuration but rewards it’s users with an almost completely transparent authentication implementation that simply works. Steps described in this post should enable you to connect your application with a secured HDP cluster. For further support read the links listed at the end of this writing. A sample project is provided on github for hands-on exercises.

Setup the Environment

You can build a quick test environment using a custom vagrant environment, download the latest HDP Sandbox, or use a Docker image (very basic setup). In all cases this article helps you in kerberizing your environment with a local installed KDC. Once kerberized you will also have to install Tomcat of course. This uses CentOS 7 with Tomcat 7 with config files being located under /etc/tomcat and webapps under /var/lib/tomcat/webapps:

$ yum install -y tomcat
$ ll /etc/tomcat/
insgesamt 208
drwxrwxr-x 3 root   tomcat     22  4. Feb 18:00 Catalina
-rw-rw-r-- 1 tomcat tomcat  12257 12. Mai 2015  catalina.policy
-rw-rw-r-- 1 tomcat tomcat   6294 12. Mai 2015  catalina.properties
-rw-rw-r-- 1 tomcat tomcat   1394 12. Mai 2015  context.xml
-rw-rw-r-- 1 tomcat tomcat    547 12. Mai 2015  log4j.properties
-rw-rw-r-- 1 tomcat tomcat   3288 12. Mai 2015  logging.properties
-rw-rw-r-- 1 tomcat tomcat   6536 12. Mai 2015  server.xml
-rw-rw-r-- 1 tomcat tomcat   1568 12. Mai 2015  tomcat.conf
-rw-rw---- 1 tomcat tomcat   1998 12. Mai 2015  tomcat-users.xml
-rw-rw-r-- 1 tomcat tomcat 163385 12. Mai 2015  web.xml

$ systemctl start tomcat
$ systemctl status tomcat
tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled)
   Active: active (running) since Do 2016-02-04 21:38:13 UTC; 7s ago
  Process: 1774 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
 Main PID: 1819 (java)
   CGroup: /system.slice/tomcat.service
           └─1819 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dc...

Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.startup.Catalina load
Feb 04 21:38:14 one.hdp server[1819]: INFO: Initialization processed in 372 ms
Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.core.StandardService startInternal
Feb 04 21:38:14 one.hdp server[1819]: INFO: Starting service Catalina
Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.core.StandardEngine startInternal
Feb 04 21:38:14 one.hdp server[1819]: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.startup.HostConfig deployWAR
Feb 04 21:38:14 one.hdp server[1819]: INFO: Deploying web application archive /var/lib/tomcat/webapps/hdp-web.war
Feb 04 21:38:14 one.hdp server[1819]: Feb 04, 2016 9:38:14 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
Feb 04 21:38:14 one.hdp server[1819]: INFO: validateJarFile(/usr/share/tomcat/webapps/hdp-web/WEB-INF/lib/jsp-api-2.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offendin...pression.class
Hint: Some lines were ellipsized, use -l to show in full.
$ ll /var/lib/tomcat/webapps/
insgesamt 42452
drwxr-xr-x 4 tomcat tomcat       51  4. Feb 21:35 hdp-web
-rw-r--r-- 1 root   root   43468411  4. Feb 21:35 hdp-web.war

Since Ambari already runs on port 8080 I changed the port to 8099 for this demo.

A Sample WebApp (WebHDFS)

As a sample web application connecting to the cluster we are going to use a basic HDFS implemention that let’s it’s users define a path to browse the file system. This is the code we need for that:

public class WebHdfsServlet extends HttpServlet implements Servlet {

    public WebHdfsServlet(){};

    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

        String path = request.getParameter("path");
        if (path == null) path = "/";

        Configuration conf = new Configuration();
        conf.set("fs.defaultFS","webhdfs://one.hdp:50070");

        FileSystem fs = FileSystem.get(conf);

        FileStatus[] fsStatus = fs.listStatus(new Path(path));

        PrintWriter out = response.getWriter();
        out.println("<html>");
        out.println("<head><title>hdfs dfs -ls /</title></head>");
        out.println("<body><h1>/</h1><ul>");

        for(int i = 0; i < fsStatus.length; i++){
            out.println("<li>" + fsStatus[i].getPath().toString() + "</li>");
        }
        out.println("</ul></body>");
        out.println("</html>");
        out.close();
    }
}

The web.xml contains the following resource description:

<web-app>
    <display-name>HDP Web Sample</display-name>

    <servlet>
        <servlet-name>WebHdfsServlet</servlet-name>
        <servlet-class>hdp.webapp.WebHdfsServlet</servlet-class>
    </servlet>

    <servlet-mapping>
        <servlet-name>WebHdfsServlet</servlet-name>
        <url-pattern>/webhdfs</url-pattern>
    </servlet-mapping>
</web-app>

Once deployed and with a none secured cluster the result of opening http://one.hdp:8099/hdp-web/webhdfs?path=/user in your browser the result should look like this:

webhdfs_sample

The result will look very different once the cluster is secured using Kerberos:

webapp_webhdfs_auth_required

Configuring Tomcat for Kerberos

In a next step to make the web application work with the secured cluster we will have to configure Tomcat for Kerberos authentication. If configured correctly this would be enough to make our web application work again. There is no need to make any changes on the code. It is worth mentioning that in this scenario of course the actions performed on the cluster will always be executed by the Tomcat user. Of course this is only sufficient for simple scenarios, but in general a proxy user setup would be advised. Please see my post about a secure HDFS client for that.

Creating a Tomcat Principal

Here Tomcat will to the authentication with the cluster transparently to the applications being deployed on the webserver. As, if you followed the steps in mentioned early to kerberize the cluster, the KDC does not have any credentials stored for a tomcat user, we first need to create a tomcat principal in the KDC. Additionally as we obviously can not type in the password every time the service wants to authenticate, we need to download a so called keytab, which is basically an encrypted file containing the password of the user. The file needs to be properly secured using stand measures like POSIX access rights. The steps required are as following:

$ kadmin.local -q "addprinc -randkey tomcat/one.hdp@MYCORP.NET"
Authenticating as principal root/admin@MYCORP.NET with password.
WARNING: no policy specified for tomcat/one.hdp@MYCORP.NET; defaulting to no policy
Principal "tomcat/one.hdp@MYCORP.NET" created.
$ kadmin.local -q "xst -k /etc/tomcat/tomcat.keytab tomcat/one.hdp@MYCORP.NET"
Authenticating as principal root/admin@MYCORP.NET with password.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
Entry for principal tomcat/one.hdp@MYCORP.NET with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/tomcat/tomcat.keytab.
$ chown tomcat. /etc/tomcat/tomcat.keytab 
$ ll /etc/tomcat/tomcat.keytab 
-rw------- 1 tomcat tomcat 394  6. Feb 21:24 /etc/tomcat/tomcat.keytab
$ klist -kt /etc/tomcat/tomcat.keytab 
Keytab name: FILE:/etc/tomcat/tomcat.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET
   2 06.02.2016 21:24:42 tomcat/one.hdp@MYCORP.NET

With the steps above we create the tomcat principal in the KDC with a random generated password. Next we download the keytab and place it under /etc/tomcat in that way, that only the tomcat user is able to read it.

All is left now is to configure Tomcat to use the created principal to authenticate requests against the secured cluster.

Configure JAAS and GSS-API

The most common approach to this is to use Java’s Authentication and Authorization Service (JAAS) which is a pluggable authentication mechanism just like PAM. This does not require the import of additional packages as JAAS is build in since JDK 1.4. In order to use JAAS for authentication in requires a login configuration file describing the type of method being used and it’s parameters applicable. For the Tomcat service we will create the jaas.conf  file under /etc/tomcat/jaas.conf  enabling GSS-API for Kerberos.

GSS-API was designed as a common service being able to access different security services. The GSS-API/Kerberos subsystem allows a Java application to authenticate to Kerberos once, and then use the acquired security credentials to access a whole array of services securely, including directory services.

com.sun.security.jgss.krb5.initiate {
    com.sun.security.auth.module.Krb5LoginModule required
    doNotPrompt=true
    principal="tomcat/one.hdp@MYCORP.NET"
    useKeyTab=true
    keyTab="/etc/tomcat/tomcat.keytab"
    storeKey=true;
};

Will have to make this configuration known to the Tomcat services, so it is aware of the security context to use. Adding the following Java options to the runtime creation of the service is enough to accomplish a kerberized security context. File /etc/tomcat/tomcat.conf :

...
# You can pass some parameters to java here if you wish to
JAVA_OPTS="-Djava.security.auth.login.config=/etc/tomcat/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false"
...

Restarting tomcat will make the sample application work with the secured cluster.

There are a few ways to validate the authentication is working and Tomcat is configured correctly. For once you need to check if Tomcat was started with the right options. So please check by for example using the status command of systemctl:

$ systemctl status tomcat
tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; disabled)
   Active: active (running) since Sa 2016-02-06 21:57:02 UTC; 3s ago
  Process: 13784 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
 Main PID: 13829 (java)
   CGroup: /system.slice/tomcat.service
           └─13829 /usr/lib/jvm/jre/bin/java -Djava.security.auth.login.config=/etc/tomcat/jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -classpath ...

Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.startup.Catalina load
Feb 06 21:57:02 one.hdp server[13829]: INFO: Initialization processed in 380 ms
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.core.StandardService startInternal
Feb 06 21:57:02 one.hdp server[13829]: INFO: Starting service Catalina
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.core.StandardEngine startInternal
Feb 06 21:57:02 one.hdp server[13829]: INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.startup.HostConfig deployWAR
Feb 06 21:57:02 one.hdp server[13829]: INFO: Deploying web application archive /var/lib/tomcat/webapps/hdp-web.war
Feb 06 21:57:02 one.hdp server[13829]: Feb 06, 2016 9:57:02 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
Feb 06 21:57:02 one.hdp server[13829]: INFO: validateJarFile(/usr/share/tomcat/webapps/hdp-web/WEB-INF/lib/jsp-api-2.1.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offendi...pression.class
Hint: Some lines were ellipsized, use -l to show in full.

Authentication attempts are logged at the KDC, so if everything works as expected you would see the tomcat user appear in the logs:

# tail /var/log/krb5kdc.log 
Feb 06 21:56:07 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:08 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454788741, etypes {rep=18 tkt=18 ses=18}, nn/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:50 one.hdp krb5kdc[8109](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:56:51 one.hdp krb5kdc[8109](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.33.100: ISSUE: authtime 1454795810, etypes {rep=18 tkt=18 ses=18}, ambari-qa-n1-hdp-basic@MYCORP.NET for HTTP/one.hdp@MYCORP.NET
Feb 06 21:57:41 one.hdp krb5kdc[8109](info): AS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454795861, etypes {rep=18 tkt=18 ses=18}, tomcat/one.hdp@MYCORP.NET for krbtgt/MYCORP.NET@MYCORP.NET
Feb 06 21:57:41 one.hdp krb5kdc[8109](info): TGS_REQ (4 etypes {18 17 16 23}) 192.168.33.100: ISSUE: authtime 1454795861, etypes {rep=18 tkt=18 ses=18}, tomcat/one.hdp@MYCORP.NET for HTTP/one.hdp@MYCORP.NET

Further Readings

Advertisements

One thought on “Connecting Tomcat to a Kerberized HDP Cluster

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s