A Secure HDFS Client Example

It takes about 3 lines of Java code to write a simple HDFS client that can further be used to upload, read or list files. Here is an example:

This file system API gives the developer a generic interface to (any supported) file system depending on the protocol being use, in this case hdfs. This is enough to alter data on the Hadoop Distributed Filesystem, for example to list all the files under the root folder:

For a secured environment this is not enough, because you would need to consider these further aspects:

  1. A secure protocol
  2. Authentication with Kerberos
  3. Impersonation (proxy user), if designed as a service

What we discuss here for a sample HDFS client can in variance also be applied to other Hadoop clients.

A Secure HDFS Protocol

One way to secure the communication between clients and Hadoop services in general is to use SSL encryption for all RPC calls. This does have a sever impact on the overall cluster performance in general. To avoid this and still ensure a secure communication it can be enough to just encrypt HTTP endpoints. If doing so swebhdfs (SSL+webhdfs) can be used as the protocol. Example:

Authentication with Kerberos

A secure client would need to use Kerberos, which is the only authentication method currently supported by Hadoop. Kerberos does require very thoughtful configuration but rewards it’s users with an almost completely transparent authentication implementation that simply works.

Making use of Kerberos authentication in Java is provided by the Java Authentication and Authorization Service (JAAS) which is a pluggable authentication method similar to PAM supporting multiple authentication methods. In this case the authentication method being used is GSS-API for Kerberos.

For JAAS a proper configuration of GSS would be needed in addition to being in possession of proper credentials, obviously. Some credentials can be created with MIT Kerberos like this:

The last line is not necessarily needed as it creates us a so called keytab – basically an encrypted password of the user – that can be used for password less authentication for example for automated services. We will make use of that here as well.

Additionally we create a JAAS configuration, we can use for authentication:

We now have multiple ways to use authentication and here I will start with probably the most simple approach regarding required code changes:

1. Authentication with Keytab

Authentication web based access to HDFS with a keytab requires almost no code changes despite the use of (s)webhdfs protocol and change of authentication method:

The above is enough if executed in a JAAS context. Creating the secure context can be done be using the above JAAS and keytab.

2. Using UserGroupInformation

For authentication in Hadoop there exists a wrapper class around a JAAS Subject to provide methods for user login. The UserGroupInformation wrapper without a specific setup uses the system security context, in case of Kerberos this exist in the ticket cache (klist shows the existing security context of a user). This is demonstrated under “With Existing Security Context” below. Further a custom security context can be used for login, either with by using a keytab file or even with credentials. Both approaches are also demonstrated here under “Providing Credentials from Login” and “Via Keytab”.

With Existing Security Context

First we would need to authenticate and make sure we have a proper security context:

With this the following can HDFS client implementation can be used in a secured environment:

Creating the JAAS context during run-time the client could be executed like this:

Providing Credentials from Login

Providing login credentials at execution requires the creation of a javax.security.auth.Subject with username and password. This means that we will have to use the GSS-API to do a kinit like this:

We still have to configure the JAAS login module referenced by the name that we provide in the above implementation. The name applied in the example above is set to be HdfsMain.class.getSimpleName(), so our module configuration should look like this:

Having this in place we can now login with username and password:

Via Keytab

In the first part we injected the security context via the JAAS security context ( -Djava.security.auth.login.config=/home/hdfs-user/jaas.conf) configuring keytab authentication. We can also achieve this using JAAS security wrapper UserGroupInformation provided by Hadoop.

The complete code being used:

Please note that it is not required to configure the JAAS context, as we are using UserGroupInformation:

Impersonation

A last aspect when writing clients for a secure HDP cluster is the proxy user setting especially if you are designing a service implementation. The proxy user functionality enables services to access resources on the cluster on behalf of another user. Hence the service is impersonating the user.

A good example of such a service is the HiveServer2. The HS2 recieves SQL requests and creates an execution plan using a execution engine like MapReduce, Tez, or Spark. The plan is being executed on the cluster on behalf of the user doing the SQL request.

Of course it is important to be in control of who is able to impersonate whom and from where. This can be configured by adding a proxyuser config to Hadoop:

If for example we have a Tomcat service running on host web.mycorp.net the following example configuration would enable the service to impersonate users in the group web-users from host web.mycorp.net.

It is important to make sure service are not able to impersonate hdfs or other service account that have special privileges. Only trusted services should be added to the proxyuser setup.

Clients can also use the UserGroupInformation class to impersonate other users. With the doAs implementation can be wrapped into the context of the user to be impersonated.

Here the user parameter defines the user (as String) in which context the call should be executed. The proxyUser is the current service or system user running the client. Be aware of the fact that not in all circumstances the proxyUser might be equal to UserGroupInformation.getCurrentUser().

Further Readings

  1. Wire Encryption in Hadoop
  2. Difference between trustStore and keyStore in Java – SSL
  3. Kerberos and Hadoop
  4. Kerberos (Protocol)
  5. Java Authentication and Authorization Service
  6. GSS-API/Kerberos v5 Authentication
  7. JAAS Login Configuration File

12 thoughts on “A Secure HDFS Client Example”

  1. Very important to keep in mind is that the hadoop hdfs api(s) rely on Java’s ServiceLoader to load the org.apache.hadoop.security.AnnotatedSecurityInfo service specified in the META-INF/services/ of the hadoop-common.jar.

    If you use tools to create an uberjar or repackaging, the authentication will fail with:

    Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null

    Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:null

    To solve either fix the META-INF/services/… or just set programmatically using SecurityUtil.setSecurityInfoProviders

    I made a post rant about this: http://funclojure.tumblr.com/post/155129283948/hdfs-kerberos-java-client-api-pains

  2. Is this different if I am running my code in Intellij on my local machine against a kerberized remote hdfs? Was not able to get it to work through my IDE.

    1. Should basically work in Windows environments as well.

      Tow things might be useful to consider for Windows environments.
      1. Kinit and klist utils can be found in the bin folder of your java distribution
      2. You need to research where your system expects the krb5.ini or krb5.conf file and adapt it to your Kerberos settings

      1. Thanks for your reply! I mean I’m running this code from windows client but I still have my kerberized hadoop cluster on an Ubuntu server. Do I need to install Kerberos on Windows as well ?
        I used System.setProperty to set the KDC and the realm name but my code is still not working..

  3. Hello,

    very nice and clear post. But there is something that, maybe because of my novelty in Hadoop, is not completely clear to me. That is, in the impersonation part, when you set the hosts and groups:

    hadoop.proxyuser.tomcat.groups=web-users
    hadoop.proxyuser.tomcat.hosts=web.mycorp.net

    where is this done? Is it in the core-site.xml file?

    And, more precisely, where is this group of users (web-users) coming from? I mean where and how did you create this group?

    Thanks in advance

  4. Hi, I’m curious if it is possible to configure JAAS from UserGroupInformation? My specific scenario is that I need to perform SPNEGO negotiation with a web server from within a Spark executor (which is already running inside of a UserGroupInformation.doAs(…) from supplying the –principal and –keytab arguments to spark-submit.

Leave a Reply

Your email address will not be published. Required fields are marked *