Hadoop Credential API

In Hadoop 2.6 a fundamental feature was introduced that did not get much attention but will play an important role moving forward – the Credential API. Looking ahead the Credential Management Framework (CMF) will play an important role for the pluggable token authentication framework, column encryption in ORC files, or the transparent data encryption. But not only future components but applications build for Hadoop can benefit from it.

Background

Many applications around Hadoop store passwords in configuration files in clear text format. This is often not acceptable for most users that share security concerns. Many application like sqoop have a very limited approach to storing sensitive database credentials.

For long developers had to come up with their own solution for their implementations. Already existing mechanism were not supported and still aren’t even with the Credential API.

1. Password Indirection or Aliasing
2. Management of identity and trust keystores
3. Rolling of key pairs and credentials
4. Discovery of externally provisioned credentials
5. Service specific CMF secret protection
6. Syntax for Aliases within configuration files

How it works

Store credentials in in keystore:

user:///,jceks://file/tmp/test.jceks,jceks://hdfs@nn1.example.com/my/path/test.jceks

Use CLI to get and store credentials:

Example: -provider jceks://file/tmp/test.jceks

Usage: hadoop credential <subcommand> [options]

Use the Java API:

import org.apache.hadoop.security.alias.CredentialProvider;
import org.apache.hadoop.security.alias.CredentialProviderFactory;
import org.apache.hadoop.conf.Configuration;

CredentialProvider.CredentialEntry credEntry = provider.getCredentialEntry(alias);