Kafka Security with Kerberos

Apache Kafka developed as a durable and fast messaging queue handling real-time data feeds originally did not come with any security approach. Similar to Hadoop Kafka at the beginning was expected to be used in a trusted environment focusing on functionality instead of compliance. With the ever growing popularity and the widespread use of Kafka the community recently picked up traction around a complete security design including authentication with Kerberos and SSL, encryption, and authorization. Judging by the details of the security proposal found here the complete security measures will be included with the 0.9 release of Apache Kafka.

The releases of HDP 2.3 already today support a secure implementation of Kafka with authentication and authorization. Especially the integration with the security framework Apache Ranger this becomes a comprehensive security solution for any Hadoop deployment with real-time data demands. In this post we by example look at how working with a kerberized Kafka broker is different from before. Here working with the known shell tools and a custom Java producer.

A Secure Kafka Broker

A kerberized HDP cluster or secured Kafka broker has some new parameters that are worth covering before continuing. Probably the most important change is the used broker (security.inter.broker.protocol) protocol PLAINTEXTSASL instead of PLAINTEXT before. Starting with the release of the security proposal Kafka will support the following protocols:

  • PLAINTEXT (non-authenticated, non-encrypted)
  • SSL (SSL authentication, encrypted)
  • SASL+PLAINTEXT (authentication, non-encrypted)
  • SASL+SSL (encrypted authentication, encrypted transport)

The use of PLAINTEXTSASL indicates the use of Kerberos without transport encryption. Further a secure broker defines a super user granted with all required priviledges. Typically this users is intended to administrer the broker. By default the only super user ( super.users) is kafka ( user:kafka)

The authorization in Kafka is a plugable implementation which can for example be replaced with a Apache Ranger authorization implementation. Kafka by default comes with the  kafka.security.auth.SimpleAclAuthorizer defined by the parameter authorizer.class.name .

Last but not least as any one somewhat familiar with Kerberos might know a translation from principal to user name has to be configured somehow. In Kafka this is defined by the parameter principal.to.local.class which is set to  kafka.security.auth.KerberosPrincipalToLocal by default.

Authentication and Authorization

Working with a secure Kafka broker going forward requires a proper authentication as well as proper permission being setup for the user. If for example authentication is not provided the following GSSException would be the reuslt when wrting to a topic:

Two things are required for a proper authentication with Kafka tools moving forward. At first you will require to kinit and obtain a valid TGT for a user principal existing in the used KDC. That can be archived like this for the user data_artist01:

Next it might be required to set the security protocol to be used explicitly by using the --security-protcol config as shown below:


As we can see from the command executed above no ACLs are currently assigned to our topic. For this Kafka provides the  kafka-acls.sh shell script to adjust access permissions of topics which we are going to use to provide our user data_artist01 with proper access rights to the topic test. The help section of the script already gives a good idea of how it can be used for that:

For now only the super user kafka has the rights to make any changes to the permissions in Kafka. To become the kafka user we simply use it’s keytab:

Being the kafka user we can now adjust the access rights to our test topic and give the user data_artist01 READ, WRITE, DELTE, DESCRIBE privileges from any host (*).

Listing the permission of the test topic again:

Writing to the topic should now be feasible for the user data_artist01:

As already mentioned before the authorization can easily also be done be Apache Ranger using a comprehensive UI and it’s auditing capabilities. For details see here.

Further Readings

One thought on “Kafka Security with Kerberos”

Leave a Reply

Your email address will not be published. Required fields are marked *