Simple Spark Streaming & Kafka Example in a Zeppelin Notebook

Apache Zeppelin is a web-based, multi-purpose notebook for data discovery, prototyping, reporting, and visualization. With it’s Spark interpreter Zeppelin can also be used for rapid prototyping of streaming applications in addition to streaming-based reports.

In this post we will walk through a simple example of creating a Spark Streaming application based on Apache Kafka.

Creating a Notebook

For our example we first need to create a new notebook, which we’ll name “Simple Spark Streaming Kafka Example”:

Naming the notebook:

Adding Dependencies to Spark Interpreter

For our Kafka example we rely on dependencies not necessarly included with the SparkContext created by the Zeppelin interpreter. Zeppelin allows to import arbitrary packages available in any Maven repository. For any none standard/public repository it needs be configured with Zeppelin.

We can reach the interpreter setting through the notebook options as shown in the picture above. Here we can set the default interpreter for our notebook, but we can also enter the settings page of all interpreters:

Find the Spark interpreter in the list of available Zeppelin interpreters for editing:

Scroll to the bottom were you should find the appropriate Dependencies sections, where you can add additional packages:

This are the packages needed depending on your Kafka distribution as well as the Scala release you are using for Spark.

Save and restart the interpreter.

Preparing Kafka Topic

For this example we create a simple topic named “spark-test-topic” with just one partition:

In our example we will use a Spark Streaming app to read the key-value messages send to this topic’s partition and split it by space to simply print out the individual words of one interval.

For this we don’t need a specific producer but simply reuse the existing console producer, which sends each line after a line break to the topic:

Simple Spark Streaming Application

For our example we do need a couple of dependencies not already imported into the created SparkContext. We do need the Kafka message serializes as well as the KafkaUtils of the Kafka streaming package for Spark.

Our SparkStreamingContext will only require ERROR logging to prevent INFO logging output. The context will run in a 5 second interval:

For our streaming context we need to know the Kafka configuration of the topic to subscribe to, which we will hold in our kafkaConf variable:

Finally we are creating a D-Stream and map it to individual words separated by space based :

The complete notebook:

The complete code example:

Done.

Further Reading

7 thoughts on “Simple Spark Streaming & Kafka Example in a Zeppelin Notebook”

  1. Hi !

    Thank you very much for your guideline, I find it very useful.
    Although I can run all the code, I think I am missing something else because I can not see the output of the script on my Zeppelin Notebook. In fact, I try to run the same code on the spark-shell and it does not print out any result neither.

    First I though it was due to communications issues, however my Zeppelin can (docker container) can reach Spark, Kafka and Zookeeper (also other containers).
    My second though is that I connects but it does not get the data inside.
    Kafka works fine. I can use producers and consumers in other applications.

    Would you have any idea why is this happening?

    Find below my versions:
    Spark: 2.1.0
    Kafka: 0.9.0.1
    Zeppelin: 0.7
    Zookeeper: 3.4.9
    org.apache.spark:spark-streaming-kafka_2.11:2.1.0
    org.apache.kafka:kafka_2.11:0.9.0.1
    org.apache.kafka:kafka-clients:0.9.0.1

    Thank you very much !!

  2. Yeah, thank you for the wonderful tutorial!

    I also got the the same problem: no output.

    I added ssc.awaitTermination() but that did not help.

    Any suggestion?

    Thanks

    John Lee

  3. The same issue for me as well . However we I run the the notebook, there are no errors, it just hangs, although the topic has data streaming and I can see that in the kafka consumer console. Any help would be great.

  4. The same issue here. After adding ssc.awaitTermination() at the end, I can see a timestamp printed per 5 second, but obviously it doesn’t read messages from Kafka. With the exactly same code in Eclipse Scala IDE, it can be run with messages consumed from Kafka successfully. Weird. Any advice?

Leave a Reply