In a recent post I demonstrated how easy it is to connect to a REST API like the one of Fitbit with Scala to collect JSON data. Taking up the results of that post here, I would like to demonstrate how Apache Zeppelin can be used to also fetch but in the end visualize the data. Based on the once collected data Zeppelin allows to easily visualize the output through different graphs.
Apache Zeppelin itself is a notebook like, web-based data analytic tool with a specific focus on exploratory data analysis in modern BigData architectures supporting multiple interpreters like Tajo, Spark, Hive, HBase and more. Saying this, it is important to point out, that in this here described case only Scala is being used to display the received data. But this use case could easily be extended to include Apache Hive or Spark.
Zeppelin Interpreter
As previously mentioned does Zeppelin support multiple interpreters. Since we are using Scala for this use case we are mainly relying on the existing Spark interpreter.
By default this is configured for a local SparkContext which make it work out of the box once you download Zeppelin.
The really other interpreter used here is the Markdown interpreter for some introductory notes. Having a Markdown interpreter like this makes for a good reporting tool:
%md # Fitbit Notebook This is a sample Zeppelin notebook for getting activity records from a fitbit account. ### Authorization You need to get authorization code to furhter proceed. [https://www.fitbit.com/oauth2/authorize?response_type=code&client_id=227P25&redirect_uri=http%3A%2F%2Fhenning.kropponline.de&scope=activity](https://www.fitbit.com/oauth2/authorize?response_type=code&client_id=227P25&redirect_uri=http%3A%2F%2Fhenning.kropponline.de&scope=activity) Copy the code from the address line of your browser, for example: [https://henning.kropponline.de/?code=78dc0adf7f4123d80b7x31c8f7446f337e0f4ad#successs](https://henning.kropponline.de/?code=78dc0adf7f4123d80b7x31c8f7446f337e0f4ad#successs) contains the code __*78dc0adf7f4123d80b7x31c8f7446f337e0f4ad*__ ### Resources Please read this for some background: [https://henning.kropponline.de/2016/03/27/access-to-fitbit-api-with-scala/](https://henning.kropponline.de/2016/03/27/access-to-fitbit-api-with-scala/)
Which results in the following web output:
Adding Dependencies
To request the Fitbit API we are using scalaj-http and for parsing the JSON output json4s is being used. Both libraries do not ship with the Zeppelin Spark interpreter so that we need to load them as additional packages to the interpreter. Note that %dep is deprecated.
%dep
z.load("org.scalaj:scalaj-http_2.10:2.2.1")
z.load("org.json4s:json4s-native_2.10:3.2.11")
Doing this requires a restart of the Spark interpreter. Check the Interpreter tab for doing a reset of interpreters. The above code is placed in a separate paragraph:
Getting the Authentication Token
Accessing user content on Fitbit an application needs to use OAuth2 for authorization. This requires a 2-way connection between the application and Fitbit, so that the resulting verification token can be send to the application. As this notebook is executed as a client application, the user is required to retrieve the the token in a separate step by calling the URL manually.
Once the token is received it can be used by the below paragraph to finally get the access token which valid for at least 1 day.
%spark
import org.json4s.{DefaultFormats, JString}
import org.json4s.JsonAST.{JArray, JObject}
import scalaj.http.{Http, Token, HttpResponse}
import org.json4s.native.JsonMethods._
var access_token:String = ""
var refresh_token:String = ""
var user_id:String = ""
val app_name:String = "hk-scala-test"
val app_client_id:String = "XXXX"
val app_client_secret:String = "xxxxxxxxxxxxxxxxxxx"
val client_cred:String = java.util.Base64.getEncoder
.encodeToString(s"$app_client_id:$app_client_secret".getBytes)
// Provide the token from the fitbit authorize request here
val token:String = "c488a46f5a3aeb0efce46e8f72ecabea09cfecfa"
val response: HttpResponse[String] = Http("https://api.fitbit.com/oauth2/token")
.headers(Seq("Authorization" -> s"Basic $client_cred",
"Content-Type" -> "application/x-www-form-urlencoded"))
.postForm(Seq("name" -> "jon",
"client_id" -> app_client_id,
"grant_type" -> "authorization_code",
"redirect_uri" -> "http://henning.kropponline.de",
"code" -> token)
).asString
val jsonResponse = parse(response.body)
val JString(access_token) = jsonResponse "access_token"
val JString(refresh_token) = jsonResponse "refresh_token"
val JString(user_id) = jsonResponse "user_id"
The variables access_token, refresh_token, and user_id can be reused in other paragraphs so we only need to execute this paragraph once.
Fetching the Activity Steps
Fetching the data just once an storing storing it in an across paragraph shareable object ActivitiesStep:
%spark
case class Steps(dateTime:String, value:String)
case class ActivitiesStep(`activities-steps`: Seq[Steps])
val response: HttpResponse[String] =
Http(s"https://api.fitbit.com/1/user/$user_id/activities/steps/date/today/3m.json")
.headers("Authorization" -> s"Bearer $access_token")
.asString
val jsonResponse = parse(response.body)
implicit val formats = DefaultFormats
val activitiesStep = jsonResponse.extract[ActivitiesStep]
In our workbook this would look like this:
Check the Fitbit API documentation to see what other activities can be received: https://dev.fitbit.com/docs/activity/
Visualizing the Result
Our previously populated data object activitiesStep can now be furhter used in other paragraphs for visualization. Zeppelin uses AngularJS as part of a it’s display system.
As a simple predefined display output %table is defined for use. It uses n (new line) as a row and t (tab) as a column delimiter.
%spark
println("%table DatetStepstSteps2")
activitiesStep.`activities-steps`.foreach( x =>
println(s"${x.dateTime}t${x.value.toInt}t${x.value.toInt-20}")
)
First we print the table statement to than further iterate through the activitiesStep printing out each result. The below graphic show the resulting visualization as a bar chart.
henning.kropponline.de 