Fitbit Visualization with Apache Zeppelin

In a recent post I demonstrated how easy it is to connect to a REST API like the one of Fitbit with Scala to collect JSON data. Taking up the results of that post here, I would like to demonstrate how Apache Zeppelin can be used to also fetch but in the end visualize the data. Based on the once collected data Zeppelin allows to easily visualize the output through different graphs.

Apache Zeppelin itself is a notebook like, web-based data analytic tool with a specific focus on exploratory data analysis in modern BigData architectures supporting multiple interpreters like Tajo, Spark, Hive, HBase and more. Saying this, it is important to point out, that in this here described case only Scala is being used to display the received data. But this use case could easily be extended to include Apache Hive or Spark.

fitbit_zeppelin Zeppelin Interpreter

As previously mentioned does Zeppelin support multiple interpreters. Since we are using Scala for this use case we are mainly relying on the existing Spark interpreter.

zeppelin_interpreter
List of Zeppelin Interpreter

By default this is configured for a local SparkContext which make it work out of the box once you download Zeppelin.

The really other interpreter used here is the Markdown interpreter for some introductory notes. Having a Markdown interpreter like this makes for a good reporting tool:

%md
# Fitbit Notebook

This is a sample Zeppelin notebook for getting activity records from a fitbit account.

### Authorization
You need to get authorization code to furhter proceed.
[https://www.fitbit.com/oauth2/authorize?response_type=code&client_id=227P25&redirect_uri=http%3A%2F%2Fhenning.kropponline.de&scope=activity](https://www.fitbit.com/oauth2/authorize?response_type=code&client_id=227P25&redirect_uri=http%3A%2F%2Fhenning.kropponline.de&scope=activity)

Copy the code from the address line of your browser, for example: [https://henning.kropponline.de/?code=78dc0adf7f4123d80b7x31c8f7446f337e0f4ad#successs](https://henning.kropponline.de/?code=78dc0adf7f4123d80b7x31c8f7446f337e0f4ad#successs) contains the code __*78dc0adf7f4123d80b7x31c8f7446f337e0f4ad*__

### Resources
Please read this for some background: [https://henning.kropponline.de/2016/03/27/access-to-fitbit-api-with-scala/](https://henning.kropponline.de/2016/03/27/access-to-fitbit-api-with-scala/)

Which results in the following web output:

zeppelin_fitbit_md_out

Adding Dependencies

To request the Fitbit API we are using scalaj-http and for parsing the JSON output json4s is being used. Both libraries do not ship with the Zeppelin Spark interpreter so that we need to load them as additional packages to the interpreter. Note that %dep is deprecated.

%dep
z.load("org.scalaj:scalaj-http_2.10:2.2.1")
z.load("org.json4s:json4s-native_2.10:3.2.11")

Doing this requires a restart of the Spark interpreter. Check the Interpreter tab for doing a reset of interpreters. The above code is placed in a separate paragraph:

zeppelin_fitbit_load_dep

Getting the Authentication Token

Accessing user content on Fitbit an application needs to use OAuth2 for authorization. This requires a 2-way connection between the application and Fitbit, so that the resulting verification token can be send to the application. As this notebook is executed as a client application, the user is required to retrieve the the token in a separate step by calling the URL manually.

Once the token is received it can be used by the below paragraph to finally get the access token which valid for at least 1 day.

%spark
import org.json4s.{DefaultFormats, JString}
import org.json4s.JsonAST.{JArray, JObject}
import scalaj.http.{Http, Token, HttpResponse}
import org.json4s.native.JsonMethods._

var access_token:String = ""
var refresh_token:String = ""
var user_id:String = ""
val app_name:String = "hk-scala-test"
val app_client_id:String = "XXXX"
val app_client_secret:String = "xxxxxxxxxxxxxxxxxxx"

val client_cred:String = java.util.Base64.getEncoder
      .encodeToString(s"$app_client_id:$app_client_secret".getBytes)

// Provide the token from the fitbit authorize request here
val token:String = "c488a46f5a3aeb0efce46e8f72ecabea09cfecfa"

val response: HttpResponse[String] = Http("https://api.fitbit.com/oauth2/token")
      .headers(Seq("Authorization" -> s"Basic $client_cred",
                   "Content-Type" -> "application/x-www-form-urlencoded"))
      .postForm(Seq("name" -> "jon",
                    "client_id" -> app_client_id,
                    "grant_type" -> "authorization_code",
                    "redirect_uri" -> "http://henning.kropponline.de",
                    "code" -> token)
      ).asString

val jsonResponse = parse(response.body)
val JString(access_token) = jsonResponse  "access_token"
val JString(refresh_token) = jsonResponse  "refresh_token"
val JString(user_id) = jsonResponse  "user_id"

The variables access_token, refresh_token, and user_id can be reused in other paragraphs so we only need to execute this paragraph once.

zeppelin_fitbit_get_auth_token

Fetching the Activity Steps

Fetching the data just once an storing storing it in an across paragraph shareable object ActivitiesStep:

%spark

case class Steps(dateTime:String, value:String)
case class ActivitiesStep(`activities-steps`: Seq[Steps])

val response: HttpResponse[String] =
      Http(s"https://api.fitbit.com/1/user/$user_id/activities/steps/date/today/3m.json")
        .headers("Authorization" -> s"Bearer $access_token")
        .asString
        
val jsonResponse = parse(response.body)
implicit val formats = DefaultFormats

val activitiesStep = jsonResponse.extract[ActivitiesStep]

In our workbook this would look like this:

zeppelin_fitbit_fetch_activity_steps

Check the Fitbit API documentation to see what other activities can be received: https://dev.fitbit.com/docs/activity/

Visualizing the Result

Our previously populated data object activitiesStep can now be furhter used in other paragraphs for visualization. Zeppelin uses AngularJS as part of a it’s display system.

As a simple predefined display output %table is defined for use. It uses n (new line) as a row and t (tab) as a column delimiter.

%spark
println("%table DatetStepstSteps2")
activitiesStep.`activities-steps`.foreach( x => 
    println(s"${x.dateTime}t${x.value.toInt}t${x.value.toInt-20}")
)

First we print the table statement to than further iterate through the activitiesStep printing out each result. The below graphic show the resulting visualization as a bar chart.

zeppelin_fitbit_vis_activity_steps

Further Reading

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s