Install Apache Zepplin via REST & Ambari

The Ambari server offers a comprehensive REST API to install a complete HDP cluster or manage all parts of it. In this post we are going to explore the possibilities of installing a new service. With Ambari it is fairly easy to define custom services for management. With YARN being a general purpose execution engine Ambari can be seen as the general purpose management service. For this example we are using the Apache Zeppelin service provided here. A more general documentation of how to install a new service to an existing cluster can be found here.

Setup Zeppelin Service

In order to make use of the management hooks defined in the description provided by Ali we have to place it under our current HDP stack definition in Ambari server. In this case we use the HDP stack definition 2.3 of an Ambari 2.1 installation:

$ yum install -y git
$ cd /var/lib/ambari-server/resources/stacks/HDP/2.3/services
$ git clone https://github.com/abajwa-hw/zeppelin-stack.git

Installing Zeppelin to Existing Cluster

# Add a Service to the Cluster

The metainfo.xml describes the components of a service to be added to Ambari. Looking at the Zeppelin metainfo we see that it is named ZEPPELIN and consists of one host component ZEPPELIN_MASTER. First we have to add the ZEPPELIN service to the cluster:

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST 
-d '{"ServiceInfo":{"service_name":"ZEPPELIN"}}' 
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services

 

Additionally the component of the services as to be added as well

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST 
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>
/services/ZEPPELIN/components/ZEPPELIN_MASTER"

With the above steps we’ve successfully setup a new service with it’s components in Ambari server. Finally we have to create necessary configurations for it and assign these configs to the cluster. Also the host component needs to be assigned to a host.

# Create Configuration

Zeppelin needs one configurations as again described in its metainfo.xml – zeppelin-config. Important is to define a type and tag for the configuration to reference later. The type is name of the configuration, while the tag is it version/reference. In Ambari every configuration is immutable! Changes are reflected by new tags.

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST 
-d @zeppelin.conf 
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/configurations

What is in the zeppelin.conf file:

{
    "type": "zeppelin-config", 
    "tag": "install", 
    "properties" : {
        "zeppelin.download.prebuilt" : "true",
        "zeppelin.executor.mem" : "512m",
        "zeppelin.install.dir" : "/opt",
        "zeppelin.interpreter.dir" : "interpreter",
        "zeppelin.interpreters" : "org.apache.zeppelin.spark.SparkInterpreter,org.apache.zeppelin.spark.PySparkInterpreter,org.apache.zeppelin.spark.SparkSqlInterpreter,org.apache.zeppelin.spark.DepInterpreter,org.apache.zeppelin.markdown.Markdown,org.apache.zeppelin.angular.AngularInterpreter,org.apache.zeppelin.shell.ShellInterpreter,org.apache.zeppelin.hive.HiveInterpreter,org.apache.zeppelin.tajo.TajoInterpreter,org.apache.zeppelin.flink.FlinkInterpreter",
        "zeppelin.notebook.dir" : "notebook",
        "zeppelin.server.port" : "9995",
        "zeppelin.spark.jar.dir" : "hdfs:///apps/zeppelin",
        "zeppelin.ssl" : "false",
        "zeppelin.ssl.client.auth" : "false",
        "zeppelin.ssl.key.manager.password" : "change me",
        "zeppelin.ssl.keystore.password" : "change me",
        "zeppelin.ssl.keystore.path" : "conf/keystore",
        "zeppelin.ssl.keystore.type" : "JKS",
        "zeppelin.ssl.truststore.password" : "change me",
        "zeppelin.ssl.truststore.path" : "conf/truststore",
        "zeppelin.ssl.truststore.type" : "JKS",
        "zeppelin.websocket.port" : "-1"
    }    
}

# Apply Configuration to the Cluster

Applying this to the cluster, means to choose the create config type and it’s tag as the current configuration for the service.

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT 
-d '{ "Clusters" : {"desired_configs": {"type": "zeppelin-config", 
"tag" : "install" }}}'  
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>

# Create Host Components

Now that we have the configuration and components in place we can assign them to hosts part of the cluster:

$ curl -u ambari:ambari -i -H 'X-Requested-By: ambari' -X POST 
-d '{"host_components" : [{"HostRoles":{"component_name":"ZEPPELIN_MASTER"}}] }' 
http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts?Hosts/host_name=$HOST_FQDN

# Install & Start the Service

Last but not least we install the configured service:

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT 
-d '{"ServiceInfo": {"state" : "INSTALLED"}}'  
http://<AMBARI_HOST>:8080/api/v1/clusters/<Cluster_Name>/services/ZEPPELIN

$ curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT 
-d '{"ServiceInfo": {"state" : "STARTED"}}'  
http://<AMBARI_HOST>:8080/api/v1/clusters/<Cluster_NAME>/services/ZEPPELIN

Further Readings

• Installing Services via REST – Ambari
• Ambari Views, Stacks, and Blueprint Workshop
• Ambari REST API
• Ambari Blueprint Hostname Topology