Services and State with Ambari REST API

The Ambari management tool for Hadoop offers among other handy tools a comprehensive REST API for cluster administration. Logically a cluster is divided into hosts, services and service components. While the UI might not always has support for all needed scenarios sure the REST API can be used to achieve it. For example moving a master component of a service from one host to another.

In this post we are going to look a little closer at the way the Ambari API can be used to manage Hadoop services. At the end of this post you will find a list of all the currently supported Hadoop services with all the needed master, slave and client components that can be manged and administrated within your HDP stack. Also this posts contains the possible states and state transitions a component might have which could become useful when facing problems like Host config is in invalid state.

The possible states a service component can be in is listed below. To change or get the state of a service component the following API calls can be issued.

Among the need to change the state of a service component a very common issue is the need to actually move one service – most likely a master service – to different host. Users faced with this problems often need to issue the existing API due to a lack of support by the existing UI – at least with older releases of Ambari Web UI. This is an example of moving the OOZIE_SERVER to a new host.

Removing a complete service from the cluster would require all the removal of all it’s components together with the service itself. For example for removing the Knox service we would need to remove the KNOX_GATEWAY component from the cluster, but before let’s get their state:

Possible states of a component:

State Description
INIT Initial/Clean state.
INSTALLING In the process of installing.
INSTALL_FAILED Install failed.
INSTALLED State when install completed successfully.
STARTING In the process of starting.
STARTED State when start completed successfully.
STOPPING In the process of stopping.
UNINSTALLING In the process of uninstalling.
UNINSTALLED State when uninstall completed successfully.
WIPING_OUT In the process of wiping out the install.
UPGRADING In the process of upgrading the deployed bits.
DISABLED Disabled master’s backup state.
UNKNOWN State could not be determined.

Allowed State transitions of a component:

Desired State Start State
INSTALLED <- INIT, UNINSTALLED, INSTALLED, INSTALLING, STARTED, INSTALL_FAILED, UPGRADING, STOPPING, UNKNOWN, DISABLED
STARTED <- INSTALLED, STARTING, STARTED
UNINSTALLED <- INSTALLED, UNINSTALLED, UNINSTALLING
INIT <- UNINSTALLED, INIT, WIPING_OUT
DISABLED <- INSTALLED, INSTALL_FAILED, UNKNOWN

Services

A Hadoop cluster contains of multiple services that individually have either master, slave or client components. Below you will find a list of all currently supported components as part of a HDP stack divided in master, slave, or client groups. The cardinality notates the number of components that can exist at the same time in one cluster.

Master Services

Name Ambari Component Name
Service
Cardinality
NameNode NAMENODE HDFS 1-2
Secondary NameNode SECONDARY_NAMENODE HDFS 1
ResourceManger RESOURCEMANAGER YARN 1-2
Application Timeline Server APP_TIMELINE_SERVER YARN 1
HistoryServer HISTORYSERVER MAPREDUCE2  1
Hive Metastore HIVE_METASTORE HIVE 1-2
HiveServer2 HIVE_SERVER HIVE 1-2
WebHcat Server WEBHCAT_SERVER HIVE 1
HBase Master HBASE_MASTER HBASE 1+
Spark Job History Server SPARK_JOBHISTORYSERVER SPARK 1
Nimbus Server NIMBUS STORM 1
Storm REST Server STORM_REST_API STORM 1
Storm UI STORM_UI_SERVER STORM 1
DRPC Server DRPC_SERVER STORM 1
Falcon Server FALCON_SERVER FALCON 1
Zookeeper ZOOKEEPER_SERVER ZOOKEEPER 1+ (odd #)
Kafka Broker KAFKA_BROKER KAFKA 1+
Knox Gateway KNOX_GATEWAY KNOX 1+
Ranger Admin Server RANGER_ADMIN RANGER 1-3
Ranger User Sync RANGER_USERSYNC RANGER 1
Ranger Key Management Server RANGER_KMS_SERVER RANGER_KMS 1+
Oozie Server OOZIE_SERVER OOZIE 1
Ganglia Server GANGLIA_SERVER GANGLIA 1
Nagios Server NAGIOS_SERVER NAGIOS 1
Ambari Metrics Service METRICS_COLLECTOR AMS 1
Zeppelin Server ZEPPELIN_MASTER SPARK / HIVE 1

Slave Services

Name Ambari Component Name
Service Cardinality
DataNode DATANODE HDFS 1+
Journale Nodes for NameNode HA JOURNALNODE HDFS 0+ (odd #)
Zookeeper Failover Service ZKFC HDFS 0+
Secondary NameNode NFS_GATEWAY HDFS 0+
Node Manager NODEMANAGER YARN 1+
HBase RegionServer HBASE_REGIONSERVER HBASE 1+
Phoneix Query Server PHOENIX_QUERY_SERVER HBASE 0+
Storm Supervisor SUPERVISOR STORM 1+
Ganglia Metrics Collector GANGLIA_MONITOR GANGLIA ALL
Ambari Metrics Collector METRICS_MONITOR AMS ALL

Clients

Name Ambari Component Name
Service Cardinality
HDFS Client HDFS_CLIENT HDFS 1+
YARN Client YARN_CLIENT YARN 1+
MapReduce Client MAPREDUCE2_CLIENT MAPREDUCE2 1+
Spark Client SPARK_CLIENT SPARK 1+
Falcon Client FALCON_CLIENT FALCON 1+
HBase Client HBASE_CLIENT HBASE 1+
Hive Client HIVE_CLIENT HIVE 1+
HCat Client HCAT HIVE 1+
Mahout Client MAHOUT MAHOUT 0+
Oozie Client OOZIE_CLIENT OOZIE 1+
Sqoop Client SQOOP SQOOP 1+
Zookeeper Client ZOOKEEPER_CLIENT ZOOKEEPER 1+

Further Readings

8 thoughts on “Services and State with Ambari REST API”

  1. Thanks for the awesome content. I was looking for some answers on how to start/stop and Ambari service(hive, yarn etc.) using REST API on an single node. I can see many documentations related to this but only at cluster level not individual node level. Can you please advice?

Leave a Reply