Ambari Augeas Module and Configuration

Operating large to small size clusters needs automation; as well does installation. Going repeatedly through installation instructions, typing the same commands over and over again is really no fun. Worse it’s a waste of resources. The same holds true for keeping the configuration of 10 or 100 nodes in sync. A first step is to use a distributed shell and the next to turn to tools like Puppet, Cobblerd, Ansible, or Chef, which promise full automation of cluster operations.

As with most automation processes this tools tend to make the common parts easy and the not so common steps hard. The fact that every setup is different requires concepts of  adaptability that are easy to use. One of this tools for adapting configuration files could be seen in Augeas. Although hard to understand at time Augeas is easy to use, as it turns arbitrary configuration files into trees on which typical tree operations can be performed.

This post highlights the possibilities to us Augeas in a Puppet setup to install and configure a HDP Ambari cluster. Ambari agents are part of every node in a cluster, so when adding new nodes to a cluster in an automated fashion you would want to be prepared.

Ambari Agent Augeas Module

An Ambari setup mainly consists of properties, XML, and ini configuration files. For properties Files a properties lens exists. The same is also true for XML configuration files by a XML lens. Bot XML and properties file manipulation for Ambari setups will be covered below.

To manipulate ini files Augeas only provides a generic IniFile lens as a general interface. As ini files sometime follow their own specification one has to provide it’s own implementation base on that interface. The following lens is based on the PHP.ini lens and once place under /usr/share/augeas/lenses/ it can be used to configure ambari-agent.ini files under /etc/ambari-agent/conf:

(* AmbariAgent module for Augeas                                          *)
(* Author: Henning Kropp hrkopp at hortonworks.com                        *)
(*                                                                        *)
(* /usr/share/augeas/lenses/ambariagent.aug                               *)
(*                                                                        *)

module AmbariAgent =
  autoload xfm

(************************************************************************
 * INI File settings
 *************************************************************************)

let comment  = IniFile.comment IniFile.comment_re IniFile.comment_default
let sep      = IniFile.sep IniFile.sep_re IniFile.sep_default
let empty    = IniFile.empty


(************************************************************************
 *                        ENTRY
 *************************************************************************)
let entry    = IniFile.entry_multiline IniFile.entry_re sep comment


(************************************************************************
 *                         TITLE
 *************************************************************************)
let title    = IniFile.title IniFile.record_re
let record   = IniFile.record title entry

(************************************************************************
 *                         LENS & FILTER
 *************************************************************************)
let lns       = IniFile.lns record comment

let filter = (incl "/etc/ambari-agent/conf/*.ini")
             . Util.stdexcl

let xfm = transform lns filter

Place this under /usr/share/augeas/lenses/ in a file named ambariagent.aug and test it by running:

augtool print  /files/etc/ambari-agent/conf/ambari-agent.ini

If nothing or an error happens, check the concrete output with:

augtool print /augeas//error

Ambari Properties Configuration

The properties lens provided by Augeas is also just a generic one similar to the ini file lens. In contrast to providing a custom lense like we did above, we will in this case use the possibility to override autoload filters in the puppet module by providing an incl statement. This will automatically map it to a /files/* mapping in augeas.

The following snippet in Puppet can be used to override the Ambari server user under /etc/ambari-server/conf/ambari.properties:

augeas { 'server-properties':
   require => Augeas[agent_update],
   lens => 'Properties.lns',
   incl => "/etc/ambari-server/conf/ambari.properties",
   changes => ["set ambari-server.user ambari_root"]
}

Ambari XML Configuration

One use case that would require the manipulation of a XML file in an Ambari setup is to change the repository URLs for your OS type. The following code snippet would exactly achieve this by using the Xml.lns in a puppet script:

augeas { 'repolist':
  require => Package[ambari-server],
  lens => "Xml.lns",
  incl => '/var/lib/ambari-server/resources/stacks/HDP/2.2/repos/repoinfo.xml',
  onlyif => "match dir[. = '/var/lib/ambari-server/resources/stacks/HDP/2.2/repos'] size != 0",
  changes => [
    "set reposinfo/latest/#text https://henning.kropponline.de/wp-content/uploads/hdpqe_urlinfo-7.json",
    "set reposinfo/os[#attribute/type='redhat6']/repo[repoid/#text='HDP-2.2']/baseurl/#text http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Champlain-Preview/2.2.0.0-7/centos6",
    "set reposinfo/os[#attribute/type='redhat6']/repo[repoid/#text='HDP-UTILS-1.1.0.20']/baseurl/#text http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.19/repos/centos6"]
}

Further Readings

• Augeas Book (GitHub free)
• What is DevOps? (free ebook / Amazon)
• Ops, DevOps and PaaS (NoOps) at Netflix
• Augeas – Stock Lenses
• Puppet Augeas Resource Type
• Alternative Augeas-based providers for Puppet