Azure VNet for Your HDP Cluster

In a series of blog posts I demonstrated how to create a custom OS Image for automatic provisioning of HDP with Vagrant on the Azure Cloud. On GitHub I share the result of a first layout among other provisioning scripts. Until now the setup was done without the proper network configuration required for the communication of the individual components of the cluster.

With the new release of the vagrant-azure plugin it will be possible to setup the cluster in a dedicated VNet. This is the last missing peace in the series of work I published to allow the automated provisioning of HDP in Azure. Unfortunately this is not quite true, as the current Ruby SDK of Azure does not allow the passing of IP addresses to the machines. We therefor have to create host entries currently by hand. It could be possible to setup a DNS or use Puppet to conduct the host mapping in a automated fashion, but I at least was not able to do so as part of this work here.

Creating a Proper VNet

First we would have to setup the VNet in which we plan to launch our cluster in.

Also currently not supported by the Ruby SDK of Azure is the creation of machines within a VNet without a affinity group. Affinity group used to be a way to make sure machines are physically located close to each other within Azure. This is in general a good idea. For some reasons the Azure team changed their design in that way that from now on the location is used instead of a separate affinity group. The creation of a VNet with affinity group is no longer possible in the Managing Portal, as for we use the Node SDK to create the VNet. The Managing Portal can be used to create the affinity group.

For creating the affinity group in the Managing Portal you go to the settings page. From there create an affinity group in the desired location:

Create Affinity Group - AzureThe created affinity group will show up in the settings like so:

Affinity Group Settings - AzureNow that we have the needed affinity group, we can proceed creating the VNet using the Node SDK. To do this, call the following command after login in to Azure:

$ azure login
$ azure network vnet create --vnet hwx-net --affinity-group hwx-hdp-22-affinity
info:    Executing command network vnet create
info:    Using default address space start IP:
info:    Using default address space cidr: 8
info:    Using default subnet start IP:
info:    Using default subnet cidr: 11
+ Getting network configuration                                                
+ Getting or creating affinity group                                           
+ Getting affinity groups                                                      
info:    Using affinity group hwx-hdp-22-affinity
+ Updating Network Configuration                                               
info:    network vnet create command OK

For a complete list of options simply call create with the –help option.

$ azure network vnet create --help                                             
help:    Create a Virtual Network
help:    Usage: network vnet create [options] <vnet>
help:    Options:
help:      -h, --help                       output usage information
help:      -v, --verbose                    use verbose output
help:      --json                           use json output
help:      --vnet <vnet>                    the name of the virtual network
help:      -e, --address-space <ipv4>       the address space for the virtual network
help:      -m, --max-vm-count <number>      the maximum number of VMs in the address space
help:      -i, --cidr <number>              the address space network mask in CIDR format
help:      -p, --subnet-start-ip <ipv4>     the start IP address of subnet
help:      -n, --subnet-name <name>         the name for the subnet
help:      -c, --subnet-vm-count <number>   the maximum number of VMs in the subnet
help:      -r, --subnet-cidr <number>       the subnet network mask in CIDR format
help:      -l, --location <name>            the location
help:      -f, --create-new-affinity-group  creates a new affinity group at the location specified in --location
help:      -a, --affinity-group <name>      the affinity group
help:      -d, --dns-server-id <dns-id>     the name identifier of the DNS server
help:      -s, --subscription <id>          the subscription id

Installing Vagrant Azure Plugin from Source

The future release of the vagrant-azure plugin will contain the possibility to launch machines inside a VNet. Hopefully also the possibility to assign IP addresses to the individual machines will be part of that release. For now to follow the here described procedure you would have to clone my version of the plugin, build and install it from that source. For details you can follow the submitted issues in the original azure-vagrant plugin here and here.

With the following steps you can clone the project from GitHub, build and install it to vagrant.

$ git clone
$ cd vagrant-azure/
$ bundle install
$ bundle exec rake build
$ vagarant plugin install pkg/vagrant-azure-1.0.5.gem

The plugin should now show up in the list of installed vagrant plugins:

$ vagrant plugin list
vagrant-azure (1.0.5)
  - Version Constraint: 1.0.5
vagrant-cachier (1.2.0)
vagrant-hosts (2.3.0)
vagrant-omnibus (1.4.1)
vagrant-openstack-provider (0.6.0)
  - Version Constraint: 0.6.0
vagrant-proxyconf (1.5.0)
vagrant-share (1.1.3, system)

Once finished you would need to add the following configuration in the Vagrantfile to make all machines present in the given VNet. Please check out this post and my collection of vagrant scripts on GitHub to follow along.

azure.vm_user = 'hdpuser'
azure.vm_virtual_network_name = 'hwx-net'
azure.vm_name = opts[:name].to_s
azure.vm_location = 'West Europe'

The cluster will be created within the VNet of which you can assure yourself from the Azure Management Portal. Each machine can reach all others as part of the VNet. Automated provisioning of HDP through Ambari can be established after configuring proper host resolution.

Assigning specific IP addresses to the created machines is yet not possible but fairly desired. This feature will probably come soon. If the use of the vagrant-hosts plugin will be possible along this, is yet to find out.

Further Readings


One thought on “Azure VNet for Your HDP Cluster

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s