Azure VNet for Your HDP Cluster

In a series of blog posts I demonstrated how to create a custom OS Image for automatic provisioning of HDP with Vagrant on the Azure Cloud. On GitHub I share the result of a first layout among other provisioning scripts. Until now the setup was done without the proper network configuration required for the communication of the individual components of the cluster.

With the new release of the vagrant-azure plugin it will be possible to setup the cluster in a dedicated VNet. This is the last missing peace in the series of work I published to allow the automated provisioning of HDP in Azure. Unfortunately this is not quite true, as the current Ruby SDK of Azure does not allow the passing of IP addresses to the machines. We therefor have to create host entries currently by hand. It could be possible to setup a DNS or use Puppet to conduct the host mapping in a automated fashion, but I at least was not able to do so as part of this work here.

Creating a Proper VNet

First we would have to setup the VNet in which we plan to launch our cluster in.

Also currently not supported by the Ruby SDK of Azure is the creation of machines within a VNet without a affinity group. Affinity group used to be a way to make sure machines are physically located close to each other within Azure. This is in general a good idea. For some reasons the Azure team changed their design in that way that from now on the location is used instead of a separate affinity group. The creation of a VNet with affinity group is no longer possible in the Managing Portal, as for we use the Node SDK to create the VNet. The Managing Portal can be used to create the affinity group.

For creating the affinity group in the Managing Portal you go to the settings page. From there create an affinity group in the desired location:

Create Affinity Group - AzureThe created affinity group will show up in the settings like so:

Affinity Group Settings - AzureNow that we have the needed affinity group, we can proceed creating the VNet using the Node SDK. To do this, call the following command after login in to Azure:

For a complete list of options simply call create with the  --help option.

Installing Vagrant Azure Plugin from Source

The future release of the vagrant-azure plugin will contain the possibility to launch machines inside a VNet. Hopefully also the possibility to assign IP addresses to the individual machines will be part of that release. For now to follow the here described procedure you would have to clone my version of the plugin, build and install it from that source. For details you can follow the submitted issues in the original azure-vagrant plugin here and here.

With the following steps you can clone the project from GitHub, build and install it to vagrant.

The plugin should now show up in the list of installed vagrant plugins:

Once finished you would need to add the following configuration in the Vagrantfile to make all machines present in the given VNet. Please check out this post and my collection of vagrant scripts on GitHub to follow along.

The cluster will be created within the VNet of which you can assure yourself from the Azure Management Portal. Each machine can reach all others as part of the VNet. Automated provisioning of HDP through Ambari can be established after configuring proper host resolution.

Assigning specific IP addresses to the created machines is yet not possible but fairly desired. This feature will probably come soon. If the use of the vagrant-hosts plugin will be possible along this, is yet to find out.

Further Readings

1 thought on “Azure VNet for Your HDP Cluster”

Leave a Reply