Azure VNet for Your HDP Cluster

In a series of blog posts I demonstrated how to create a custom OS Image for automatic provisioning of HDP with Vagrant on the Azure Cloud. On GitHub I share the result of a first layout among other provisioning scripts. Until now the setup was done without the proper network configuration required for the communication of the individual components of the cluster.

With the new release of the vagrant-azure plugin it will be possible to setup the cluster in a dedicated VNet. This is the last missing peace in the series of work I published to allow the automated provisioning of HDP in Azure. Unfortunately this is not quite true, as the current Ruby SDK of Azure does not allow the passing of IP addresses to the machines. We therefor have to create host entries currently by hand. It could be possible to setup a DNS or use Puppet to conduct the host mapping in a automated fashion, but I at least was not able to do so as part of this work here. Continue reading “Azure VNet for Your HDP Cluster”

Creating a HDP Ready CentOS Image for Azure

There already exist some collections of VM images in Windows Azure gallery. Additionally on VM Depot the community can share custom provisioned images. Looking for a HDP ready image based on CentOS I could not find one suitable for my needs. In this post I would like to describe how I created my first HDP ready image based on CentOS for Windows Azure. The image will be created within Azure itself, as I don’t have access to Hyper-V and any attempts to create a CentOS based VHD with VirtualBox failed for unknown reasons. Continue reading “Creating a HDP Ready CentOS Image for Azure”