In a series of blog posts I demonstrated how to create a custom OS Image for automatic provisioning of HDP with Vagrant on the Azure Cloud. On GitHub I share the result of a first layout among other provisioning scripts. Until now the setup was done without the proper network configuration required for the communication of the individual components of the cluster.
With the new release of the vagrant-azure plugin it will be possible to setup the cluster in a dedicated VNet. This is the last missing peace in the series of work I published to allow the automated provisioning of HDP in Azure. Unfortunately this is not quite true, as the current Ruby SDK of Azure does not allow the passing of IP addresses to the machines. We therefor have to create host entries currently by hand. It could be possible to setup a DNS or use Puppet to conduct the host mapping in a automated fashion, but I at least was not able to do so as part of this work here.
Creating a Proper VNet
First we would have to setup the VNet in which we plan to launch our cluster in.
Also currently not supported by the Ruby SDK of Azure is the creation of machines within a VNet without a affinity group. Affinity group used to be a way to make sure machines are physically located close to each other within Azure. This is in general a good idea. For some reasons the Azure team changed their design in that way that from now on the location is used instead of a separate affinity group. The creation of a VNet with affinity group is no longer possible in the Managing Portal, as for we use the Node SDK to create the VNet. The Managing Portal can be used to create the affinity group.
For creating the affinity group in the Managing Portal you go to the settings page. From there create an affinity group in the desired location:
The created affinity group will show up in the settings like so:
Now that we have the needed affinity group, we can proceed creating the VNet using the Node SDK. To do this, call the following command after login in to Azure:
$ azure login ... $ azure network vnet create --vnet hwx-net --affinity-group hwx-hdp-22-affinity info: Executing command network vnet create info: Using default address space start IP: 10.0.0.0 info: Using default address space cidr: 8 info: Using default subnet start IP: 10.0.0.0 info: Using default subnet cidr: 11 + Getting network configuration + Getting or creating affinity group + Getting affinity groups info: Using affinity group hwx-hdp-22-affinity + Updating Network Configuration info: network vnet create command OK
For a complete list of options simply call create with the –help option.
$ azure network vnet create --help help: Create a Virtual Network help: help: Usage: network vnet create [options] <vnet> help: help: Options: help: -h, --help output usage information help: -v, --verbose use verbose output help: --json use json output help: --vnet <vnet> the name of the virtual network help: -e, --address-space <ipv4> the address space for the virtual network help: -m, --max-vm-count <number> the maximum number of VMs in the address space help: -i, --cidr <number> the address space network mask in CIDR format help: -p, --subnet-start-ip <ipv4> the start IP address of subnet help: -n, --subnet-name <name> the name for the subnet help: -c, --subnet-vm-count <number> the maximum number of VMs in the subnet help: -r, --subnet-cidr <number> the subnet network mask in CIDR format help: -l, --location <name> the location help: -f, --create-new-affinity-group creates a new affinity group at the location specified in --location help: -a, --affinity-group <name> the affinity group help: -d, --dns-server-id <dns-id> the name identifier of the DNS server help: -s, --subscription <id> the subscription id
Installing Vagrant Azure Plugin from Source
The future release of the vagrant-azure plugin will contain the possibility to launch machines inside a VNet. Hopefully also the possibility to assign IP addresses to the individual machines will be part of that release. For now to follow the here described procedure you would have to clone my version of the plugin, build and install it from that source. For details you can follow the submitted issues in the original azure-vagrant plugin here and here.
With the following steps you can clone the project from GitHub, build and install it to vagrant.
$ git clone git@github.com:/hkropp/vagrant-azure.git $ cd vagrant-azure/ $ bundle install $ bundle exec rake build $ vagarant plugin install pkg/vagrant-azure-1.0.5.gem
The plugin should now show up in the list of installed vagrant plugins:
$ vagrant plugin list vagrant-azure (1.0.5) - Version Constraint: 1.0.5 vagrant-cachier (1.2.0) vagrant-hosts (2.3.0) vagrant-omnibus (1.4.1) vagrant-openstack-provider (0.6.0) - Version Constraint: 0.6.0 vagrant-proxyconf (1.5.0) vagrant-share (1.1.3, system)
Once finished you would need to add the following configuration in the Vagrantfile to make all machines present in the given VNet. Please check out this post and my collection of vagrant scripts on GitHub to follow along.
... azure.vm_user = 'hdpuser' azure.vm_virtual_network_name = 'hwx-net' azure.vm_name = opts[:name].to_s azure.vm_location = 'West Europe' ...
The cluster will be created within the VNet of which you can assure yourself from the Azure Management Portal. Each machine can reach all others as part of the VNet. Automated provisioning of HDP through Ambari can be established after configuring proper host resolution.
Assigning specific IP addresses to the created machines is yet not possible but fairly desired. This feature will probably come soon. If the use of the vagrant-hosts plugin will be possible along this, is yet to find out.
henning.kropponline.de 
Azure VNet for Your HDP Cluster http://t.co/XInaXxdF7m
LikeLike