Browsing HDP Public Repo with s3cmd

From time to time it can be very useful to be able to search for HDP repository release directly from the public repo. Especially if you want to search for a recent development or technical preview version. Also this can become handy if you need to create an offline repository for your company intranet.

The HDP repositories are available through Amazon’s S3 storage layer. A tool quite convenient to use it s3cmd.

After downloading it s3cmd can easily be installed based on python:

// requires python-setuptools
$ cd ~/Downloads/
$ tar xfz s3cmd-1.5.2.tar.gz
$ cd s3cmd-1.5.2
$ more INSTALL // to read INSTALL guide
$ sudo python setup.py install

Browing HDP repo:

$ s3cmd ls s3://public-repo-1.hortonworks.com/HDP/centos6/

Using wildcards:

$ s3cmd ls s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.0-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.1-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.2-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.3-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/GA/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/
2013-07-09 00:06  0  s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/

Wildcards can be used for filtering:

$ s3cmd ls s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.*
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.0.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.1.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.3.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.5.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.6.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.8.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.99.0

For help:

$ s3cmd --help

Further Readings

• s3cmd Usage http://s3tools.org/usage
• INSTALL https://github.com/s3tools/s3cmd/blob/master/INSTALL
• s3cmd Download http://s3tools.org/download
• For Windows: S3Express http://www.s3express.com/
• S3: HDP Public Repo s3://public-repo-1.hortonworks.com/
• S3: HDP Dev Repp: s3://dev.hortonworks.com/