Browsing HDP Public Repo with s3cmd

From time to time it can be very useful to be able to search for HDP repository release directly from the public repo. Especially if you want to search for a recent development or technical preview version. Also this can become handy if you need to create an offline repository for your company intranet.

The HDP repositories are available through Amazon’s S3 storage layer. A tool quite convenient to use it s3cmd.

After downloading it s3cmd can easily be installed based on python:

// requires python-setuptools
$ cd ~/Downloads/
$ tar xfz s3cmd-1.5.2.tar.gz
$ cd s3cmd-1.5.2
$ more INSTALL // to read INSTALL guide
$ sudo python setup.py install

Browing HDP repo:

$ s3cmd ls s3://public-repo-1.hortonworks.com/HDP/centos6/

Using wildcards:

$ s3cmd ls s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.0-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.1-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.2-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/2.3-latest/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/GA/
DIR   s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/
2013-07-09 00:06  0  s3://public-repo-1.hortonworks.com/HDP/centos6/2.x/

Wildcards can be used for filtering:

$ s3cmd ls s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.*
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.0.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.1.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.2.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.3.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.4.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.5.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.6.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.8.0/
DIR   s3://dev.hortonworks.com/HDP/centos6/2.x/updates/2.3.99.0

For help:

$ s3cmd --help

Further Readings

Advertisements

One thought on “Browsing HDP Public Repo with s3cmd

  1. Hello,
    After I installed s3cmd, it seems to be need Access and Secret Keys to connect to the s3 HDP repo link. How can I obtain these keys?

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s