This is the End of Hadoop as We Know It (And I Feel Fine!)
The stocks of both companies soar instantly after market and as soon the announcement of the merger was made. Overall market participants seemed pleased by the outlook of a companied company. Across the media widely positive reactions sprawled and Forrester for example is sure that this is a “A Win-Win For All”. So all fine!?
Well, there is at least one that strongly disagrees with this assessment, although this might mainly be because his job title suggests so. In this regards did the CEO of MapR, John Schroeder, say:
“I can’t find any innovation benefits to customers in this merger”
Apart from this almost sole opinion, is this deal really the success story everyone believes it to be? No, it is not in my opinion and as the dust settles it be comes more and more obvious that this deal is surrounded by dark clouds.
Continue reading “Why the Hortonworks-Cloudera Merger Is a Big Defeat?”
In a restricted setup YARN executes task of computation frameworks like Spark in a secured Linux or Window Container. The task are being executed in the local context of the user submitting the application and are not being executed in the local context of the yarn or some other system user. With this come certain constraints for the system setup.
How is YARN actually able to impersonate the calling user on the local OS level? This posts aims to give some background information to help answer such questions about secure containers. Only Linux systems are considered here, no Windows.
Continue reading “YARN Secure Container”
Ambari Management Packs are a new convenient way to integrate various services to the Ambari stack. As an example in this post we are using the Solr service mpack to install HDP on top of a newly installed cluster.
The HDP search mpack is available on the Hortonworks public repository for download. A mpack essentially is tar balls containing a mpack.json file specification and related binaries. Continue reading “Installing HDP Search with Ambari”
Over two years ago in March 2014 I joined the Iron Blogger community in Munich, which is one of the largest, still active Iron Blogger communities worldwide. You can read more about my motivation behind it here in one of the 97 blog posts published to date: Iron Blogger: In for a Perfect Game.
The real fact is that I write blogs solely for myself. It’s my own technical reference I turn to. Additionally writing is a good way to improve once skills and technical capabilities, as Richard Guindon puts it in his famous quote:
“Writing is nature’s way of letting you know how sloppy your thinking is.”
What could be better suited to improve something than by leaning into the pain, how the great Aaron Swartz, who died way too early, once described it? And it is quite a bit of leaning into the pain publishing a blog post every week. Not only for me, but also for those close to me. But I am going to dedicate a separate blog post to a diligent retrospection in the near future. This post should all be about NUMBERS. Continue reading “2016 in Numbers”
Some notes for Kerberos debugging in a secure HDP setup:
- Setting Debug Logs
To enable debug logs in Java for Kerberos sun.security.krb5.debug needs to be set to true. Doing this for Hadoop can be done in the hadoop-env.sh file by adding it to the HADOOP_OPTS environment variable:
Additionally the HADOOP_JAAS_DEBUG variable can be set also:
Receiving traces in bash/shell can be enabled by setting the following environment variable:
- Testing auth_to_local Settings
Setting the auth_to_local rules correclty can be quite crucial. This is especially true for KDS trust environments. The rules can be easily tested with the HadoopKerberosName call of Hadoop security. You can run it as:
$ hadoop org.apache.hadoop.security.HadoopKerberosName pinc@REALM.COM
In this Sunday Read with Horton edition we take a closer look at the selection of papers about Distributed Consensus provided by Camille Fournier (Zookeeper PMC) as part of the RfP (Research for Practice) of the ACM. For Hadoop practitioners distributed consensus is best know as Apache Zookeeper, which supports most critical aspects of almost all Hadoop components. Continue reading “Sunday Read: Distributed Consensus”
In any HDP cluster with a HA setup with quorum there are two NameNodes configured with one working as the active and the other as the standby instance. As the standby node does not accept any write requests, for a client try to write to HDFS it is fairly important to know which one of the two NameNodes it the active one at any given time. The discovery process for that is configured through the hdfs-site.xml.
For any custom implementation it’s becomes relevant to set and understand the correct parameters if a current hdfs-site.xml configuration of the cluster is not given. This post gives a sample Java implementation of a HA HDFS client. Continue reading “Sample HDFS HA Client”