It’s scary… but sometimes things go really well!

If you’ve been in IT for more than a few minutes, you know that things seldom go as planned.  Sometimes they go sideways and you’re left to clean up an abomination.  Been there, done that, expect it at least 15% of the time.  And that’s with 30+ years of experience.

However, sometimes, occasionally, the worlds align, karma is a babe and stuff just works like you envisioned it.  The Pizza Delivery Boy had a 45,000 ft. vision of Hadoop Clusters that he shared with a Grease Monkey.  Long story short; crazy visions and monkeys (pseudo intelligent creatures) can get lucky.

All self-deprecation  aside; we’ve created something that might be a new major break in Hadoop Cluster deployments.  We’ve learned some solid lessons along the way, have been corrected about some of our assumptions, and generally kicked some serious ass in the Enterprise, Secure, Rapid Deployment, Hadoop World.   Is that enough classifications?

We now have Datanodes that boot to a fixed IP (so we don’t need to restart NN  services to accept new IP Addr), can switch between Dev, Prod, etc. environments by rebooting, can be added to a data center by simply  being powered on w/ network connectivity and are still secured against rogue systems.  If we didn’t know we’re getting new systems, they don’t get access to the cluster, regardless of their IP addr.

I don’t need to maintain IP Address reservations in DHCP,  I don’t need host definitions for DHCP. I can use 1 default setting for all environments and they’ll still only load the software for their own config.  Using existing open source software (oneSIS) and capitalizing on PXE boot parameters, we can now deliver a very solid, consistent and expandable cluster; nearly on demand.

Datanode add-ons (such as R)  can be distributed to DataNodes without downtime.

I am very seldom impressed with my own accomplishments.  I think I’ll make an exception this time.


The Humble Grease Monkey.

About Grease Monkey

30+ Years of IT Geekiness, Linux Fanboy and Open Source patriot.
This entry was posted in Administration, Deployment, Development, Tuning and tagged , , . Bookmark the permalink.

One Response to It’s scary… but sometimes things go really well!

  1. Pizza Boy says:

    Nice job – now let’s see how scalable it can be as we expand. We now need Integration, Backup, and to expand prod and dev. 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.