If you’ve been in IT for more than a few minutes, you know that things seldom go as planned. Sometimes they go sideways and you’re left to clean up an abomination. Been there, done that, expect it at least 15% of the time. And that’s with 30+ years of experience.
However, sometimes, occasionally, the worlds align, karma is a babe and stuff just works like you envisioned it. The Pizza Delivery Boy had a 45,000 ft. vision of Hadoop Clusters that he shared with a Grease Monkey. Long story short; crazy visions and monkeys (pseudo intelligent creatures) can get lucky.
All self-deprecation aside; we’ve created something that might be a new major break in Hadoop Cluster deployments. We’ve learned some solid lessons along the way, have been corrected about some of our assumptions, and generally kicked some serious ass in the Enterprise, Secure, Rapid Deployment, Hadoop World. Is that enough classifications?
We now have Datanodes that boot to a fixed IP (so we don’t need to restart NN services to accept new IP Addr), can switch between Dev, Prod, etc. environments by rebooting, can be added to a data center by simply being powered on w/ network connectivity and are still secured against rogue systems. If we didn’t know we’re getting new systems, they don’t get access to the cluster, regardless of their IP addr.
I don’t need to maintain IP Address reservations in DHCP, I don’t need host definitions for DHCP. I can use 1 default setting for all environments and they’ll still only load the software for their own config. Using existing open source software (oneSIS) and capitalizing on PXE boot parameters, we can now deliver a very solid, consistent and expandable cluster; nearly on demand.
Datanode add-ons (such as R) can be distributed to DataNodes without downtime.
I am very seldom impressed with my own accomplishments. I think I’ll make an exception this time.
THIS STUFF IS AWESOME!
The Humble Grease Monkey.