It’s Friday evening before our Hadoop Administrator leaves for a week of vacation in New Hampshire and about an hour before he leaves he says “it’s turning into a pumpkin in an hour”. Of course we wanted to go live with a new project on Friday afternoon before he leaves. About a week before he left we reminded everyone he was leaving and made it clear he didn’t have a “real” backup.
So now me – the “Ugly Step Sister” is filling in for Cinderella and of course things don’t go as planned….it never does. I’m fond of saying if things “just worked” we wouldn’t need high priced engineers/architects, so I suppose that’s a good thing. Now after a week of filling in for Cinderella we have had a number of last minute things that needed fixed and I have figured out that I have been living in an ivory tower a bit and I have forgotten how to sweep floors.
When moving into architecture ten years ago my biggest fear was my IT capabilities would atrophy and I would be less employable. I know that I’m actually very good at strategy and I am far more value strategically driving millions of dollars in savings and large strategic shifts in IT, but boy this week was a humbling experience. Now that I’m through the week and I have overcome a number of obstacles (some of them slowly) here is what I have learned:
First – I need to get back to my roots. Every architect needs a sandbox to get hands on keyboard and they need to allocate time to get their hands dirty. I’m creating a sandbox for myself to tinker and I’m going to spend some time doing some Hadoop Automation work – I really like getting my hands dirty.
Second – We need more automation – many of the things we do in Hadoop are manual today. Right now the Hadoop Engineering team (Grease Monkey) is so busy fighting fires…or hair on fire developers that he isn’t automating much. While he was out I put together a rough user add script – that adds users includes integrating with active directory, creating Hadoop environments, adjusting permissions, rolling changes from access to name nodes, creating pub, sub, bus directories, creating hive table, testing…and more testing. (Grease Monkey took it to the next level when he returned)
Third – Developers are crazy…hair on fire crazy and giving them a deadline drives crazy blame everyone behavior. We need some queuing mechanism that gives us the ability to track, plan, and prioritize work. We put a preliminary easy to use one in place, but nobody is using it. We need to go back and address this in a non-corporate, fast moving startup sort of fashion….but we NEED IT!
Fourth – While we are doing some amazing innovative things that are game changing in the admin space we are also very early in the Operationalization of the platform and we have a ton of solidification work to do before we can breathe easy. As fast as Hadoop is evolving I’m not sure we will ever breathe easy though.