There were only a few things that had been determined. We were using SuSE Linux, we would (eventually) have Active Directory for Authentication/Authorization and were running on HP/Compaq servers. We started with physical, book style systems and used SAN for large storage requirements.
I learned how to use YaST in place of kickstart and started building the first project. We were running IBM Websphere and our first test was setting up new system, that would have otherwise been run on AIX. I didn’t realize until later this was more of a POC than a genuine effort. I highly expect the VP of my area was expecting Linux to bomb so he could keep AIX.
Fortunately, the project was a huge success and highly cost effective. The next few projects were large enough we needed to higher (rent) some additional administrators. And since I was the only full time Linux employee, I got to be in charge.
Given that we had no legacy BS to hold us back, we started doing things the right way. No servers were hand built. There was a minimal base YaST script and we had templates to add required libraries for things like Websphere (WAS). Working with the networking teams, I was able to get a “next server” added to certain networks so we could have PXE kick off the build. PXE has this cool feature where it will look for a filename based on it’s MAC address and work it’s way to less specific files if it can’t find it. Finally it will look for a file called “default” where we tell it to boot to local HD. On a new server this triggers a reboot and the process continues until we’re ready to build.
After we’d set up some common NAS mount point for tools and home dirs, etc. we we’re able to “push the button” and build a server in about 7 minutes. Gathering all of the data before pushing the button took a while longer. Someone had to assign an IP Address. We had to determine a hostname (using a naming convention of meaningful letters and numbers). We had to get SAN Allocated, etc. etc. We usually built using the shared NAS drives to ensure consistency. Someone challenged our numbers to I ran a special build going directly to a local FTP server and it built in just over 3 minutes.
Once we have all of the details specified, we use the YaST Templates to build a custom YaST file and put it in the appropriate PXE “next server” directory. We started off with some bash scripts to do this and stored all of the data in a MySQL Database. Brainstorming with some of the hired help, we built out a pretty cool Web UI system call SPAN. Server Provisioning and Notes (or Notebook).
SPAN became the tool for all of our gathering stuff. We specified templates, hostnames, update schedules, business owners and basically anything we thought we’d need to deal with a server. It was pretty awesome. We had change management and consistency tools, we had monitoring, version control, templates and all of the good IT things a system should have. We even ran an LDAP Server for the first several months until the AD integration could be worked out.
In short, it was a top tier platform. Our very first Audit was rated “Generally Acceptable” by the auditors, which if you’ve ever been audited, you know is an A+. One of the issues was something like “Former employees should be removed from Groups they were members of.” 1. Their accounts were disabled, so the groups don’t matter. 2. We weren’t in charge of managing those groups. There were 2 or 3 other items, but they were all a nit picky.
The system continued to cruise along until the VP was ready to strike. I was moved to a new team that was supposed to help with automation. The manager was on a PIP and we only had $500k budget for a tool that was supposed to cost $750k or so. We spent a few weeks installing and learning the tool and were able to demonstrate self serve host provisioning on both Linux and Windows. Eventually they spent our budget on something else and I … Moved on to Hadoop, which is where this whole blog started.