Data for Profit $$

There were only a few things that had been determined. We were using SuSE Linux, we would (eventually) have Active Directory for Authentication/Authorization and were running on HP/Compaq servers. We started with physical, book style systems and used SAN for large storage requirements.

I learned how to use YaST in place of kickstart and started building the first project. We were running IBM Websphere and our first test was setting up new system, that would have otherwise been run on AIX. I didn’t realize until later this was more of a POC than a genuine effort. I highly expect the VP of my area was expecting Linux to bomb so he could keep AIX.

Fortunately, the project was a huge success and highly cost effective. The next few projects were large enough we needed to hire (rent) some additional administrators. And since I was the only full time Linux employee, I got to be in charge.

Given that we had no legacy BS to hold us back, we started doing things the right way. No servers were hand built. There was a minimal base YaST script and we had templates to add required libraries for things like Websphere (WAS). Working with the networking teams, I was able to get a “next server” added to certain networks so we could have PXE kick off the build. PXE has this cool feature where it will look for a filename based on it’s MAC address and work it’s way to less specific files if it can’t find it. Finally it will look for a file called “default” where we tell it to boot to local HD. On a new server this triggers a reboot and the process continues until we’re ready to build.

After we’d set up some common NAS mount points for tools and home dirs, etc. we we’re able to “push the button” and build a server in about 7 minutes. Gathering all of the data before pushing the button took a while longer. Someone had to assign an IP Address. We had to determine a hostname (using a naming convention of meaningful letters and numbers). We had to get SAN Allocated, etc. etc. We usually built using the shared NAS drives to ensure consistency. Someone challenged our numbers to I ran a special build going directly to a local FTP server and it built in just over 3 minutes.

Once we have all of the details specified, we use the YaST Templates to build a custom YaST file and put it in the appropriate PXE “next server” directory. We started off with some bash scripts to do this and stored all of the data in a MySQL Database. Brainstorming with some of the hired help, we built out a pretty cool Web UI system call SPAN. Server Provisioning and Notes (or Notebook).

SPAN became the tool for all of our gathering stuff. We specified templates, hostnames, update schedules, business owners and basically anything we thought we’d need to deal with a server. It was pretty awesome. We had change management and consistency tools, we had monitoring, version control, templates and all of the good IT things a system should have. We even ran an LDAP Server for the first several months until the AD integration could be worked out.

In short, it was a top tier platform. Our very first Audit was rated “Generally Acceptable” by the auditors, which if you’ve ever been audited, you know is an A+. One of the issues was something like “Former employees should be removed from Groups they were members of.” 1. Their accounts were disabled, so the groups don’t matter. 2. We weren’t in charge of managing those groups. There were 2 or 3 other items, but they were all a nit picky.

The system continued to cruise along until the VP was ready to strike. I was moved to a new team that was supposed to help with automation. The manager was on a PIP and we only had $500k budget for a tool that was supposed to cost $750k or so. We spent a few weeks installing and learning the tool and were able to demonstrate self serve host provisioning on both Linux and Windows. Eventually they spent our budget on something else and I … Moved on to Hadoop, which is where this whole blog started.

I got my first Linux CD ROM in 1995, Walnut Creek from 1994. My previous experience with DOS and CP/M left me longing for true multitasking. I had a few web hosts over the years but nothing super serious. I basically wanted to control my own mail and web servers. The mail server lost luster when I got my Gmail invite. But I had experience.

That experience got honed working for an online check system company. I was running a backup data center. We were installing VMWare hosts and then RHEL VM’s as needed. We had a lot of custom web servers running that we built using kickstart. I learned a lot about manipulating kickstart scripts and dealing with hardware. After about 18 months I ran across an opportunity to build a Linux platform “green field” style. The large, Fortune 20, enterprise was running AIX and Windows. The closest thing they had to a mainframe was an IBM AS/400 system or two.

So… I took the new gig.

During the dot com boom, I was working in the software distribution world. At the time the world was worried about running out of IP addresses. I was irritated by this as IPv6¹ had been around for quite some time and I didn’t understand why more people weren’t using it. All of the then current operating systems supported it, so….

I put a plan together to build a second internet using IPv6. My idea was to build a “virtual backbone” using some existing networking and firing up some dark fiber. I wanted to partner with local cable companies to run the “last mile” and deal with managing customers. We were going to use a federated Identity Management system so you could tell who was who. The idea was fleshed out to bring in other players to incentive-ize adoption and expansion.

Taking advantage of the features in IPv6 we could use QOS to stream TV and Audio, plus keep all traffic secure using it’s built in encryption, IPSec. And relieve the IPv4 IP issues.

It was good enough to warrant a “shark tank” like trip to Charlotte, NC. An investor there had several VC connections for whom he sourced ideas. His usual process was a 2 1/2 hour lunch with 7 – 10 VC’s where ideas were given 15 minutes to make their pitch. For me, he invited his VC’s and we had a 3 hour dinner where I was the only pitch. By the end, I was offered $5M to do a test in a local market.

Within 1 week of that meeting, the dot com bubble popped and No one wanted anything to do with the idea.

I also pitched the idea that we could set up our own Master DNS server to offer more domains than were currently available. At the time the .com domain selling business was 100’s of millions of dollars per year. We were going to do all the fun ones. .mail, .xxx, .mall, .person, .family, etc. etc. Setting your own DNS is pretty trivial and Google and Cloudflare are late to the party IMHO.

¹ https://en.wikipedia.org/wiki/IPv6