First I’m on a configuration management panel run by James Turnbull at the Open Source Bridge in Portland, OR.  I haven’t been on a multiple-tool panel in a while, so this should be interesting.  I’m also excited to be back in Portland – I absolutely love the town, so much that I’m moving back there this summer.  And for those non-Americans who think we don’t make great beer:  Get thyself to Portland, and see how wrong you are.  I should also be running a Puppet BoF there.

From there I fly down to San Jose to give a workshop on Puppet at the O’Reilly Velocity conference.  This is its second year, and while it claims to have a web operations/performance focus, I think it stands to quickly become the best general operations conference around.  LISA is too academic and just moves too slowly, and there isn’t a whole lot of other competition.

Last year’s was great, and this year’s is a day longer and looks to be even better.

Finally, the day after Velocity, I attend Om Malik’s Structure.  This conference is more focused on operations executives, which is a very different focus for me, but I met a lot of interesting and knowledgeable people there last year, including some from completely outside the technology space.  I normally diss executive events, but this one seems to stand out.

Hopefully you’ll have a chance to track me down at one of these events.

As mentioned previously, I’d concluded that ISconf wasn’t sufficient, because it was theoretically bogus (maintaining order didn’t guarantee you all that much), it precluded refactoring, and it didn’t actually make my sysadmin work easier, really — I was still writing tons of little shell scripts.  As I described it in my paper: “In choosing between ISconf and most other [configuration management systems], this is largely a choice between a tool which easily organizes work and a tool which makes the work itself easier.”

Order Matters

Given the situation at the time, and the fact that Cfengine’s author, Mark Burgess, was having a kind of war of words with the author of ISconf, Steve Traugott,  it made sense to look at Cfengine.  I want to step aside for a moment, though, and discuss this war of words.  The summary of the war is “order matters”, and when I hear that it’s like “where’s the beef” or “it has what plants crave” — it’s a simple phrase but one that brings up context, hilarity, and stupidity all at once.  Steve maintained that order matters when managing systems, and Mark theoretically maintained that it did not.  These respective beliefs were evident in that Steve’s tool, ISconf, was really just a means of guaranteeing order and nothing more, and Mark’s tool, Cfengine, made it ridiculously difficult to maintain order.

The truth, of course, is that order does matter, but only as a prerequisite, not the end-all be all.  You obviously can’t start a service if you haven’t installed its package yet, but there is such a thing as orthogonal services, and you definitely, definitely don’t want to build all of your systems with the assumption that every chunk of work requires every other chunk of work ever performed.

My hopefully-reasonable summary should not imply that their conversations were reasonable; they were not. It was an infantile and annoying conversation all around, and to me, the phrase “order matters” will always evoke laughter, pity, and a desire to drink heavily while spouting epithets. When I was first exposed to this conversation, I was an ISconf user (well, *the* ISconf user — one other person adopted it, and I suppose Steve used it, but…), so everyone assumed I was on Steve’s side, but I’m not much of an ideologue, and I certainly wasn’t committed that early.

Integrating Cfengine

Anyway. So, I decided to look at Cfengine.  It’s a completely different kind of tool than ISconf — it’s organized around what it calls “actions”, although none of them are verbs, for some reason.  These are things like files, shellcommands, processes, and many others.  The immediate, obvious difference between the two is that Cfengine is a tool focused on what you’re trying to do, and ISconf is a tool that just cares about in what order you do it.

The other thing about Cfengine is it has this whole “control” section that’s all about setting up variables and making decisions, and you can set booleans (which Cfengine calls classes) that can then be used to make decisions throughout the whole configuration.  I immediately imported a lot of the convoluted logic I had hidden in ISconf make files into Cfengine and saved myself a ton of headache; then I integrated the two tools, such that ISconf could take advantage of all of the knowledge and context that Cfengine had, and Cfengine could take advantage of all of the information I’d already input into ISconf.

This, of course, meant I was going to hell. I wrote an article describing my experiences with each tool, plus how I integrated them, and there were some interesting conversations as a result.  Initially, the integration was focused on not throwing away the work I’d done with ISconf, but over time, I stopped adding code to ISconf and largely just focused on Cfengine, and in the end, I found it was more effective to just do my work directly through Cfengine.

At the same time, I started a consulting company to focus on helping other companies with with configuration management problems, since so few organizations seemed to have a decent solution, and I was more interested in continuing to refine my solution than I was in moving on to more mundane sysadmin problems like backups or monitoring.

After a few months of consulting, though, I found that Cfengine could be painfully low-level.  It could do files, commands, and processes, but it couldn’t do users, groups, or packages.  Also, I was maintaining Cfengine configurations for multiple platforms, and it was a real nightmare managing Cfengine code that did the same thing in different ways, or even just used different paths.

Even worse, though, I found I couldn’t keep promises to my customers.  Cfengine often wouldn’t compile on a given platform across releases, I couldn’t consistently get bug fixes or refactorings accepted into the code base, and I had no ability to influence the direction of its development.  Mark Burgess was the main developer, and he was very defensive of any attempt to refactor significant parts of the code.  He also, somewhat notoriously, said he considered version control to be overhead, which meant we kept magically getting bug regressions.

Looking for other options

So, frustrated with Cfengine and its lack of a developer community, I started looking around for other options. I tried out Quattor, LCFG, SmartFrog, Bcfg2, psgconf, and probably more, but none of them seemed like they would allow the kind of extension and abstraction that I was looking for.  I was afraid I was going to have to make my own tool, but I really didn’t want to.  As an interim step, in the summer of 2004 I decided to fork Cfengine and fix some of its problems.  I called this fork cfng (it’s completely dead now).

One of the things that drove me most crazy about Cfengine was that it had 25 different syntaxes for its 33 or so different actions — ‘processes’ was *slightly* different from ‘shellcommands’, for no good reason.  Really, there were all kinds of scary implementation decisions.  My fork was meant to fix these problems. Unfortunately, I never got it to work; it was just too much work pushing that much spaghetti C code around, at least for me.

Around the time I gave up on rewriting Cfengine, my friend Kent Skaar decided to go work for BladeLogic, and he somehow convinced me to join and them to hire me.  However, I was, for good reason, concerned about retaining the rights to ideas I’d already developed, so I took some time before joining to write a prototype roughly implementing the product I was thinking of building.  At the time, the product was code-named ‘Blink’.

The LISA Configuration Management Community

I have to make another aside here and mention that through this whole period I was attending the configuration management workshops at the USENIX LISA conference.  On the one hand, these were great workshops, because they were full of smart people talking about interesting problems, but on the other hand, they were just horrible.  It seemed that no one was interested in making a truly useful, simple tool, and there was this ridiculous, inane focus on theory in lieu of practice.  I was told multiple times that it just wasn’t possible to write a good tool until the theory was solved, and I should just wait until it was.  Any scientist could tell you that, in nearly all cases, theory comes from data, and data comes from wandering around randomly, so this theory-then-tool idea was just BS.

Over the course of my years in the workshops, I’d come to the conclusion that building some kind of resource abstraction layer was the key to building a great tool.  I reasoned that this layer would function as a means of abstracting the operating system and allowing the sysadmin to focus on the what rather than the how, and I figured the abstraction layer could be the first step in a real sysadmin tool stack.

My ideas were met with outright derision.  I was told the theory was utter crap, unworkable *and* stupid, and uninteresting to boot.  As time went on, the criticism from certain parties got downright ridiculous, to the point where half of why I built Puppet in the end was to prove these jackasses wrong.

I don’t think it was coincidental that I was the only person in the room out selling my services as a consultant, rather than sitting in a stable job and just working on config-mgmt a limited amount of time.  This sales requirement put a very different spin on how I thought about things, and my understanding of the requirement for simplicity and reuse seemed unique in the group.  While Bcfg2′s author has bragged that you can get an install up and running in as little as a few days, I’ve always said if it takes you more than an hour to do something useful with Puppet, I’ve screwed up.

So anyway, I’m going to these conferences, sharing my ideas, and getting absolutely nowhere.  Everyone in the room has his (yes, his) own tool, and no one’s really moving anywhere.  I’m feeling pretty insulted most of the time and getting pretty tired of people just categorically saying I’m wrong while being unwilling to listen to me or, really, even try other tools out (at that point, I was the only person who had downloaded and tried out every other tool in the group).

The Prototype

So, for all these reasons and more, I decided to see if I could make a prototype that had the functionality I was looking for.  I knew I wanted:

• High-level resources:  Users, not useradd; packages, not apt-get; and hosts, not /etc/hosts.
• Simple cross-platform naming: The ability to use different names for the same resource on different platforms; e.g., the ssh package or the location of the sshd configuration file
• Easy extensibility: I wanted users to be able to add custom resource types.

Beyond that, there were some more specific bits I knew I wanted, but that was the main list.  When I started on this, I was a perl developer, so I tried it in that, but I just couldn’t conceive of how to implement the class relationships I was envisioning.  This was 2004, when everyone was hotting up Python, so I decided to give it a try, but Python just makes my eyes bleed.  No, it’s not the whitespace — that’s just silly, not painful.  It was things like ‘len’ being a function, rather than a method, and the fact that ‘print’ with a trailing comma behaves differently than without one.

I had a friend at the time who’d heard Ruby was cool (although never tried it himself), so I decided to give that a try, and in four hours I went from having never seen a line of Ruby to having a functional prototype of the resource abstraction layer.  Boom, I was off.

BladeLogic and Beyond

Now that I had the prototype and I was comfortable that I had code that outlined the ideas I was coming to the table with, I accepted the job at BladeLogic.  My wife was still working on her Phd in cancer biology, so we couldn’t move to Boston (where they are located); instead, I would get an apartment there and work there 2/3 or 3/4 of the time and work from Nashville the rest of the time.

My understanding of my role there was to help them design configuration management tools, but that didn’t seem to be why they actually hired me — they were far more interested in my helping to set up a test lab or some junk.  I was pretty impatient with the job, given that I was never seeing my wife, so I was pushing until it either became what I wanted or broke, and it only took 5 months for me to decide it wasn’t worth it and quit.

I went back home and took stock.  One of my main complaints about BladeLogic was that I didn’t think they were producing software that I would want to use or even recommend to others, yet I still thought there was a clear opportunity.  Cfengine had the widest deployment, but, well, it was a piece of crap.  No version control, horrible implementation, no marketing, no commercial backing, etc.

In addition to the opportunity, I thought I actually had some decent ideas, and I was pretty confident in my ability to make a living selling them. Yes, selling — I’m a very technical person, but I completely understand the value and necessity of selling my ideas, whether it be to colleagues, coworkers, employees, or customers.

So, in March of 2005, I decided to take my paltry savings and a small loan from family and start a software company focused on producing a better configuration management tool.  After realizing that there were already multiple software projects called ‘Blink’, and much hemming and hawing and thinking, I decided to call this project Puppet.

The next installation in this series will actually cover the history of Puppet itself, hopefully all the way to the present day.

Fundamentally, the main reasons I’m keeping it are that I couldn’t see a huge differentiator pushing me toward the iPhone, I didn’t really want to switch from TMobile to ATT (TMobile will unlock my phones for me and has said they don’t mind tethering apps), and I like the more open nature of the gPhone.  Assuming someone actually makes a tethering app at some point, it could save me and everyone at my company about $60/month, since we’d otherwise need to buy a separate device to get constant ‘net access for our laptops.

Here are some things I particularly like about the gPhone:

• The Status bar: I really like this.  I like one place for notifications and information, and a single consistent interface for all of them, along with my ability to either answer them or ignore them.
• The Dialer Application: I think this is great.  I don’t know that it’s a ton different from the iPhone’s, but I really liked this version.  I liked the previous calls list, the favorites list, and easy searching for contacts.
• The mid-call menu: I didn’t get a chance to try this out on the iPhone, but I loved this on the gPhone.  Press the menu button and you can enable/disable your bluetooth headset, switch calls, put someone on hold, and more.  Call switching and conference calling was astoundingly faster than doing so with any other phone I’ve had, which is pretty awesome considering how often I make conferences calls between myself, Andrew, and Teyo.
• Task switching: You can hold the ‘home’ button down for a second or so to get a list of the last six applications you’ve used, and you can switch right to the application without having to go to the home screen.  On the iPhone, you always have to go to the home screen to get to another application.  Welcome to 1987, people — this is MacOS 6, but with no excuse.  Android also has a Task Switcher application that just switches across all installed applications; it mostly sucks, but it’s encouraging that task switching is one of the first problems attacked in the OS.
• Notification lights: This seems like a small thing, but the phone has three small, colored LEDs; a green LED for message notification, a red light for low-battery, and an orange light for charging.  I really like these — they’re small enough that they don’t stab you like so many LEDs do these days, and they provide a great, simple way to know what’s going on with your phone, without having random vibrations every who-knows-how often.  I was using a friend’s iPhone, and it literally vibrated every three or four minutes without any indication why.  I’m assuming it was a notification I was supposed to track down, but there was no way to figure it out without going to the home screen and searching for red numers.
• Home screen vs. App Management: Android has a home screen with three panes, and you can arrange them however you want — apps, links to URLs, etc.  It also has an applications tab pull with every app on the system, arranged heirarchically.  This is great because you can set up a subset of your applications on the home screen, arranged how you want, and then you can peruse the whole app list alphabetically.  The iPhone, on the other hand, only has the home screen — you add more apps, and it adds more panes.  You apparently can’t change the four “most common” apps on the bottom, and there’s apparently no easy way to browse the apps alphabetically.

The combination of app management and task switching go a long ways toward making this phone feel more like a real operating system on a phone, rather than the iPhone which feels more like a phone trying to be a real operating system.  It’s weird, because I know the iPhone has a real OS under there somewhere; it’s just not acting like one.

I kinda like the keyboard, although I don’t know if it’s that big of a difference for me (I’m hopefully never going to be much of an emailer with my phone), and I like how responsive the phone is.

There are some things I really don’t like about it, though.  Browsing is a generally crappy experience — the zoom thingie completely blows, in that it inconsistently accepts touches and it’s just a crappy interface in general, and it’s also not-quite-impossible to actually click the links you want.  However, using apps like  Google Reader works fine, and I expect it will only get better.  And, again, this isn’t what I mainly expect to use the phone for.  It’s fine for stop-gap, like sitting in the Doctor’s office waiting room; it’s just not something I’d want to do a ten day trip with.

I’m pretty concerned about the durability of the hardware; I could already store my credit cards between the sliding screen and the keyboard.  Hopefully I’ll be able to replace the phone as necessary for the first year, and by then I hope they’ve got it figured out.

I do really wish the phone supported multi-touch, and with it the multi-finger gestures like the iPhone pinch.

I also wish there were just a Quicksilver-like interface for the whole phone — give me a single gesture to bring up Quicksilver, and I’ll never go to the home screen again.

An Aside

This is neither iPhone nor Android, but it must be said.  What is up with how unimaginative people are being with the touchscreen support?  Why is it that the only way to interact with either of these things is directly touching visible objects, or a traditional keyboard?  Where are my full-screen gestures that I can define, where are my accelerometer-based navigation, where, basically, is anything that really takes advantage of the form factor and touchscreen to be different?

The gPhone has pretty cool gesture recognition for unlocking — you get a grid of nine dots and have to make a specific gesture to unlock the phone.  Why can’t this same theme be used for, well, everything?  I’d like global gesture support for shortcuts to my most common applications, for instance — a Z to open one app, a loop to open another.  I haven’t seen a single situation where I need to drag in multiple directions without lifting my finger, so I don’t see what it would interfere with.

Both of these phones basically suck for context and task management.  Each phone has a “back” button, where the iPhone’s is on-screen and the gPhone’s is physical.  What?  A back button?  Back buttons suck in browsers, why would anyone ever think they make sense as the primary navigational paradigm in, well, anything, ever?  Sure, if you’re stuck in a completely linear world, sure, but on a phone where I regularly use 5-10 applications a day, and could conceivably use 20 or more, a back button just doesn’t suffice.

It bugs the hell out of me that people aren’t taking the Wii as their example — do something different, something weird.  If people aren’t laughing at your ideas, then you’re not trying hard enough.

Cheers.