Links:

Categories

Recent Posts

Bookmarks

Recent Status

Upcoming Events

I'm speaking at Open Source World in San Francisco, August 12-13 I'm speaking at The Ruby Hoedown in Nashville, TN, August 28-29

Site search

Categories

November 2008
M T W T F S S
« Oct   Dec »
 12
3456789
10111213141516
17181920212223
24252627282930

Tags

Blogroll

Navbar

Related

A Short Puppet History, pt. 2: Cfengine

Now that I’ve gone through the ISconf years (although it was really less than a year), it’s time to move on to how Cfengine affected Puppet.  There seems to be a pervasive belief that Puppet is an attempt to make a Cfengine++, and I’m hoping that this article will dispel that silly notion.

As mentioned previously, I’d concluded that ISconf wasn’t sufficient, because it was theoretically bogus (maintaining order didn’t guarantee you all that much), it precluded refactoring, and it didn’t actually make my sysadmin work easier, really — I was still writing tons of little shell scripts.  As I described it in my paper: “In choosing between ISconf and most other [configuration management systems], this is largely a choice between a tool which easily organizes work and a tool which makes the work itself easier.”

Order Matters

Given the situation at the time, and the fact that Cfengine’s author, Mark Burgess, was having a kind of war of words with the author of ISconf, Steve Traugott,  it made sense to look at Cfengine.  I want to step aside for a moment, though, and discuss this war of words.  The summary of the war is “order matters”, and when I hear that it’s like “where’s the beef” or “it has what plants crave” — it’s a simple phrase but one that brings up context, hilarity, and stupidity all at once.  Steve maintained that order matters when managing systems, and Mark theoretically maintained that it did not.  These respective beliefs were evident in that Steve’s tool, ISconf, was really just a means of guaranteeing order and nothing more, and Mark’s tool, Cfengine, made it ridiculously difficult to maintain order.

The truth, of course, is that order does matter, but only as a prerequisite, not the end-all be all.  You obviously can’t start a service if you haven’t installed its package yet, but there is such a thing as orthogonal services, and you definitely, definitely don’t want to build all of your systems with the assumption that every chunk of work requires every other chunk of work ever performed.

My hopefully-reasonable summary should not imply that their conversations were reasonable; they were not. It was an infantile and annoying conversation all around, and to me, the phrase “order matters” will always evoke laughter, pity, and a desire to drink heavily while spouting epithets. When I was first exposed to this conversation, I was an ISconf user (well, *the* ISconf user — one other person adopted it, and I suppose Steve used it, but…), so everyone assumed I was on Steve’s side, but I’m not much of an ideologue, and I certainly wasn’t committed that early.

Integrating Cfengine

Anyway. So, I decided to look at Cfengine.  It’s a completely different kind of tool than ISconf — it’s organized around what it calls “actions”, although none of them are verbs, for some reason.  These are things like files, shellcommands, processes, and many others.  The immediate, obvious difference between the two is that Cfengine is a tool focused on what you’re trying to do, and ISconf is a tool that just cares about in what order you do it.

The other thing about Cfengine is it has this whole “control” section that’s all about setting up variables and making decisions, and you can set booleans (which Cfengine calls classes) that can then be used to make decisions throughout the whole configuration.  I immediately imported a lot of the convoluted logic I had hidden in ISconf make files into Cfengine and saved myself a ton of headache; then I integrated the two tools, such that ISconf could take advantage of all of the knowledge and context that Cfengine had, and Cfengine could take advantage of all of the information I’d already input into ISconf.

This, of course, meant I was going to hell. I wrote an article describing my experiences with each tool, plus how I integrated them, and there were some interesting conversations as a result.  Initially, the integration was focused on not throwing away the work I’d done with ISconf, but over time, I stopped adding code to ISconf and largely just focused on Cfengine, and in the end, I found it was more effective to just do my work directly through Cfengine.

At the same time, I started a consulting company to focus on helping other companies with with configuration management problems, since so few organizations seemed to have a decent solution, and I was more interested in continuing to refine my solution than I was in moving on to more mundane sysadmin problems like backups or monitoring.

After a few months of consulting, though, I found that Cfengine could be painfully low-level.  It could do files, commands, and processes, but it couldn’t do users, groups, or packages.  Also, I was maintaining Cfengine configurations for multiple platforms, and it was a real nightmare managing Cfengine code that did the same thing in different ways, or even just used different paths.

Even worse, though, I found I couldn’t keep promises to my customers.  Cfengine often wouldn’t compile on a given platform across releases, I couldn’t consistently get bug fixes or refactorings accepted into the code base, and I had no ability to influence the direction of its development.  Mark Burgess was the main developer, and he was very defensive of any attempt to refactor significant parts of the code.  He also, somewhat notoriously, said he considered version control to be overhead, which meant we kept magically getting bug regressions.

Looking for other options

So, frustrated with Cfengine and its lack of a developer community, I started looking around for other options. I tried out Quattor, LCFG, SmartFrog, Bcfg2, psgconf, and probably more, but none of them seemed like they would allow the kind of extension and abstraction that I was looking for.  I was afraid I was going to have to make my own tool, but I really didn’t want to.  As an interim step, in the summer of 2004 I decided to fork Cfengine and fix some of its problems.  I called this fork cfng (it’s completely dead now).

One of the things that drove me most crazy about Cfengine was that it had 25 different syntaxes for its 33 or so different actions — ‘processes’ was *slightly* different from ’shellcommands’, for no good reason.  Really, there were all kinds of scary implementation decisions.  My fork was meant to fix these problems. Unfortunately, I never got it to work; it was just too much work pushing that much spaghetti C code around, at least for me.

Around the time I gave up on rewriting Cfengine, my friend Kent Skaar decided to go work for BladeLogic, and he somehow convinced me to join and them to hire me.  However, I was, for good reason, concerned about retaining the rights to ideas I’d already developed, so I took some time before joining to write a prototype roughly implementing the product I was thinking of building.  At the time, the product was code-named ‘Blink’.

The LISA Configuration Management Community

I have to make another aside here and mention that through this whole period I was attending the configuration management workshops at the USENIX LISA conference.  On the one hand, these were great workshops, because they were full of smart people talking about interesting problems, but on the other hand, they were just horrible.  It seemed that no one was interested in making a truly useful, simple tool, and there was this ridiculous, inane focus on theory in lieu of practice.  I was told multiple times that it just wasn’t possible to write a good tool until the theory was solved, and I should just wait until it was.  Any scientist could tell you that, in nearly all cases, theory comes from data, and data comes from wandering around randomly, so this theory-then-tool idea was just BS.

Over the course of my years in the workshops, I’d come to the conclusion that building some kind of resource abstraction layer was the key to building a great tool.  I reasoned that this layer would function as a means of abstracting the operating system and allowing the sysadmin to focus on the what rather than the how, and I figured the abstraction layer could be the first step in a real sysadmin tool stack.

My ideas were met with outright derision.  I was told the theory was utter crap, unworkable *and* stupid, and uninteresting to boot.  As time went on, the criticism from certain parties got downright ridiculous, to the point where half of why I built Puppet in the end was to prove these jackasses wrong.

I don’t think it was coincidental that I was the only person in the room out selling my services as a consultant, rather than sitting in a stable job and just working on config-mgmt a limited amount of time.  This sales requirement put a very different spin on how I thought about things, and my understanding of the requirement for simplicity and reuse seemed unique in the group.  While Bcfg2’s author has bragged that you can get an install up and running in as little as a few days, I’ve always said if it takes you more than an hour to do something useful with Puppet, I’ve screwed up.

So anyway, I’m going to these conferences, sharing my ideas, and getting absolutely nowhere.  Everyone in the room has his (yes, his) own tool, and no one’s really moving anywhere.  I’m feeling pretty insulted most of the time and getting pretty tired of people just categorically saying I’m wrong while being unwilling to listen to me or, really, even try other tools out (at that point, I was the only person who had downloaded and tried out every other tool in the group).

The Prototype

So, for all these reasons and more, I decided to see if I could make a prototype that had the functionality I was looking for.  I knew I wanted:

  • High-level resources:  Users, not useradd; packages, not apt-get; and hosts, not /etc/hosts.
  • Simple cross-platform naming: The ability to use different names for the same resource on different platforms; e.g., the ssh package or the location of the sshd configuration file
  • Easy extensibility: I wanted users to be able to add custom resource types.

Beyond that, there were some more specific bits I knew I wanted, but that was the main list.  When I started on this, I was a perl developer, so I tried it in that, but I just couldn’t conceive of how to implement the class relationships I was envisioning.  This was 2004, when everyone was hotting up Python, so I decided to give it a try, but Python just makes my eyes bleed.  No, it’s not the whitespace — that’s just silly, not painful.  It was things like ‘len’ being a function, rather than a method, and the fact that ‘print’ with a trailing comma behaves differently than without one.

I had a friend at the time who’d heard Ruby was cool (although never tried it himself), so I decided to give that a try, and in four hours I went from having never seen a line of Ruby to having a functional prototype of the resource abstraction layer.  Boom, I was off.

BladeLogic and Beyond

Now that I had the prototype and I was comfortable that I had code that outlined the ideas I was coming to the table with, I accepted the job at BladeLogic.  My wife was still working on her Phd in cancer biology, so we couldn’t move to Boston (where they are located); instead, I would get an apartment there and work there 2/3 or 3/4 of the time and work from Nashville the rest of the time.

My understanding of my role there was to help them design configuration management tools, but that didn’t seem to be why they actually hired me — they were far more interested in my helping to set up a test lab or some junk.  I was pretty impatient with the job, given that I was never seeing my wife, so I was pushing until it either became what I wanted or broke, and it only took 5 months for me to decide it wasn’t worth it and quit.

I went back home and took stock.  One of my main complaints about BladeLogic was that I didn’t think they were producing software that I would want to use or even recommend to others, yet I still thought there was a clear opportunity.  Cfengine had the widest deployment, but, well, it was a piece of crap.  No version control, horrible implementation, no marketing, no commercial backing, etc.

In addition to the opportunity, I thought I actually had some decent ideas, and I was pretty confident in my ability to make a living selling them. Yes, selling — I’m a very technical person, but I completely understand the value and necessity of selling my ideas, whether it be to colleagues, coworkers, employees, or customers.

So, in March of 2005, I decided to take my paltry savings and a small loan from family and start a software company focused on producing a better configuration management tool.  After realizing that there were already multiple software projects called ‘Blink’, and much hemming and hawing and thinking, I decided to call this project Puppet.

The next installation in this series will actually cover the history of Puppet itself, hopefully all the way to the present day.

Posted: November 6th, 2008 under Uncategorized.

« Previous post Next post »

Viewing 79 Comments

 

Trackbacks

(Trackback URL)

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus