Unfortunately, we could not travel to Japan to receive the award in person, but I was able to give a talk via Skype the night of the award ceremony.  It was around 2am my time on a day I’d traveled with my wife and twins, so it was a long day, but it was worth it.  I only hope my talk was worth anything.

Andrew did all of the hard work around getting the submission in and organizing the talk itself, so I definitely have him to thank for that.

Now I just need to figure out how to get my bank to accept those postal money orders from Japan.

Development Workflow

We led the discussion with a conversation about how the development workflow will change now that we’re finally releasing the code in master as 0.25.  After much discussion, we largely concluded that the least-surprise solution was to use the ‘master’ branch for stable development, and have something like a ‘next’ branch for new features and major refactorings.

The main goal is for new developers to be able to clone our repository and get to developing code immediately without having to worry about switching branches or any such thing.  More experienced developers will know where their development should be done, so it’s mostly a question of least-surprise for newcomers.

There was also discussion of fly-by-night developers, people who show up, produce a patch or two, and wander off.  James Turnbull (who handles 99% of the merging and release management) said he was fine accepting patches over email, rather than forcing people to publish everything via git.

Other than those two changes, our development workflow has worked pretty well.  However, the fly-by-night patches led into discussion of…

Core vs. Modules

Puppet’s core is starting to get many non-core features, like Nagios and Zenöss integration.  It’s essentially impossible for the core team to maintain all of these extentions, but many of them were written as a one-off solution and won’t be maintained by their authors.  Because of this, we  as a project need to develop a means of splitting Puppet’s core away from all of these modules.

Discussion was had of using a Nagios-style ‘plugins’ repository, but I don’t like that idea because it leaves basically the same problem – a centralized repository that no one wants to maintain.

Instead, we all basically agreed that we want to focus on modules as the means of adding functionality to Puppet, but the big problem there is that the only way to do so is to add package-like behaviours to modules, and none of us wants to make our own package manager.  Nonetheless, it was agreed that this was the only real option, so it just remains to do it.

Roadmap and Release Status

Against all odds, it looks like we’ll get 0.25 out in February – I’ll be merging in the last major refactor this week, and then it’s just a question of getting as many bug-fixes in as we can and dropping the release.

In addition, there are (annoyingly) a few important bugs in 0.24.7, so we’re unfortunately going to have to be put out a 0.24.8 release.  One thing that’s become clear in the last couple of releases is that we need to get more community testing of release candidates – many of the bugs would have been caught by basic testing in the community.

Facter Design and Shadow{Facter,Puppet}

RailsMachine released ShadowPuppet and ShadowFacter in the last few weeks, and it’s led to a flurry of design conversations.  We didn’t cover too much of it in the dev call, other than my reiteration that I’m excited by them and would like to merge them into Puppet and Facter.

Conclusion

And that was it.  Please join us on the next call if you can.

As the post title suggests, though, I think ‘foil ball’ is a more appropriate term.

You have significant problems when you rely on golden images:  Image sprawl, updating your images, and image state vs. running state.

Image sprawl is what you get when the number of images (not running virtual machines) you have grows to an essentially unmaintainable figure.  Let’s start with a simple LAMP stack:  At the least, you’ll have a separate image for your web, database, and application servers.  Oh, except you probably need a load balancer image.  If you have any support services like DNS, you need an image for those.  And so on.  You soon find that you have a separate image for every service you provide.

Now that you’ve got this image sprawl, you run into the next issue:  Updating these images is relatively expensive, and nearly always results in redundancy.  It’s expensive because even trivial changes require a full image rebuild, which is itself a bit complicated.  The redundancy comes because you *still* have to do some work on the image once it’s booted as a server, even if it’s minimal.  So now you’ve got this complicated image generation process that has some kind of overlap with a simple on-server management process.  Another kind of redundancy arrives when you make a change that affects multiple images (e.g., upgrading the same package, or performing the same configuration change): you have to make this change to each of these images separately.

Oh, and by the way – this updating process is usually completely unrelated to the process you use to update your non-image machines.  Because hey, if a little bit of redundancy is good, then redundant redundancy is especially awesome.

Say you managed all of that, though, and all of your images are correctly updated all of the time.  Great, now you just have to reboot every machine on your network to take advantage of the new changes.  Of course, this isn’t exactly feasible for every machine all the time, which means you’ve got drift between the desired and actual configuration state.

This is why I think maintaining these images is more like managing a foil ball:  It’s difficult to pull apart, difficult to press back together, and if you get too many of them they just get into the way.

If, instead, you use a single, base image for all of your work — I call these images stem cell images for what are hopefully obvious reasons – and then use a tool like Puppet to configure them once they’re running, you avoid all of the above problems:  You have one image to maintain and it’s necessarily simplistic, you use the same tool and the same configuration base across all images, and Puppet keeps your machines updated within 30 minutes of any central change.

So, if someone tries to sell you a golden image, don’t buy it – instead choose a tool you can use for every machine in your organization, and push every configuration operation possible into that tool, rather than spreading tasks around to your provisioning, image management, and configuration management tools.  This is just as true for tools like Jumpstart and Kickstart – they should do as little as possible, and hand off immediately to a tool like Puppet; well, really, just Puppet.