We’re going to take a tour of the newly released Puppet Dashboard web front-end.  Puppet Dashboard is (or will be) a web front end that keeps you informed and in control of everything going on in your Puppet ecosystem. It currently functions as a reporting dashboard and an external node repository and will soon do much more, including having better marketing copy.

Fundamentally, Dashboard lets you do two things: configure nodes using parameters, classes and groups for use as an external nodes tool; and monitor the status of nodes through real-time reporting and versioned change tracking. The main dashboard page shows you the status of recent Puppet runs, displays important information like pass/failure statistics, and alerts you of important failures, errors and unexpected events. Let’s take a look at how it does this.

Go check it out.

In the meantime, here’s my quarterly blog post.

I was finally able to attend an Ubuntu Developer Summit last month, this time in Dallas (not quite as nice a location as the last one – Barcelona).  My reason for being there was that they had 3 or 4 sessions devoted to Puppet-related concepts, and they wanted upstream (that would be me) to provide some insight and get involved in getting the work done.

It was a really interesting and different conference.  I’ve actually participated in other UDSes in the past via phone and IRC, and now that I’ve been to the conference, I realize how structural that remote participation is, and I’d love to copy it as we hold more Puppet conferences (e.g., the PuppetCamp Europe we’re planning for next year).  Every room had a session-specific IRC channel, plus live streaming video and audio, so you really could participate from anywhere in the world.  The last UDS, I could hear an audio stream and would type on IRC my responses to questions.  It was a bit awkward, but it meant I could actually be involved.

I think the focus on coming away with action plans is another thing that sets UDS apart.  So many other conferences are the destination – get your talk written, your presentation presented, and then drink and talk all week.  While there’s lots of drinking and talking at UDS – there was a really lively bar scene, I think partially because there’s so little hallway track — the sessions are really focused on “we’re all in the same city for a week, let’s knock this out”.

That’d be another great takeaway, but it’s hard to know how to copy it without having thousands of motivated, active, involved community members, like Ubuntu does.  I might decide to try for something like its blueprints in a one-day pre-PuppetCamp session or something; I like the idea of a “get shit done” focus for at least part of any conference.

In terms of how it affects Puppet, it’s a couple of different ways.  Since Puppet is now in Ubuntu Main, they are concerned about the long-term support of whatever we release, because they have to support it for five years.  They’re also looking at integrating it into various aspects of the system, and in particular, etckeeper and the installer.  So far, Ubuntu is the only OS out there that I’ve seen really look for ways to integrate a CM system into the heart of the OS.  As they should be, they’re taking baby steps, but those steps are clearly harbingers for what’s to come.

The etckeeper integration is not something you’d use in a normal environment, but I could see how it could be useful for some cases.  It’d kind of be a filebucket on steriods.

The goal of the installer integration would be, at the least, to make sure that you could have a complete functional machine once out of the installer, so Puppet would need to be able to run chroot’d by the installer.  We also talked about converting the catalog into a format that the installer can understand — essentially a preseed file — which would allow the installer to take care of the majority of the package installs, which would be much faster on the first big run.

Hopefully both of these integrations will yield great results, and either way, it was a great conference and I hope that we and Canonical/Ubuntu continue finding interesting things to work on together.

We’ve had talks by Ohad Levy on The Foreman, Brice Figureau on StoreConfigs, Paul Nasrat on Facter, and James Turnbull on developing Puppet.  After the talks yesterday, we split up into self-organized sessions, which are always my favorite — everyone’s far more involved, and you know just about everyone is getting something from every session.

The conference has been successful enough that I’m confident we’ll have more of them.  Next time, though, we’ll skip the catering and spend the extra money on getting a location closer to the city center.

Anyway, the goal here is to help you understand how the imminent release of Puppet 0.25 affects you.  And by “you” here I am, of course, making some assumptions – if you don’t use or care about Puppet, um, not so much.  But if you use Puppet, or, even better, if you care about how well Puppet solves your problems, then this post is for you.  I hope.

I’ve no short-term memory and don’t feel like planning this blog post out, so I’m going to just do what I can to ad-lib the reasons that Puppet’s new-found RESTfulness kicks ass.  And stuff.

Smaller and Faster

First and foremost, 0.25 does not have many what you would call features.  It’s mostly a refactoring release.  This refactoring in general results in a faster system that takes less memory.  Experiments have borne this out:  People are seeing 10-40x speedups in fileserving, at least 30% less ram consumed on client and server, and much faster run times on the client.

So, look not for features but for benefits.

What is REST?

REST is more of a principle than a standard, but the things that matter in it for Puppet are:

• All data transferred is treated as a stand-alone ‘resource’ (and here ‘resource’ means something different than a normal Puppet resource).  For instance, Puppet 0.25 treates catalogs, file content, file metadata, certificates, certificate requests, fact collections, and probably a bit more as resources.
• All resources have a unique URI.  E.g., a given host’s catalog can be found at ‘http://puppet/$environment/catalog/$hostname’.
• Standard HTTP infrastructure is used as the ‘api’ – HTTP ‘put’, ‘get’, and ‘delete’ are the primary operations, and HTTP content-type handling is used for serialization negotiation.

Of course, none of these are really helpful to you, whomever you are.  What’s helpful is how we’ve made these basic ideas useful in Puppet.  So here are some of the things you get:

Faster and Smaller

The big problem with our current network protocol (XMLRPC, which is essentially XML over HTTPS) is that everything has to be considered XML, which means that anything that isn’t actually XML (which is, um, everything in Puppet) has to be encoded and escaped so it can act like XML.  For file serving, this means that every file has a minimum of three copies in memory – plaintext, encoded, and escaped.  This is time- and memory-expensive.

Oops I Did It Again

Sorry, couldn’t resist.  One of the benefits of switching to REST is that we’re breaking compatibility, so we can fix some design problems we (i.e., I) made early-on.  The biggest benefits have been around fileserving and the catalog.

In pre-0.25 releases, recursive file-serving involves a minimum of one call per directory and one call per file.  This is obviously slow and bad.  0.25 fixes this so it’s one call for an entire directory structure, with possibly one call for each file that actually needs to be transferred.  The combination of this plus the lack of encoding and escaping has resulted in most people seeing a 10x speedup in recursive fileserving.

For catalogs, well, pre-0.25 releases don’t really talk about them.  They transfer this serialized recursive tree structure, rather than the whole catalog (which is really mostly a graph).  This matters a bit in 0.25, because it’s allowed us to remove some extraneous stupidity that resulted from the limitations of the tree structure (relationships no longer propagate to all contained resources – see the ticket for more information), but it’s going to be a bigger deal going forward as we can start relying on the complete graph being passed around.

Both of these refactors could have been done with XLMRPC, but they would have required breaking compatibility, which is always a difficult step to take, so doing it all at once makes the most sense.

Easier Integration

The big deal in our use of REST is really an internal system that enables REST – this is the all-powerful Indirector.  It’s essentially a plugin system that makes it easy to add new sources or destinations for data we care about.  For example, when we wanted to queue the storage of catalogs to a database (rather than forcing the client to wait until it’s done), we had to develop the queueing infrastructure, of course, but integrating it with Puppet was less than 100 lines of code that knew how to accept catalogs and send them to the queue.

This ease of integration opens up all kinds of possibilities, like moving our CA to be database-backed, but the main thing is that if you come up with some cool integration, it’s similarly easy for you — you’re not dependent on us.

We’re still missing one piece – a user-configurable routing system to help you configure which plugin is used – but we’ve got that mostly done and it’ll be in the release after 0.25.

Note that this integration was often possible in earlier releases, but it was much harder because you had to know a lot more about every individual class you were integrating with.  Because we use the same methods and the same interface and the same location pattern for everything, you don’t have to know as much.

Enforced Good Citizenship

Without the Indirector at the heart of everything, and an assumption that other people are going to be integrating with our code, we are forced to build more decoupled, portable components.  This isn’t much, but it does always enforce a kind of good citizenship, which is something.

One of the things that separates Moonshine from other solutions like Chef and Sprinkle is that out of the box, Moonshine comes with recipes for the same Ubuntu/Ruby Enterprise Edition/Apache/Passenger/MySQL stack that’s in production use at Rails Machine.

We’re pretty excited about this, for multiple reasons.  It’s another Ruby company developing in and around Puppet, and it’s a great, simple way for Rails developers to take advantage of both Puppet and the RailsMachine stack.

The next step is to get the ShadowPuppet pure Ruby interface imported into Puppet.  It will complement the existing language rather than replacing it, but we haven’t really settled all of the details yet.

Unfortunately, we could not travel to Japan to receive the award in person, but I was able to give a talk via Skype the night of the award ceremony.  It was around 2am my time on a day I’d traveled with my wife and twins, so it was a long day, but it was worth it.  I only hope my talk was worth anything.

Andrew did all of the hard work around getting the submission in and organizing the talk itself, so I definitely have him to thank for that.

Now I just need to figure out how to get my bank to accept those postal money orders from Japan.

Development Workflow

We led the discussion with a conversation about how the development workflow will change now that we’re finally releasing the code in master as 0.25.  After much discussion, we largely concluded that the least-surprise solution was to use the ‘master’ branch for stable development, and have something like a ‘next’ branch for new features and major refactorings.

The main goal is for new developers to be able to clone our repository and get to developing code immediately without having to worry about switching branches or any such thing.  More experienced developers will know where their development should be done, so it’s mostly a question of least-surprise for newcomers.

There was also discussion of fly-by-night developers, people who show up, produce a patch or two, and wander off.  James Turnbull (who handles 99% of the merging and release management) said he was fine accepting patches over email, rather than forcing people to publish everything via git.

Other than those two changes, our development workflow has worked pretty well.  However, the fly-by-night patches led into discussion of…

Core vs. Modules

Puppet’s core is starting to get many non-core features, like Nagios and Zenöss integration.  It’s essentially impossible for the core team to maintain all of these extentions, but many of them were written as a one-off solution and won’t be maintained by their authors.  Because of this, we  as a project need to develop a means of splitting Puppet’s core away from all of these modules.

Discussion was had of using a Nagios-style ‘plugins’ repository, but I don’t like that idea because it leaves basically the same problem – a centralized repository that no one wants to maintain.

Instead, we all basically agreed that we want to focus on modules as the means of adding functionality to Puppet, but the big problem there is that the only way to do so is to add package-like behaviours to modules, and none of us wants to make our own package manager.  Nonetheless, it was agreed that this was the only real option, so it just remains to do it.

Roadmap and Release Status

Against all odds, it looks like we’ll get 0.25 out in February – I’ll be merging in the last major refactor this week, and then it’s just a question of getting as many bug-fixes in as we can and dropping the release.

In addition, there are (annoyingly) a few important bugs in 0.24.7, so we’re unfortunately going to have to be put out a 0.24.8 release.  One thing that’s become clear in the last couple of releases is that we need to get more community testing of release candidates – many of the bugs would have been caught by basic testing in the community.

Facter Design and Shadow{Facter,Puppet}

RailsMachine released ShadowPuppet and ShadowFacter in the last few weeks, and it’s led to a flurry of design conversations.  We didn’t cover too much of it in the dev call, other than my reiteration that I’m excited by them and would like to merge them into Puppet and Facter.

Conclusion

And that was it.  Please join us on the next call if you can.

As the post title suggests, though, I think ‘foil ball’ is a more appropriate term.

You have significant problems when you rely on golden images:  Image sprawl, updating your images, and image state vs. running state.

Image sprawl is what you get when the number of images (not running virtual machines) you have grows to an essentially unmaintainable figure.  Let’s start with a simple LAMP stack:  At the least, you’ll have a separate image for your web, database, and application servers.  Oh, except you probably need a load balancer image.  If you have any support services like DNS, you need an image for those.  And so on.  You soon find that you have a separate image for every service you provide.

Now that you’ve got this image sprawl, you run into the next issue:  Updating these images is relatively expensive, and nearly always results in redundancy.  It’s expensive because even trivial changes require a full image rebuild, which is itself a bit complicated.  The redundancy comes because you *still* have to do some work on the image once it’s booted as a server, even if it’s minimal.  So now you’ve got this complicated image generation process that has some kind of overlap with a simple on-server management process.  Another kind of redundancy arrives when you make a change that affects multiple images (e.g., upgrading the same package, or performing the same configuration change): you have to make this change to each of these images separately.

Oh, and by the way – this updating process is usually completely unrelated to the process you use to update your non-image machines.  Because hey, if a little bit of redundancy is good, then redundant redundancy is especially awesome.

Say you managed all of that, though, and all of your images are correctly updated all of the time.  Great, now you just have to reboot every machine on your network to take advantage of the new changes.  Of course, this isn’t exactly feasible for every machine all the time, which means you’ve got drift between the desired and actual configuration state.

This is why I think maintaining these images is more like managing a foil ball:  It’s difficult to pull apart, difficult to press back together, and if you get too many of them they just get into the way.

If, instead, you use a single, base image for all of your work — I call these images stem cell images for what are hopefully obvious reasons – and then use a tool like Puppet to configure them once they’re running, you avoid all of the above problems:  You have one image to maintain and it’s necessarily simplistic, you use the same tool and the same configuration base across all images, and Puppet keeps your machines updated within 30 minutes of any central change.

So, if someone tries to sell you a golden image, don’t buy it – instead choose a tool you can use for every machine in your organization, and push every configuration operation possible into that tool, rather than spreading tasks around to your provisioning, image management, and configuration management tools.  This is just as true for tools like Jumpstart and Kickstart – they should do as little as possible, and hand off immediately to a tool like Puppet; well, really, just Puppet.