RESTful Puppet and You
No, I’m actually not dead, as much as it might appear from my lack of updates. Fortunately there are other means to verify my vitality.
Anyway, the goal here is to help you understand how the imminent release of Puppet 0.25 affects you. And by “you” here I am, of course, making some assumptions – if you don’t use or care about Puppet, um, not so much. But if you use Puppet, or, even better, if you care about how well Puppet solves your problems, then this post is for you. I hope.
I’ve no short-term memory and don’t feel like planning this blog post out, so I’m going to just do what I can to ad-lib the reasons that Puppet’s new-found RESTfulness kicks ass. And stuff.
Smaller and Faster
First and foremost, 0.25 does not have many what you would call features. It’s mostly a refactoring release. This refactoring in general results in a faster system that takes less memory. Experiments have borne this out: People are seeing 10-40x speedups in fileserving, at least 30% less ram consumed on client and server, and much faster run times on the client.
So, look not for features but for benefits.
What is REST?
REST is more of a principle than a standard, but the things that matter in it for Puppet are:
- All data transferred is treated as a stand-alone ‘resource’ (and here ‘resource’ means something different than a normal Puppet resource). For instance, Puppet 0.25 treates catalogs, file content, file metadata, certificates, certificate requests, fact collections, and probably a bit more as resources.
- All resources have a unique URI. E.g., a given host’s catalog can be found at ‘http://puppet/$environment/catalog/$hostname’.
- Standard HTTP infrastructure is used as the ‘api’ – HTTP ‘put’, ‘get’, and ‘delete’ are the primary operations, and HTTP content-type handling is used for serialization negotiation.
Of course, none of these are really helpful to you, whomever you are. What’s helpful is how we’ve made these basic ideas useful in Puppet. So here are some of the things you get:
Faster and Smaller
The big problem with our current network protocol (XMLRPC, which is essentially XML over HTTPS) is that everything has to be considered XML, which means that anything that isn’t actually XML (which is, um, everything in Puppet) has to be encoded and escaped so it can act like XML. For file serving, this means that every file has a minimum of three copies in memory – plaintext, encoded, and escaped. This is time- and memory-expensive.
Oops I Did It Again
Sorry, couldn’t resist. One of the benefits of switching to REST is that we’re breaking compatibility, so we can fix some design problems we (i.e., I) made early-on. The biggest benefits have been around fileserving and the catalog.
In pre-0.25 releases, recursive file-serving involves a minimum of one call per directory and one call per file. This is obviously slow and bad. 0.25 fixes this so it’s one call for an entire directory structure, with possibly one call for each file that actually needs to be transferred. The combination of this plus the lack of encoding and escaping has resulted in most people seeing a 10x speedup in recursive fileserving.
For catalogs, well, pre-0.25 releases don’t really talk about them. They transfer this serialized recursive tree structure, rather than the whole catalog (which is really mostly a graph). This matters a bit in 0.25, because it’s allowed us to remove some extraneous stupidity that resulted from the limitations of the tree structure (relationships no longer propagate to all contained resources – see the ticket for more information), but it’s going to be a bigger deal going forward as we can start relying on the complete graph being passed around.
Both of these refactors could have been done with XLMRPC, but they would have required breaking compatibility, which is always a difficult step to take, so doing it all at once makes the most sense.
Easier Integration
The big deal in our use of REST is really an internal system that enables REST – this is the all-powerful Indirector. It’s essentially a plugin system that makes it easy to add new sources or destinations for data we care about. For example, when we wanted to queue the storage of catalogs to a database (rather than forcing the client to wait until it’s done), we had to develop the queueing infrastructure, of course, but integrating it with Puppet was less than 100 lines of code that knew how to accept catalogs and send them to the queue.
This ease of integration opens up all kinds of possibilities, like moving our CA to be database-backed, but the main thing is that if you come up with some cool integration, it’s similarly easy for you — you’re not dependent on us.
We’re still missing one piece – a user-configurable routing system to help you configure which plugin is used – but we’ve got that mostly done and it’ll be in the release after 0.25.
Note that this integration was often possible in earlier releases, but it was much harder because you had to know a lot more about every individual class you were integrating with. Because we use the same methods and the same interface and the same location pattern for everything, you don’t have to know as much.
Enforced Good Citizenship
Without the Indirector at the heart of everything, and an assumption that other people are going to be integrating with our code, we are forced to build more decoupled, portable components. This isn’t much, but it does always enforce a kind of good citizenship, which is something.
Posted: June 4th, 2009 under Development, Puppet.
Tags: api, design, internals, Puppet, refactoring, rest
Viewing 15 Comments
Thanks. Your comment is awaiting approval by a moderator.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Do you already have an account? Log in and claim this comment.
Trackbacks
(Trackback URL)
June 5, 2009 at 7:05 am
[...] The Madstop » RESTful Puppet and You - I’ve no short-term memory and don’t feel like planning this blog ...