One of the things that separates Moonshine from other solutions like Chef and Sprinkle is that out of the box, Moonshine comes with recipes for the same Ubuntu/Ruby Enterprise Edition/Apache/Passenger/MySQL stack that’s in production use at Rails Machine.

We’re pretty excited about this, for multiple reasons.  It’s another Ruby company developing in and around Puppet, and it’s a great, simple way for Rails developers to take advantage of both Puppet and the RailsMachine stack.

The next step is to get the ShadowPuppet pure Ruby interface imported into Puppet.  It will complement the existing language rather than replacing it, but we haven’t really settled all of the details yet.

Unfortunately, we could not travel to Japan to receive the award in person, but I was able to give a talk via Skype the night of the award ceremony.  It was around 2am my time on a day I’d traveled with my wife and twins, so it was a long day, but it was worth it.  I only hope my talk was worth anything.

Andrew did all of the hard work around getting the submission in and organizing the talk itself, so I definitely have him to thank for that.

Now I just need to figure out how to get my bank to accept those postal money orders from Japan.

Anyway, so I’m once again wrestling with data lifetime in Puppet.  This is one of those problems that always seems licked but then crops up again.

See, there’s plenty of data in a Puppet transaction whose lifetime should only be that transaction:  File stats, user name to gid mappings, and all kinds of information about the current state of the machine.  We need to collect this information, and we don’t want to collect it more than once a transaction (e.g., we need a file’s uid, gid, and mode, but we can get them all from a single Stat instance); but we also don’t want the data lying around for the next transaction.

So Puppet has support for a ‘flush’ method throughout most of the RAL:  By default, the transaction calls ‘flush’ on each resource, and the resource calls ‘flush’ on its provider.  This makes it easy to get rid of data that should be cleared up after the transaction.

Kind of.  See, the *real* reason for the ‘flush’ method is actually to flush changes to disk; e.g., the provider might have multiple attributes changed, and then you call ‘flush’ on it to make all of those changes at once.  It’s just that it’s also a convenient place to clean up data because, well, it’s the only place to do so.  So some time in the last few months or years or decades, my brain decided it does both things, but it seemed to hide this conclusion from me until yesterday.

But yesterday I was trying to fix all of the broken tests resulting from my file serving refactoring, and I was finding that, not surprisingly, I kept having these cached stat instances lying around — but *only* if the file hadn’t changed.  E.g., consider this code:

assert_events([], resource)
File.unlink(file)
assert_events([:file_created], resource)

Ignore, please, whether this is a good idea or what; the point is that the first line runs a transaction that results in a cached stat but no changes; and because there are no changes, there’s nothing to flush to disk; and because there’s nothing to flush to disk, ‘flush’ isn’t called; and because ‘flush’ isn’t called, the ‘stat’ is lying around still.  Which means the next transaction uses the cached stat, but of course, reality has changed in the meantime.

So, I need something that will make it easy for my data to match the lifetimes I want.  I made this Cacher Module that purportedly solves this problem for me, but noooo, it solves a different data-lifetime problem:  I have a lot of initialization code that generally only runs once but ends up running many times during testing, so I needed a clean way to remove the initialized data after every test.  So, this module has a single, global boolean that defines whether a given chunk of data is expired or still valid.  That’s all fine and dandy for one-time configuration data, but it doesn’t work for transactions.

So, now I’m trying to enhance that module to support either the global expiration marker or a per-instance marker, so that I can have all of the resources use a timestamp in their catalog to determine whether their cached data is expired.

One of the hard problems here is that you don’t want to find yourself maintaining a global list of anything anywhere.  My first design for the Cacher module involved it keeping a reference to all the cached data, which would have made it easy to clear the cache but would have had lots of code accessing tons of global data; stupid.  Instead, everyone keeps references to their own data, and the only global data is the expiration marker.  This works great for global things.

But resources and catalogs aren’t global.  Even worse, I’m (reasonably) using an internal class to check expiration.  So, that internal class needs to have access to the catalog’s timestamp in order to figure out if its cached data is still valid.  This is automatically ugly — an instance you shouldn’t even need to know exists now needs access to a big collection of resources.  Yuck.  Without this, the internal class doesn’t have a reference to anything except the data it’s managing, but there’s just no escaping it needing access to that timestamp somehow, so you either give it a reference, or you pass it one every time you check for a value.

The *stupid* thing is that this isn’t the problem I want to solve.  I just want the old fileserving behaviour but now with gooey RESTfulness.  That, plus the fact that I’ve apparently decided that I can no longer use spackle and sheetrock mud during development means that I have to fix this even though it wouldn’t normally be on my critical path.  Yuck.

ISconf

Puppet was heavily influenced by two tools, but almost more importantly, it was heavily influenced by what I couldn’t do with those tools.  I attended LISA in 2001, where I saw Steve Traugott give his talk on automating infrastructures.  I was impressed enough that I convinced my then-company to bring him in for a week of consulting.  During that week, he convinced me to give a tool he’d written, ISconf, a try.

ISconf is an insanely simple tool — it’s just a way to apply a set of make stanzas in the same order every time.  Steve’s theory was that the only way to get consistency from your computers was to do exactly the same thing on them every time.  So, he wrote up everything he did as a make stanza, and then built ISconf as a way of executing them in a consistent order.

Or at least, that was the theory.  When I got his code for ISconf, it was, em, not terribly usable.  It was written in shell and perl, with shell calling perl calling shell calling shell calling perl then exploding.  I had already bought into the basic theory, but I couldn’t see how to his code (which he was calling ISconf 2), so I rewrote it with some pretty significantly different functionality but following, as closely as possible, his basic theory. Naturally, I called this ISconf 3. On the ISconf web site, you can read more of this history, along with Steve’s relatively bitter version of how this portion of the history goes. (Incidentally, I wrote the majority of the content on that site.)

I won’t go into the detail of how it worked, except that all of the actual work happened with make stanzas.  This meant that, really, ISconf was a huge collection of very small shell scripts. This seems straightforward, until you start thinking about things like package installation — the command to install one package is almost the same as that to install a different package, which meant that a lot of my make stanzas had a lot of duplication.

To make my life easier, I started abusing the hell out of make, switching to gmake, and starting to make simple rules using ‘%’ to avoid duplication:

pkg/%:
...

I can’t even remember how this worked, thankfully; I just know you can extract what the ‘%’ matches and use it to choose the package name when installing a package.

This worked great for packages, since in most cases (and all the cases I was dealing with), you can easily specify a package by name.  It didn’t work so well for things like cron jobs, though — you’ve got between 1 and 6 fields, and the last field can have spaces and all manner of crap.  So, I started maintaining a separate file of all of these cron jobs, written in a perl Data::Dumper format, with each cron job having a unique name.  Then my make stanza would just refer to the arbitrary name I’d chosen.

This at least made ISconf more usable, but there were still a lot of problems.  You can read my writeup from around then in my LISA paper, but there were some serious problems with using ISconf.  First and foremost, it wasn’t a real solution — it was just a way to organize billions of little shell scripts, and as most anyone knows by now, I’m not so fond of shell scripts.  Also, the whole theory was bogus — there’s just no way you can guarantee consistency, not through order or any other mechanism.  Your dev servers and test servers can never be exactly the same, so stupidly replaying the same scripts in order also can’t guarantee sameness.  As a ridiculous example, I’ve seen printers that don’t work if their hostnames have an odd number of characters, and there’s just no way you can test this kind of case.

Beyond the theoretical problems, though, there were even worse practical issues.  You were supposed to never touch any code once it’d been deployed to a machine, because otherwise you couldn’t guarantee consistent behaviour, but this is just silly, because it means that you can never refactor code or change how you manage a given aspect of your system.  Imagine a solution that required immediate calcification of anything you ever did — once you deploy it, it’s deployed like that forever, even if it’s stupid.  No reason to learn; even worse, you get punished for learning, because you just add a new system to maintain every time you learn anything.

So, I wanted the ability to refactor my solutions, and ISconf couldn’t deliver that, so I started looking for something else that could.  Check back for part 2 to see what I did next.