Sometimes, though, multiple series of commits are related to each other, which can easily get lost.  Even worse, multiple series of commits developed in tandem can cause downright painful development histories.

For example, we’re currently working on refactoring Puppet’s ActiveRecord integration while at the same time we’re adding the ability to queue the database store operations.  We’ve had two development teams (of 1 to 3 people each) working on each feature, constantly publishing and rebasing against each others’ work.  You could certainly argue that this isn’t the right model (we should have worked in serial rather than parallel, probably), time constraints didn’t allow this.

On Friday, we got to the point where we’re nearly done, and I started sending my code to the -dev list for review.  That was straightforward enough, because I’d done my development in a separate branch and I still had those branches.  The other team, though, had been pushing code around like made, tuning and modifying their code over time, so sending their commits out for review was harder.  To top it off, our final branch was already a mixture of the different efforts.

So, I decided to see if I could clean up the development history. We had 80 commits spread all around, and I needed to reorder and squash them so they made easy sense during code review.  (This is a pseudo-process without actual code, because the reality was too messy to reproduce here.)  First I needed a list of the commits in the branch; git rebase is our tool through this process, and using it interactively (with ‘-i’) is the key.  So, I made a new branch and started my rebasing:

git checkout dev

git checkout -b clean_dev

git rebase -i 0.24.8

This opens up my editor with an ordered list of all of the commits in my branch that aren’t in 0.24.8.  There are three things you can do with commits in this list:  Leave them alone, combine them with another commit, and delete them (which deletes them from the branch).  Because I knew this would be a long complicated process, I saved the whole list to a separate file to start.

Some of the commits in our branch were backports of fixes we needed from the ‘master’ branch, so these were the only commits I left in the commit list in my first rebase.  This added about six commits, and was pretty easy to merge.  All of the other commits were just deleted from my clean branch.

The next step was to add my indirected ActiveRecord code.  This was only four or five commits, but should have been collapsed into fewer than that (e.g.,  one of the commits fixed a misspelling in a test).  There are multiple ways you could do this, including cherry-picking, but rebasing is definitely the most powerful.  I created a new temporary branch in which to do my rebase:

git checkout -b clean_indirected_activerecord

git rebase -i clean_dev

This actually results in a noop, because I’m rebasing against a branch that’s a complete duplicate of my current branch.  However, because I’m in interactive mode, I can do whatever I want from here, so I pasted in my four or five commits, and s/pick/squash/ where appropriate.  Once I dealt with any merge conflicts, I then merged back into my clean branch:

git checkout clean_dev

git merge –no-ff clean_indirected_activerecord

I decided to force the merge commit to exist because, um, actually I don’t have a good reason.  It seemed like a good idea to have a clear milepost saying that a given branch is merged, like that first email in a code review describing a patch series.

So now I’ve got a branch that has a series of patches that prepare the branch for us (with backported fixes, mostly), then a series of patches providing the first set of development.  Now I just need to repeat this process for the other three development stages, one done by me and two done by others.

I had no problem with my other code; it was 8 or so patches, but I wrote them all, so I could easily handle merge conflicts.  I also had no problem with one of the other chunks of code, because it was only three commits, so simple cherry-picking would have sufficed.

The last bit is where rebasing eventually broke down.  In the end, I had 35 commits that I thought contributed something  to the code (we had some duplicate commits in there, somehow, and some other commits that got cancelled out by later commits), but it looked like they should have been reduced to as little as four or five commits, because that’s about how many components were added in the code.  However, I don’t know this code as well as the people who wrote it, so I decided to punt here and told them they needed to clean up their development path and send me some patches that had no duplication and no code that gets deleted in a later patch.  My expectation is that they’ll create entirely new commits from the current state of the files, because the current commit history reflects a process rather than the desired state.

This whole process made me think of a discussion we had a while back on the -dev list (and a related thread started by Brice Figureau).  Apparently Linux development maintains every commit series separately, and only merges them when it’s time for release.  Or rather, the release involves a final merge.  They maintain multiple ongoing development branches; one of them has all of the proposed patch series, but it’s never merged directly into the release branch.  Instead, when a given patch series is accepted, it’s merged separately into the main branch.

This rebasing I did above made me realize the benefit of that approach – if the four development chunks had each remained separate branches, rather than merging early and merging often, it would have been *much* easier to keep them clean, and the developer responsible for a given chunk could always easily rebase just his or her own commits without affecting anyone else.  Then, when it was time for release, I could just merge them all in the appropriate order and release.

This is pretty easy for four patch series, but obviously gets more complicated as we have tens of sets.  I think for now, it’s too much work to maintain the patch sets in appropriate merge order without actually merging, but I think at some point, it really will make sense.  At the very least, this process has taught me the value of rebasing early and rebasing often.

Development Workflow

We led the discussion with a conversation about how the development workflow will change now that we’re finally releasing the code in master as 0.25.  After much discussion, we largely concluded that the least-surprise solution was to use the ‘master’ branch for stable development, and have something like a ‘next’ branch for new features and major refactorings.

The main goal is for new developers to be able to clone our repository and get to developing code immediately without having to worry about switching branches or any such thing.  More experienced developers will know where their development should be done, so it’s mostly a question of least-surprise for newcomers.

There was also discussion of fly-by-night developers, people who show up, produce a patch or two, and wander off.  James Turnbull (who handles 99% of the merging and release management) said he was fine accepting patches over email, rather than forcing people to publish everything via git.

Other than those two changes, our development workflow has worked pretty well.  However, the fly-by-night patches led into discussion of…

Core vs. Modules

Puppet’s core is starting to get many non-core features, like Nagios and Zenöss integration.  It’s essentially impossible for the core team to maintain all of these extentions, but many of them were written as a one-off solution and won’t be maintained by their authors.  Because of this, we  as a project need to develop a means of splitting Puppet’s core away from all of these modules.

Discussion was had of using a Nagios-style ‘plugins’ repository, but I don’t like that idea because it leaves basically the same problem – a centralized repository that no one wants to maintain.

Instead, we all basically agreed that we want to focus on modules as the means of adding functionality to Puppet, but the big problem there is that the only way to do so is to add package-like behaviours to modules, and none of us wants to make our own package manager.  Nonetheless, it was agreed that this was the only real option, so it just remains to do it.

Roadmap and Release Status

Against all odds, it looks like we’ll get 0.25 out in February – I’ll be merging in the last major refactor this week, and then it’s just a question of getting as many bug-fixes in as we can and dropping the release.

In addition, there are (annoyingly) a few important bugs in 0.24.7, so we’re unfortunately going to have to be put out a 0.24.8 release.  One thing that’s become clear in the last couple of releases is that we need to get more community testing of release candidates – many of the bugs would have been caught by basic testing in the community.

Facter Design and Shadow{Facter,Puppet}

RailsMachine released ShadowPuppet and ShadowFacter in the last few weeks, and it’s led to a flurry of design conversations.  We didn’t cover too much of it in the dev call, other than my reiteration that I’m excited by them and would like to merge them into Puppet and Facter.

Conclusion

And that was it.  Please join us on the next call if you can.

ISconf

Puppet was heavily influenced by two tools, but almost more importantly, it was heavily influenced by what I couldn’t do with those tools.  I attended LISA in 2001, where I saw Steve Traugott give his talk on automating infrastructures.  I was impressed enough that I convinced my then-company to bring him in for a week of consulting.  During that week, he convinced me to give a tool he’d written, ISconf, a try.

ISconf is an insanely simple tool — it’s just a way to apply a set of make stanzas in the same order every time.  Steve’s theory was that the only way to get consistency from your computers was to do exactly the same thing on them every time.  So, he wrote up everything he did as a make stanza, and then built ISconf as a way of executing them in a consistent order.

Or at least, that was the theory.  When I got his code for ISconf, it was, em, not terribly usable.  It was written in shell and perl, with shell calling perl calling shell calling shell calling perl then exploding.  I had already bought into the basic theory, but I couldn’t see how to his code (which he was calling ISconf 2), so I rewrote it with some pretty significantly different functionality but following, as closely as possible, his basic theory. Naturally, I called this ISconf 3. On the ISconf web site, you can read more of this history, along with Steve’s relatively bitter version of how this portion of the history goes. (Incidentally, I wrote the majority of the content on that site.)

I won’t go into the detail of how it worked, except that all of the actual work happened with make stanzas.  This meant that, really, ISconf was a huge collection of very small shell scripts. This seems straightforward, until you start thinking about things like package installation — the command to install one package is almost the same as that to install a different package, which meant that a lot of my make stanzas had a lot of duplication.

To make my life easier, I started abusing the hell out of make, switching to gmake, and starting to make simple rules using ‘%’ to avoid duplication:

pkg/%:
...

I can’t even remember how this worked, thankfully; I just know you can extract what the ‘%’ matches and use it to choose the package name when installing a package.

This worked great for packages, since in most cases (and all the cases I was dealing with), you can easily specify a package by name.  It didn’t work so well for things like cron jobs, though — you’ve got between 1 and 6 fields, and the last field can have spaces and all manner of crap.  So, I started maintaining a separate file of all of these cron jobs, written in a perl Data::Dumper format, with each cron job having a unique name.  Then my make stanza would just refer to the arbitrary name I’d chosen.

This at least made ISconf more usable, but there were still a lot of problems.  You can read my writeup from around then in my LISA paper, but there were some serious problems with using ISconf.  First and foremost, it wasn’t a real solution — it was just a way to organize billions of little shell scripts, and as most anyone knows by now, I’m not so fond of shell scripts.  Also, the whole theory was bogus — there’s just no way you can guarantee consistency, not through order or any other mechanism.  Your dev servers and test servers can never be exactly the same, so stupidly replaying the same scripts in order also can’t guarantee sameness.  As a ridiculous example, I’ve seen printers that don’t work if their hostnames have an odd number of characters, and there’s just no way you can test this kind of case.

Beyond the theoretical problems, though, there were even worse practical issues.  You were supposed to never touch any code once it’d been deployed to a machine, because otherwise you couldn’t guarantee consistent behaviour, but this is just silly, because it means that you can never refactor code or change how you manage a given aspect of your system.  Imagine a solution that required immediate calcification of anything you ever did — once you deploy it, it’s deployed like that forever, even if it’s stupid.  No reason to learn; even worse, you get punished for learning, because you just add a new system to maintain every time you learn anything.

So, I wanted the ability to refactor my solutions, and ISconf couldn’t deliver that, so I started looking for something else that could.  Check back for part 2 to see what I did next.

We’ve had a Roadmap for a while, but it’s pretty simplistic, and I haven’t had enough of a plan to be able to reasonably maintain it.  In our call today, though, we came up with a pretty clear plan for what we’re going to do next, when we’re going to do it, and what version numbers it will have.  Here’s a summary of what we decided (which I’ll then be importing into the roadmap document, where it will actually be maintained).

• 0.24.7: We’re going to get this release out as soon as possible (most likely around the middle of November), with any critical fixes resulting from the 0.24.6 release.  This will be the last stable release in the 0.24.x line.
• 0.25: This release will be the first release from the master branch, and is planned for the end of the year.  Its primary changes will be all of the plumbing to support REST, plus converting, at a minimum, file serving and catalog retrieval to using REST.
• 0.26: This release will convert any remaining XMLRPC network APIs to REST, and will be the last release before 1.0. It should be out in the second quarter of 2009.
• 1.0: This release stabilizes the network API:  According to the plan, projects developed against its API should work going forward.

We’ve got less detail on the 2.0 and 3.0 releases, but we’ve got main targets for them.  The 2.0 release will focus on refactoring the transactional system, and the 3.0 release will focus on refactoring the RAL extension interfaces.

Now that we’ve got a version roadmap, we can begin doing realistic ticket assignments to those different versions.

Of course, we’ll still need a lot of help from the community to get all of the tickets done that we want to complete.  The versions will have a key deliverable, and that’s what I’ll be focusing on, but we’ll want a lot more in each release, and that’s where we’ll need help.