Sometimes, though, multiple series of commits are related to each other, which can easily get lost.  Even worse, multiple series of commits developed in tandem can cause downright painful development histories.

For example, we’re currently working on refactoring Puppet’s ActiveRecord integration while at the same time we’re adding the ability to queue the database store operations.  We’ve had two development teams (of 1 to 3 people each) working on each feature, constantly publishing and rebasing against each others’ work.  You could certainly argue that this isn’t the right model (we should have worked in serial rather than parallel, probably), time constraints didn’t allow this.

On Friday, we got to the point where we’re nearly done, and I started sending my code to the -dev list for review.  That was straightforward enough, because I’d done my development in a separate branch and I still had those branches.  The other team, though, had been pushing code around like made, tuning and modifying their code over time, so sending their commits out for review was harder.  To top it off, our final branch was already a mixture of the different efforts.

So, I decided to see if I could clean up the development history. We had 80 commits spread all around, and I needed to reorder and squash them so they made easy sense during code review.  (This is a pseudo-process without actual code, because the reality was too messy to reproduce here.)  First I needed a list of the commits in the branch; git rebase is our tool through this process, and using it interactively (with ‘-i’) is the key.  So, I made a new branch and started my rebasing:

git checkout dev

git checkout -b clean_dev

git rebase -i 0.24.8

This opens up my editor with an ordered list of all of the commits in my branch that aren’t in 0.24.8.  There are three things you can do with commits in this list:  Leave them alone, combine them with another commit, and delete them (which deletes them from the branch).  Because I knew this would be a long complicated process, I saved the whole list to a separate file to start.

Some of the commits in our branch were backports of fixes we needed from the ‘master’ branch, so these were the only commits I left in the commit list in my first rebase.  This added about six commits, and was pretty easy to merge.  All of the other commits were just deleted from my clean branch.

The next step was to add my indirected ActiveRecord code.  This was only four or five commits, but should have been collapsed into fewer than that (e.g.,  one of the commits fixed a misspelling in a test).  There are multiple ways you could do this, including cherry-picking, but rebasing is definitely the most powerful.  I created a new temporary branch in which to do my rebase:

git checkout -b clean_indirected_activerecord

git rebase -i clean_dev

This actually results in a noop, because I’m rebasing against a branch that’s a complete duplicate of my current branch.  However, because I’m in interactive mode, I can do whatever I want from here, so I pasted in my four or five commits, and s/pick/squash/ where appropriate.  Once I dealt with any merge conflicts, I then merged back into my clean branch:

git checkout clean_dev

git merge –no-ff clean_indirected_activerecord

I decided to force the merge commit to exist because, um, actually I don’t have a good reason.  It seemed like a good idea to have a clear milepost saying that a given branch is merged, like that first email in a code review describing a patch series.

So now I’ve got a branch that has a series of patches that prepare the branch for us (with backported fixes, mostly), then a series of patches providing the first set of development.  Now I just need to repeat this process for the other three development stages, one done by me and two done by others.

I had no problem with my other code; it was 8 or so patches, but I wrote them all, so I could easily handle merge conflicts.  I also had no problem with one of the other chunks of code, because it was only three commits, so simple cherry-picking would have sufficed.

The last bit is where rebasing eventually broke down.  In the end, I had 35 commits that I thought contributed something  to the code (we had some duplicate commits in there, somehow, and some other commits that got cancelled out by later commits), but it looked like they should have been reduced to as little as four or five commits, because that’s about how many components were added in the code.  However, I don’t know this code as well as the people who wrote it, so I decided to punt here and told them they needed to clean up their development path and send me some patches that had no duplication and no code that gets deleted in a later patch.  My expectation is that they’ll create entirely new commits from the current state of the files, because the current commit history reflects a process rather than the desired state.

This whole process made me think of a discussion we had a while back on the -dev list (and a related thread started by Brice Figureau).  Apparently Linux development maintains every commit series separately, and only merges them when it’s time for release.  Or rather, the release involves a final merge.  They maintain multiple ongoing development branches; one of them has all of the proposed patch series, but it’s never merged directly into the release branch.  Instead, when a given patch series is accepted, it’s merged separately into the main branch.

This rebasing I did above made me realize the benefit of that approach – if the four development chunks had each remained separate branches, rather than merging early and merging often, it would have been *much* easier to keep them clean, and the developer responsible for a given chunk could always easily rebase just his or her own commits without affecting anyone else.  Then, when it was time for release, I could just merge them all in the appropriate order and release.

This is pretty easy for four patch series, but obviously gets more complicated as we have tens of sets.  I think for now, it’s too much work to maintain the patch sets in appropriate merge order without actually merging, but I think at some point, it really will make sense.  At the very least, this process has taught me the value of rebasing early and rebasing often.

Of course, in the open core model there must be “commercially-available extensions” in order to get companies to sign a “commercial license”. Why is this? Because the open core product has been intentionally hobbled to force companies to buy the closed software product in order to get it to do the things that customers need it to do, and thus to generate revenue for the software company.

I find this article interesting, because he seems to have taken the opposite tack from me in terms of deciding what causes the most dilution of an open source product.

I’ve always figured that requiring copyright attribution is a greater sin than providing commercial add-ons, with the strong assumption that the product completely stands on its own without those add-ons.

As far as I see it, requiring copyright attribution restricts the developer community, while providing commercial add-ons doesn’t restrict anyone anywhere, it just says that some of my code isn’t free.

Sure, I can see if I produce a crippled, useless free product that *requires* the commercial add-ons that it’d be evil, but if the OSS portions actually do kick ass on their own, then what’s the issue?

To me, community is the big differentiator.  If a given OSS project doesn’t actually care enough about a community, then open core is basically a sales tool.  But if it *really* cares about community, then open core always has to worry that the community can thrive without the commercial portions.  I basically think of this as requiring that the open core company always open source anything that’s required by the project or that’s essentially commoditized, but it leaves plenty of room for using a commercial license on those projects that not everyone needs, and especially on those areas that are clearly not commoditized and not many people need.

I think Tarus has a special perspective on OSS success because he’s in what amounts to an entirely commoditized space – there are so many monitoring tools with such a similar feature set that it’s crazy to think anyone would choose anything but an entirely open source solution.  In less commoditized spaces, it likely doesn’t make quite as much sense to stick to entirely open licenses.

This bit in particular sticks out:

If you like your users why not provide all of the features as open source? Red Hat does it, JBoss does it, OpenNMS does it. The answer will always be that their business model can’t survive unless they sell closed source software. In that aspect, I can see little difference between open core companies and Microsoft.

Well, I can tell you that the reason that Reductive Labs is considering producing commercial software is that we can’t afford to produce much more software unless someone pays for the development, and at this point, we have a thriving, healthy community that largely gets huge benefit from Puppet without ever needing help from us.  So, our options are to grow so slowly that all of the interesting opportunities pass us by, or to start producing software that allows us to, at the least, recoup our development costs.

I agree with Tarus’s basic sentiment: A lot of these open core companies aren’t actually producing open source software that you could reasonably use on its own.  But that doesn’t invalidate the model, in my opinion.

His model of providing supported binaries, as Red Hat and others do, only works for those who use compiled languages, which means it’s right out for us.  Maybe we should stick some C in there, just to make it easier to charge for support?

So what do you think:  Is it a greater sin to only accept patches to your product if the contributor is willing to assign copyright to your commercial company, or to produce some closed-source code?

Development Workflow

We led the discussion with a conversation about how the development workflow will change now that we’re finally releasing the code in master as 0.25.  After much discussion, we largely concluded that the least-surprise solution was to use the ‘master’ branch for stable development, and have something like a ‘next’ branch for new features and major refactorings.

The main goal is for new developers to be able to clone our repository and get to developing code immediately without having to worry about switching branches or any such thing.  More experienced developers will know where their development should be done, so it’s mostly a question of least-surprise for newcomers.

There was also discussion of fly-by-night developers, people who show up, produce a patch or two, and wander off.  James Turnbull (who handles 99% of the merging and release management) said he was fine accepting patches over email, rather than forcing people to publish everything via git.

Other than those two changes, our development workflow has worked pretty well.  However, the fly-by-night patches led into discussion of…

Core vs. Modules

Puppet’s core is starting to get many non-core features, like Nagios and Zenöss integration.  It’s essentially impossible for the core team to maintain all of these extentions, but many of them were written as a one-off solution and won’t be maintained by their authors.  Because of this, we  as a project need to develop a means of splitting Puppet’s core away from all of these modules.

Discussion was had of using a Nagios-style ‘plugins’ repository, but I don’t like that idea because it leaves basically the same problem – a centralized repository that no one wants to maintain.

Instead, we all basically agreed that we want to focus on modules as the means of adding functionality to Puppet, but the big problem there is that the only way to do so is to add package-like behaviours to modules, and none of us wants to make our own package manager.  Nonetheless, it was agreed that this was the only real option, so it just remains to do it.

Roadmap and Release Status

Against all odds, it looks like we’ll get 0.25 out in February – I’ll be merging in the last major refactor this week, and then it’s just a question of getting as many bug-fixes in as we can and dropping the release.

In addition, there are (annoyingly) a few important bugs in 0.24.7, so we’re unfortunately going to have to be put out a 0.24.8 release.  One thing that’s become clear in the last couple of releases is that we need to get more community testing of release candidates – many of the bugs would have been caught by basic testing in the community.

Facter Design and Shadow{Facter,Puppet}

RailsMachine released ShadowPuppet and ShadowFacter in the last few weeks, and it’s led to a flurry of design conversations.  We didn’t cover too much of it in the dev call, other than my reiteration that I’m excited by them and would like to merge them into Puppet and Facter.

Conclusion

And that was it.  Please join us on the next call if you can.

…this step is another milestone for the openQRM project which now includes the automatic configuration management features for the managed appliances powered by Puppet. With integrating Puppet into openQRM the mission is to provide a generic web-based user interface for the Puppet manifest. The current state already provides some pre-made classes like web-server, database-server, lamp etc. and automatically sets up Puppet for the openQRM environment in best-practice  manner by keeping the manifest in an snv-repository.

John Willis had a say, too:

The important thing, in my opinion, about the the openQRM Puppet integration transaction is that it exposes the exciting and beautiful things about open source. From a discussion about the importance of the integration of provisiong and configuration management, (i.e., openQRM and Puppet), to a fully developed integrated solution, only took 3 weeks. Imagine this discussion happening between even the most agile of proprietary vendors and the time line of something like this to happen. In the proprietary example they would still be playing phone tag just to figure out how their lawyers could talk. I am pretty sure Matt never even called Luke once during that whole process.

I haven’t yet had the chance to give OpenQRM a try, but hopefully this will encourage others to try it, and maybe one of those others will let me know how it goes for them.

One thing some people apparently don’t realize is that Puppet couldn’t have been created without Reductive Labs — there’s no way I would have had the time or motivation to create Puppet without a company devoted to funding its development.  All of the money that Reductive Labs makes goes back into funding Puppet, mostly in the form of development, marketing, and community management (although less of the latter, recently).  Almost all of the money I spend on marketing is in the form of conference speaking, so don’t think I’m spending big coin on TV spots or anything, but these conferences have been *huge* sources of community growth.

Strangely, I’ve heard complaints that I’m focusing too much on the company and not enough on the project, and as a result the product is suffering.  I find this strange because the project doesn’t pay my bills, feed my children, or buy my beer, the company does, and without those things, the project itself couldn’t exist.  If, instead, I had taken some corporate sysadmin job, I’d have been spending maybe 5-10 hours a week on Puppet, instead of between 10 and 60 hours a week like I’ve been doing for the last three and a half years.  This, I think, is the thing that people tend to forget about open source — yes, you can make money doing open source, but in doing so you almost always put yourself in conflict between making money and writing code.

The key is that Reductive Labs doesn’t make money when I write Puppet code, it makes money by helping people with existing code; yet, my business model is based on plowing that money into writing code, on the assumption that the code will make the product better, increasing the market, and hopefully allowing both project and company to grow.

This gets especially complicated when my own goals show up.  I’ve always seen Puppet as the first step toward building a significantly enhanced toolchain for system administration, and it was always my goal to use Puppet as a launchpad to having Reductive Labs fund other great projects.  (This phrasing is important — I need to make money to do development full time, so Reductive Labs (currently) needs to make money to fund that development.)  This is why I’ve always resisted billing us as “the Puppet company” — I don’t want to be pigeonholed in the event we have other successes, which I hope we will.

These longer-term goals are also why I’ve avoided joining other companies and just doing part-time development on Puppet for them.  Leaving aside that none of them would have considered hiring me until I’d already had significant success with Puppet (which puts us in a strange Catch-22, where they’d only offer to hire me once I’d shown I didn’t need their help), no one has ever expressed an interest in hiring me to build a product group, which is what I’m trying to do here.

Going back to finding that balance, one of the things I’ve foolishly allowed myself to do in the last year is get too caught up in fixing bugs, rather than adding functionality or doing interesting refactoring.  It’s amazing how disheartening and depressing it is to spend all of your days fixing bugs in code you wrote, all the while struggling to make enough money to pay people to help you fix those bugs.  Having more people at the company has helped bring focus to my efforts and those of the company as a whole, and having many more people involved in the project has helped bring focus to the needs of the project.

I think that’s about all I have to say on the subject right now — a presentation of the problem, a statement of the status quo, and a wandering finish.  Now that I’ve introduced the topic, though, I hope to be able to post more lucidly and usefully about it in the future.