Testing Cached Values

I'm currently in the middle of the largest refactoring effort I've ever done, while simultaneously learning tons about how to be a better developer. I'm constantly feeling a bit overwhelmed, a bit behind the curve, and like someone's going to look at my code one day and say, "hey nimwit, you just pull this string here and suddenly 2/3 of your code just goes away."

However, one thing I've really grokked recently is that if you're fighting your tools too much, you are on the wrong track, and one way in which it seems I'm constantly fighting my tools is the combination of cached values and testing.

For instance, I have an HttpPool class that knows how to set up Net::HTTP instances with all of the SSL information they need. This class caches SSL information like the certificate and key, so that each connection doesn't hit the disk for this info, which is obviously a pretty decent use of the cache. This caching generally takes this simple form:

def ssl_host
    unless defined?(@ssl_host)
        @ssl_host = Puppet::SSL::Host.new
    end
    @ssl_host
end

Yes, you could just do @ssl_host ||= Puppet::SSL::Host.new, but I've gotten some weird behaviour out of that in the past, and it also throws a warning for undefined variables.

So anyway, this works fine in unit tests; I test the caching, and I use mocks everywhere else. When I get to integration tests, though, it starts to really hurt, especially since I try not to do much mocking in my integration tests.

For instance, say I've got two unrelated tests that do an ssl connection. They each create some certificates, start a daemon, and try to connect. In this situation, the first one caches the ssl information, and the second one uses the cached values instead of its own new certificate, and you get invalid certificates.

After talking to Rick Bradley on #nashdl on IRC (gotta represent!), I've decided on at least an initial course of action. I'm going to create a module that provides both a caching and cache-clearing interface; anyone using cached values would use this caching interface instead of caching the values themselves, and the module itself would give you a single point of entry for clearing all caches on the system.

My first instinct was to create a Cache class or struct and keep a list of them in the caching module, so they can be cleared as necessary, but my recent work with TTLs has made me realize that time-based concepts of cache dirtiness are much better than actively cleaning.

So now, I'm thinking that the caching module will just have a timestamp, and only cached values created after that time will be valid. Before the Cache struct returns any values, it will always check that time, and it will know whether the value it has is still good or should be discarded.

This keeps us from maintaining global lists of caches, and it also makes clearing caches insanely cheap -- just reset a timestamp. Given that Puppet already sometimes has onerous memory requirements, I also like that it makes the caches themselves more likely to get garbage collected, since only the caching instance ever knows about the actual cache itself.

So my new method would look something like this:

def ssl_host
    cache(:ssl_host) { Puppet::SSL::Host.new }
end

That bit with the block is something I just thought of; if we've got a value, it's not called, but if we need a value, it's there for us. Pretty sweet.

Now just to implement it.

Wed, 07 May 2008 | Tags: programming, testing, caching

Jay and I converge on testing

Those four people who have been reading this blog for a while know I've been struggling to think and program like Jay Fields. In particular, he seems to have presented a few rules in the past that don't like to be used together:

• Inline setup code: You apparently should not use before or setup blocks with your code; instead, you should have all of the setup code in every method.
• Mocks instead of Stubs: Here he claims that it's always better to use expects instead of stubs.
• One expectation per test: With only one expectation, you're only testing one feature at a time, and you're guaranteed that each behaviour is tested separately.

Now, let's do a simple combinatorial exercise, and put these three rules together:

• You should have all related code in your test
• You should only use Mocks (expects), not Stubs (stubs)
• You should only have on expectation (expects) per test

It's pretty clear that, like the old saw about programming ("All programs can be reduced to one line of code with a bug in it"), Jay is pointing us toward tests that can largely only be one line of code. Yeah, I know sometimes setup methods don't involve any mocking, but often they do.

And, since your tests can only be one line of code, they can't test very much, which means that all of your methods need to also be one line of code, else they aren't testable. (Yes, I'm being a touch extreme here, but that is where the arrow is pointed, anyway.)

You can see how this would kinda drive me bonkers. Some local dev friends have been trying to help me see the error of my ways, mostly so my code would stop looking like such crap; I've learned a helluva lot in the last 8 months or so, and most of it has actually made my code look more like Jay would recommend. I will say it's blindingly obvious Jay is doing internal development at enterprises, rather than developing as part of a consistent team producing software that is distributed to the wider world.

But one can only go so far, and the three rules above, in combination, are just way too far. I've often kind of sputtered expasperatedly at Jay's posts, especially his announcement of his new testing tool, expectations. Again, I can kind of see where he's going with that, but you've got another thing coming if you think I'm using it, especially given how happy (at least, relative to test/unit) I am with RSpec.

Also, I think it's just stupid having all setup code inline. DRY ("don't repeat yourself") is just as true in your test code as anywhere else, and having a maintainable test code base is, IMO, more important than having your normal code base be maintainable, because tests are kind of unnecessary. If you have good, readable, maintainable tests, then people who contribute will also contribute tests. If your tests are all 50 lines long and have lots of repetition, then 1) you've got 5x the amount of code you should, which is wicked expensive, and 2) you've got so much code no one will look at it. Yay, never getting patches with tests in them. My favorite example of this is Steve Yegge's rant Code's Worst Enemy; he describes his 500k line Java project with no tests, which is a lot of code but much less code than if it had tests. I've experienced in Puppet that test code seems to be much harder to maintain that normal code (although maybe it's just own crap test code, not normal test code), and having 5x test code than normal code would make me just quit writing unit tests entirely.

So, I am absolutely overjoyed to announce that Jay has changed one of his rules: He now recommends stubs over mocks. This is clearly just for setup code and such, but it's a big step. He even goes into using stub_everything, which I find is the only way to build tests that aren't fragile. For instance, say you start with this code:

class MyClass
    def go
        start()
        finish()
    end
end

describe MyClass do
    before
        @me = MyClass.new
        @me.stubs(:start)
        @me.stubs(:finish)
    end

    it "should start when going" do
        @me.expects(:start)
        @me.go
    end

    it "should finish when going" do
        @me.expects(:finish)
        @me.go
    end
end

Now you find you need a validation method, so you add this test:

it "should validate when going" do
    @me.expects(:validate)
    @me.go
end

Update: Fixed code to actually call @me.go in the validate test.

Oops. Now your single test passes, but your two old tests break, because you were only stubbing start and finish, instead of using stub_everything. Your setup code needs to be modified to take this new call into account (or, if you're Jay, you need to modify every test in your suite; yay). This comes up constantly. If you specifically mock or stub methods during setup, then you are almost guaranteed to have cascading failures when you expand your code.

Anyway, the point is, if I tried to follow Jay's rules, then the above trivial change -- I add one line of code to a very simple method -- would result in me adding a test for that line, plus at least one line of code in every other test in that suite. Instead, if I use stub_everything, then I add my new test and I'm done. (Well, kind of; notice I'm not actually testing the order of the method calls, which is actually pretty tough.)

My recommendation is to read Jay, since he's clearly thinking and talking about aspects of testing that not many others are, but read him with a skeptical eye, and be willing to say "That's just nuts!" and write your own relatively abstracted test code. And if you're working with people who can't think to look at their setup code when a test fails, then you need to find a different job.

Tue, 06 May 2008 | Tags: ruby, testing, programming, bdd, tdd, jayfields

Ruby has a distribution problem

I've been doing a better job of reading development books recently (e.g., Domain Driven Design), and something has really begun to stick out at me. There seems to be a split between those developers who write software that is expected to run in one place and those who write software that is expected to run in many places.

If you, as a developer, know that your software will really only be installed at a single customer (whether that customer is your employer, a consulting client, or whatever), then your life is drastically easier -- you don't usually have to worry about cross-platform issues, and you don't have to worry about different users having different needs, because you only have one user.

Obviously there's no inherent problem with having the simpler life of a developer with only one user, but it seems to me that the Ruby community is, as a group, largely adopting that perspective as the default. This is worrying to me, because I'm building an application that I expect to be installed in thousands of locations (in fact, it's probably already installed in thousands of locations). I'd like to take as much advantage of existing Ruby code as possible, but it's not exactly easy.

For example, rubygems (probably my least favorite ruby software of all time) basically require that you always try to load them, because their design stupidly requires that you know whether a given piece of software is installed via rubygems or some other mechanism. For instance, if you've installed the Facter gem, then this code doesn't work:

ruby -rfacter -e 'Facter.to_hash'

Instead, you have to do this:

ruby -rubygems -rfacter -e 'Facter.to_hash'

The reason is that rubygems installs in a location that Ruby doesn't search by default. The reason for that is that apparently this one guy, somewhere, wanted to have multiple versions of a given package installed at once. Who wants this? Let's just say it's not the guys who are distributing hundreds and thousands of copies of their software.

The truth is, most Rubyists don't even seem to use gems this way -- they tend to create a vendor subdirectory in their project, and then install their gems there. This is a clear example of how little they expect to have their projects distributed. These gems might be compiled, they might conflict with installed software, they might require installed software -- you have no idea, because it's an entirely separate repository of packages.

This is basically anathema to how I think about management, yet it's the standard, recommended practice in the Rails community, because it makes it easy to "guarantee" behaviour in a given environment. Of course, your guarantee is only good if no one ever tries to run the software anywhere except an exact duplicate of where you run it.

I tried to have a conversation about this at the Ruby Hoedown last year -- my claim was it was difficult to turn a Rails project into a native package, especially with the tendency toward requiring all kinds of random gems. Quite a few people kind of stared blankly at me and said, multiple times, "I just put it in vendor." Since then, this has become my go-to phrase for describing the Ruby way of solving distribution problems: "I just put it in vendor." I keep waiting for someone to try to put their kernel or web browser in vendor: "We only support the Firefox copy in vendor, sorry."

I don't know if other communities are any better at this. From what I can tell, this is basically how the Java community behaves, too. They have pathologically bad distribution systems, and as a language it seems to be most influenced by consulting shops developing huge, worthless software projects for large enterprises, rather than developing distributed applications that will be installed in thousands of locations.

I'd like to think that Puppet would have some counter-affect to this. It's one of the largest and most sophisticated publicly available Ruby projects, it's already installed in at least hundreds and probably thousands of places, and it does a pretty good job of working nearly everywere. However, I keep getting blank stares when I talk about this with other Rubyists, half the time I'm called a troll for even bringing it up, and when I explain why Puppet exists to most Rubyists, they just say, "I just put it in vendor", or, maybe, "Why not just use Capistrano?" To that I ask, how do you install Capistrano, but you know what they say to that.

I think Rails is a big part of the problem. Rails is clearly created by a company that will never distribute its software, and the Rails philosophy is again almost pathologically opposed to the idea of turning your software into a package. Imagine trying to make a Rails project LSB compliant -- your database.yaml file would need to be in /etc, your log directory would need to be in /var, and your actual code would need to be in /usr. There went all of your fancy Rails "convention over configuration", and you're suddenly fighting Rails instead of using it, and everyone you ask for help just tells you to "put it in vendor".

I'm looking at creating a new application that I'm planning on distributing, and one of my big goals is to be able to distribute the core in one package and various additional pieces of functionality as separate packages. I'll need to simultaneously support as many of my customer platforms as I can and provide a consistent operating environment for my packages. The only way to do this is to have supported operating environments with well-defined dependencies, such as you can almost trivially build in Debian or Red Hat.

For those of you who are thinking, "you could just put it in vendor", or "you could at least use gems", No, I couldn't. Take a trivial example: Say I want to use RRD support in my application (which is likely, in this case). There is Ruby support for RRD, but not in Gem form. Even if there were a gem, though, it would require a native RRDTool package, and, of course, Gems can't specify dependencies on native packages, so I'd be telling my customers, "well, install X gems and Y packages".

Instead, if I use native packages (say, those for Debian and Red Hat, to cover most cases), I can define clear dependencies for all cases. I know Debian provides everything I need, and in the rare case it doesn't, I can provide my own apt repository (and the same for yum). Gems, on the other hand, can really only do Ruby stuff. No, I don't actually want to put glibc in vendor, thanks.

I don't see a solution to this, other than getting more Rubyists distributing their software, but I'd really like to see this issue begin to be approached by the community. I feel like a wolf howling in the wilderness at this point, and if often feels like I'm fighting against my community in order to produce software that hundreds or thousands of people will install, as opposed to just use over the web.

Mon, 05 May 2008 | Tags: ruby, industry, distribution, package

Git branch in your bash prompt

Kevin Barnes has posted his mechanism for getting the current branch of the git repository into his bash prompt.

He mentions color in his article, and it turns out I'm the person who added the color, so I figured I'd post my version.

Here are the functions I have:

git_current_branch()
{
  git branch 2>/dev/null | sed -n '/^\*/ s/^\* //p'
}

git_display()
{
  br=$(git_current_branch)
  if [ -n br ]; then
      echo $br | BRANCH="$br" GIT_COLOR=$(git_color) awk '{if ($1) { print ENVIRON["GIT_COLOR"] ENVIRON["BRANCH"] " " } }'
  fi
}

git_color()
{
    git status 2>/dev/null | grep -c : | awk '{if ($1 > 0) { print ENVIRON["ORANGE"] } else { print ENVIRON["PINK"] } }'
}

And then here's my prompt:

title="033]0;h:W007" PS1="$title[$(git_display)$GREENw$NOCOLOR]nu@h("'$?'") $ "

First, the git bits. As Kevin mentions, I color the branch name; I use orange if I've got modified files, and pink if I don't (these are names that I map elsewhere to terminal codes). The three functions provide the three, um, functions: See what branch I'm on, see whether there are uncommitted files, and colorize the branch name.

Now, the bash bits.

First, you'll notice I have a multi-line prompt. I first started this when I switched to a color prompt, because for a while there bash didn't like the hidden characters that add color. I got to choose between a multi-line prompt, or a prompt that wrapped in broken ways. Since I really only wanted color in the path, I put that on the upper line, avoiding most of the wrapping problems. It was worth it, because (especially when I was still a sysadmin by trade) having color, almost any color, in the prompt makes it easy to pick out my commands from command output.

Now that I've had my prompt this way for about 4 years, I'm pretty fond of it. I think the bash wrapping problems are mostly fixed now, but I'm keepin git.

The $title sets the terminal title (which is slightly useful).

So, that's how I add the git branch, colored by whether I have uncommitted files, to my prompt, with a bit more prompt info thrown in for kicks.

Mon, 05 May 2008 | Tags: tools, git, bash

Reductive Labs should fill out the Little 4

Michael Cote and John Willis have been talking for a while about the Little 4 in management software, and it looks like Qlusters is no longer on the list.

I'd love to be able to comfortably say that Reductive Labs deserves to fill that fourth slot (you can have your say too). The truth is, of course, that all of those companies are far larger than Reductive Labs, and they've all successfully gotten investment while we have not (although we haven't tried all that hard). The products of the companies are pretty dissimilar, too -- they're mostly more focused on monitoring rather than what I would call management, as far as I can tell.

On the other hand, Puppet has a lot of traction, and is a clear leader in its space. We've been profitable since almost the beginning (which is to say, profitable enough to pay my meagre wages), and we've got a great and growing community. Now that Andrew Shafer has joined the company full-time as a partner, I do think we're going to start growing, and it's well-timed in terms of how the community is developing.

I do hope we grow this year, but I don't really know how we will. I'm still considering how hard we should be seeking investment, but it seems that VCs are pretty uninterested in infrastructure (or maybe they're just uninterested in me). Really, I'm hoping I can just get a big enough customer base that Andrew and I can build a bigger development team and start doing some of the almost-obvious but really interesting projects to enhance the Puppet ecosystem, like change control applications.

But the summary is, I want to deserve to be in the Little 4, and I think Puppet is popular enough that we just might, but there's still lots more to do.

Mon, 21 Apr 2008 | Tags: industry, software

Closing the Whisky Experiment

It's been a bit more than a month since I started my Double Blind Whisky Taste Test, and I've decided it's time to call it. I've drunk a scarily large amount of the whisky, but science is a harsh mistress and she will not be denied. Those of you who are sitting at home complaining that I haven't fixed your bug, it's because I was drunk from too much experimentation.

I've posting my reviews to Twitter, which in some ways adds a third blindness: I didn't know how I reviewed a given Scotch previously.

I'm writing this having not yet looked at the key that will tell me which Scotch is which. Here are my reviews so far (using my arbitrary decanter names). I'm actually taking a small drink of each as I write, just to remind myself and do one last review. These reviews are in order of perceived strength, so that I'm going from weakest to strongest.

My comments will also try to mention what I would do with these whiskies; this is because I've still got decanters with a lot of whisky in them, and really, I probably won't drink them all straight, which is all I would normally do with a single malt.

My primary goal in this review was to find that single peaty Scotch that I could put in my liquor cabinet (currently virtual, since the liquor is kind of scattered wherever there is storage).

• Teardrop: Very drinkable and very smooth, but relatively mild. It almost doesn't belong in this review. Once I have closed the experiment, I look forward to comparing it to Macallan, which is my old stand-by in terms of smooth Scotch. A little bit of peat, some harshness if you aren't careful, and just a bit of roundness. It's like all the edges are filed off and nothing really stands out. I expect this is a great Scotch if you're not drinking it near anything else, but near the others, it seems weak. It's almost a Scotch for people who don't drink Scotch.

  I like it, but it's not peaty enough to be considered a great peaty whisky.

• Oval: This one definitely has some peat in it, but it's got a hint of sourness that I don't really like (that's sour as in "something went sour" as opposed to "puckery"). This is apparently a consistent review, since I've mentioned that in my twitter microreviews, too. I expect that this would make a great whisky sour, but you would always want to hide the whisky's own sour, rather than focusing on it. Hopefully it would meld with the citrus, rather than having some sort of weird sour harmonic.

  At least this one's peaty, and it's got an overall good flavour. I wouldn't recommend it to most people, though, because of the sour note. I do remember drinking this multiple times without really noticing the sour, for what it's worth.

• The D: [1] This is definitely my favorite. It's got peat, a bit of spice, no real harshness unless you snort it, and none of the sourness of the oval. It has an almost mustiness to it, like a good blue cheese. It's really an even, peaty whisky; drinkable neat, but strong. It should mix well if you wanted to, given how much flavour it has, and go well in soda or with ice, but it deserves to be neat, or maybe on ice to keep the fumes down.

• Square: Very spicy, almost with a Bourbon spice, but quite harsh, also. Strangely, there isn't a lot of other flavour to it. There's some peat to it (I expect that's what adds the spice), but the harshness stands out so much that I'm not real fond of this whisky. I'm not even convinced it would taste that good with soda (which is often how I drink my bourbon), since it doesn't have much depth.

Okay, so now we know what I think about each of the whiskies, it's time to get the key out and see what is what. First, though, I will give my guesses as to what is what. Here's my estimate:

• Laphroaig: Square
• Teardrop: Caol Ila
• Ardbeg: Oval
• Lagavulin: The D

And now, without further ado, here's the real key:

• Laphroaig: Oval
• Teardrop: Caol Ila
• Ardbeg: Square
• Lagavulin: The D

So, I got two correct but mixed two of them. The clear winner in the "best peaty Islay malt" category is Lagavulin, which is unfortunate, since it's about $20 more per bottle than any of the others (assuming you can find Caol Ila, which I had to buy at the Duty Free in Melbourne). The Caol Ila gets an honorable mention as the easiest to drink.

Looks like I've been harshing on the Ardbeg this whole time, and it's the Laphroaig that has the sourness I didn't particularly like.

And with that, the experiment closes. Now I can finally drink some of the Caol Ila 18yr I have -- I couldn't drink it while the experiment was on, since it would likely have clued me into which decanter had the 12yr.

In the name of Science (and great whisky), Luke

Sun, 13 Apr 2008 | Tags: whisky, science

ArsTechnica Launches Forum on Large-Scale IT

(by Luke) ArsTechnica, my favorite tech news site, has just announced the launch of a forum dedicated to the discussion of large-scale IT:

The forum has been in beta for a few weeks, and already there are some great discussions happening. We envision The Server Room as a place to discuss IT topics that don't already have a dedicated forum, and are of an IT nature. Users are already talking about virtualization, storage, disaster recovery, and systems design. IT hardware discussion is appropriate as a function of the plan.

So go join the Server Room and tell 'em how great Puppet is.

Wed, 09 Apr 2008 | Tags: industry, automation, arstechnica, forum

Mountain West Ruby Conference

(By Drew)

I've been meaning to post this for a while, but we didn't have a good way to make this a multi-author blog. (We still don't, which is only slightly less lame than the fact that I haven't put anything up about the conference)

I spent the weekend before last at the Mountain West Ruby Conference (March 28-29).

This was the first language centric conference I have attended and I would recommend getting involved to anyone who wants to expand their understanding of Ruby and programming.

These conferences aren't going to help someone find all the answers, but they will help you start asking the right questions, and you couldn't beat the price of MWRC. $100 for a t-shirt, 2 lunches, 2 full days of talks, plus 2 party/hackfests with a fridge packed full of redbull and a room packed with passionate programmers.

Just doesn't get much better than that. . .

The speakers topics ranged from whimsical to deep magic. Some of my favorites were Giles Bowkett on metprogramming/code generation(who had everyone monkey patching sombreros into the presentations), the keynote by Jim Weirich, and Patrick Farley on Ruby internals. The most practical presentation, at least for me and Puppet, was Philippe Hanrigou's talk on using GDB and DTrace to troubleshoot running processes.

There are some excellent resources on Phillipe's website. Anyone interested in getting insight into running Ruby processes should take advantage of what he has available.

All the speakers are online. (Thanks Confreaks and the sponsors)

These aren't the $10 dollar burritos you are looking for

Tue, 08 Apr 2008 | Tags: Ruby, mtnwestrubyconf, Darth Vader, Sombrero

Checking whether classes have been evaluated

Another snippet I used for testing.

Bug #1165 requests the ability to check for whether a class exists, but we've already got that:

class yay {
    notify { "This is a test": }
}

class other {
    if defined(Class[yay]) {
        notify { "Defined works just fine": }
    }
}

include yay, other

Prints:

notice: This is a test
notice: Defined works just fine

Works for me. :)

Thu, 03 Apr 2008 | Tags: puppet, bug, class, snippet

Caching and REST

(by Luke)

One of the things I was supposed to write about last week was how I'm rethinking some of Puppet's internal caching. This rethinking is a direct result of listening to ThoughtWork's IT Matters Podcast on REST (I've only listened to part 1 so far). I actually listened to the episode three times, because it's only about 20 minutes and I listened to it on a 60 minute bike ride, which worked well because it was so windy that day that I didn't hear the whole thing any of those listenings.

I'll hopefully write later about how this podcast made me rethink how environments are used in fileserving, but for now, I'm going to focus on caching.

Puppet::Node.find("mynode")

Wed, 02 Apr 2008 | Tags: programming, thoughtworks, podcast, rest, caching, design, ruby, api, luke

Links

• Contact
• Reductive Labs
• Puppet
• Puppet Presentations
• Luke's Shared Blogroll

Categories

• Industry
• Puppet
• SysAdmin
• Tools
• Test Code
• Travel

Tags

Syndication

• RSS
• Atom