At CDD we use a fairly strange stack to run our automated in-browser (Selenium RC) integration tests.

When we started writing these tests back in 2007, we used the best technology we knew of that was then available, a Rails plugin called funkytown, since-abandoned brainchild of Pivotal Labs. Funkytown’s code is a little weird, and we’ve had to fix bugs and inflexibilities from time to time. We monkeypatched its rake tasks. Over the years we updated the bundled version of the selenium RC jar file several times.

We wrote another library (selenium-spec, which we open sourced once a while back) on top of funkytown that integrated selenium tests into rspec, provided the ability to chain examples (see this blog post), as well as some webrat-like helper methods, except better, enabling code like element('results').should be_visible.

So that’s what we’ve been using for a few years now. It’s not pretty, but it works. When cucumber and webrat came out, all the buzz made me think that we’d inevitably switch over some day in the beautiful future when everything will be perfect. However, I recently had the opportunity to try them out on a another pet project of mine, and I was surprised to learn that even though they have a much bigger install base than our install base of one, our crazy stack turns out to be better.

There are three major problems with cucumber/webrat/selenium RC, as far as I can tell:

1. You can’t really use fixtures.
2. Debugger breakpoints in your application code don’t work.
3. I swear there was a number three.

All of those (well, at least the first two) aren’t true of our crazy stack, and I wouldn’t want to give them up.

There are some nice things about cucumber/webrat. The annotations are neat, and it’s kind of cool that you can run your tests as both Rails integration and selenium RC tests (although I imagine once your tests get realistically complex at all that is no longer so assured).

Perhaps a good opportunity, one day in that beautiful future, for us to clean up our stack, rebase it on some more modern infrastructure (or rewrite our own), and open source what for us has been a really useful tool.

From the RSpec documentation:

It is very tempting to use before(:all) and after(:all) for situations in which it is not appropriate. before(:all) shares some (not all) state across multiple examples. This means that the examples become bound together, which is an absolute no-no in testing. You should really only ever use before(:all) to set up things that are global collaborators but not the things that you are describing in the examples.

Well-known conventional wisdom says that different test cases (in spec-speak, “examples”) should not depend on one another for state, should be runnable in any order, etc. I certainly agree with this wisdom in general, but I think there’s one case where this rule should be broken. We’ve been writing a fair number of Selenium RC tests lately for our app, using RSpec to drive Selenium RC. When writing integration tests like this, each example (in test speak, “test method”) is often a very long script with lots of shoulds/asserts in it. We lose the nice descriptive power of small examples with specific, descriptive text, and instead are faced with a choice between vague and high-level example descriptions and really long example descriptions using ugly here documents that can easily fall out of synch with the example code.

Instead, we want to be able to do something like this:

    describe "A user customizing a car" do
      use_chained_examples
 
      before(:all) do
        @model = models(:spiffy)
        log_in
        go_to_car_customization_start_page
      end
 
      it "should first be required to select car model" do
        page_title.should == "Select a model"
        droplist("model_id").should be_present
        droplist("model_id").options.should == [
          "(Please select a model)",
          models(:spiffy).name,
          models(:sporty).name
        ]
        droplist("model_id").selected_value.should == ""
 
        droplist("model_id").select(@model.id)
        click_next_button
      end
 
      it "should then be required to select paint color" do
        page_title.should == "Select #{@model.name} color"
        droplist("color_id").should be_present
        # etc.
      end
    end

A before(:each) block can be used to reset the page between each example, which we’ve found useful when testing a bunch of validations or something similar. Note too that using chained examples is not the default behavior, and must be explicitly specified by the developer, who should “know what they are doing” if they do this.

Anyway, obviously we did figure out how to make this happen, and after we’ve refined it a bit if people are interested we’ll open source our selenium rspec stuff as a plugin. Note that our selenium specs use a different spec_helper.rb than the rest of our normal specs, so we’re keeping the ability to chain examples out of our standard specs, as conventional wisdom would recommend. Let me know what you think.