Behavior-Driven Development (BDD) in Python

We've adopted Behave over Lettuce for our BDD needs here at Djaodjin. Below I'll present a summary of the motivations for this, and some criticisms of Behave.

Why Behave?

The Behave people have done an excellent job of fairly comparing their project to others' on their own site. For me, the only real alternative was Lettuce.

When I first read Behave's criticism of Lettuce, the lack of automatic cleanup for world had zero impact on me. I hadn't yet written many features. In fact, I assumed this was a largely irrelevant feature in Behave, since context was essentially a place for global variables, and global variables are bad. It wasn't until I had some more first-hand experience that I realized something important about working with typical BDD tools.

BDD is unforgivingly stateful

Most typical BDD tools - even those in Ruby like Cucumber - saddle you with this basic limitation: you must manage your own state between steps. Because they all match lines of the Domain Specific Language (steps) to bits of code (usually functions), there isn't much variation in how you can pass state between these functions. If all steps' formal args come from patterns extracted from the DSL step strings, and a step function depends on something being communicated from a prior step function, then it must do so via what is essentially a global variable.

An example: let's use BDD to test a sign-up procedure. A Lettuce-style feature might look like this:

Feature: Sign-up
In order to register with the site
As an unregistered user
I'll register an account

Scenario: register a new user
	Given my name is "Rutherford"
	And I am on the registration page
	When I fill out and submit the registration form
	Then I should be on the profile page
	And I should see my name

We might implement these steps like the following:

from lettuce import *
import mechanize
import os.path

@step('my name is "(.*)"')
def set_my_name(step, name):
    # save our name for subsequent steps
    world.reg_name = name

@step('I am on the registration page')
def on_reg_page(step):
    world.br = mechanize.Browser()
    # open login.html in the parent dir
    world.br.open('file://%s' % os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'signup.html')))
    # world.br is now primed for subsequent steps

@step('I fill out and submit the registration form')
def fill_out_form(step):
    # save our name for subsequent steps to use
    world.reg_name = 'Rutherford'

    # interact with the form and submit
    world.br.select_form(nr=0)
    world.br.form['username'] = world.reg_name
    world.br.form['password'] = 'hello'
    world.br.submit()

@step('I should be on the profile page')
def should_be_on_profile(step):
    # depends on world.br being seeded
    assert 'profile.html' in world.br.geturl()

@step('I should see my name')
def should_see_my_name(step):
    # depends on world.reg_name and world.br being seeded
    assert world.reg_name in world.br.response().read()
    world.br.response().seek(0)

In this example, login.html is a simple HTML document with a form that submits to profile.html. The latter document contains the sought-after string "Rutherford", and everything passes.

The point of this exercise is that this style of feature requires that step implementations do behind-the-scenes bookkeeping to support it. The step I should see my name assumes we know the name we're talking about, although this wasn't passed in to the step function via the feature language. Thus, under the restrictions we described above, there's no other way to get the name we set earlier, except via global-ish variables. world is exactly this.

Aside: I previously wrote that this passing mechanism is essentially a global variable, and not exactly a global variable. I point this out because we can imagine that step functions are methods of an object, and the object serves as context. However, this change is entirely syntactic; it does nothing to help the step function guarantee that it is invoked with the correct preconditions in the context, which in this case, is now the calling object.

You might say: "of course! You should be using step parameters, and putting everything into the feature language!" By that, I mean that the scenario body would look like this:

Scenario: register a new user
	Given I am on the registration page
	When I put "Rutherford" in the "username" field
	And I put "hello" in the "password" field
	And I submit the registration form
	Then I should be on the profile page
	And I should see "Rutherford"

This solves the problem by moving everything into the feature language, but once you start requiring larger sets of data, this quickly becomes untenable. Consider a form with a name, billing address, credit card info, and shipping address. That could be twelve or more fields, depending on how you represent addresses and names. Given that we've got to repeat each of those twice in the scenario (once to type it into a form, and once to assert it was echoed), we've blown our scenario out to something more than twenty lines. It may work, but it has defeated one of the purposes of using a DSL to emulate natural language specifications: it is no longer readable by anyone. It might as well be written in code.

My point is this: in order to use natural (meaning both "human" and "not stilted") language for features, we need to allow for step definitions to share state, i.e., some steps need to be able to communicate to subsequent steps. Further, current BDD toolkits don't do much to help with this.

Wait, wasn't this a shootout?

Oh yeah. Here's the catch with Lettuce: it never cleans up world.

Never. It just assumes anything you put there should be left there forever. This allows you to easily confuse downstream steps with upstream global pollution. Your triage procedure will probably rely on using hooks to unset individual variables in world.

Behave improves on this by passing a context object to each step function. This object acts a lot like a variable scope - it supports layered lookups at different levels. Thus, anything you set in the context at the "scenario" level will get popped off when the current scenario is exited. If you have some values that should be available for all features, then you can use the environment hooks to set the underlying context at a lower level.

Conclusion: use Behave

BDD offerings in Python are still relatively few and immature. I have several beefs with them, especially when applied to testing web applications. Overall, the state of BDD tools at the moment just doesn't feel right - integrating it with something as popular as Django is surprisingly fiddly and often requires undocumented off-roading (thanks, Mechanize). However, it can provide useful acceptance tests, to guarantee the user gets what they need, and systems tests, to guarantee units work together nicely. If those goals appeal to you, I'd suggest looking into Behave.