Behavior-Driven Development (BDD) in Pythonby Michael Armida on Fri, 12 Oct 2012
The Behave people have done an excellent job of fairly comparing their project to others' on their own site. For me, the only real alternative was Lettuce.
When I first read Behave's criticism of Lettuce, the lack of automatic
world had zero impact on me. I hadn't yet written many
features. In fact, I assumed this was a largely
irrelevant feature in Behave, since
context was essentially a
place for global variables, and global variables are bad. It wasn't until I
had some more first-hand experience that I realized something important about
working with typical BDD tools.
BDD is unforgivingly stateful
Most typical BDD tools - even those in Ruby like Cucumber - saddle you with this basic limitation: you must manage your own state between steps. Because they all match lines of the Domain Specific Language (steps) to bits of code (usually functions), there isn't much variation in how you can pass state between these functions. If all steps' formal args come from patterns extracted from the DSL step strings, and a step function depends on something being communicated from a prior step function, then it must do so via what is essentially a global variable.
An example: let's use BDD to test a sign-up procedure. A Lettuce-style feature might look like this:
Feature: Sign-up In order to register with the site As an unregistered user I'll register an account Scenario: register a new user Given my name is "Rutherford" And I am on the registration page When I fill out and submit the registration form Then I should be on the profile page And I should see my name
We might implement these steps like the following:
from lettuce import * import mechanize import os.path @step('my name is "(.*)"') def set_my_name(step, name): # save our name for subsequent steps world.reg_name = name @step('I am on the registration page') def on_reg_page(step): world.br = mechanize.Browser() # open login.html in the parent dir world.br.open('file://%s' % os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'signup.html'))) # world.br is now primed for subsequent steps @step('I fill out and submit the registration form') def fill_out_form(step): # save our name for subsequent steps to use world.reg_name = 'Rutherford' # interact with the form and submit world.br.select_form(nr=0) world.br.form['username'] = world.reg_name world.br.form['password'] = 'hello' world.br.submit() @step('I should be on the profile page') def should_be_on_profile(step): # depends on world.br being seeded assert 'profile.html' in world.br.geturl() @step('I should see my name') def should_see_my_name(step): # depends on world.reg_name and world.br being seeded assert world.reg_name in world.br.response().read() world.br.response().seek(0)
In this example,
login.html is a simple HTML document with a
form that submits to
profile.html. The latter document contains
the sought-after string "Rutherford", and everything passes.
The point of this exercise is that this style of feature requires that step
implementations do behind-the-scenes bookkeeping to support it. The step
I should see my name assumes we know the name we're talking about,
although this wasn't passed in to the step function via the feature language.
Thus, under the restrictions we described above, there's no other way to get
the name we set earlier, except via global-ish variables.
is exactly this.
Aside: I previously wrote that this passing mechanism is essentially a global variable, and not exactly a global variable. I point this out because we can imagine that step functions are methods of an object, and the object serves as context. However, this change is entirely syntactic; it does nothing to help the step function guarantee that it is invoked with the correct preconditions in the context, which in this case, is now the calling object.
You might say: "of course! You should be using step parameters, and putting everything into the feature language!" By that, I mean that the scenario body would look like this:
Scenario: register a new user Given I am on the registration page When I put "Rutherford" in the "username" field And I put "hello" in the "password" field And I submit the registration form Then I should be on the profile page And I should see "Rutherford"
This solves the problem by moving everything into the feature language, but once you start requiring larger sets of data, this quickly becomes untenable. Consider a form with a name, billing address, credit card info, and shipping address. That could be twelve or more fields, depending on how you represent addresses and names. Given that we've got to repeat each of those twice in the scenario (once to type it into a form, and once to assert it was echoed), we've blown our scenario out to something more than twenty lines. It may work, but it has defeated one of the purposes of using a DSL to emulate natural language specifications: it is no longer readable by anyone. It might as well be written in code.
My point is this: in order to use natural (meaning both "human" and "not stilted") language for features, we need to allow for step definitions to share state, i.e., some steps need to be able to communicate to subsequent steps. Further, current BDD toolkits don't do much to help with this.
Wait, wasn't this a shootout?
Oh yeah. Here's the catch with Lettuce: it never cleans up
Never. It just assumes anything you put there should be left there forever.
This allows you to easily confuse downstream steps with upstream global
pollution. Your triage procedure will probably rely on using hooks to unset
individual variables in
Behave improves on this by passing a context object to each step function. This object acts a lot like a variable scope - it supports layered lookups at different levels. Thus, anything you set in the context at the "scenario" level will get popped off when the current scenario is exited. If you have some values that should be available for all features, then you can use the environment hooks to set the underlying context at a lower level.
Conclusion: use Behave
BDD offerings in Python are still relatively few and immature. I have several beefs with them, especially when applied to testing web applications. Overall, the state of BDD tools at the moment just doesn't feel right - integrating it with something as popular as Django is surprisingly fiddly and often requires undocumented off-roading (thanks, Mechanize). However, it can provide useful acceptance tests, to guarantee the user gets what they need, and systems tests, to guarantee units work together nicely. If those goals appeal to you, I'd suggest looking into Behave.