API testing with Cucumber

In our project we have a REST API using JSON that is used both internally by our mobile clients and offered externally to third parties.  Since we were consolidating all functional testing to Cucumber, it was natural to experiment what would be the best way to test API’s in Cucumber.

The common first instinct is to write out the requests, payload and response expectations directly in the feature files.  After all, a major point of Cucumber and ATDD is specification by example.  The problem is that this results in very verbose descriptions that are hard to follow.

Consider the following example given in the json_spec gem:

Feature: User API
  Scenario: Index action includes full user JSON
    Given the following user exists:
      | id | first_name | last_name |
      | 1  | Steve      | Richert   |
    And I visit "/users/1.json"
    And I keep the JSON response as "USER_1"
    When I visit "/users.json"
    Then the JSON response should be:
      """
      [
        %{USER_1}
      ]
      """

Consider that for a moment.  Do you understand what’s happening in this use case?  Next imagine giving this over to your business owner for a feature review.

One of the goals of Cucumber and specification by example is to provide a common language between developers and business. The above example contains so much detail that the intent of the use case is lost.  The code reminds me a lot of an example that Aslak Hellesøy gave of using the web_steps.rb “training wheels” originally included in the Cucumber-Rails gem:

Scenario: Successful login
  Given a user "Aslak" with password "xyz"
  And I am on the login page
  And I fill in "User name" with "Aslak"
  And I fill in "Password" with "xyz"
  When I press "Log in"
  Then I should see "Welcome, Aslak"

This kind of detail should be abstracted out of the scenarios into higher-level steps and by use of page objects.  Such refactoring makes the tests read better and much less brittle.

So if you don’t write explicit operations, what do you write?  Consider what a business owner would write as individual requirements.  Describe the intent of the API, not the explicit details.

I came up with a quite nice format in which to write the API tests.  As an example of registration:

Scenario:  Successful registration
   When I perform registration with the required parameters
   Then the request should be successful
    And I should receive a valid access token

This describes the essential functionality of the API, without any of the gory details.  The corresponding step definitions could be something like:

When /^I perform registration with the required parameters$/ do
  # Populate request information
  @request_method = "POST"
  @request_url = "/rest/register"
  @request_body = {email: generateEmail, password: "pass123",
      terms_accepted: true}
  @successful_response_code = 201
  @error_response_code_email_reserved = 409
end

Then /^the request should be successful$/ do
  # Perform the request and assert the response code
  @response = RestApi.call(@request_method, @request_url,
      body: @request_body, expect: @successful_response_code)
end

Then /^I should receive a valid access token$/ do
  # ... test that the token in @response is valid
end

The interesting part is that the actions (“When”) store fields to the World, and only the post-condition “Then the request should be successful” performs the request.  This allows adding more directives that modify the request, resulting in very natural language:

Scenario:  Failed registration; email address in use
   When I perform registration with the required parameters
    But the email address is already reserved
   Then the request should fail due to email reserved

Here the step definition for “But the email address is already reserved” would either register the email address in @request_body.email or set the email address to something that is known to be reserved.  The corresponding “should fail” step expects the response code in @error_response_code_email_reserved.

This approach also abstracts the setting up the request parameters.  When we needed to make a change to the mandatory registration parameters, there was only one method that needed to be changed.

This structure has been a great success in our project.  Very often I’ve noticed that after having the feature file defined, it takes only 5-10 minutes to implement the steps.  There’s a lot of step reuse, and often you only need to write the request setup step, and possibly some custom post-condition steps.

One caveat of this approach is that the exact methods, URLs and parameter names are not presented in the feature file, and thus it does not function as documentation on its own.  We have a separate API document, and we’ve been considering moving the corresponding parts to the free-text description area in the feature files.  This would not be executable specification, but I think it’s an acceptable compromise.  I feel it more important to keep the scenarios succinct and readable, and thus more maintainable.

About these ads
This entry was posted in Coding, Cucumber and tagged , , . Bookmark the permalink.

2 Responses to API testing with Cucumber

  1. Tris says:

    This is an interesting post with regards to the level of abstraction you are applying. I’ve been using Cucumber heavily for a couple of years for acceptance testing on web UI’s using Selenium, and based on that have always followed a similar approach – attempting to describe the interactions at a higher level using only language that would make sense to an average person.

    I am currently looking at testing a REST API (which has led me to this blog post), and would by inclination follow the same path you have described here.
    However, I’ve also just taken a look at the Cucumber book (written by Matt Wynne and Aslak Hellesoy) and their examples of REST API testing contain expected JSON as a part of the scenario. The reason given is that the purpose of a Cucumber scenario is to be readable by the stakeholder, which in the case of an API you can expect to be a technical person.

    I have to admit, I find that to be something of an about face on all the guidelines on correct Cucumber usage that can be found elsewhere.

    • Sampo N. says:

      There is a valid argument for having the API details in the scenarios. However, I find that using Cucumber / ATDD has two primary benefits:

      Initially, it facilitates and encourages communication between developers and business owners. A business owner wants to know that you can register an account through the API and what the validation rules are; they are not interested in the gory details.

      Later on, the feature files function as living documentation. But documentation is useful only if it is read. In the case of APIs, you cannot replace proper API documentation with examples. You don’t want to read through scenarios looking for the proper input and ouput parameters, you want them to be in a single table with descriptions.

      Whenever I try to read through a scenario containing query URLs and JSON responses, my eyes start to glaze over. One or two examples with full responses may be useful, but for demonstrating validation rules they are just in the way. You can also do both ways: it’s easy to write a shared step definition “When I perform a GET request to /xyz” that initializes the World parameters accordingly, and in latter scenarios use the higher abstraction layer.

      Ideally, we would have a DSL that allows you to write “traditional” API documentation with parameter descriptions in tables, tie those to usage scenarios and run those as executable specifications. I’m not sure if the Gherkin language is flexible enough for that. Until we have such a language, I find it better to keep the scenarios on a use-case detail level, and describe the details elsewhere (for example the comment section).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s