Acceptance Criteria vs. Acceptance Tests: Minimize your documentation

To know that a story is done, we create a set of acceptance tests (story tests) that, when they pass, we know the story is complete and functions as expected.

The process we follow for defining our acceptance tests is comprised of two steps.

1. We first come up with high level acceptance criteria.
2. Right before, or during development, we use the criteria to define the actual acceptance tests.

You’ll notice that we make a distinction between Acceptance Criteria and the Acceptance Tests. Here are the definitions as I see them:

• Acceptance Criteria is the minimal documentation that ensures a sufficient implementation of acceptance tests.
• Acceptance Tests are the detailed specification of the system’s behavior for all meaningful scenarios, used to assert its correctness.

Acceptance Criteria

• Ensures the team knows when they are done
• Ensures the team does not forget important edge cases and considerations
• Produced through collaboration of the developers, testers and product owners (3 amigos)
• Created prior to development, during planning phase
• Expressed at a high level (conceptual, not detailed)
• Expressed in whatever form works best for the team…keep it minimal
• Considers edge cases and failure scenarios
• Keep it concise (minimum documentation needed by team…may be more for one team, less for another)

Acceptance Criteria is the minimal amount of documentation you need to specify the expected behavior of the feature and any edge cases that need to be considered. Agile favors working software over comprehensive documentation. This is an expression of that principle. We don’t flesh out every possible example, but provide “just enough” documentation to ensure the correct tests get written. The amount of documentation necessary may vary from team to team. We use a simple bulleted list to capture edge cases and things we need to consider. If your team is distributed, you may need more documentation. Teams that are co-located with testers and product owners (like us) need less.

Acceptance Tests

• Defines behavior of system
• Ensures the feature works as expected
• Code implemented by developers and testers
• Test definition can include product owners or customers if you are using a DSL that decouples the definition of a test from its implementation code (like with Gherkin or FitNesse)
• Test definition can happen before development if using a DSL as mentioned above
• Test implementation occurs during development (ideally in a test-first manner)
• Tests are usually implemented in code, but if testing manually (hopefully only rarely), the “implementation” can be a list of steps/expectations

The Acceptance Tests are the fully detailed expression of the tests….their implementation. We use SpecFlow (which is the .NET equivalent of Cucumber). It allows us to specify Given-When-Then scenarios in the Gherkin syntax. We work with our product owners to create the actual test definitions in Gherkin, then developers pair with testers to implement them. When a test is not feasible to automate, we still develop the Gherkin and can simply follow it as a list of steps to run the test manually.

I have found it useful to conceptually separate the documentation of the tests (Acceptance Criteria) from the implementation (the actual Acceptance Tests). It helps me to remember not to over-specify the tests. As long as the documentation is enough to ensure sufficient tests will be implemented, I don’t need to add more detail.