I this series, I want to touch on one of the biggest traps people fall into with test automation: writing too many high-level tests. I have made this painful mistake and struggled with constant test failures and spent many hours troubleshooting things that weren’t even problems in production. They were just bad (flaky) tests. I finally found my way out of that mess, and hopefully I can help you do the same.
For the beginners here, I’ll start with what levels you can write tests at and why lower level tests are more valuable. I’ll show you why testing through the UI layer is so painful, and how to push higher level tests down.
I’ll try to keep each post short and to the point. Without further ado, here is part 1.
Levels of Testing
There many different ways to test your code, but they can all be boiled down into three main categories or levels.
FYI: The names of the levels may differ based on who you are talking to, but the underlying concepts are the same. Seems no one can agree on what the best names are for these things.
Unit Tests
Unit tests are the lowest level tests. These tests are very focused and usually only test a few lines of code. They run completely in-memory and never access external resources like disk or network so they are very fast. Because of this, you can run them often and get very fast feedback on errors. I run mine every few minutes. When they fail, I can usually just hit CTRL-Z to undo my last few changes and they are passing again. No need to debug!
Even when they fail later (when I can’t just hit CTRL-Z), the problem is usually obvious. There’s only a few lines of code it tests, so I don’t have to look far for the problem. For the same reason, there’s only a few things that could actually cause the test to fail, so they don’t fail that often.
Unit Tests are very low cost. Easy to write. Easy to maintain.
Integration Tests
Integration tests validate the interactions of multiple classes with each other, and the environment (like the disk, network, databases, etc.). These tests are inherently slower than unit tests, so you can’t run them as often. This means it may take longer before you realize you introduced an error.
Plus, since they cover much more code, there are more reasons these tests can fail, so they tend to fail more often than unit tests. And when they do, sifting through more code means it takes longer to figure out where the problem is.
Integration tests take more effort to build and maintain than unit tests. They are harder to debug, and take longer to identify issues.
UI or End-to-End Tests
These tests are sometimes called “functional” tests as well. They test the fully deployed system as a black box, just like a user would use it. These usually interact directly with the UI.
These tests exercise all the code in the system, from the UI down to the database. It also exercises all the third party resources or external systems as well. So if anything in this chain breaks, the test will fail. And when they fail, because so many things can cause it to fail, it’s often very hard to determine what caused the failure. I often find myself sifting through log files and logging in to remote servers to figure out what the heck broke. Not fun.
These tests are also the slowest to run, so they aren’t run very often at all. So when you introduce an error, it may be a long time before you realize it. By then you have moved on to something else so it takes additional effort to get your head back around the problem so you can debug it.
These tests are brittle, and very difficult to maintain. They have the highest cost of all the types of tests you can write. They do have value, but it comes with a cost.
The Obvious Conclusion
So, given what you just read, where is the most valuable place to focus your testing efforts? Yeah, I don’t even need to give you the answer. It’s pretty self-evident:
Always test at the lowest level possible.
When you have tests at a high level, it’s best to “push them down” as far as you can. That’s what this series is all about.
So, take a look at your tests. Where have you focused all your efforts? Are you fighting to keep the tests running? Is there any correlation between the two? If so, stay tuned, we’ll look at ways to fix this.
