Traditionally, testing followed a test plan. A mighty document listing what would be tested, and by whom, and how.
Recently I got asked: what might a DevOps test plan look like?
An interesting question really, since the testing landscape has changed considerably. How would a test plan fit in there?
Would there even still be a test plan?
Well, yes and no.
There wouldn’t be a proper Very Important Document headlined “Software Test Plan”, as was the case when I had my first job as a tester, fresh out of university.
But there would be an implicit test plan: it just takes the shape of automated test scripts, CI configuration files, or similar description through code.
This is a very nice kind of test plan of course, since it’s always complete and true by definition – after all, this is exactly what the test automation would execute.
But this is just a superficial difference. There are much more important ones:
- test automation has changed the way we look at tests
- continuous integration has increased the testing frequency
- since the scope of what a DevOps team works on, the system under test became a lot larger
Let’s examine those aspects in more detail.
test automation and CI
If all (or most) tests are automated, then your test plan suddenly becomes very simple: just run everything, always. There’s really no reason not to. OK, I’m being a bit glib there, because it might be too expensive or resource-intensive or time-consuming to run all the tests, but surely there’s a meaningful subset that could be run many many times per day.
And surely there’s no reason the full test suite can’t at least be run nightly.
So figuring out when to run the tests is replaced by figuring out what to do about that firehose of feedback you’re subjecting yourself to. Instead of test planning, you’ll do something like test result planning.
The System Under Test just grew by an order of magnitude
Maybe all the system components are still the same. But if you’re a DevOps team, and take responsibility for successfully delivering functionality to the customer, you’re not just testing your application code anymore.
Now you’re also testing
- your operations infrastructure
- your deployment automation
- your monitoring
- your feedback loops
- your organisation’s communications structures
This means that not only has the sheer amount of “stuff” grown, but you’ve also inherited new non-code aspects to your system which require testing. A large process aspect has been added. Different systems or components working together to create the product, deliver it along the pipeline, observe and tend to it in production.
Notably, humans are now part of the system, as those who watch it, tend to it, improve it or fix things that turn out to be broken.
The DevOps test plan
So, finally, test planning takes a different shape: it’s not about distributing the work anymore (that gets done by machines principally, anyway). It’s about delineating the increasingly fuzzy boundaries of the growing system under test, and trying to make sure that all of the important questions are being asked.
The questions of what would be tested, and by whom, and how are still asked through a test plan (by whatever name you give it), but the answers are very different now than they used to be.