When you have a test plan with lots of different tests, does it matter what order you run them in?

The answer to this question is that it depends. Firstly, if you don't expect to have time to run all of the tests then you should run the more important tests first. That way when you do run out of time the tests that are still outstanding that you didn't have time to run should be those that test for conditions that will hopefully never arise.


The order you run your tests in might also make a difference to how long the testing will take. The reason for this is that if you have to amend the script because one of your tests failed to produce the correct result then you don't just need to rerun that test, you also need to rerun all of the other tests that might be impacted by the changed code.

Ideally then you should run the tests that are going to fail first so that you reduce the need to rerun the tests that passed prior to rectifying the failure. Of course you don't expect any of the tests to fail which makes it impossible to run the tests in that order.

We need some other way to order our tests so as to try to minimise the number of reruns even though we do not know prior to testing just which tests will fail and require the code to be modified.

Another consideration is that if we list all of the tests in their order of importance (based on how likely the particular path is to occur and how easily that our visitor can force the code to follow a less common path) then we may end up with tests at the end of that list where it will not be worth fixing the code if the test fails.

With those tests we are simply running the test so as to know whether the test will pass or fail as the input conditions for the test are assumed to be such that it should never follow that path anyway.

Ideally we will want to run the entire series of tests through once without needing any changes to be made to the code that will require any tests to be rerun. So let's start working out our testing order backwards by deciding that all of the tests will be run once in decreasing order of importance after we have rectified all of the code needed to resolve failed tests that we decide are not allowed to fail.

So if that will be our last series of tests then if we are fortunate to have no failures then that also needs to be our only series of tests. So we'll run our tests in decreasing order of importance and provided that any failed tests are sufficiently far down the list to not require fixing then we will be done once we do the one sequence of tests.

What do we do though if a test higher up in the sequence fails and we need to change the code to fix it? Well in that case we are going to need to run all of the tests again and assuming our change doesn't cause a prior test that passed to now fail we will only need to rerun those prior tests the once.

Of course if we return to the top of the list of tests and restart from there after fixing each failed test then the tests at the top of the list will be rerun after we fix the code to resolve each failed test. To reduce the number of times we need to rerun tests we should not rerun the tests from the start after fixing a failed test so that it now passes. Instead we should continue down the list and run the tests that haven't yet been run. These tests will be far more likely to uncover further failures than rerunning the tests that already passed since we are more likely to have issues with tests that haven't yet been run than we are with tests that passed before the (hopefully) minor code changes required to resolve a failed test.

Once we have run all the tests once and have rerun only those tests which failed and required coding changes to make them pass then we need only rerun the tests from the top of the list down to the test that immediately preceded the last code change. As long as all of those tests pass we will have achieved our objective of running all the tests once against the final code with their being no failures that require the code to be further modified before it goes live.

Of course if we are unfortunate enough to have another test fail during this rerun of the more important tests and we need to make further code changes then we'll need to run the entire series through to the end and then again from the start through to the test immediately preceding the failure but this should only happen if the correction we made to fix a failure in the first pass through the tests was incorrect.

By following this sequence for our planned testing we should keep the number of times that we need to rerun tests that pass to a minimum.