How to reduce regression testing effort by 50%?
Examples of a manual regression test strategy when automated tests are not in place
Two months moving to a corporate development environment, I had my first-hand experience doing whole month of nothing except manual regression.
Project Status
A complex video streaming service which was recently taken back from a vendor. The code quality is poor. Almost no coverage of automated tests. The in-house team owns the server code and desktop code. The other vendors own code for Big Screen(Samsung, Sony, LG TV…etc) and Mobile platforms iOS, and Android.
Before the releases from other vendors go live, the in-house team is expected to perform regression tests on the given releases.
Effort for Full Regression
For a recent Big Screen Release, there were 2000+ test cases in total to run from:
- 300+ test cases/ TV
- 7–8 different TVs
Team
A few testers and other are of developer background. Therefore detailed test cases were written so that every person can help regression testing (manual checking) given this release was done by an outside vendor.
Team Mood
Genuinely unhappy. Everyone found running those test cases tedious. People fighting for every opportunity to work on anything other than regression testing, that contest further drives down team morale.
Questions worth pondering
How do we make this regression testing less painful?
Some may argue that the pain is necessary before automated tests are in place and therefore we need to run the whole suite of regression test case to have the confidence.
In this case, another question worth asking is:
Is running the whole suite of regression test cases enough?
The regression test cases for any given reasonable sized software are no where near exhaustive, complete full scenarios. It is impossible to test everything.
There are always test scenarios not captured in the existing test cases. Are the test scenarios not captured in the regression test cases not important? If 300 test cases were written in the regression suite, we run 300 for this release. If we had more time to write 400 test cases so we run 400 test cases to complete regression test suite?
Is the scope of regression testing determined by the number of pre-written test cases?
How about the test scenarios not covered by existing test cases? Are they not as important? How do we know?
Every release is different. The testing effort depends on the context.
Could it be that some unwritten tests cases are more relevant for this release but not executed?
Yes I believe it is totally possible. The prewritten regression suite does not capture the difference in the risks in each release. There could be unwritten yet important test scenarios not explored simply because the test executors ‘only follow test cases’.
It is impossible to write test cases covering all possible scenarios. However, we can reduce the likelihood by making a conscious evaluation for each release to delegate more effort on the relevant, risky area.
In my opinion, I do not believe release testing scope is determined by the number of pre-written test cases in the regression suite. Every release is different. It is better to make a conscious decision for each release on the potential
How do we plan testing effort for each release?
Testing is an activity with trade-off between time, money (resources) and system covered. For a given release, we want to know we uncover the most risky issues early to allow time for the bug to be fixed without delaying the release.
Given the diminishing returns in the testing effort, we need to draw a line to stop testing and release by accepting a certain level of risk and minor bugs.
To put theory into practice with our Big Screen release:
Risk:
- The fundamental core feature such as login/ signup/ playing video
- The code written specifically for each device such as Network issues handling, video playback
The risk area identified requires testing across each and every device otherwise we would not have the confidence knowing those area work properly.
Less Risk:
- The system behaviour written with common code for all the devices
- The test scenario where it concerns more in terms of existing unchanged business logic, less concerning CSS rendering
The less risky area are the logics written with common code for all the devices, that is less to do with CSS rendering on the screen. We could be 95% confident (made the number up) that once we tested concurrency handling, forget password etc on one device that they will function as expectedly on all other devices. By spending additional efforts on 1 more device could probably only add 1% more confidence (again the number was made up).
That is the idea. By analyzing each area of interest in the system, we could quickly identify the risky area where we should test early, across all devices to find potential issues. We could also identify less risky area where it is less cost effective to test across all devices value and reduce the overall testing effort by more than 50% while getting similar level of confidence.
Saying “we need to run these test cases because they are in the regression test suite” is a bad justification why you spend this testing effort.
Saying “Due to X changes and Y risks, we made the decision to test A, B, C area to release with confidence within the given period” is a much more convincing reason.
All in all, I believe it is beneficial to have a test entrance meeting with relevant stakeholders (3 amigos!) before devoting huge resources for running full regression tests across all devices for 1 month.
Developers provide technical insights into risks and PMs provide risk tolerance indication from business perspectives. By creating a test strategy with inputs from devs and PMs (and other relevant parties), and agreeing on the test coverage, risk intake for each release; we could optimise team resources to deliver most the value.
With careful planned test strategy, you could strike a better balance between cost and value and potentially save 50% regression testing effort
More Resources
- A simple visualisation showing testing process — reality vs what we think we know https://thetestinggraph.net/
- James Bach’s reasoning on common misconceptions on testing — There are lots of gems in his blogs. I am always amazed on how he could presenting his arguments on different facets of testing so well. https://www.satisfice.com/blog/archives/1728