Filed under: Performance Monitoring
Finding bugs can be an expensive exercise. When you factor when during the SDLC that the bug was found and who found it, the cost of a bug goes up. Bugs found early in the development process are the cheapest. Bugs found later in a development cycle have more of an impact as there has been a greater amount of time and people spent to expose the bug. Bugs found by customers are the most expensive of all.
Bugs found early in the SDLC are fixed early and they are generally more isolated. This can begin with Requirements and the fix is to make a change to the requirements. Bugs found in the construction phase are usually found by the developer coding or even during a code review. These bugs only take the developers time to find and fix. Sometimes, more importantly, the code around the bug is still top of mind for the developer. The time from finding to fixing is short… the cost is minimal.
Bugs found during the testing stage are more expensive to find and fix. This is especially true if it takes a team of testers running manual tests. These test runs, although broader in coverage, often take days to complete. Bugs found must be then prioritized and fixed which again will require developers time. The fix must be integrated and verified which requires more time, and not to mention the time spent to regress the code to insure that the fix did not cause more bugs… more people and more time means a greater cost.
The most expensive bug is a bug found by a customer. Bug reports from users usually involve a support specialist’s time (to figure out what the issues actually is) as well as testers and developers to analyze, fix and verify. Also, since it was a customer that found a bug, it has the potential to change their perception of the product, which can lead to them leaving the product altogether.
In order to reduce the cost of finding bugs they need to be found early and as quickly as possible. If testing is done manually, it tends to be the bottleneck in the process. As the loop between development and customer usage lengthens, bugs found later in the cycle actually become more expensive, as new work is built on defective code. To avoid this, testing needs to be broad and deep, and executed as early as possible. The only method to achieve this is through automated testing.
To implement automated testing can also be costly and a difficult exercise but can reduce the cost of bugs and improve the quality of the product. We’ve made this journey over the last 3 months, and while we’re certainly not done, we’d like to share with you what we’ve learned. Tomorrow, we’ll look at how to set up Jenkins to test our Java web app and some of the lessons learned while doing so.