ASI

Question: “Have you been involved in calculating the ROI on Automation testing projects. Generally it has been found that the right ROI can be achieved only in long term projects spanning 2-5 years. As it takes a long time to understand the application and then develop a suite of test scripts, which provides benefits in later stages of the projects. I would like to know that what was the size of your projects and what kind of benefits you gained after some years of automation work in the project from time, money etc.?”

My Response: “This answer is based on Functional Test Automation tools like QuickTest Pro and WinRunner – not load testing which is different. I’ve implemented HP/Mercury tools since 1992. Yes you can gain ROI on short term projects! The major areas of benefit are: 1. Data driven testing 2. Regression testing 3. Multiple test cycles. For the ROI calculation simply time a manual tester performing a single iteration of the testing task that the test script will execute and do this a couple of times to get an average. Then average out the cost for the QA staff. Now run a couple of iterations with the tool to come up with the average speed the tool is taking. On average I have always found the automation tool (QuickTest Pro is my tool) to run a minimum of 33% faster than manual testers, and as fast as 74% in some cases. How is this possible? The more detail that the tester has to check, the faster the tool will perform in comparison to the effort a manual tester takes to perform the exact same checks/tasks. A very simple test of just entering data with no other checks equates to 33% faster. For a test where the tester has to do a lot of test validation for each test step will slow down a tester and the tool will normally out perform the tester in speed. Example data: I will use the average salary for a Software Quality Assurance professional to be $75,000 annual salary => about $288 a day (based on 260 workdays in a year) => which comes out to cost about $36 an hour. Real-world example: I had a data driven test script that had 960 rows of data (iterations) to execute. The reality is, an actual person would never take the time or have the patience to run through the application 960 times using different data each time. However this one test within the first 16 hours of execution revealed two major defects in the application that the team was having a real hard time duplicating, but now they had the actual steps and data that caused the issues. Now if we apply the 33% time savings against this example, it would come out like this:
  – 960 iterations in 16 hours equals 60 iterations an hour which is 1 minute per iteration by the tool.
  – If this was a 33% savings in time, then a manual tester takes 33% longer to do the same task which is about 1 minute and 20 seconds (80 seconds total).
  – 960 iterations x 80 seconds = 76800 seconds / 60 = 1280 minute / 60 = 21.3 hours for a manual tester.
  – In Project Management, normally a human resource (developer, tester, DBA, SysAdmin) will be forecasted at 6 hours of actual work a day due to meetings and other responsibilities consuming time. 21.3 hours / 6 hours a day = 3.5 days to get this same test effort accomplished.
  – Although the automation tool savings was only 33%, the test effort was actually accomplished in a single day, versus a manual tester would still take 2.5 more days to finish the work. Based on the financial cost mentioned above, the 2.5 additional days is a savings of $720 (calculated at (($288 * 2)+($288/2))). So basically this individual test script would save the company $720 every time it was run. If this test script is normally run multiple times due to multiple test cycles and future projects, then this can be a substantial savings. So you have a decent savings for the short-term ($720 for a single execution) and it could then equate to a lot of money if there are multiple test cycles and then when it is used in regression testing during other projects.
 The ROI on test automation can be very surprising and a lot of people initially want to dismiss the answer. But if you are honest with how much is truly being tested, how long it really takes people to test then you will recognize the real savings. Also keep in mind that it is not only saving you money, but most likely it is performing testing that would never get down otherwise!!  Finally there is a cost savings for the defect found.  This is company and defect specific so it can not be averaged but in our example the two defects were substantial as they ultimately lead to two different backend servers crashing.  This could be costing the company substantial money in support costs and reputation.  So now it is possible that the very first test execution run saving you $720 may actually be saving you substantially more.  Also keep in mind that this is also just “1″ test script saving you money, when you start adding up the savings on all the other test scripts you develop the savings increases significantly.
 Note: I have also automated processes that when using this formula actually produced a savings as great as $9,390 per process run which equated to a monthly savings of $150,240 based on how many times they ran the test script.