ASI

Question: “Have you been involved in calculating the ROI on Automation testing projects. Generally it has been found that the right ROI can be achieved only in long term projects spanning 2-5 years. As it takes a long time to understand the application and then develop a suite of test scripts, which provides benefits in later stages of the projects. I would like to know that what was the size of your projects and what kind of benefits you gained after some years of automation work in the project from time, money etc.?”

My Response:  I’ve implemented HP/Mercury tools since 1994, and “YES” you can gain ROI on short term projects!  This answer is based on Functional Test Automation tools like QuickTest Pro and WinRunner – not load testing which is different.  The major areas of benefit are:

1. Data driven testing
2. Regression testing
3. Multiple test cycles.

For the ROI calculation, you need to perform a few tasks:

#1 - Simply time a manual tester performing two or three iterations of the testing task that the test script will execute, in order to get an average.

#2 – Average out the cost for your QA staff. 

#3 – Run a couple of iterations of the test script that has been automated performing the same test case in order to come up with the average speed the tool is taking to execute the test case.

#4 – Finally do the math that I describe below to find your cost savings.

On average I have always found that the automation tool (QuickTest Pro is my tool of choice) to run a minimum of 33% faster than manual testers, and significantly faster in many cases.  How is this possible?  The more detail that the tester has to check, the faster the tool will perform in comparison to the effort a manual tester takes to perform the exact same checks/tasks.  A very simple test of just entering data with no other checks equates to 33% faster.  For a test where the tester has to do a lot of test validation or perform verification calculations for each test step, this will greatly slow down a tester versus the tool will normally out perform the tester in speed in these tasks.  A classic example of an automation script I recently developed was to validate an output text file that contained numerous lines of data that were various sub-totals and grand-totals of various categories from the data that was processed on the backend coming from the database.  The manual tester had to validate many of these files each day and process through the file making sure all the totals were correct based and the sub-totals added up to the total that was displayed.  On average the files contained 700 lines of data and averaged out to 371 checks that had to be made.  This task was very monotonous, tedious and took the manual tester about 40 minutes to perform.  Once automated with QuickTest Pro it averaged 17 seconds to run and perform all these checks.  This is a real world example with huge time savings!

Data Formula Example to find the ROI:  I will use the average salary for a Software Quality Assurance professional to be $75,000 annual salary => about $288 a day (based on 260 workdays in a year) => which comes out to cost about $36 an hour.  Another real-world example: I had a data driven test script that had 960 rows of data (iterations) to execute.  The reality is, an actual person would never take the time or have the patience to run through the application 960 times using different data each time.  However this one automation test script within the first 16 hours of execution revealed two major defects in the application that the team was having a real hard time duplicating, but now they had the actual steps and data that caused the issues.  Now if we apply the 33% time savings against this example, it would come out like this:
  – 960 iterations in 16 hours equals 60 iterations an hour which is 1 minute per iteration by the tool.
  – If this was a 33% savings in time, then a manual tester takes 33% longer to do the same task which is about 1 minute and 20 seconds (80 seconds total).
  – 960 iterations x 80 seconds = 76800 seconds / 60 = 1280 minute / 60 = 21.3 hours for a manual tester.
  – In Project Management, normally a human resource (developer, tester, DBA, SysAdmin) will be forecasted at 6 hours of actual work a day due to meetings and other responsibilities consuming their time.   So 21.3 hours / 6 hours a day = 3.5 days to get this same test effort accomplished.
  – Although the automation tool savings was only 33%, the test effort was actually accomplished in a single day, versus a manual tester would still take 2.5 more days to finish the work.  Based on the financial cost mentioned above, the 2.5 additional days is a savings of $720 (calculated at (($288 * 2)+($288/2))). So basically this individual test script would save the company $720 every time it was run.  If this test script is normally run multiple times due to multiple test cycles and future projects, then this can be a substantial savings.  You end up with a decent savings for the short-term ($720 for a single execution) and it could then equate to a lot of money if there are multiple test cycles when it is used in regression testing during other projects.

The ROI on test automation can be very surprising and a lot of people initially want to dismiss the answer since it seems too big to be true.  But if you are honest with how much is truly being tested, how long it really takes people to test, then you will recognize the real savings.  Also keep in mind that it is not only saving you money, but most likely it is performing testing that would never get done otherwise!!  Finally there is a cost savings for any defects found.  This savings is company and defect specific so it can not be averaged but in our example the two defects were substantial as they ultimately lead to two different backend servers crashing that they couldn’t reproduce but it happened enough to create concern.  This kind of issue could be costing the company substantial money in support costs and reputation.  But now it is possible that the very first test execution run saving you $720 may actually be saving you substantially more.  Also keep in mind that this is also just “1″ test script saving you money, when you start adding up the savings on all the other test scripts you develop the savings increases significantly.

Savings on other real world projects:

1. I automated a process that produced a savings of $9,390 per process run (due to the huge time savings) which equated to a monthly savings of $150,240 based on how many times they ran the test script.
2. In regards to the first example at the top of this article about the data file being validated in 17 seconds, it came out to a cost savings of $2,016 a month as this test effort was performed 4 times a day, 7 days a week.  This equated to an annual savings of $24,192.

So “YES” you can gain ROI in the short-term with smaller projects!