ASI

An interesting Test Management topic came up on LinkedIn that I responded to and thought I’d expand on it on my blog site. The question was “How can a test manager provide an accurate estimation like time, cost, resources for the test case creation, testing like Functional, regression and system testing for a large project?”

For big projects I like to follow the process of time estimate refinement where you start with an “E0″ estimate which is +/- 100% (because this is based off of the initial business request (BRD) and lacks detail), then it is followed with an “E1″ which is typically +/- 50% because it should be based off detailed Functional Requirements and a Software Requirements Specification document (SRS). Once a Design Document and a technical approach discussed the estimate can be finalized with an “E2″ which is +/- 10% (some environments don’t bother with the E2 and require the E1 to be more specific because they will jump to the tech spec).

Some people talk about using “industry averages” in estimating because they don’t want to take the time to proper define the requirements, design and approach. They just want to move as quick as possible into the Coding phase; this is another discussion in itself, so back to the topic…I don’t like using “industry averages” as they tend to be too generic and quite often recommend less time than what is really needed. Applications today are more complex than ever and if a Test Engineer is to do a thorough in-depth test effort, it will take a lot of time to test. A lot of companies will make no effort to really figure out the true test effort and go with the industry average that QA will take 30-40% of the development time. However a developer could take 10 minutes to make a change in a critical part of the system (like the report engine of a financial app) and this would imply that QA time would only be 4 minutes which everyone would laugh at. Although they should laugh at the fact that the developer only spent 10 minutes developing before turning the project over to QA – what happened to the Unit Testing? So obviously industry averages are not very useful unless it is a big project and you are pressured to throw out an estimate for some meeting – then go with the 40% of development time. Otherwise if you have the ability to do your job correctly then put some meaningful effort into looking at past test efforts.

My opinion is that proper time estimation is best done by diligently tracking actual time for these elements in your current environment in a test management tool. I’ve always used the Quality Center and in that tool I have customized it to track various test execution times. The bulk of your “time estimation” consists of actual test execution time and if you have a good modular design for your application you can then draw on actual past experience to know how long a task like regression testing took in the past and the various functional testing tasks. Every application & environment is different so when managers change jobs they have to start this process over again to establish the duration times for the application at hand. Couple of pointers:
- Test Cases: if you use a test management tool, then you should create a modular structure that enables you to create and REUSE test cases. Therefore your test case development time estimate will change for each project. For enhancement and maintenance releases this time estimate should be minimal since you should be able to capitalize on existing test cases already created in the past and reuse them. For bigger projects that have lots of new functionality then you should refer back to similar projects in the past and see how much time was needed back then to create the variety of new test cases.
- Test Execution: In the Quality Center I have customized it to track overall project test effort times besides the individual test case execution times. This way I can once again refer back to actual past test efforts and see how long something took to test (especially when you have a project that modifies functionality that you have tested in the past). So during our process, when we write the Test Plan we include what testing will be done in each of the test phases. Once that has been established we then reference past test runs and see how long it took to test the same or similar functionality in the past and that becomes our basis for the current time estimates. This way our time estimates are not “feels like” times but timings that are based on real test efforts.
- Bug fixing: this effort is based on past experience with specific development teams. Every developer is different so you will need to apply past project experiences with specific developers to current efforts. Once again using a tool like the Quality Center with some customization in place, we can generate reports on specific project efforts and see how many defects were generated and come up with averages on what we expect with similar projects (based on functionality and developer). This helps with estimating the additional test cycles required during the bug fixing phase. By tracking defects generated by developer, you will have a good idea how “clean”/bug free the code will probably be based off past trends. This really helps you to estimate how many test cycle iterations you may have to execute.

Bottom line is that a test management tool that is customized to your environment can greatly help provide very useful time estimates; but it doesn’t do this on its own. The test team as a whole (testers, team leads and managers), have to all be diligent in documenting all the necessary details during all the various testing tasks. A little extra time and effort during the testing process greatly benefits the team down the road in planning future test efforts.

Now having more accurate time estimates will enable you to apply cost to the process much more realistically because you now know how long a project will take (“man-hours”) and you can then decide on the number of resources you want to apply to reduce the timeline (as long as they are available in the timeframe you are looking at).