ASI

This is part of my series on “What is QA really all about?”  I find that even today after all these years of sophistication in software development that organizations still don’t really understand what QA is really all about and what is truly involved in verifying that an application is ready for Production deployment.  As part of this series, I want to address the basic types of testing that ideally would be performed in a normal test effort. 

Lots of organizations think that “testing” is not difficult and that you could just grab someone off the street, sit them down in front of the computer and let them play with the application.  Some technology shops may do this (and I know some actually do) but if you want to do a professional job then there is significantly more work involved – therefore the reason why there are very experienced professional career testers.

The large the project, the more testing involved.  For small enhancement projects and Production Support fixes, the life cycles are shorter and the test efforts tend to not be as involved (however realize there is also some business risk in reducing the amount of testing performed).  What is listed below are the standard Test Phases that may or may not be performed.  For instance, some companies don’t do User Acceptance Testing and some don’t do Load Testing because they don’t have the software or hardware environments to conduct it in.  The only “test effort” not performed by the QA Test Team is “Unit Testing” which is the responsibility of the developers and should be performed before turning the software over to the test team.  Unfortunately this is either quickly skimmed over or completely ignored by the developers.

In an ideal test effort the following would occur:

Functional Testing: Functional testing exercises test cases that target each desirable and undesirable function of the software. The focus is normally on the new features/functionality added to the application.  Each new function of the software is targeted and tested in separation from other functions of the software.  This is normally performed manually as it can be challenging to develop automated scripts in a timely manner on brand new functionality that may or may not be buggy and still make the time-lines for the project (normally the new features are automated after the release and added to the regression test bed).
1. Positive Testing – This involves running through test cases using valid data.
2. Negative Testing – This involves running through test cases using invalid data; data intended to generate error messages.  Remember that when running negative test cases, when you get the error message displayed that you are looking for that this becomes “an expected result” not an error.
3. Data Driven Testing – This involves running through the same functionality over and over again with data variations.  This is not a true load test, but a mini load test to make sure the application can support users doing the same thing over and over again – like submitting requests or ordering parts or doing searches.  You want to run through the functionality using different data (and normally this is all valid data) one iteration after another.  If your application is to support 1000 requests a day, then your application better be able to support that requirement.  Likewise the other benefit is to run through the functionality using various data combinations.  Simple test cases have a tendency to use similar data, so by running a data driven test you can see if the application has an issue with various formats or data combinations.  This is an extremely valuable type of testing, but very tedious and monotonous so using an automation tool for this is very important.
4. Boundary Testing – This is the process of testing the “boundaries” of acceptable data – the min’s and max’s.  Once again in testing, lots of testers get caught in the process of using similar data and don’t change it up much.  So you may have a field that accepts two digit numbers in the range of 1-99 so you would want to not only test entering in “1″ and “99″ but do a few valid deviations in between (like “9″, “24″ and “75″) but also jump outside the boundary and test “0″, “01″, and “100″ and see if the application works as expected per the requirements.
5. Single User Performance Testing – This test effort verifies that the user experience for an individual with no or minimal load on the system is acceptable per the requirements.  This is best done by running through many iterations to average the performance time and is best done using an automated test tool.
6. SQL Penetration – This type of testing checks to see if a hacker can execute SQL commands in an edit field.  A trick hackers use is by entering SQL queries in an edit field to see if they can retrieve data from your database using a field that actually access the database in its normal design.  Make sure a hacker can’t penetrate your database this way.
7. Security Testing – This is to verify that the web sites and applications are not vulnerable to security attacks and hacks.
8. Exploratory Testing – is the random testing of the entire system done in creative and unusual ways to uncover defects found on the system’s edges – issues between modules, exploiting hidden dependencies, or unexpected state combinations.  This should be the last form of testing done as it is very unstructured and what is more like the idea of “playing with the application”.  It is useful but not structured.  Many times when a defect is uncovered the tester really has no idea what they did to create the defect since they are not tracking their actions so it becomes difficult to reproduce the defect because they were just “winging it”.

The next two test phases can either be executed separately or together.  For large scale projects they can easily be executed separately and typically are.  For small scale projects, feature enhancements or bugs this test phase is normally done together as one testing process -

Integration Testing:  Integration testing exercises several functional units that have been combined to form a module, subsystem, or system.  Integration testing focuses on the interfaces between units, to make sure the units work together.  The objective of this testing is to verify that the components of the system are working as defined per the approved requirements.

System Testing:  In system testing, black box testing is performed to verify the system correctly functions per the specified requirements.  The test environment must be as close to production as possible, including factors such as hardware setup, database size and complexity.  By replicating the production environment, we can more accurately test ‘softer’ system features (performance, security and fault-tolerance).  This is an end-to-end test.

Regression Testing: In this testing phase, the Test Team attempts to execute a mix of test cases involving different functions of the software to form a regression test.  This regression test needs to be executed each time a software code deployment is made to the QA testing environment.  Outside of an application redesign which is an overhaul of the application functionality and look & feel, the majority of development efforts are enhancement projects.  The norm is that about 75% of the application is not changing and only 25% or less is changing.  Therefore that means 75% of the application functionality is “regression testing”.  Most organization don’t have the time and resources to fully regression test a major application so the Test Team attempts to exercise as much of the existing application functionality to make sure the new features do not have any negative effect on the existing functionality that exists in production today.  What is identified for testing is normally based on business risk (risk analysis) and whatever automated test scripts have been already built to verify functionality.  Note that this is where you get the biggest “bang for the buck” when it comes to investing in test automation.  Regression testing is the best candidate for automation.  If properly approached and an automation test bed is created, then you can be running automated test scripts on one (or many) PCs to verify existing functionality (taking care of your regression testing) while you are conducting Functional testing of the new functionality that has just been built.

User Acceptance Testing (UAT):  The objective of User Acceptance Testing is to evaluate the product for its readiness in the marketplace and provide our customers with the confidence and assurance that the software is ready to be used.   The major focus of acceptance testing is on the user and proving that the system works.  This testing is performed by “Business” entities – typically the business owner of the project request or a manager/supervisor of the department that will be using the new features in Production.

Load/Stress/Performance Testing:  The Performance Engineer on the Test Team will put the application(s) under various load tests when requirements dictate.  If the requirements do not specify a load test is necessary, the Performance Engineer will attempt to perform a minimal load test as time permits.  Depending on technical constraints the testing may occur in the QA Environment or in the “UAT/Staging” environment.  See our other blog post on “Load vs Stress vs Performance Testing” for more information on the details of this test phase.

Now there are a couple of other categories that occur at various times during the other phases and so I’ll list them here.

ReTest or Retesting:  This typically occurs in two ways during the process of testing a code base for release:

1. When a defect is found on a piece of functionality and the developer fixes the problem.  Once the developer resolves the defect/bug they will change the status of the defect/bug to “Fixed”.  At this point the tester needs to get the latest code base (checkout the latest branch) and “retest” the functionality that was in error before and make sure the developer has really fixed the problem.
2. Another time this is done is when the code base that has already been tested and “QA Approved” is then changed by either changing the code at the branch level or the code is merged with another branch of code for release.  In this case the testers need to “retest” the code to make sure that what worked before still works after they made coding changes.

Smoke Testing / Health Check / Validation Test / Sanity Check:  For the most part these three types are very similar in style and can occur at various points in the process.  Most of the time they relate to Production deployments, however they can also occur during a code release process when a new branch/version of the code base is received from the development team or when branches of code are merged together in one version and should be tested prior to being released to Production.

1. Smoke Testing – this is typically done when you get a major project for the first time, or you have just released your code to Production.  It is intended to run through the code real fast hitting some major functionality to make sure at least the major features work.  If the major features don’t work, then it’s not even worth continuing testing since the developers have to immediately fix some major functionality.
2. Health Check, Validation Testing, and Sanity Check – these typically mean the same thing and like Smoke Testing it is a high-level test  of major functionality of the application to determine the “health” of the system once you move the new code to Production.  At this point you are hitting major functionality that is both regression and new features.  This is not an in-depth test since you don’t have the time since this test occurs immediately after releasing the new code into Production; so you need to do a quick test to make sure you didn’t break Production.