ASI

This is the first part of my series on “What is QA really all about?”  I find that even today after all these years of sophistication in software development that organizations still don’t really understand what QA is really all about and what is truly involved in verifying that an application is ready for Production deployment.  As part of this series, I want to address the basic concepts of “load testing”.  I constantly run into the misconception of what load testing is and “why can’t we run a load test as soon as development is done code?”

The first thing to note with any one of the load testing concepts is that we are not trying to find functional defects.  The expectation is that the code being tested is already stable; so at a minimum Functional and System/Integration testing should have already been performed to identify and eliminate the bugs that would prevent an end-to-end test from executing successfully.  With load testing we need to be able to execute end-to-end tests and if there are bugs breaking the process, then how can we simulate large volumes of users coming through the system using it?  What we are trying to identify with Load testing in general is “can the application and/or system handle high volumes of load without the system slowing down or crashing”. So with a “bug free” functioning application, we can begin executing the various types of load testing in order to “tune” the application in order to best handle volume with acceptable performance times.

Load Testing Concepts:

Load Testing – this type of test is where a heavy user load is simulated to determine at what point the software system’s response time degrades or fails. The purpose is to determine the capability limitation of the system.  This is typically performed with defined scenario groups executing a large number of virtual users with increased numbers added to the test over specified periods of time.

Stress Testing – this type of load test will immediately jump to large volumes of virtual users with the intent of seeing how the system handles a large stress on it.  This type of test is usually focused on finding out if servers crash, databases crash, applications freeze or performance just bottoms out.

Performance Testing – this type of load test is designed to see at what points of load on the system will create performance hits/impacts.  Normally the load test will begin with a minimal number of virtual users and then gradually increase the number of virtual users over a defined period of time.

Load Test Types:

Scalability Test – Perform repeated stress, overload, volume, and longevity tests with different server or network hardware configurations to determine the most cost-effective options for supporting the targeted load levels.
  1. Using “Vertical scaling” (up) adds additional RAM or CPUs within individual server boxes.
  2. Using “Horizontal scaling” (out) adds additional server boxes to a cluster of servers.

Longevity Test – Test that the system can sustain a consistent number of concurrent Vusers executing transactions using near-peak capacity, over a minimum 24-hour period (note we may not always be able to execute as long as 24 hours, but that is the ideal scenario). Longevity testing uses more disk space and other system resources and can be used to measure the build-up of obsolete logs, intermediate data structures, and statistical data that need to be periodically removed. Longer runs allow for the detection and measurement of the impact of occasional events (like log truncations) and anomalies that occur infrequently. These tests verify provisions for managing space, such as log truncation jobs that only usually occur in the middle of the night.

Volume Test – Test that the system can handle a massive data entry/load to the applications affected. During the test, the pattern of the application response time is measured while more data is added. Volume tests make sure there are sufficient system resources to handle the stream of data.

Overload Sustainability Test – Test how well the hardware allocated for the application can support the number of anticipated users. The goal is to quantify the degradation in response time and resource consumption at various levels of simultaneous users. This is done by gradually ramping up the number of Vusers until the system “chokes” at a breakpoint (when the number of connections flattens out, response time degrades or times out and errors appear). During tests, the resources used by each server are measured to make sure there is enough transient memory space and that the application correctly manages the system memory.

Stress Test – Test the maximum load that can be placed on the application to measure its upper performance boundaries and how it handles peak conditions. First it tests the load that the system was planned to withstand. Then additional load is added (more Vusers) that creates an overload condition on the system for a certain period of time. After that time, the load is reduced to the previous level and the operating system parameters are monitored. Response times are also measured at this stage.