During performance testing we simulate demand on an application or system under test (SUT) and then measure it’s ability to deal with that demand.
In this context, demand can take many forms, such as simulating users navigating a web page, setting up voice calls through a telecoms network, or passing IP signalling traffic through a router. It all depends what system or application you are trying to load test.
Performance testing is quite often also called load testing, traffic testing, or stress testing, or nonfunctional testing.
But be careful when calling performance testing “nonfunctional testing”. In fact, I consider performance testing to be just one type of non-functional test. See my article called “What Non Functional Tests Should You Consider?” for a description of some other non-functional tests that you should think about when developing a comprehensive test strategy and test plan.
Types of Performance Testing
Just send some traffic through the system and you’ll be fine – right? Wrong.
Within performance testing there are quite a few sub-categories, depending on your test objectives. Who knew performance testing was so complicated!
Here are the different types of performance testing that should be considered.
Load testing is the simplest form of performance testing, and is usually run to understand the behaviour of the system under a specific load.
This load is calculated as the peak expected demand that the system will be expected to cope with under normal conditions, day in day out, week after week.
For example, a typical load test would be to simulate a number of concurrent users on the application or system performing a specific number of transactions within a certain time period. The system (signalling network, database, application server, etc) would be monitored during the test to check that response times, hardware and software resources, and other critical processes perform as expected, and that there are no bottlenecks.
You should also consider user experience while running a performance test to make sure that the service provided to the end user is satisfactory.
Also referred to as break-point testing, stress testing is used to understand the upper limits of capacity within the system.
This kind of test provides very useful information about a system that serves many objectives, such as:
- to determine the system’s robustness under extreme load and the effectiveness of system throttling and other safeguarding processes
- to ensure the system does not crash during a denial of service (DDOS) attack
- to determine the weakest link (or bottleneck) within the entire system
- to determine operability, and whether the system can still be accessed and administered under extreme load
- to help plan upgrades, i.e. if you have a duplicated or redundant system architecture, can one system handle the demand while the other is taken out of service to be upgraded.
- to discover the upper performance limits of a system to help future capacity planning
Soak testing is a type of load testing that involves testing the system under a production load for an extended duration of time, and is used to identify problems such as memory leaks, resource leaks, or response time degradation that usually only happen after the application has been running for a sustained period.
Soak testing is also sometimes known as endurance testing, long duration testing, or longevity testing.
An ideal soak test would simulate the actual load seen in production, with the normal peaks and dips in load that are typically seen in application usage throughout the day, as opposed to just generating an average load. If this is not possible, then a conservative approach would be to run the system at peak production loads for the duration of the test.
How long should you run a soak test?
One of the main challenges of running a soak test is that it is a very time-consuming process, which can be difficult to justify on projects with very strict time schedules. However, in a perfect world you should run your soak test for 5 days, but if that is not possible then aim for at least 48 hours.
Scalability testing is the testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability
Rather than testing for performance from a load perspective, tests are created to determine the effects of configuration changes to the system’s components on the system’s performance and behaviour. A common example would be experimenting with different methods of load-balancing.
Performance Test Environment
Performance testing is usually conducted in a test environment identical to the production environment before the system is permitted to go live.
Performance testing often deals with creating/scripting the work flows of key identified business processes, and there are a wide variety of performance test tools that can be used like HP LoadRunner, NeoLoad, Apache JMeter, Rational Performance Tester, Silk Performer, Gatling, and Flood.
Each of the tools just mentioned (which is in no way an exhaustive nor complete list) either employs a scripting language (C, Java, JS) or some form of visual representation (drag and drop) to create and simulate end user work flows.
Most tools also allow for “Record & Replay”, where the performance tester will launch the testing tool, hook it on a browser or thick client and capture all the network transactions which happen between the client and server. In doing so a script is developed which can be enhanced/modified to emulate various business scenarios
The following parameters can be monitored during a performance test to measure and analyze the behaviour and response characteristics of the application or system under test.
Server Hardware Parameters
As a first step, the patterns generated by these 4 parameters provide a good indication on where the bottleneck lies. To determine the exact root cause of the issue, software engineers use tools such as profilers to measure what parts of a device or software contribute most to the poor performance, or to establish throughput levels (and thresholds) for maintained acceptable response time.