What is Non-Functional Testing?
Non-functional testing is the testing of a software application or system for its non-functional requirements.
Non-functional requirements define the operation of the system under test, such as the way the system operates, how it is maintained and how resilient it is to failure conditions.
This is in contrast to functional requirements that describe the behaviour of the system.
Non-functional testing requirements are often much more difficult to define than functional testing requirements where the expected functionality and behaviour is easier to document.
Objectives of Non-Functional Testing
Non-functional testing has a number of objectives, such as:
- Prove the system delivers suitable performance under a variety of network load conditions.
- Ensure the system can cope with fluctuations and predicted increases in demand over time or to meet future capacity requirements.
- Optimize installation and delivery of the system, software and applications and identify any build and deployment issues.
- Identify KPIs and ensure that appropriate logging, data capture and network reporting are in in place to allow the system to be monitored and managed effectively.
- Ensure that the system can be easily managed and maintained during normal day-to-day operations.
- Collect and produce measurements and metrics for internal research and development, and future network planning.
- Ensure the system is resilient to failures, both internally or in any other parts of the network, and that it can recover from failures easily.
- Identify and reduce any risks and costs associated with non-functional aspects of the product.
Types of Non-Functional Testing
It’s common to find that the names of many non-functional tests are used interchangeably because of the overlap in scope between various non-functional requirements.
For example, performance is a broad term that includes many specific non-functional requirements like reliability and scalability.
The following categories of non-functional requirements will help to determine what tests are required in each area:
- Performance Testing
- Peak Load Testing
- Soak Testing
- Stress Testing
- Utilization Testing
- Volume Testing
- Scalability Testing
- Operations and Maintenance (OAM) Testing
- Resilience and Recovery Testing
- Security Testing
- Disaster Recovery Testing
- Localization Testing
- Compliance Testing
Let’s look at each type of non-functional testing in more detail.
Performance Testing
Performance testing is used to measure the speed or effectiveness of a system, network or software program.
Responsiveness, speed, scalability, and stability are some of the key performance indicates that can be measured under a variety of load conditions.
Performance testing is often performed in a lab environment using tools to model and simulate expected load in the real-world, although sometimes it may also be possible to perform limited performance testing in the production environment.
A good performance test should also consider the concept of concurrency.
For example, as well as an application being able to respond to a service request within 1000ms, it should be able to do when a defined number of concurrent users are accessing the service.
A well-defined performance test will have clear performance criteria that can be easily tested and measured.
Examples of non-functional test requirements for performance testing:
- An application must respond to a service request within 1000 milliseconds, for 1000 concurrent users each sending 10 requests per second.
- A web server must respond to an HTTP request within 500 milliseconds
- A synchronous database request must respond within 800 milliseconds
- A batch process must complete its processing within 60 minutes
Peak Load Testing
Peak load testing is run to simulate the peak volumes of load and concurrency that occur at the daily, weekly and seasonal peaks of your application.
This testing ensures that the performance, utilization and volume non-functional requirements are satisfied, as well as ensuring that the system can be maintained at peak system loads.
Soak Testing
A soak test is a performance test that is run over an extended period of time to ensure that the system is able to provide continuous and uninterrupted service. Soak testing is also sometimes called long duration testing.
A good rule of thumb is to run a soak test for at least 48 hours that cover 2 overnight cycles. This is because systems often have internal maintenance routines or batch processes that execute overnight, so it is important that the soak test runs during those periods to flush out any potential issues.
Also, a good soak test should simulate any naturally occurring daily highs and lows in traffic, and not just run an average load. If modelling the traffic profile in this way over time is not possible, then ideally the soak test should run at the production peak hour load for the duration of the test.
Soak tests are designed to uncover issues such as memory and other resource leaks, sockets or routines not being closed properly, or any other processes that may cause a gradual deterioration in system responsivity and performance over time.
Based on the results of soak testing, the system can then be reconfigured or optimized in order to avoid any such degradation or impaired performance.
One of the main drawbacks of soak testing is that it can be a very time-consuming process, and therefore difficult to schedule if the project has very tight timelines.
Stress Testing
Stress testing is performed to test the stability and reliability of the system under extremely heavy load conditions.
Stress testing is also often used to discover the absolute limits of the system under test, by increasing the amount of load incrementally until the system or application can no longer cope with the extreme demand.
Stress testing focuses on the system’s stability and it’s ability to function in both normal and abnormal conditions.
Examples of non-functional test requirements for stress testing:
- A Firewall must be able to withstand and recover from a denial-of-service (DoS or DDoS) attack, where external systems attempt to flood the bandwidth or resources of the target firewall.
Utilization Testing
Utilization testing is a subset of performance testing that is run to assess the impact of high volumes of transactions on the hardware, memory and physical components of a system, and other constrained network resources such as bandwidth and number of TCP connections.
Examples of non-functional test requirements for utilization testing:
- The application server shall not exceed 50% CPU consumption under levels of peak volume and concurrency
- The database server shall not exceed 90% of its available memory during peak load
- A maximum number of 20 TCP connections will be setup between the client and server during peak load
Volume Testing
Volume testing ensures that the application or system under test is able to meet the required performance with a specified amount of data.
This could refer to the size of a database or it could also refer to the size of an input file.
For example, if you want to volume test your application with a specific database size, you will create data in your database to that size and then test the application’s performance to make sure it still meets requirements.
Examples of non-functional test requirements for volume testing:
- Ensure that a database with 10 million entries still meets the required transaction response rate
- Ensure an input .dat data file used by an application can be processed correctly when it contains the predicted amount of data that will be used in production
- Ensure that a system build process can still process an .xml input file when the file doubles in size to include additional configuration items
Scalability Testing
Scalability testing gradually increases the level of load and concurrency to ensure the application or system can support the future predicted growth over a period of time.
Examples of non-functional test requirements for scalability testing:
- The application must be able to support an annual transactional growth rate of 10%, and still meet all defined transactional performance requirements
- The database must be able to an annual growth rate of 20%, with no degradation in database performance
- The application must be able to support a 10% growth in user concurrency, and still meet all defined transactional performance requirements
Operations and Maintenance (OAM) Testing
OAM testing is performed to make sure that the application can be operated and maintained in the production environment.
This includes making sure that monitoring is setup to detect application failures and error logs, statistics are logged to provide KPIs and other system information, and any maintenance or clean-up routines run as expected to keep the system running at peak performance.
Examples of non-functional test requirements for operability testing:
- All alarms are sent to a centralized alarm monitoring system, such as Netcool.
- All system log files are regularly backed up and archived to ensure disk space does not get filled up
- An audit record for each business transaction is written to the audit log
- Operations are able to login to the application and perform critical maintenance actions even when the system is under extreme load.
Resilience and Recovery Testing
Resilience testing is performed to ensure that an application is able to withstand stress and other problems in one or more parts of a system while continuing to perform its core functions, avoid loss of data and provide an acceptable level of service.
Recovery testing is performed to ensure that an application is able to recover from stressful events and failure conditions.
Examples of non-functional test requirements for resilience and recovery testing:
- Ensure that a database replicates in real-time to a standby instance, and that in the event of a failure the standby instance can takeover with no loss of service
Security Testing
Security testing is a large and wide ranging subject, and is critical to the operations of the business and to maintain business continuity.
Security testing can be grouped into the following 4 areas:
- Network security: Testing for vulnerabilities in the network infrastructure, resources and policies.
- System software security: Testing for weaknesses in core system software (operating system, database system, and other software) that the application depends on.
- Client-side application security: Testing to ensure that the client (such as a browser or other such tool) cannot be manipulated.
- Server-side application security: Testing to ensure that the server software and its technologies are robust and prevent intrusion.
Examples of non-functional test requirements for security testing:
- Ensure that a system can only be accessed by authorized users who must provide and username and a password
- Ensure a web browser cannot be used for cross-site scripting (XSS) vulnerability exploitation to allow attackers to bypass access controls
- Ensure that system software, firmware and patch levels are all up to date
- Ensure that any sensitive personal data is encrypted and transmitted using SSL
Some tests, such as penetration tests, are often outsourced to specialist companies who provide specialist knowledge and are able to simulate service attacks from outside the companies network and infrastructure.
Disaster Recovery Testing
Disaster recovery testing is performed to ensure that an organization can recover and restore business operations after a catastrophic systems failure or complete disruption.
A disaster could be man-made or natural events, such as a hurricane, earthquakes, floods, acts of terrorism, computer vandalism, sabotage, and inadvertent mishaps such as hardware misconfigurations and accidentally deleted files.
Typically, an organization will develop a disaster recovery plan (DRP) that will detail all the steps required to recover from a disaster, and this can often be used as the basis for testing.
Disaster recovery tests should be run regularly throughout the year to ensure that the DRP works, and that the steps to perform the process are well documented and up-to-date.
Examples of non-functional test requirements for disaster recovery testing:
- Ensure that the customer billing database runs a full backup once a week with incremental backups daily
- Ensure that business critical data is backed up each week and archived off-site in a flood and fire proof facility
- Ensure standby systems in a different location are able to takeover IT, network and other business operations in case the primary systems are destroyed by an earthquake.
Localization Testing
Localization testing is performed to make sure that a product or application can be configured or adapted to meet the cultural, lingual and other requirements of a specific country, region or locale.
Localization aims to adapt a product or service to look and feel like it has been specifically developed to meet their needs of a specific target audience.
The following important areas should be considered when testing localization:
- Date and time formats (including numeric formats)
- Currency
- Keyboard types and layouts
- Color schemes, symbols, and icons
- Text and graphics which may be viewed as sensitive in certain culture
- Different legal requirements
Examples of non-functional test requirements for disaster recovery testing:
- Ensure that the application can be configured to render characters in various languages using Unicode and that the database supports Unicode characters
- Ensure that a user can select a different currency from a dropdown menu on the website home page and that the selected currency is displayed in the correct format and the correct currency conversion rate is applied
- Ensure that the time and date and phone number formats are properly formatted for target region
- Ensure that when accessing the online help pages that they are presented and formatted in the correct local language
Note that localization is often abbreviated to “L10N” as there are 10 characters between L & N in the word localization.
Compliance Testing
Compliance testing, or conformance testing, ensures that a process, product, or service complies with the requirements of a specification, technical standard, contract, business policy or regulation.
Conformance testing may be undertaken by the producer of the product or service, by a user, or by an accredited independent organization, such as the author of the standard being used.
Below are some organization who have set standards that might be relevant to software testing:
- Institute of Electrical and Electronics Engineers (IEEE)
- World Wide Web Consortium (W3C)
- General Data Protection Regulation (GDPR)
- Health Insurance Portability and Accountability Act (HIPAA)
- International Organization for Standardization (ISO)
- Payment Card Industry Security Standard Council (PCI SSC)
Note that the standards set by these bodies encompass many areas including processes, products and personnel; standards specific to software testing is a tiny part.
Examples of non-functional test requirements for compliance testing:
- Ensure that personal data is captured and stored in compliance with GDPR regulations
- A food company that displays organic labelling undergoes testing to ensure that its products do not contain any synthetic chemicals
- An newly installed electrical distribution board is tested to make sure it is in compliance with IEEE electrical safety regulations
