The objective of a performance test is to demonstrate that the system meets requirements for transaction throughput and response times simultaneously.
This infrastructure is an asset and an expensive one too, so it pays to make as much use of this infrastructure as possible. Fortunately, this infrastructure is a test bed, which can be re-used for other tests with broader objectives. A comprehensive test strategy would define a test infrastructure to enable all these objectives be met.
The performance testing goals are:
•End-to-end transaction response time measurements.
•Measure Application Server components performance under various loads.
•Measure database components performance under various loads.
•Monitor system resources under various loads.
•Measure the network delay between the server and clients
Pre-Requisites for Performance Testing:
We can identify five pre-requisites for a performance test. Not all of these need be in place prior to planning or preparing the test (although this might be helpful), but rather, the list defines what is required before a test can be executed.
First and foremost thing is
The design specification or a separate performance requirements document should :
•Defines specific performance goals for each feature that is instrumented.
•Bases performance goals on customer requirements.
•Defines specific customer scenarios.
Quantitative, relevant, measurable, realistic, achievable requirements
As a foundation to all tests, performance requirements should be agreed prior to the test. This helps in determining whether or not the system meets the stated requirements. The following attributes will help to have a meaningful performance comparison.
•Quantitative - expressed in quantifiable terms such that when response times are measured, a sensible comparison can be derived.
•Relevant - a response time must be relevant to a business process.
•Measurable - a response time should be defined such that it can be measured using a tool or stopwatch and at reasonable cost.
•Realistic - response time requirements should be justifiable when compared with the durations of the activities within the business process the system supports.
•Achievable - response times should take some account of the cost of achieving them.
Stable system:
A test team attempting to construct a performance test of a system whose software is of poor quality is unlikely to be successful. If the software crashes regularly, it will probably not withstand the relatively minor stress of repeated use. Testers will not be able to record scripts in the first instance, or may not be able to execute a test for a reasonable length of time before the software, middleware or operating systems crash.
Realistic test environment:
The test environment should ideally be the production environment or a close simulation and be dedicated to the performance test team for the duration of the test. Often this is not possible. However, for the results of the test to be realistic, the test environment should be comparable to the actual production environment. Even with an environment which is somewhat different from the production environment, it should still be possible to interpret the results obtained using a model of the system to predict, with some confidence, the behavior of the target environment. A test environment which bears no similarity to the actual production environment may be useful for finding obscure errors in the code, but is, however, useless for a performance test.
Performance Requirements:
Performance requirements normally comprise three components:
•Response time requirements
•Transaction volumes detailed in ‘Load Profiles’
•Database volumes
Response time requirements:
When asked to specify performance requirements, users normally focus attention on
response times, and often wish to define requirements in terms of generic response times.
A single response time requirement for all transactions might be simple to define from the user’s point of view, but is unreasonable. Some functions are critical and require short response times, but others are less critical and response time requirements can be less stringent.
Load profiles:
The second component of performance requirements is a schedule of load profiles. A load profile is the level of system loading expected to occur during a specific business scenario.
Business scenarios might cover different situations when the users’ organization has different levels of activity or involve a varying mix of activities, which must be supported by the system.
Database volumes:
Data volumes, defining the numbers of table rows which should be present in the database after a specified period of live running complete the load profile. Typically, data volumes estimated to exist after one year’s use of the system are used, but two year volumes or greater might be used in some circumstances, depending on the business application.
Tuesday, September 30, 2008
Monday, September 29, 2008
Performance Testing
The performance testing is a measure of the performance characteristics of an application. The main objective of a performance testing is to demonstrate that the system functions to specification with acceptable response times while processing the required transaction volumes in real-time production database. The objective of a performance test is to demonstrate that the system meets requirements for transaction throughput and response times simultaneously. The main deliverables from such a test, prior to execution, are automated test scripts and an infrastructure to be used to execute automated tests for extended periods.
What is Performance testing?
Performance testing of an application is basically the process of understanding how the web application and its operating environment respond at various user load levels. In general, we want to measure the latency, throughput, and utilization of the web site while simulating attempts by virtual users to simultaneously access the site. One of the main objectives of performance testing is to maintain a web site with low latency, high throughput, and low utilization.
Why Performance testing?
Performance problems are usually the result of contention for, or exhaustion of, some system resource. When a system resource is exhausted, the system is unable to scale to higher levels of performance. Maintaining optimum Web application performance is a top priority for application developers and administrators.
Performance analysis is also carried for various purposes such as:
•During a design or redesign of a module or a part of the system, more than one alternative presents itself. In such cases, the evaluation of a design alternative is
the prime mover for an analysis.
•Post-deployment realities create a need for the tuning the existing system. A systematic approach like performance analysis is essential to extract maximum benefit from an existing system.
•Identification of bottlenecks in a system is more of an effort at troubleshooting. This helps to replace and focus efforts at improving overall system response.
•As the user base grows, the cost of failure becomes increasingly unbearable. To
increase confidence and to provide an advance warning of potential problems in
case of load conditions, analysis must be done to forecast performance under load.
Typically to debug applications, developers would execute their applications using different execution streams (i.e., completely exercise the application) in an attempt to find errors.
When looking for errors in the application, performance is a secondary issue to features;however, it is still an issue.
What is Performance testing?
Performance testing of an application is basically the process of understanding how the web application and its operating environment respond at various user load levels. In general, we want to measure the latency, throughput, and utilization of the web site while simulating attempts by virtual users to simultaneously access the site. One of the main objectives of performance testing is to maintain a web site with low latency, high throughput, and low utilization.
Why Performance testing?
Performance problems are usually the result of contention for, or exhaustion of, some system resource. When a system resource is exhausted, the system is unable to scale to higher levels of performance. Maintaining optimum Web application performance is a top priority for application developers and administrators.
Performance analysis is also carried for various purposes such as:
•During a design or redesign of a module or a part of the system, more than one alternative presents itself. In such cases, the evaluation of a design alternative is
the prime mover for an analysis.
•Post-deployment realities create a need for the tuning the existing system. A systematic approach like performance analysis is essential to extract maximum benefit from an existing system.
•Identification of bottlenecks in a system is more of an effort at troubleshooting. This helps to replace and focus efforts at improving overall system response.
•As the user base grows, the cost of failure becomes increasingly unbearable. To
increase confidence and to provide an advance warning of potential problems in
case of load conditions, analysis must be done to forecast performance under load.
Typically to debug applications, developers would execute their applications using different execution streams (i.e., completely exercise the application) in an attempt to find errors.
When looking for errors in the application, performance is a secondary issue to features;however, it is still an issue.
Tuesday, September 23, 2008
Load Testing
Introduction:
Load Testing is creation of a simulated load on a real computer system by using virtual users who submit work as real users would do at real client workstations and thus testing the systems ability to support such workload.
Testing of critical web applications during its development and before its deployment should include functional testing to confirm to the specifications, performance testing to check if it offers an acceptable response time and load testing to see what hardware or software configuration will be required to provide acceptable response time and handle the load that will created by the real users of the system
1.Why is load testing important ?
Load Testing increases the uptime for critical web applications by helping you spot the bottlenecks in the system under large user stress scenarios before they happen in a production environment
2.When should load testing be done?
Load testing should be done when the probable cost of the load test is likely less than the cost of a failed application deployment.
Thus a load testing is accomplished by stressing the real application under simulated load provided by virtual users.
Load Testing Process
This is the first step when the project decides on load testing for its system. Evaluation of the requirements and needs of a system, prior to load testing will provide more realistic test conditions. For this one should know all key performance goals and objectives like number of concurrent connections, hits per second etc.,
Another important analysis of the system would also include the appropriate strategy for testing applications. It can be load testing or stress testing or capacity testing.
Load Testing is used to test the application against a requested number of users. The objective is to determine whether the site can sustain a requested number of users with acceptable response times. Stress testing is nothing but load testing over extended periods of time to validate an application’s stability and reliability. Similarly capacity testing is used to determine the maximum number of concurrent users that an application can manage. Hence for businesses capacity testing would be the benchmark to say that the maximum loads of concurrent users the site can sustain before the system fails.
Finally it should also be taken into consideration of the test tool which supports load testing by determining its multithreading capabilities and the creation of number of virtual users with minimal resource consumption and maximal virtual user count.
User Scripts
Once the analysis of the system is done the next step would be the creation of user scripts. A script recorder can be used to capture all the business processes into test scripts and this more often referred as virtual users or virtual user scripts. A virtual user is nothing but an emulated real user who drives the real application as client. All the business process should be recorded end to end so that these transactions will assist in breakdown of all actions and the time it takes to measure the performance of business process.
Settings
Run time settings should be defined the way the scripts should be run in order to accurately emulate real users. Settings can configure the number of concurrent connections, test run time, follow HTTP redirects etc., System response times also can vary based on the connection speed. Hence throttling bandwidth can emulate dial up connections at varying modem speeds (28.8 Kbps or 56.6 Kbps or T1 (1.54M) etc.
Performance Monitoring
Every component of the system needs monitoring :the clients, the network, the webs server, the application server, the database etc., This will result in instantly identifying the performance bottle necks during load testing. But if the tools support real time monitoring then testers would be able to view the application performance at any time during the test.
Thus running the load test scenario and monitoring the performance would accelerate the test process thereby producing a more stable application
Analyzing Results
The last but most important step in load testing is collecting and processing the data to resolve performance bottlenecks. The reports generated can be anything ranging from Number of hits, number of test clients, requests per second, socket errors etc.,
Hence analyzing the results will isolate bottle necks and determine which changes are needed to improve the system performance. After these changes are made the tests must re run the load test scenarios to verify adjustments.
Load Testing with WAST
Web Application Stress is a tool to simulate large number of users with a relatively small number of client machines. Performance data on an web application can be gathered by stressing the website and measuring the maximum requests per second that the web server can handle. The next step is to determine which resource prevents the requests per second from going higher, such as CPU, memory, or backend dependencies.
Conclusion
Load testing is the measure of an entire Web application's ability to sustain a number of simultaneous users and transactions, while maintaining adequate response times. It is the only way to accurately test the end-to-end performance of a Web site prior to going live.
Two common methods for implementing this load testing process are manual and automated testing.
Manual testing would involve
• Coordination of the operations of users
• Measure response times
• Repeat tests in a consistent way
• Compare results
As load testing is iterative in nature, the performance problems must be identified so that system can be tuned and retested to check for bottlenecks. For this reason, manual testing is not a very practical option.
Today, automated load testing is the preferred choice for load testing a Web application. The testing tools typically use three major components to execute a test:
• A console, which organizes, drives and manages the load
• Virtual users, performing a business process on a client application
• Load servers, which are used to run the virtual users
With automated load testing tools, tests can be easily rerun any number of times and the results can be reported automatically. In this way, automated testing tools provide a more cost-effective and efficient solution than their manual counterparts. Plus, they minimize the risk of human error during testing.
Load Testing is creation of a simulated load on a real computer system by using virtual users who submit work as real users would do at real client workstations and thus testing the systems ability to support such workload.
Testing of critical web applications during its development and before its deployment should include functional testing to confirm to the specifications, performance testing to check if it offers an acceptable response time and load testing to see what hardware or software configuration will be required to provide acceptable response time and handle the load that will created by the real users of the system
1.Why is load testing important ?
Load Testing increases the uptime for critical web applications by helping you spot the bottlenecks in the system under large user stress scenarios before they happen in a production environment
2.When should load testing be done?
Load testing should be done when the probable cost of the load test is likely less than the cost of a failed application deployment.
Thus a load testing is accomplished by stressing the real application under simulated load provided by virtual users.
Load Testing Process
This is the first step when the project decides on load testing for its system. Evaluation of the requirements and needs of a system, prior to load testing will provide more realistic test conditions. For this one should know all key performance goals and objectives like number of concurrent connections, hits per second etc.,
Another important analysis of the system would also include the appropriate strategy for testing applications. It can be load testing or stress testing or capacity testing.
Load Testing is used to test the application against a requested number of users. The objective is to determine whether the site can sustain a requested number of users with acceptable response times. Stress testing is nothing but load testing over extended periods of time to validate an application’s stability and reliability. Similarly capacity testing is used to determine the maximum number of concurrent users that an application can manage. Hence for businesses capacity testing would be the benchmark to say that the maximum loads of concurrent users the site can sustain before the system fails.
Finally it should also be taken into consideration of the test tool which supports load testing by determining its multithreading capabilities and the creation of number of virtual users with minimal resource consumption and maximal virtual user count.
User Scripts
Once the analysis of the system is done the next step would be the creation of user scripts. A script recorder can be used to capture all the business processes into test scripts and this more often referred as virtual users or virtual user scripts. A virtual user is nothing but an emulated real user who drives the real application as client. All the business process should be recorded end to end so that these transactions will assist in breakdown of all actions and the time it takes to measure the performance of business process.
Settings
Run time settings should be defined the way the scripts should be run in order to accurately emulate real users. Settings can configure the number of concurrent connections, test run time, follow HTTP redirects etc., System response times also can vary based on the connection speed. Hence throttling bandwidth can emulate dial up connections at varying modem speeds (28.8 Kbps or 56.6 Kbps or T1 (1.54M) etc.
Performance Monitoring
Every component of the system needs monitoring :the clients, the network, the webs server, the application server, the database etc., This will result in instantly identifying the performance bottle necks during load testing. But if the tools support real time monitoring then testers would be able to view the application performance at any time during the test.
Thus running the load test scenario and monitoring the performance would accelerate the test process thereby producing a more stable application
Analyzing Results
The last but most important step in load testing is collecting and processing the data to resolve performance bottlenecks. The reports generated can be anything ranging from Number of hits, number of test clients, requests per second, socket errors etc.,
Hence analyzing the results will isolate bottle necks and determine which changes are needed to improve the system performance. After these changes are made the tests must re run the load test scenarios to verify adjustments.
Load Testing with WAST
Web Application Stress is a tool to simulate large number of users with a relatively small number of client machines. Performance data on an web application can be gathered by stressing the website and measuring the maximum requests per second that the web server can handle. The next step is to determine which resource prevents the requests per second from going higher, such as CPU, memory, or backend dependencies.
Conclusion
Load testing is the measure of an entire Web application's ability to sustain a number of simultaneous users and transactions, while maintaining adequate response times. It is the only way to accurately test the end-to-end performance of a Web site prior to going live.
Two common methods for implementing this load testing process are manual and automated testing.
Manual testing would involve
• Coordination of the operations of users
• Measure response times
• Repeat tests in a consistent way
• Compare results
As load testing is iterative in nature, the performance problems must be identified so that system can be tuned and retested to check for bottlenecks. For this reason, manual testing is not a very practical option.
Today, automated load testing is the preferred choice for load testing a Web application. The testing tools typically use three major components to execute a test:
• A console, which organizes, drives and manages the load
• Virtual users, performing a business process on a client application
• Load servers, which are used to run the virtual users
With automated load testing tools, tests can be easily rerun any number of times and the results can be reported automatically. In this way, automated testing tools provide a more cost-effective and efficient solution than their manual counterparts. Plus, they minimize the risk of human error during testing.
Saturday, September 6, 2008
SYSTEM TESTING
Introduction to SYSTEM TESTING:
For most organizations, software and system testing represents a significant element of a project's cost in terms of money and management time. Making this function more effective can deliver a range of benefits including reductions in risk, development costs and improved 'time to market' for new systems.
Systems with software components and software-intensive systems are more and more complex everyday. Industry sectors such as railway,telecom , automotive, and space and aeronautical, are good examples. It is often agreed that testing is essential to manufacture reliable products. However, the validation process does not often receive the required attention. Moreover, the validation process is close to other activities such as conformance, acceptance and qualification testing.
The difference between function testing and system testing is that now the focus is on the whole application and its environment . Therefore the program has to be given completely. This does not mean that now single functions of the whole program are tested, because this would be too redundant. The main goal is rather to demonstrate the discrepancies of the product from its requirements and its documentation. In other words, this again includes the question, ``Did we build the right product?'' and not just, ``Did we build the product right?''
However, system testing does not only deal with this more economical problem, it also contains some aspects that are orientated on the word ``system'' . This means that those tests should be done in the environment for which the program was designed, like a mulituser network or whetever. Even security guide lines have to be included. Once again, it is beyond doubt that this test cannot be done completely, and nevertheless, while this is one of the most incomplete test methods, it is one of the most important.
A number of time-domain software reliability models attempt to predict the growth of a system's reliability during the system test phase of the development life cycle. In this paper we examine the results of applying several types of Poisson-process models to the development of a large system for which system test was performed in two parallel tracks, using different strategies for test data selection.
we will test that the functionality of your systems meets with your specifications, integrating with which-ever type of development methodology you are applying. We test for errors that users are likely to make as they interact with the application as well as your application’s ability to trap errors gracefully. These techniques can be applied flexibly, whether testing a financial system, e-commerce, an online casino or games testing.
System Testing is more than just functional testing, however, and can, when appropriate, also encompass many other types of testing, such as:
1.security
2.load/stress
3.performance
4.browser compatibility
5.localisation
Need for System Testing:
Effective software testing, as a part of software engineering, has been proven over the last 3 decades to deliver real business benefits including:
Reduction of costs increased productivity:Reduce rework and support overheads
More effort spent on developing new functionality and less on "bug fixing" as quality increases
Reduce commercial risks:If it goes wrong, what is the potential impact on your commercial goals? Knowledge is power, so why take a leap of faith while your competition step forward with confidence?
These benefits are achieved as a result of some fundamental principles of testing, for example, increased independence naturally increases objectivity.Your test strategy must take into consideration the risks to your organisation, commercial and technical. You will have a personal interest in its success in which case it is only human for your objectivity to be compromised.
System Testing Techniques:
Goal is to evaluate the system as a whole, not its parts
Techniques can be structural or functional
Techniques can be used in any stage that tests the system as a whole (acceptance, installation, etc.)
Techniques not mutually exclusive
Structural techniques
stress testing - test larger-than-normal capacity in terms of transactions, data, users, speed, etc.
execution testing- test performance in terms of speed, precision, etc.
recovery testing - test how the system recovers from a disaster, how it handles corrupted data, etc.
operations testing - test how the system fits in with existing operations and procedures in the user organization
compliance testing - test adherence to standards
security testing - test security requirements
Functional techniques
requirements testing - fundamental form of testing - makes sure the system does what it’s required to do
regression testing - make sure unchanged functionality remains unchanged
error-handling testing - test required error-handling functions (usually user error)
manual-support testing - test that the system can be used properly - includes user documentation
intersystem handling testing - test that the system is compatible with other systems in the environment
control testing - test required control mechanisms
parallel testing - feed same input into two versions of the system to make sure they produce the same output
Functional techniques:
input domain testing - pick test cases representative of the range of allowable input, including high, low, and average values
equivalence partitioning - partition the range of allowable input so that the program is expected to behave similarly for all inputs in a given partition, then pick a test case from each partition
boundary value - choose test cases with input values at the boundary (both inside and outside) of the allowable range
syntax checking - choose test cases that violate the format rules for input
special values - design test cases that use input values that represent special situations
output domain testing - pick test cases that will produce output at the extremes of the output domain
Structural techniques
statement testing - ensure the set of test cases exercises every statement at least once
branch testing - each branch of an if/then statement is exercised
conditional testing - each truth statement is exercised both true and false
expression testing - every part of every expression is exercised
path testing - every path is exercised (impossible in practice)
Error-based techniques
basic idea is that if you know something about the nature of the defects in the code, you can estimate whether or not you’ve found all of them or not
fault seeding - put a certain number of known faults into the code, then test until they are all found
mutation testing - create mutants of the program by making single changes, then run test cases until all mutants have been killed
historical test data - an organization keeps records of the average numbers of defects in the products it produces, then tests a new product until the number of defects found approaches the expected number
Conclusion:
Hence the system Test phase should begin once modules are integrated enough to perform tests in a whole system environment. System testing can occur in parallel with integration test, especially with the top-down method.
For most organizations, software and system testing represents a significant element of a project's cost in terms of money and management time. Making this function more effective can deliver a range of benefits including reductions in risk, development costs and improved 'time to market' for new systems.
Systems with software components and software-intensive systems are more and more complex everyday. Industry sectors such as railway,telecom , automotive, and space and aeronautical, are good examples. It is often agreed that testing is essential to manufacture reliable products. However, the validation process does not often receive the required attention. Moreover, the validation process is close to other activities such as conformance, acceptance and qualification testing.
The difference between function testing and system testing is that now the focus is on the whole application and its environment . Therefore the program has to be given completely. This does not mean that now single functions of the whole program are tested, because this would be too redundant. The main goal is rather to demonstrate the discrepancies of the product from its requirements and its documentation. In other words, this again includes the question, ``Did we build the right product?'' and not just, ``Did we build the product right?''
However, system testing does not only deal with this more economical problem, it also contains some aspects that are orientated on the word ``system'' . This means that those tests should be done in the environment for which the program was designed, like a mulituser network or whetever. Even security guide lines have to be included. Once again, it is beyond doubt that this test cannot be done completely, and nevertheless, while this is one of the most incomplete test methods, it is one of the most important.
A number of time-domain software reliability models attempt to predict the growth of a system's reliability during the system test phase of the development life cycle. In this paper we examine the results of applying several types of Poisson-process models to the development of a large system for which system test was performed in two parallel tracks, using different strategies for test data selection.
we will test that the functionality of your systems meets with your specifications, integrating with which-ever type of development methodology you are applying. We test for errors that users are likely to make as they interact with the application as well as your application’s ability to trap errors gracefully. These techniques can be applied flexibly, whether testing a financial system, e-commerce, an online casino or games testing.
System Testing is more than just functional testing, however, and can, when appropriate, also encompass many other types of testing, such as:
1.security
2.load/stress
3.performance
4.browser compatibility
5.localisation
Need for System Testing:
Effective software testing, as a part of software engineering, has been proven over the last 3 decades to deliver real business benefits including:
Reduction of costs increased productivity:Reduce rework and support overheads
More effort spent on developing new functionality and less on "bug fixing" as quality increases
Reduce commercial risks:If it goes wrong, what is the potential impact on your commercial goals? Knowledge is power, so why take a leap of faith while your competition step forward with confidence?
These benefits are achieved as a result of some fundamental principles of testing, for example, increased independence naturally increases objectivity.Your test strategy must take into consideration the risks to your organisation, commercial and technical. You will have a personal interest in its success in which case it is only human for your objectivity to be compromised.
System Testing Techniques:
Goal is to evaluate the system as a whole, not its parts
Techniques can be structural or functional
Techniques can be used in any stage that tests the system as a whole (acceptance, installation, etc.)
Techniques not mutually exclusive
Structural techniques
stress testing - test larger-than-normal capacity in terms of transactions, data, users, speed, etc.
execution testing- test performance in terms of speed, precision, etc.
recovery testing - test how the system recovers from a disaster, how it handles corrupted data, etc.
operations testing - test how the system fits in with existing operations and procedures in the user organization
compliance testing - test adherence to standards
security testing - test security requirements
Functional techniques
requirements testing - fundamental form of testing - makes sure the system does what it’s required to do
regression testing - make sure unchanged functionality remains unchanged
error-handling testing - test required error-handling functions (usually user error)
manual-support testing - test that the system can be used properly - includes user documentation
intersystem handling testing - test that the system is compatible with other systems in the environment
control testing - test required control mechanisms
parallel testing - feed same input into two versions of the system to make sure they produce the same output
Functional techniques:
input domain testing - pick test cases representative of the range of allowable input, including high, low, and average values
equivalence partitioning - partition the range of allowable input so that the program is expected to behave similarly for all inputs in a given partition, then pick a test case from each partition
boundary value - choose test cases with input values at the boundary (both inside and outside) of the allowable range
syntax checking - choose test cases that violate the format rules for input
special values - design test cases that use input values that represent special situations
output domain testing - pick test cases that will produce output at the extremes of the output domain
Structural techniques
statement testing - ensure the set of test cases exercises every statement at least once
branch testing - each branch of an if/then statement is exercised
conditional testing - each truth statement is exercised both true and false
expression testing - every part of every expression is exercised
path testing - every path is exercised (impossible in practice)
Error-based techniques
basic idea is that if you know something about the nature of the defects in the code, you can estimate whether or not you’ve found all of them or not
fault seeding - put a certain number of known faults into the code, then test until they are all found
mutation testing - create mutants of the program by making single changes, then run test cases until all mutants have been killed
historical test data - an organization keeps records of the average numbers of defects in the products it produces, then tests a new product until the number of defects found approaches the expected number
Conclusion:
Hence the system Test phase should begin once modules are integrated enough to perform tests in a whole system environment. System testing can occur in parallel with integration test, especially with the top-down method.
Subscribe to:
Posts (Atom)
