Manual Testing Course-Day 5
A TEST PLAN is a document describing software testing scope and activities. It is the basis for formally testing any software / product in a project.
ISTQB Definition
- test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice,and any risks requiring contingency planning. It is a record of the test planning process.
- master test plan: A test plan that typically addresses multiple test levels.
- phase test plan: A test plan that typically addresses one test phase.
Test Plan is influenced by the following factors:
- Organizational test policy and test strategy
- Software development life cycle model
- Scope of testing
- Availability of resources.
At the beginning of the project, the test plan can be a draft or with very little details. But, as the project progresses and more information becomes available, the test plan needs to be fleshed out. Test planning is a continuous activity and is performed throughout the product’s life cycle.
Types
Test plans can be of the following types:
- Master Test Plan: A single high-level test plan for a project / product that unifies all other test plans.
- Testing Level Specific: Test plans for each level of testing:
- Unit Test Plan
- Integration Test Plan
- System Test Plan
- Acceptance Test Plan
- Testing Type Specific: Test plans for specific types of testing like Performance Test Plan and Security Test Plan.
Test Plan Template
The format and content of a software test plan vary depending on the processes, standards, and test management tools being implemented. Nevertheless, the following format, which is based on IEEE standard for software test documentation, provides a summary of what a test plan can / should contain.
Test Plan Identifier:
- Provide a unique identifier for the document. (Adhere to the Configuration Management System if you have one.)
Introduction:
- Provide an overview of the test plan.
- Specify the goals / objectives.
- Specify any constraints.
References:
- List the related documents, with links to them if available, including the following:
- Project Plan
- Configuration Management Plan
Test Items:
- List the test items (software / products) and their versions.
Features to be Tested:
- List the features of the software / product to be tested.
- Provide references to the Requirements and/or Design specifications of the features to be tested.
Features Not to Be Tested:
- List the features of the software / product which will not be tested.
- Specify the reasons these features won’t be tested.
Approach:
- Mention the overall approach to testing.
- Specify the following:
- Testing levels [if it’s a Master Test Plan],
- Testing types
- Testing methods
Item Pass / Fail Criteria:
- Specify the criteria that will be used to determine whether each test item has passed or failed testing.
Suspension Criteria and Resumption Requirements:
- Specify criteria to be used to suspend the testing activity.
- Specify what is required before testing can resume.
Test Deliverables:
- List test deliverables, and links to them if available, including the following:
- Test Plan (this document itself)
- Test Cases
- Test Scripts
- Test Data
- Defect Reports
- Test Reports
Test Environment:
- Specify the properties of test environment: hardware, software, network, etc.
- List any testing or related tools.
Estimate:
- Provide a summary of test estimates (cost or effort) and/or provide a link to the detailed estimation.
Schedule:
- Provide a summary of the schedule, specifying key test milestones, and/or provide a link to the detailed schedule.
Staffing and Training Needs:
- Specify staffing needs by role and required skills.
- Identify training that is necessary to provide those skills, if not already acquired.
Responsibilities:
- List the responsibilities of each team / role / individual.
Risks:
- List the risks that have been identified.
- Specify the mitigation plan and the contingency plan for each risk.
Assumptions and Dependencies:
- List the assumptions that have been made during the preparation of this plan.
- List the dependencies.
Approvals:
- Specify the names and roles of all persons who must approve the plan.
- Provide space for signatures and dates. (If the document is to be printed.)
Test Plan Guidelines
- Make the plan concise. Avoid redundancy and superfluousness. If you think you do not need a section that has been mentioned in the template above, go ahead and delete that section in your test plan.
- Be specific. For example, when you specify an operating system as a property of a test environment, mention the OS Edition / Version as well, not just the OS Name.
- Make use of lists and tables wherever possible. Avoid lengthy paragraphs.
- Have the test plan reviewed a number of times prior to baselining it or sending it for approval. The quality of your test plan speaks volumes about the quality of the testing you or your team is going to perform.
- Update the plan as and when necessary. An out-dated and unused document stinks and is worse than not having the document in the first place.
Unit Testing
UNIT TESTING, also known as COMPONENT TESTING, is a level of software testing where individual units / components of a software are tested. The purpose is to validate that each unit of the software performs as designed.
Definition by ISTQB
- unit testing: See component testing.
- component testing: The testing of individual software components.
Elaboration
A unit is the smallest testable part of any software. It usually has one or a few inputs and usually a single output. In procedural programming, a unit may be an individual program, function, procedure, etc. In object-oriented programming, the smallest unit is a method, which may belong to a base / super class, abstract class or derived / child class. (Some treat a module of an application as a unit. This is to be discouraged as there will probably be many individual units within that module.) Unit testing frameworks, drivers, stubs, and mock / fake objects are used to assist in unit testing.
Defects are usually fixed as soon as found and they are not formally reported and tracked.
Analogy
During the process of manufacturing a ballpoint pen, the cap, the body, the tail, the ink cartridge and the ballpoint are produced separately and unit tested separately.
Method
Unit Testing is usually performed by using the White Box Testing method and is normally automated.
Tasks
- Unit Test Plan [Prepare >> Review >> Rework >> Baseline]
- Unit Test Cases/ Test Scripts/ Test Data [Prepare >> Review >> Rework >> Baseline]
- Unit Test [Execute >> Re-execute]
When is it performed?
Unit Testing is the first level of software testing and is performed prior to Integration Testing. Though unit testing is normally performed after coding, sometimes, specially in test-driven development (TDD), automated unit tests are written prior to coding.
Who performs it?
It is normally performed by software developers themselves or their peers. In rare cases, it may also be performed by independent software testers but they will need to have access to the code and have an understanding of the architecture and design.
Benefits
- Unit testing increases confidence in changing / maintaining code. If good unit tests are written and if they are run every time any code is changed, we will be able to promptly catch any defects introduced due to the change. Also, if codes are already made less interdependent to make unit testing possible, the unintended impact of changes to any code is less.
- Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse.
- Development is faster. How? If you do not have unit testing in place, you write your code and perform that fuzzy ‘developer test’ (You set some breakpoints, fire up the GUI, provide a few inputs that hopefully hit your code and hope that you are all set.) But, if you have unit testing in place, you write the test, write the code and run the test. Writing tests takes time but the time is compensated by the less amount of time it takes to run the tests; You need not fire up the GUI and provide all those inputs. And, of course, unit tests are more reliable than ‘developer tests’. Development is faster in the long run too. How? The effort required to find and fix defects found during unit testing is very less in comparison to the effort required to fix defects found during system testing or acceptance testing.
- The cost of fixing a defect detected during unit testing is lesser in comparison to that of defects detected at higher levels. Compare the cost (time, effort, destruction, humiliation) of a defect detected during acceptance testing or when the software is in production.
- Debugging is easy. When a test fails, only the latest changes need to be debugged. With testing at higher levels, changes made over the span of several days / weeks / months need to be scanned.
- Codes are more reliable. Why? I think there is no need to explain this to a sane person.
Tips
- Create a proper unit test plan. [If not documented, at least in your head.]
- Find a test automation tool / framework for your language.
- Create test cases focusing on areas that impact the behavior of the system the most.
- Isolate the development environment from the test environment.
- Use test data that is close to that of production.
- Before fixing a defect, write or modify a test that exposes the defect. Why? First, you will later be able to catch the defect if you do not fix it properly. Second, your test suite is now more comprehensive. Third, you will most probably be too lazy to write the test after you have already fixed the defect.
- Write test cases that are independent of each other. For example, if a class depends on a database, do not write a case that interacts with the database to test the class. Instead, create an abstract interface around that database connection and implement that interface with a mock object.
- Aim at covering all paths through the unit. Pay particular attention to loop conditions.
- Make sure you are using a version control system to keep track of your test scripts.
- In addition to writing cases to verify the behavior, write cases to ensure the performance of the code.
- Perform unit tests continuously and frequently.
One more reason
Lets say you have a program comprising of two units and the only test you perform is system testing. [You skip unit and integration testing.] During testing, you find a bug. Now, how will you determine the cause of the problem?
- Is the bug due to an error in unit 1?
- Is the bug due to an error in unit 2?
- Is the bug due to errors in both units?
- Is the bug due to an error in the interface between the units?
- Is the bug due to an error in the test or test case?
Unit testing is often neglected but it is, in fact, the most important level of testing.
_____________________________________________________________________________________________
INTEGRATION TESTING is a level of software testing where individual units / components are combined and tested as a group. The purpose of this level of testing is to expose faults in the interaction between integrated units. Test drivers and test stubs are used to assist in Integration Testing.
ISTQB Definition
- integration testing: Testing performed to expose defects in the interfaces and in the
interactions between integrated components or systems. See also component integration
testing, system integration testing. - component integration testing: Testing performed to expose defects in the interfaces and
interaction between integrated components. - system integration testing: Testing the integration of systems and packages; testing
interfaces to external organizations (e.g. Electronic Data Interchange, Internet).
Analogy
During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. For example, whether the cap fits into the body or not.
Method
Any of Black Box Testing, White Box Testing and Gray Box Testing methods can be used. Normally, the method depends on your definition of ‘unit’ and what exactly you are integrating. The test can be either manual or automated.
Tasks
- Integration Test Plan [Prepare >> Review >> Rework >> Baseline]
- Integration Test Cases/ Test Scripts/ Test Data [Prepare >> Review >> Rework >> Baseline]
- Integration Test [Execute >> Re-execute]
When is Integration Testing performed?
Integration Testing is the second level of testing performed after Unit Testing and before System Testing [Caveat: See system integration testing below].
Who performs Integration Testing?
Unit / Component Integration Testing is usually conducted by developers and System Integration Testing by or independent testers.
Types
There are mainly two types of integration testing:
Unit / Component Integration Testing
This type of integration testing focuses on the interactions and interfaces between integrated units / components. Unit integration testing is performed after unit testing, and is generally automated. In iterative and incremental development, unit integration tests are usually part of the continuous integration process.
System Integration Testing
This type of integration testing focuses on the interactions and interfaces between systems (A system is a group of interacting or interrelated entities that form a unified whole). The other system that needs to be integrated with can either be internal or external (developed by someone else over which the organization has no control). Such systems are normally exposed via an Application Programming Interface (API) or a Microservice.
System Integration testing may be done after System Testing or in parallel with it.
Approaches
There are basically four approaches to Integration Testing:
Big Bang
This is an approach to Integration Testing where all or most of the units are combined together and tested at one go. This approach is taken when the testing team receives the entire software in a bundle. So what is the difference between Big Bang Integration Testing and System Testing? Well, the former tests only the interactions between the units while the latter tests the functionalities in the entire system.
Top Down
This is an approach to Integration Testing where top-level units are tested first and lower level units are tested step by step after that. This approach is taken when top-down development approach is followed. Test Stubs are needed to simulate lower level units which may not be available during the initial phases.
Bottom Up
This is an approach to Integration Testing where bottom level units are tested first and upper-level units step by step after that. This approach is taken when bottom-up development approach is followed. Test Drivers are needed to simulate higher level units which may not be available during the initial phases.
Sandwich/ Hybrid
This is an approach to Integration Testing which is a combination of Top Down and Bottom Up approaches.
Tips
- Ensure that you have a proper Architecture / Technical Design document where interactions between each units are clearly defined. In fact, you will not be able to perform Integration Testing without this information.
- Ensure that you have a robust Software Configuration Management system in place. Or else, you will have a tough time tracking the right version of each unit, especially if the number of units to be integrated is many.
- Make sure that each unit is unit tested before you start Integration Testing.
- As far as possible, automate your tests, especially when you use the Top Down or Bottom Up approach, because regression testing is important each time you integrate a unit and manual regression testing can be tedious / inefficient.
SYSTEM TESTING is a level of software testing where a complete and integrated software is tested. The purpose of this test is to evaluate the system’s compliance with the specified requirements.
ISTQB Definition
- system testing: The process of testing an integrated system to verify that it meets specified requirements.
Analogy
During the process of manufacturing a ballpoint pen, the cap, the body, the tail, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. When the complete pen is integrated, System Testing is performed.
Method
Usually, Black Box Testing method, in which the internal structure of the system under test is unknown, is used. Tests are normally done manually but the trend of test automation, specially for Regression Testing, is picking up.
Tasks
- System Test Plan [Prepare >> Review >> Rework >> Baseline]
- System Test Cases/ Test Scripts/ Test Data [Prepare >> Review >> Rework >> Baseline]
- System Test [Execute >> Re-execute]
When is it performed?
System Testing is the third level of software testing performed after Integration Testing and before Acceptance Testing.
Who performs it?
Normally, independent software testers perform System Testing.
Types
System Testing is the most comprehensive level of testing and many types of tests are performed. Some of them are mentioned below:
- Smoke Testing to ensure that the most important functions work and to decide whether the build is fit for further testing.
- Functional Testing to ensure that features work as per the functional requirements.
- Regression Testing to ensure that changes (enhancements or defect fixes) to the software have not adversely affected it.
- Usability Testing to determine if the system is easily usable from an end-user’s perspective.
- Performance Testing to determine how the system performs in terms of responsiveness and stability under a certain load.
- Security Testing to uncover vulnerabilities of the system and determine that its data and resources are protected from possible intruders.
- Compliance Testing to determine the compliance of the system with internal or external standards.
Test Environment
Since System Testing involves testing the software as a whole, the goal should be to make the test environment as close to the production environment as possible. Else, you will be hurled with production defects that the sophisticated you failed to catch whereas the simple end-users instantly found. What an embarrassment!
Confusion with System Integration Testing
Sometimes, System Testing is confused with System Integration Testing, which is just a type of Integration Testing focusing on the interactions and interfaces between systems. System Testing, on the other hand, takes a more holistic approach and focuses on the entire system’s ability to fulfill the business / software requirements.
___________________________________________________________________________________________
ACCEPTANCE TESTING is a level of software testing where a system is tested for acceptability. The purpose of this test is to evaluate the system’s compliance with the business requirements and assess whether it is acceptable for delivery (or writing that big check).
ISTQB Definition
- acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system.
Analogy
During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. When the complete pen is integrated, System Testing is performed. Once System Testing is complete, Acceptance Testing is performed so as to confirm that the ballpoint pen is ready to be made available to the end-users (students, poets, lovers, and so on).
Method
Acceptance Testing normally uses the Black Box Testing method and is executed manually. Mostly, the testing does not follow a strict procedure and is not scripted but is rather ad-hoc.
Tasks
- Acceptance Test [Execute >> Re-execute]
Test Plan and Test Cases are seldom prepared or followed though it is advised specially for Internal Acceptance Testing.
When is it performed?
Acceptance Testing is the fourth and last level of software testing performed after System Testing and before making the system available in production for actual use.
Types
Acceptance Testing can be categorized into two types (Internal and External):
Internal Acceptance Testing
This type of Acceptance Testing, also known as Alpha Testing, is performed by members of the organization that developed the software but who are not directly involved in the project (Development or Testing). Usually, it is the members of Product Management, Sales and / or Customer Support.
External Acceptance Testing
This type of Acceptance Testing is performed by people who are not employees of the organization that developed the software.
Customer Acceptance Testing
This type of Acceptance Testing is performed by the customers of the organization that developed the software. They are the ones who asked the organization to develop the software. [This is in the case of the software not being owned by the organization that developed it.]
User Acceptance Testing (UAT)
This type of Acceptance Testing, also known as Beta Testing, is performed by the end users (either existing or potential) of the software. They can be the customers themselves or the customers’ customers or the general public.
Note
In a sequential software development model, Acceptance Testing is only conducted at the end of the life cycle whereas in an iterative and incremental development model, it is conducted at the end of each iteration / increment, which helps in boosting the level of confidence in the software at an early stage.
____________________________________________________________________________________________________
Comments
Post a Comment