Leveraging Pairwise Test Technique For Cross Browser Testing
Cross Browser Testing ⢠Miscellaneous â˘
155928 Views | 19 Min Read

The primary responsibility of the test lead or the test analyst is to come up with âTest case design & Planningâ in order to attain maximum test coverage. As it is nearly impossible to achieve 100% test coverage (unless and until your product has very simple requirements & implementation), the test team should come up with test cases/test suites that can help in testing the core functionalities of the product, thereby achieving the best coverage results. Along with test coverage, another Key Performance Indicator (KPI) that is critical as far as software testing is concerned is âDefect Yield Rateâ. Higher the test coverage and defect yield rate, better the quality of the product.
Some things are easier said than done and the same can be said about âWebsite Testingâ. Due to factors like reduced Time To Market, growing business competition, tight deadlines, and other varying factors; the development and test teams are always under intense pressure to maximize their productivity (thereby improving the Return on Investment [ROI]). For the development team, the criteria could be LOC (Lines of Code) pushed to the project/product repository; for the test team, the criteria could be number of tests executed/bugs raised/number of tests developed.
Browser compatibility testing as we know is practice where the permutations and combinations could turn out to be overwhelming. It is important to invest your efforts in planning & executing the right tasks. In order to achieve the same, your test team should make use of a smart testing approach i.e design, plan & execute test cases that can help in achieving the best results. âPairwise Testingâ is one such testing methodology that can be used to get the best test coverage by writing effective test cases.
Here, I will be portraying a pairwise testing tutorial, I will be doing so by demonstrating a pairwise testing example. We will also look on how to leverage pairwise testing algorithm for performing cross browser testing.
Introduction To Pairwise Testing
Pairwise Testing, also called as all-pairs testing is a black-box test design technique that can deliver nearly 100% test coverage. As per the ISTQB (International Software Testing Qualifications Board), the official definition of Pairwise Testing is as mentioned below
âPairwise Testing is a black box test design technique in which test cases are designed to execute all possible discrete combinations of each pair of input parameters.â
In a complex project, the output for majority of the test cases may not depend on one single parameter. There could be multiple factors like state transitions/state machines, input parameters, shared parameters, variable factors like accessible memory/user preferences, etc. depending on the type & domain of the project. Though there are methodologies like Boundary Value Analysis & Equivalence Partitioning to derive values of these variable factors; their usage can only be limited to arrive at the âprobable valuesâ that these factors can have.
Though you may come up with the possible values for these variable factors, come up with test cases/test suites that can cover all the combinations can be a herculean task. Hence, it becomes critical to come up with a âsubset of combinationsâ which when inputted to the test cases result in achieving the best results i.e. maximum test coverage.
The lingering question could be âHow do you give precedence to certain test combinations over others?â and what if you do not test some input combinations that may have affected the overall test coverage. This is where âPairwise Testingâ or âAll-pairs Testingâ can be used for coming up with effective tests that have multiple input parameters. The mechanism used by pairwise testing to come up with discrete test combinations will be discussed in subsequent sections.
Pairwise Testing – An Example
Pairwise testing should be used in scenarios where the application/web-app being developed has huge number of entries and large test configurations. For instance, if you are working on a website/web-application that can be downloaded & used by consumers from across the globe, you need to make sure that it works seamlessly on different browsers/devices/operating systems. Your job is to devise a foolproof cross browser testing strategy so that the application is tested on different combinations that yield best possible test results.
In order to come up with different test combinations for browser compatibility testing using the pairwise testing algorithm, techniques like âOrthogonal Arrayâ and tools that make use of âClassification Treeâ concept can be used.
Before we had a look at a âcross browser testingâ use case; to start with, let us have a look at a rather simpler pairwise testing example.
Pairwise Testing Example – Problem Statement (Mobile E-Commerce O2O Application)
- For a pairwise testing example, we will consider an e-commerce store (or a store which is an integration of O2O i.e. Online to Offline) which caters to âbuying & sellingâ of mobile phones. Consumers can âBuyâ new/old phones from the online medium or from the physical outlet/store.
- Apart from buying phones of the brand (Samsung, One Plus, Oppo), customers can also sell their old phones online or by visiting their store.
- An online store is open 365/7 whereas the store is open only during Business working Hours (9 AM ~ 8 PM).
- Customers should have multiple touch points in order to have a look & feel of the product (primarily phones from Samsung, One Plus, Oppo) and there should be the facility to âbuy/sellâ old/new phones. Due to business reasons, the store (online, as well as offline) is only operational in cities of Hyderabad & Mumbai only.
Once we have the requirements, it is time to have a look at the different parameters (fixed & variable) involved in development & testing of the application
Variables & Operations Involved In The Process Of Pairwise Testing Example
- Transaction category – Buy and Sell (2 combinations)
- Model category – Samsung, One Plus, Oppo (3 combinations)
- Location category – Mumbai, Hyderabad (2 combinations)
- Booking type/category – In-store, online (2 combinations)
- Timings – Working (9 AM ~ 8 PM), Non-Working (365/7, only in case of online category) (2 combinations)
- IMEI Validity – Valid (phones with Indian IMEI for only Samsung, Oppo, One Plus phones), Invalid (international unlocked phones) (âNâ combinations for phones with valid IMEI, letâs assume âNâ = 1000)
Now, that we have the details about the requirements. Can you imagine the number of possible combinations.
(2 * 3 * 2 * 2 * 2 * 1000) = (48,000) test combinations
In order to cater to these many combinations, the test team might need to invest a significant amount of time required for test case development & testing. If we look closely, IMEI can be categorized in a simplistic manner and classified in only âtwo main categoriesâ (valid and invalid) i.e. 2 combinations.
We again have a look at the number of valid combinations.
(2 * 3 * 2 * 2 * 2 * 2) = 96 combinations
To arrive at the best possible test combinations, we make use of the âOrthogonal Arrayâ technique, using which we get pre-defined arrays with respect to combination for every generation of the test case. To start with, the number of columns in the array is normally number of entries + one (One extra column is added for serial number).
In the application which we are planning to test, our columns would be Model, Transaction, IMEI, Location, booking type, timings.
The number of entries that each column can have is below
Before we generate the combination of test cases, we have to make sure that the column which has the maximum number of values (in our case, it is Model) should be the first column. The orthogonal array for pairwise testing example of âMobile e-store O2O Applicationâ would look like below
As you can see from the above array, Phones can be purchased/sold via online/offline medium. Irrespective of the medium being used, only customers from Hyderabad, and Mumbai can use the application. Customers can access the online store at anytime and offline store is open only during âBusiness hoursâ. We now work on the different test combinations that are possible with the available data set.
To start with, fill all the columns starting with the one which can take the maximum number of entries i.e. in our case, it is âModelâ (3 entries). The order columns are updated in decreasing chronological order. The array would look something as shown below
In order to come up with the different combinations for our pairwise testing example, we make a list of patterns that are possible for one particular model (e.g. Samsung). Phone of model Samsung can have âBuy/Sellâ option and that operation is only possible for cities like Mumbai & Hyderabad. Booking can be done âin store/onlineâ. If the IMEI is valid, the necessary operation (Buy/Sell) can go through, else the operation is rejected.
Based on the analysis, below is the table derived from our requirements. As you can see, there are two entries for âBuy + Hyderabadâ and âSell + Mumbaiâ. In order to have different combination of testcases, we have created a new combination under One Plus for âBuy + Mumbaiâ & âSell + Hyderabadâ
By looking at the entries, we observe that is not a fair distribution for âIMEI Validityâ columns & âOrder Categoryâ columns i.e. there is no entry for âBuy + Invalidâ and âSell + Validâ. Hence, we perform a swap operation for the last 2 entries in the âIMEI Validityâ.
A potential combination could be when a particular product (irrespective of the brand) is brought during the ânon-working hoursâ. We do not bother about the manner in which the product was purchased i.e. online/in store. Hence, we add two more test cases where all the column entries are âNAâ except for âTimingsâ. With only two values for Timings, we add another two test cases. The final test array has 8 entries and is shown below
As shown in the final computation table, we have narrowed down from 96 test combinations to only 8 combinations. This would reduce the overall complexity involved in execution of these test combinations. The combinations that are generated for pairwise testing might differ from one scenario to another.
You could even make use of pairwise testing online tools to create different test combinations. For the purpose of verification, we use this pairwise testing tool. The input values that were provided to the tool are shown below
Once you have keyed in the input values for your pairwise testing example, you have the option to generate âAll combinationsâ (by clicking âGenerate All Combinationsâ, as well as âPairwise combinationsâ (by clicking âGenerate Pairwiseâ). As expected, when we clicked on âGenerate All combinationsâ, it generated 96 different
combinations.
When we clicked âGenerate Pairwiseâ, there were 9 different combinations generated, most of them are inline with the combinations that were generated earlier using the âOrthogonal Arrayâ technique. Here is the permalink to the test combination generation. Below is the snapshot of the combinations created by the pairwise testing tool.
Now that you have a fair idea about the usage of pairwise testing with example mentioned above, let us have a brief look at the scenarios when using it would be highly beneficial.
Pairwise Testing – When To Use It?
Though Pairwise testing can be highly effective in locating issues/bugs in the product, the test analyst or test lead needs to decide whether it should be used for testing their product. Pairwise testing is based on the âcoupling effectâ, a research which was conducted by Jefferson Offutt in the year 1992. As per this finding, major complicated faults are implicitly tested if the product is explicitly tested for simple faults.
This means that if there is a fault that occurs due to settings applied to some configuration variables, there is a high possibility that the fault could be the setting of a âsmaller subset of configuration variablesâ.
Combinational data generation can be used to extend the capabilities of pairwise testing to triples, quartets, etc. In a nutshell, if the configuration variables/test combinations (data set) included in the pairwise test plan are âideal for testingâ (i.e. testing that combination indirectly means that many other combinations have been tested), you can achieve the best results from this test approach. Letâs have a look at some of the major advantages of Pairwise testing.
- It can be used in any type & scale of the project, irrespective of the domain being used i.e. medical, agriculture, technology, etc.
- As pairwise testing is based on the principle that majority of the bugs are triggered through an input parameter or due to an interaction between a pair of parameters, there is a significant reduction in the number of test cases being executed. The test cases for pairwise testing can be instrumental in attaining higher test coverage with better quality assurance.
- There is a significant reduction in the effort involved in designing test cases. As the focus is on the test cases that have the maximum impact, teams can achieve better test coverage with fewer efforts.
Though pairwise testing or all-pairs testing has a lot of advantages, there are certain limitations as well. Some of the limitations are below:
- Pairwise testing might not be effective if the interaction between the configuration variables is not understood while coming up with the test cases.
- As the test combinations are derived using techniques like Orthogonal Array, there are possibilities where the most important inputs are not given proper consideration during test case design & development. In case of cross browser testing; âbrowser versionâ is an important parameter and testing may falter in case certain browser versions are not tested to the full potential.
- The ROI (Return on Investment) of Pairwise testing may be less in case the input values that are used for testing are incorrect. Though boundary value testing is important to achieve maximum test coverage, it should not happen that testing is performed with incorrect input values. Take the case of testing your web application/website on different devices. Having an input combination of Linux & Internet Explorer (IE) may not fetch any results and should be ignored. If there are more test cases that focus on such category of inputs, lesser would be the advantage of using pairwise testing.
Hence, it is recommended to perform a feasibility test before opting for pairwise testing. If there are other projects in your organization where pairwise testing has been used, test leads should have a look at the learning (wrt pairwise testing) and other aspects related to testing.
Using Pairwise Testing To Speed Up Cross Browser Testing
As mentioned earlier, software products have to be tested for âcross-browser + cross-platform + cross-deviceâ compatibility before they are released to the end customers. There could be a possibility that your target customers might be using âold version of Chrome browser on Windows 7â or âlatest version of Firefox on Ubuntuâ, etc.
If the product is not verified for âoverall compatibilityâ, it could result in average customer experience as there could be issues in UI or in functionality. For example, the product explainer video is located on YouTube, but the target browser does not have âAdobe Flash supportâ. Hence, it is important that the product goes under cross browser testing against browsers/operating systems/devices that are used by your âtarget customer/audienceâ. Maintaining an in-house infrastructure to accommodate different test devices having multiple versions of browsers is impractical & non-scalable. This is where LambdaTest as a cross browser testing tool on the cloud can be beneficial for testing. You can perform Automation testing using their online Selenium grid, UI regression testing, Responsive testing, live interaction testing, and so much more on the LambdaTest platform.

The section where we discussed about the âPros & Cons of Pairwise testingâ clearly mentioned about the importance of focusing on the test combinations that can give the maximum ROI as far as pairwise testing is concerned. There are scenarios where you have to perform cross-browser testing of your product and you focus your effort on âtest combinations that are invalidâ or âignore combinations that matter the mostâ. Create a browser compatibility matrix to keep your priorities in check. Also, use a web analytics tool to track browsers which bring your the most traffic.
Letâs consider the previous pairwise testing example of âMobile e-store O2O Applicationâ where your test team has to verify the basic functionalities (customer signup, sign-in, product search, and product checkout) of the online store on different browsers (Firefox, Chrome, Internet Explorer, Safari, etc.), Operating systems (iOS, macOS, Windows, Ubuntu, Android, etc.). To keep the scenario less complex, we have not introduced the âbrowser versionsâ/âOS versionsâ/device type, etc.
Below is the input table that consists of the requirements. As we would use the orthogonal array technique for thegeneration of test combinations, âOperating Systemâ would be the first column in the array.
You need to apply the same logic which was explained in the section to generate test combinations for pairwise testing. Below is the list of the test combinations to be used for cross browser testing based on our pairwise testing example.
As seen in the combinations above, there are certain cases which can be considered as âfalse-positivesâ e.g. Internet Explorer (IE) is not present on iOS & macOS operating systems. Hence, these combinations have to be ignored even though it is a part of the test array.
Complete list of test combinations for the above use case can be found at here. As a test team or test lead, your job does not end at generation of the test combinations, but to carefully allocate âadequate resourcesâ (manpower & time) to verify combinations that are important for your project. If you focus on the wrong combinations, you may not be able to achieve adequate test coverage and your product may be released with bugs.
Once you have identified the necessary combinations, you can make use of cloud based cross browser testing provided by LambdaTest. Using LambdaTest Capabilities Generator, you can use automation test framework like Selenium with languages like C#, Python, Java, Ruby, PHP, Javascript.
The capability generator provides you with code for different combinations using parameters like Operating System/Browser type/Browser version/Resolution, etc, based upon your selections. As you already have a list of test combinations generated using the Orthogonal Array tech.
In case your team is not well equipped with automation techniques, they can make use of manual cross browser testing on LambdaTest. While planning these tests, it is important to keep a track of the test combinations generated for pairwise testing since the basis of any type of testing is the âtest combinationsâ.
Pairwise Testing tools
There are many tools available for generation of test combinations for pairwise testing. You can find the complete list of tools over here. It is necessary that you have a look at the product license and necessary terms & conditions before short listing any tool.
Conclusion
There are many testing methodologies that are used for testing different categories of products/projects. Pairwise testing is one approach that should be used in scenarios where you want to achieve the best test coverage with optimum efforts. Similar to any test approach, it is recommended that you have a look at case-studies/projects in your organization where pairwise testing was successfully deployed and fetched good results.
Combinations generated for pairwise testing example can be used to verify your product for cross browser testing by making use of LambdaTest. The core USP of pairwise testing is âachieving good test results in minimum time durationâ and using LambdaTest, you can double the benefits of pairwise testing.
Written by Himanshu Sheth
Himanshu Seth is an engineer with 15+ years of experience. He has worked with multi-national companies as well as startups. He is also an avid blogger on technology.