Learn

Share Learnings

Clear Introductions

The following materials are clear and accurate descriptions of several different kinds of combinatorial software testing.  Reading them in order will give you a good understanding of pairwise, orthogonal array-based, model-based and more comprehensive software testing methods.  All of these types of combinatorial software testing approaches seek to answer the same general question, namely, "given a particular quality (or thoroughness) objective for your testing needs, what combination of test conditions can accomplish the objective in the fewest number of test cases?" 

In addition to the articles on this page, you might also find several instructional videos helpful as well.

                                                  
 Title / Author       Illustrative Excerpts from the Introductions  Additional Comment

Elfriede Dustin
"It’s a laudable goal: to conduct a thoroughly exhaustive test of a system.  In the real world, however, it’s generally not possible, feasible, or cost effective to test a system using all the possible variations and combinations of test parameter."  "This technique is very useful for finding a small set of tests (from a large number of possibilities) that exercises key combinations."
This is a very clearly-written introduction to orthogonal array-based test planning techniques.  The author also includes "5 mistakes to avoid" which are worth a read, particularly (2) "Focusing the testing effort on the wrong area of the application", and (5) "Picking the wrong parameters to combine".


Pairwise Testing 

 Michael Bolton
"We can save time and effort and find bugs efficiently by testing variables and values in combination."  
"...Only a mathematician could appreciate that definition. Let's figure out what it means in practical terms."


A straightforward, no-nonsense, explanation of the nuts and bolts of both pairwise and Orthogonal Array-based methods.  Many people use "pairwise" and "OA" interchangeably.  In fact, as correctly explained by Michael Bolton, test conditions generated by the approaches can be quite different in terms of (a) quantity of test scripts generated and (b) the distribution of test points.  

Richard Kuhn, Raghu Kacker, Jeff Lei, Justin Hunter

A 10-project empirical study that compared the effectiveness of our pairwise/2-way testing with manual test case selection methods resulted in average improvements of 2.4 times as many bugs found per tester hour and 13% more bugs found overall. The projects were conducted at six companies (each with testing teams using different approaches working in parallel) and tested actual commercial applications in development.    
This is another good, introductory article about combinatorial software testing.  It highlights more thorough methods of software testing that go beyond simple pairwise (or 2-way) testing coverage that is found in descriptions of orthogonal array testing and pairwise software testing articles.  3-way, 4-way, 5-way, and 6-way testing methods are discussed.  As supported by "Figure 2" on the second page of the article, these n-way solutions offer more thorough coverage than 2-way tests are able to achieve (at the "cost" of needing to execute additional tests).


Justin Hunter

"My suspicion is that a significant reason that combinatorial software testing methods are not much more widely adopted is that many of the articles describing it are simply too complex and/or too abstract for many testers to understand and apply.... In this photograph-rich, numbers-light, presentation, I’ve tried to explain what combinatorial testing is all about without the wonky-ness."
This accessible and humorous presentation takes a fresh approach to describing pairwise and combinatorial software testing. It dispenses with sterile formulas and presents the user with a lot of fun, evocative images to convey the benefits of using these Design of Experiments-inspired software test design methods.


Madhav S. Phadke




"To improve customer satisfaction and reduce development costs, it is imperative that software teams reduce testing costs, reduce product introduction delays, and send fewer faults to the field. The Robust Testing TM method, based on the special properties of orthogonal arrays, can achieve these goals by generating test cases that are more efficient and thorough in finding faults. In many cases, the Robust Testing method has increased tester productivity by a factor of two or better."     
Dr. Phadke is an advocate of orthogonal-array-based test case selection methods and was one of the first experts to apply Taguchi's "Robust Design"principles to software testing.  He sees OA as generally superior to pairwise methods, in part, because OA methods are designed to achieve a more uniform  distribution of test points within a test domain.




D.M. Cohen, S.R. Dalal, M.L. Fredman, G.C. Patton 
[Combinatorial Testing] is "used in a variety of applications for unit, system, and interoperability testing. It has generated both high-level test plans and detailed test cases. In several applications, it greatly reduced the cost of test plan development....  Testers can base [their] input on detailed development requirements or on a system's high-level functional requirements, such as its user manual.... [The approach] is widely applicable and generates efficient test sets of good quality."
As we are clearly focused on spreading pragmatic, actionable insights to testing practitioners and business-minded IT professionals, we will avoid posting articles filled with equations most testers would find impenetrable.  Even though this article delves into more technical "Greek-letter-speak" than most of the other articles listed here, it is a clearly-written report that does a good job at explaining how Combinatorial Testing methods are actually applied, using several real-world testing scenarios.
Practical Combinatorial Testing Beyond Pairwise Testing

Rick Kuhn, Yu Lei, Raghu Kacker

"In the past few years, advances in covering-array algorithms, integrated with model checking or other testing approaches, have made it practical to extend combinatorial testing beyond pairwise tests."



This is an excellent introduction to efficiently testing for potential multi-modal faults that might be missed by smaller pairwise test plans. The article covers what to to when guaranteeing that all pairs of parameter settings will be tested is not sufficient to meet your quality goals.  The solution involves n-wise, covering array-based, combinatorial testing methods.