What is Orthogonal Array Testing? (Example)

Orthogonal Array Testing

Orthogonal Array Testing (OAT) is a software testing technique that uses orthogonal arrays to create test cases. It is a statistical testing approach, and it is especially useful when the system under test accepts a large number of data inputs. Orthogonal array testing maximizes test coverage by pairing and combining the inputs, so the system is exercised with comparatively fewer test cases and less execution time.

For example, when a train ticket has to be verified, factors such as the number of passengers, ticket number, seat numbers, and train numbers have to be tested. Testing each factor one by one is cumbersome. It is more efficient when the QA engineer combines several inputs together and tests them in a single run. In such cases, we can use the Orthogonal Array testing method.

This type of pairing or combining of inputs to save time is called Pairwise testing, and the OATS technique is one way of producing a pairwise test set. Because the technique works purely from input values and expected results, without any view of the internal code, it belongs to the black box testing family.

Why OAT (Orthogonal Array Testing)?

In the present scenario, delivering a quality software product to the customer has become challenging due to the complexity of the code.

In the conventional method, test suites include test cases derived from every combination of input values and pre-conditions. As a result, n number of test cases has to be covered.

But in a real scenario, the testers will not have the leisure to execute all the test cases to uncover the defects, as there are other activities such as documentation, suggestions, and feedback from the customer that have to be accommodated inside the testing phase.

Hence, the test managers wanted to optimize the number and quality of the test cases to ensure maximum Test coverage with minimum effort. This effort is called Test Case Optimization.

• Systematic and statistical way to test pairwise interactions.
• Interactions and integration points are a major source of defects.
• Execute a well-defined, concise set of test cases that are likely to uncover most (not all) bugs.
• The orthogonal approach guarantees the pairwise coverage of all variables.

The technique therefore sits beside the other input-reduction methods a team already uses. Equivalence partitioning and boundary value analysis decide which values are worth testing for a single field; orthogonal arrays decide which of those values should be tested together.

How Orthogonal Array Testing Is Represented

An orthogonal array is described by a short formula rather than by its contents. The diagram below shows how the three quantities in that formula relate to the shape of the array.

• Runs (N) – Number of rows in the array, which translates into the number of test cases that will be generated.
• Factors (K) – Number of columns in the array, which translates into the maximum number of variables that can be handled.
• Levels (V) – Maximum number of values that can be taken on any single factor.

A single factor usually has 2 to 3 inputs to be tested. That maximum number of inputs decides the Levels.

Read the array name from left to right. In L9(3⁴) the 9 is the run count, the 3 is the number of levels each factor can take, and the superscript 4 is the number of factors. The same article uses L4(2³), which is four runs for three factors that each have two levels.

Two properties make an array orthogonal, and both have to hold before the run count can be trusted:

• Balance: every level of a factor appears the same number of times in its own column.
• Pairwise completeness: for any two columns, every combination of one level from each appears the same number of times.

If the second property fails, the table is simply a shortened test set, not an orthogonal array, and the pairwise guarantee that justifies the reduction no longer applies.

How to do Orthogonal Array Testing: Examples

1. Identify the independent variable for the scenario.
2. Find the smallest array with the number of runs.
3. Map the factors to the array.
4. Choose the values for any “leftover” levels.
5. Transcribe the Runs into test cases, adding any particularly suspicious combinations that are not generated.

The two worked examples below apply those five steps, first to a small two-level problem and then to a larger three-level one.

Example 1

A Web page has three distinct sections (Top, Middle, Bottom) that can be individually shown or hidden from a user.

• No of Factors = 3 (Top, Middle, Bottom)
• No of Levels (Visibility) = 2 (Hidden or Shown)
• Array Type = L4(2³)

(4 is the number of runs arrived at after creating the OAT array)

If we go for the conventional technique of changing one section at a time, we need test cases like 2 X 3 = 6 Test Cases.

Test Cases	Scenarios	Values to be tested
Test #1	HIDDEN	Top
Test #2	SHOWN	Top
Test #3	HIDDEN	Bottom
Test #4	SHOWN	Bottom
Test #5	HIDDEN	Middle
Test #6	SHOWN	Middle

Those six runs vary one section at a time and never place two sections in a chosen state together. Testing every combination instead would take 2³ = 8 runs.

If we go for OAT Testing we need 4 Test cases as shown below:

Test Cases	TOP	Middle	Bottom
Test #1	Hidden	Hidden	Hidden
Test #2	Hidden	Visible	Visible
Test #3	Visible	Hidden	Visible
Test #4	Visible	Visible	Hidden

Check any two columns of that table and all four Hidden/Visible pairs appear exactly once, which is the orthogonality property in its smallest form.

Example 2

A microprocessor’s functionality has to be tested:

• Temperature: 100C, 150C and 200C.
• Pressure : 2 psi, 5 psi and 8 psi
• Doping Amount : 4%, 6% and 8%
• Deposition Rate : 0.1 mg/s, 0.2 mg/s and 0.3 mg/s

By using the conventional method we need 3⁴ = 81 test cases to cover all the inputs. Let us work with the OATS method instead:

No. of factors = 4 (temperature, pressure, doping amount and deposition rate)

Levels = 3 levels per factor (temperature has 3 levels – 100C, 150C, and 200C – and likewise the other factors also have 3 levels each)

Create an array as below:

1. Columns with the No. of factors

Test case #	Temperature	Pressure	Doping amount	Deposition rate

2. Enter the number of rows equal to levels per factor. i.e. temperature has 3 levels. Hence, insert 3 rows for each level of temperature,

Test case #	Temperature	Pressure	Doping amount	Deposition rate
1	100C
2	100C
3	100C
4	150C
5	150C
6	150C
7	200C
8	200C
9	200C

3. Now split up the pressure, doping amount and the deposition rates in the columns.

For eg: Enter 2 psi across temperatures 100C, 150C and 200C, likewise enter doping amount 4% for 100C, 150C and 200C and so on.

Test case #	Temperature	Pressure	Doping amount	Deposition rate
1	100C	2 psi	4%	0.1 mg/s
2	100C	5 psi	6%	0.2 mg/s
3	100C	8 psi	8%	0.3 mg/s
4	150C	2 psi	4%	0.1 mg/s
5	150C	5 psi	6%	0.2 mg/s
6	150C	8 psi	8%	0.3 mg/s
7	200C	2 psi	4%	0.1 mg/s
8	200C	5 psi	6%	0.2 mg/s
9	200C	8 psi	8%	0.3 mg/s

Hence, in OAs, we need 9 Test cases to cover.

Accuracy note: filling the remaining three columns straight down, as step 3 does, repeats the same pressure, doping and deposition triple inside every temperature block. Pressure and doping amount therefore move together, and pairs such as 2 psi with 6% never appear, so the nine rows above are a reduced test set rather than a true orthogonal array. The balanced L9(3⁴) array below keeps the same nine runs while covering all nine level pairs between every two factors.

Test case #	Temperature	Pressure	Doping amount	Deposition rate
1	100C	2 psi	4%	0.1 mg/s
2	100C	5 psi	6%	0.2 mg/s
3	100C	8 psi	8%	0.3 mg/s
4	150C	2 psi	6%	0.3 mg/s
5	150C	5 psi	8%	0.1 mg/s
6	150C	8 psi	4%	0.2 mg/s
7	200C	2 psi	8%	0.2 mg/s
8	200C	5 psi	4%	0.3 mg/s
9	200C	8 psi	6%	0.1 mg/s

Orthogonal Array Testing vs Pairwise (All-Pairs) Testing

The two terms are used interchangeably in many teams, yet they are not identical. Both aim at covering pairs of values, but they differ in how strictly the resulting table is constrained.

Aspect	Orthogonal array testing	Pairwise (all-pairs) testing
Underlying structure	Orthogonal array, borrowed from design of experiments	Covering array, generated by a search algorithm
Coverage rule	Every pair appears an equal number of times	Every pair appears at least once
Factors with unequal levels	Needs a mixed-level array, which may not exist for the shape required	Handled directly by the generator
Typical run count	Equal or slightly higher, because of the balance requirement	Equal or lower for the same inputs
Constraints between values	Difficult to express inside the array	Supported by most modern generators

In practice, a balanced orthogonal array is the clearer teaching model and the safer choice when factors share the same level count, while a covering array is what most tools produce for real projects with awkward, unequal inputs. Both are complementary to decision table testing and state transition testing, which target business rules and sequence rather than combinations of values.

Orthogonal Array Testing Tools

Building an array by hand is practical only for the small examples above. Beyond four or five factors, teams use a generator, and the tool decides the run count for them.

• PICT: an open-source command-line generator from Microsoft. Parameters and their values are written in a plain text model file, and the tool prints the combination set. Constraints and sub-models are supported, which matters when certain value pairs are invalid. See the PICT repository.
• ACTS: a research tool from NIST that generates covering arrays for two-way up to six-way coverage, and also publishes a library of pre-computed arrays. It is available from the NIST combinatorial testing project.
• Commercial combinatorial platforms: hosted tools add requirement traceability, expected results and export into a test management system, which suits larger teams that need an audit trail.
• Published array libraries: standard arrays such as L4, L8, L9, L16 and L18 are tabulated and can be copied directly when the factor and level counts match.

Whichever tool produces the table, the generated rows still have to be turned into runnable cases with real data and expected results, and then scheduled like any other suite inside software testing techniques already in use on the project.

Advantages of Orthogonal Array Testing

• Guarantees testing of the pair-wise combinations of all the selected variables.
• Reduces the number of test cases.
• Creates fewer test cases that still cover the interaction of every pair of variables.
• A complex combination of the variables can be handled.
• Is simpler to generate and less error-prone than test sets created by hand.
• It is useful for Integration Testing, and equally useful for configuration testing, where browser, operating system and device combinations multiply quickly.
• It improves productivity due to reduced test cycles and testing times.

The saving compounds during regression testing, because a smaller balanced suite is cheap enough to re-run every cycle, whereas an exhaustive combination suite is usually run once and then quietly abandoned.

Disadvantages of Orthogonal Array Testing

• As the data inputs increase, the complexity of the test case design increases. As a result, manual effort and time spent increase, and the testers have to go for Automation Testing.
• Pairwise coverage is not full coverage. A defect that only appears when three specific values occur together can pass through the array untouched.
• Factors with different numbers of levels do not map onto a standard array cleanly, so a mixed-level array or a covering array generator is needed.
• Choosing the levels is a judgement call. If a value that matters is left out of the input model, no array can recover it.
• Constraints between values, such as a payment method that is only valid in one country, are awkward to express and usually produce invalid rows that must be removed by hand.

None of these limitations rule the technique out; they simply set its boundary. Orthogonal arrays reduce the combination problem, while risk analysis and domain testing decide which factors deserve a place in the model to begin with.

Mistakes or errors while performing OAT

• Directing the testing effort at the wrong area of the application.
• Picking the wrong parameters to combine.
• Using orthogonal array testing where the input set is small enough to test exhaustively.
• Applying orthogonal array testing manually on a model large enough to need a generator.
• Relying on orthogonal array testing alone for high-risk applications, where deeper coverage is required.

The last item is the one that causes the most damage in practice. On a safety-critical or financial feature, the pairwise set is a floor rather than a ceiling, and it is normally supplemented with targeted cases at system testing level and with the wider plan described in types of software testing.

FAQs