Test a function with many options

A function with four or five options has more combinations than anyone writes by hand, and most of the faults in such code need only one or two options set a particular way. Name each option's values in a TestSpace, ask all_pairs for cases in which every pair of values appears, and loop over the cases in a @testset.

The function under test here is smooth(x; window, boundary, kernel), a moving-window filter, defined in a hidden block. Any function with options works the same way.

1. Describe the options

Give each option a name and the values worth trying, in the order you want them tried. Add a rule for each combination the function does not support.

using UnitTestDesign, Test

space = TestSpace((
        window   = [1, 3, 5],
        boundary = [:clamp, :reflect, :periodic],
        kernel   = [:box, :triangle, :gaussian],
        n        = [1, 4, 100],        # the length of the input
    );
    constraints = [
        @forbid(window == 1 && kernel != :box),
        @forbid(boundary == :reflect && n <= window ÷ 2;
                reason = "reflect needs more points than half the window"),
    ])

TestSpace with 4 parameters, 81 combinations, 2 constraints
  window:   1, 3, 5
  boundary: :clamp, :reflect, :periodic
  kernel:   :box, :triangle, :gaussian
  n:        1, 4, 100

2. Generate the cases

cases = all_pairs(space)

12 cases (lower bound 9) · strength 2 · Auto: Construction() · 4 parameters · 81 combinations
excluded: 2 pairs forbidden; see report(cases)
     window  boundary   kernel     n
  1  1       :clamp     :box       1
  2  5       :periodic  :gaussian  1
  3  3       :periodic  :box       4
  4  5       :clamp     :triangle  4
  5  3       :clamp     :gaussian  100
  6  5       :reflect   :box       100
  7  1       :reflect   :box       4
  8  1       :periodic  :box       100
  9  3       :reflect   :triangle  100
 10  5       :reflect   :gaussian  4
 11  3       :periodic  :triangle  1
 12  1       :reflect   :box       1

The summary line says what you asked for: 12 cases at strength 2 over 4 parameters, whose full product has 81 combinations, from the default engine, Auto, which here kept the catalog's array, seeded under the rules (Construction()), over IPOG's 13 cases. Every pair of values of every two options appears in at least one case, among the pairs some valid case can hold.

The excluded: line counts the pairs that no case holds because the rules exclude them: here, a window of 1 with a kernel other than :box. They are not gaps in the design, since no valid case could contain them. When a space also shows pairs "impossible under the constraints", no single rule forbids those pairs, but the rules together leave no valid case that holds them.

3. Loop over the cases

@testset "smooth" begin
    for (; window, boundary, kernel, n) in cases
        y = smooth(fill(2.0, n); window, boundary, kernel)
        @test length(y) == n
        @test all(≈(2.0), y)       # a constant input stays constant
    end
end

Test Summary: | Pass  Total  Time
smooth        |   24     24  0.1s

Each case is a NamedTuple, so (; window, boundary, kernel, n) picks the values out by name, and reordering the space cannot swap two arguments. The test needs an oracle that holds for every combination of options; a property such as "a constant stays constant" is often easier to state than an exact answer. See Choosing values and oracles.

4. Read the excluded line in full

report measures the cases again and lists each excluded combination with the rule that excludes it:

report(cases)

12 cases cover all 52 feasible pairs of an 81-combination space (2 pairs forbidden)
excluded:
  (window = 1, kernel = :triangle): forbidden by rule 1 (@forbid(window == 1 && kernel != :box))
  (window = 1, kernel = :gaussian): forbidden by rule 1 (@forbid(window == 1 && kernel != :box))
size: 12 cases; lower bound 9: the 3 × 3 = 9 combinations of window and boundary need a case each
bonus: 46 of 92 feasible triples covered
prefix curve:
  first 3 of 12 cover 34% (18 of 52)
  first 5 of 12 cover 57% (30 of 52)
  first 8 of 12 cover 80% (42 of 52)
  first 10 of 12 cover 92% (48 of 52)
  first 12 of 12 cover 100% (52 of 52)
seed: none (Auto uses no randomness)

The bonus: line is the interaction coverage the pairs give at the next strength: 50 of the 92 feasible triples already appear. The prefix curve says how much the first cases cover, for a suite that runs only some of them.

5. When to go to triples

Go to all_triples when a fault is likely to need three options set together, or when cases are cheap enough that the extra rows do not matter. To raise the strength only for the options that interact, name them in stronger:

(length(all_triples(space)),
 length(all_pairs(space; stronger = [(:window, :boundary, :n) => 3])),
 length(full_factorial(space)))

(31, 25, 57)

Every triple takes 31 cases, triples within the three named options 25, and every valid case 57. These counts are the rows the engine produced for this space, not lower bounds. design_sizes prints them side by side with the pairs and triples each design covers, as in Run a simulation campaign.

Pitfall: a rule means "never tested here"

A rule removes a combination from this test entirely. smooth with boundary = :reflect and too few points throws an error instead of returning a value. If throwing is the documented behavior, that is worth a test of its own, with the rejected combinations as its cases: see Test invalid inputs.