Audit and extend an existing suite

A package already has hand-written tests that call a function with particular options, and you want to know which combinations they leave out before adding more. Write the existing calls as rows, measure them against a space with coverage, and let the generator add only the rows that are missing.

1. Write the existing calls as rows

Suppose runtests.jl already calls an interpolation routine three ways:

interpolate(xs, ys; method = :linear, boundary = :error)                     # Float64, uniform grid
interpolate(xs, ys; method = :cubic,  boundary = :extrapolate)               # Float64, uniform grid
interpolate(xs_irregular, ys; method = :linear, boundary = :clamp)           # Float64, irregular grid

Each call becomes a NamedTuple that names every parameter of the space, including the ones the call sets implicitly, such as the grid and the element type:

using UnitTestDesign

existing = [
    (method = :linear, boundary = :error,       grid = :uniform,   T = Float64),
    (method = :cubic,  boundary = :extrapolate, grid = :uniform,   T = Float64),
    (method = :linear, boundary = :clamp,       grid = :irregular, T = Float64),
]

space = TestSpace((
        method   = [:nearest, :linear, :cubic],
        boundary = [:error, :clamp, :extrapolate],
        grid     = [:uniform, :irregular],
        T        = [Float64, Float32],
    );
    constraints = [@forbid(method == :nearest && boundary == :extrapolate)])

2. Measure them

coverage(existing, space)

covers 16 of 36 feasible pairs, 20 missing: (method = :nearest, boundary = :error), (method = :cubic, boundary = :error), (method = :nearest, boundary = :clamp), (method = :cubic, boundary = :clamp), (method = :linear, boundary = :extrapolate), (method = :nearest, grid = :uniform), (method = :nearest, grid = :irregular), (method = :cubic, grid = :irregular), (method = :nearest, T = Float64), (method = :nearest, T = Float32), and 10 more
excluded: 1 pair forbidden

The three tests hold 16 of the 36 pairs that some valid case can hold. The list names the missing pairs, and excluded: counts the pair the rule forbids, which no case needs. This is interaction coverage, a statement about combinations of values, not about which lines of code ran.

missing_interactions gives the missing pairs as data, one NamedTuple each, for writing targeted tests by hand or for a script to act on:

first(missing_interactions(existing, space), 5)

5-element Vector{NamedTuple}:
 (method = :nearest, boundary = :error)
 (method = :cubic, boundary = :error)
 (method = :nearest, boundary = :clamp)
 (method = :cubic, boundary = :clamp)
 (method = :linear, boundary = :extrapolate)

3. Top up to pairs

Pass the existing rows as must_include. They come first, in the order given, and the generator adds rows only for the pairs they miss:

pairs = all_pairs(space; must_include = existing)

8 cases (3 must-include, minimal) · strength 2 · Auto: IPOG() · 4 parameters · 36 combinations
excluded: 1 pair forbidden; see report(cases)
    method    boundary      grid        T
 1  :linear   :error        :uniform    Float64
 2  :cubic    :extrapolate  :uniform    Float64
 3  :linear   :clamp        :irregular  Float64
 4  :linear   :extrapolate  :irregular  Float32
 5  :cubic    :clamp        :uniform    Float32
 6  :nearest  :clamp        :uniform    Float64
 7  :cubic    :error        :irregular  Float32
 8  :nearest  :error        :irregular  Float32

The summary says "3 must-include": rows 1 to 3 are the existing tests, unchanged, and rows 4 to 8 are the tests to add. A must-include row that breaks a rule is an error naming the row and the rule, so a stale test cannot slip in unnoticed.

coverage(pairs)

covers 36 of 36 feasible pairs
excluded: 1 pair forbidden

4. Extend to triples

The same move takes the pairwise suite to triples. Pass the pairwise rows as must-include rows to all_triples:

triples = all_triples(space; must_include = pairs)
coverage(triples)

covers 56 of 56 feasible triples
excluded: 4 triples forbidden

triples[1:length(pairs)] == pairs

true

The first 8 rows are the pairwise suite, in its order, so the existing tests and the new pairwise rows stay where they were, and the triples add 11 more.

Pitfall: values must match the domain exactly

Values match by identity, the same type and isequal. An existing test that passes the string "linear" where the domain holds the symbol :linear, or 1 where it holds 1.0, is not in the domain, and coverage says which row and parameter:

try
    coverage([(method = "linear", boundary = :error, grid = :uniform, T = Float64)], space)
catch err
    showerror(stdout, err)
end

ArgumentError: coverage row 1: "linear" is not a value of `method`, whose domain is [:nearest, :linear, :cubic].

Write the rows with the values the space uses, or add the value to the domain if the function really accepts both.