UnitTestDesign.jl

Describe the configurations your code must handle; it tells you which combinations your tests exercise, and supplies a compact set of additional cases covering the rest.

pkg> add UnitTestDesign

That installs 0.5, which needs Julia 1.10 or later.

When to use it

Example

A solver takes a mode, a factorization and a tolerance. The factorization applies only in exact mode, and exact mode needs a tight tolerance. Write the parameters and those two rules as a TestSpace, and ask for every pair of values:

using UnitTestDesign

space = TestSpace(
    (mode = [:fast, :exact], solver = [:none, :lu, :qr], tol = [1e-3, 1e-6]);
    constraints = [
        @require(mode == :exact || solver == :none),
        forbid((mode = :exact, tol = 1e-3); reason = "exact mode needs a tight tolerance"),
    ])

cases = all_pairs(space)

5 cases (lower bound 4) · strength 2 · Auto: IPOG() · 3 parameters · 12 combinations
excluded: 3 pairs forbidden, 2 impossible under the constraints; see report(cases)
    mode    solver  tol
 1  :fast   :none   0.001
 2  :exact  :none   1.0e-6
 3  :exact  :lu     1.0e-6
 4  :exact  :qr     1.0e-6
 5  :fast   :none   1.0e-6

The five cases hold every pair of values that some valid case can hold. The excluded line counts the pairs that no valid case can hold, and explain names the rules behind one of them, a pair that neither rule mentions on its own:

explain(space, (solver = :lu, tol = 1e-3))

infeasible: no valid case contains (solver = :lu, tol = 0.001); rules 1 and 2 together exclude it (rule 1: @require(mode == :exact || solver == :none); rule 2: exact mode needs a tight tolerance)

coverage measures the interaction coverage of tests you already have:

handwritten = [(mode = :fast, solver = :none, tol = 1e-3),
               (mode = :exact, solver = :lu, tol = 1e-6)]
coverage(handwritten, space)

covers 6 of 11 feasible pairs, 5 missing: (mode = :exact, solver = :none), (mode = :exact, solver = :qr), (mode = :fast, tol = 1.0e-6), (solver = :none, tol = 1.0e-6), (solver = :qr, tol = 1.0e-6)
excluded: 3 pairs forbidden, 2 impossible under the constraints

and must_include keeps those cases first and adds cases for the missing pairs:

all_pairs(space; must_include = handwritten)

5 cases (2 must-include, lower bound 4) · strength 2 · Auto: IPOG() · 3 parameters · 12 combinations
excluded: 3 pairs forbidden, 2 impossible under the constraints; see report(cases)
    mode    solver  tol
 1  :fast   :none   0.001
 2  :exact  :lu     1.0e-6
 3  :exact  :none   1.0e-6
 4  :exact  :qr     1.0e-6
 5  :fast   :none   1.0e-6

Each case is a NamedTuple, so a test loops over them, here with solve as the function under test:

using Test

@testset "solve" begin
    for (; mode, solver, tol) in cases
        @test solve(A, b; mode, solver, tol) ≈ A \ b
    end
end

Test Summary: | Pass  Total  Time
solve         |    5      5  0.5s

The saving grows with the number of parameters. Four parameters of three values each have 81 combinations, and every pair of their values fits in 9 cases:

all_pairs([1, 2, 3], ["low", "mid", "high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim])

9 cases (minimal) · strength 2 · Auto: Construction() · 4 parameters · 81 combinations
    p1  p2      p3   p4
 1  1   "low"   1.0  :greedy
 2  2   "mid"   3.7  :greedy
 3  3   "high"  4.9  :greedy
 4  1   "mid"   4.9  :relax
 5  2   "high"  1.0  :relax
 6  3   "low"   3.7  :relax
 7  1   "high"  3.7  :optim
 8  2   "low"   4.9  :optim
 9  3   "mid"   1.0  :optim

What it promises

• Every returned case is valid: it satisfies every constraint.
• Every combination the design asks for (every pair of values at strength 2, every triple at strength 3) that at least one valid case contains appears in at least one returned case. You never write the rules that other rules imply.
• Every combination it leaves out is attributed: forbidden by rules it names, or impossible because the rules it names combine.
• An unknown is never disguised. If a search reaches its budget, generation stops with a ResourceLimitError that names the limit, and coverage reports the combination as unresolved, with its counts as bounds and no percentage. Nothing is called covered, excluded or complete that was not decided.
• The same call, under the same package and Julia versions, gives the same cases. Auto(), the default engine, uses no randomness; GND draws from a fixed default seed. To keep a list of cases across releases and edits, commit it, or pass it back as must_include.

The designs are compact, with no promise of a minimum number of cases. Each result states a proven lower bound beside its count, and says "minimal" when the count meets it. The default engine, Auto(), chooses for your space between IPOG's design and an algebraic array from a catalog, recommend says what it would choose, and design_sizes shows how many cases each strategy gives before you choose one. The full statement is the contract.

The manual

• Tutorial: the example above, built up one idea at a time, from a one-line call to checking what a design covers.
• How-to guides, one per job: test a function with many options, test generic code across types, plan a CI matrix, run a simulation campaign, audit and extend an existing suite, diagnose a failure, test invalid inputs, combine with property-based testing, and commit a design as data.
• Explanation, for why it works this way: choosing values and oracles, interaction coverage and the evidence, constraints, engines, and IPOG.
• Reference: every exported name, and the migration table from 0.4. 0.5 is a breaking release: disallow is gone in favor of constraints, and positional calls return a TestCases of tuples.
• For AI agents: the decision rule, three patterns, and the one-line check, on one page.
• Developer: the contract, the non-goals, and contributing.