Migration from 0.4

UnitTestDesign 0.5 is a breaking release. Most 0.4 code still runs, with deprecation warnings, but three things change under it: disallow is gone, positional calls return a TestCases of tuples instead of a Vector{Vector{Any}}, and GND gives the same cases on every call. This page lists every removed and deprecated spelling, with its replacement and a before-and-after example. The "after" code runs when the manual is built.

The default engine was IPOG in 0.4 and is Auto() in 0.5. Auto() keeps the smaller of IPOG's design and an algebraic array from a catalog (Construction) where the catalog applies, which is mostly where every parameter has the same number of values, and gives IPOG's design elsewhere. On the package's benchmark spaces it never has more cases than IPOG, and on 30 of 85 spaces of equal value counts or of strength + 1 parameters it has fewer. A call that names engine = IPOG() gets IPOG's design, as the 0.4 default did. IPOG's own cases change too: 0.5's IPOG finds each case's values by lookup and keeps the smallest of four designs (IPOG). Over the package's benchmark grid, 1,826 spaces and strengths, it has as many cases as the IPOG it replaced or fewer at 96.7% of them, and 2% fewer in total, but more at about 3%: usually one to four more, up to about 9% more, and 160 more (5.6%) for ten parameters of four values at strength 5. So no engine repeats 0.4's cases exactly; to keep them, save them and pass them back as must_include, which keeps every saved case (Commit a design as data). 0.5 also adds Compact, which removes rows from any engine's design. What a result shows and keeps changes too: its summary line names the engine and, for Auto, what it chose (Auto: IPOG()), it and report show a lower bound beside the count, TestCases and Report gain a record field, with the bound, the engine's configuration and the stages that ran, and DesignSizes gains engines and an engine on each row; see Engines.

Compat bounds

Under Julia's semantic versioning a minor release before 1.0 is breaking. A compat entry of UnitTestDesign = "0.4" or "^0.4" means [0.4.0, 0.5.0), so Pkg never upgrades a package that declares it to 0.5. Raise the bound to "0.5" after making the changes below. A project with no compat entry for UnitTestDesign gets 0.5 at its next update.

The table

A deprecated spelling still works. It warns through Base.depwarn, which Julia shows under --depwarn=yes, as Pkg.test runs, and it will be removed in the next breaking release. Passing a deprecated keyword together with its replacement (n_way and strength, seeds and must_include, wayness and stronger, M and candidates) is an ArgumentError, even when one of them is at its default value.

A removed spelling fails with Julia's ordinary error: disallow and Counter with a MethodError for an unsupported keyword, Excursion and generate_tuples with an UndefVarError.

disallow becomes rules on a space

In 0.4 a disallow function received every argument of a row. The generator also called it on partial rows, passing nothing for the arguments not yet chosen, so a rule that did more than compare values needed a guard:

# 0.4
disallow(n, level, value, kind) =
    value !== nothing && kind !== nothing && value > 4 && kind == :optim

all_pairs([1, 2, 3], ["low", "mid", "high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim];
          disallow = disallow)

In 0.5 the parameters have names, and a rule names the ones it reads:

using UnitTestDesign

domains = (n = [1, 2, 3], level = ["low", "mid", "high"], value = [1.0, 3.7, 4.9],
           kind = [:greedy, :relax, :optim])
space = TestSpace(domains; constraints = [@forbid(value > 4 && kind == :optim)])

cases = all_pairs(space)

10 cases (lower bound 9) · strength 2 · Auto: Construction() · 4 parameters · 81 combinations
excluded: 1 pair forbidden; see report(cases)
     n  level   value  kind
  1  1  "low"   1.0    :greedy
  2  2  "mid"   3.7    :greedy
  3  3  "high"  4.9    :greedy
  4  1  "mid"   4.9    :relax
  5  2  "high"  1.0    :relax
  6  3  "low"   3.7    :relax
  7  1  "high"  3.7    :optim
  8  3  "mid"   1.0    :optim
  9  2  "low"   4.9    :greedy
 10  2  "low"   1.0    :optim

The guard is gone. A rule receives values of the parameters it names and nothing else, and only once each of them has a value: it never sees nothing as a placeholder. If a domain lists nothing, a rule receives it as a real value.

The same rule has three other spellings. Each builds a Constraint, and each allows the same rows:

spellings = [
    forbid(:value, :kind) do v, k
        v > 4 && k == :optim
    end,
    require(:value, :kind) do v, k
        v < 4 || k != :optim
    end,
    # A whole-case rule receives the complete row, as `disallow` did. It is
    # the most direct translation and the slowest to search; prefer names.
    forbid(case -> case.value > 4 && case.kind == :optim),
]
all(full_factorial(domains; constraints = [rule]) == full_factorial(space) for rule in spellings)

true

A positional call takes no rules. Name the parameters, with a TestSpace as above or with named domains and constraints =, as in all_pairs(domains; constraints = spellings[1:1]). The same goes for full_factorial(...; disallow = f), which becomes full_factorial(space).

To check a rule, ask the space; explain names the rules that exclude a combination:

explain(space, (value = 4.9, kind = :optim))

forbidden by rule 1 (@forbid(value > 4 && kind == :optim))

Constraints describes what a rule may read and how rules combine.

all_tuples and n_way

# 0.4
all_tuples([1, 2, 3], ["a", "b"], [true, false], [:p, :q]; n_way = 3)

covering([1, 2, 3], ["a", "b"], [true, false], [:p, :q]; strength = 3)

12 cases (minimal) · strength 3 · Auto: Construction() · 4 parameters · 24 combinations
     p1  p2   p3     p4
  1  1   "a"  true   :p
  2  2   "a"  true   :q
  3  3   "a"  true   :p
  4  1   "b"  true   :q
  5  2   "b"  true   :p
  6  3   "b"  true   :q
  7  1   "a"  false  :q
  8  2   "a"  false  :p
  9  3   "a"  false  :q
 10  1   "b"  false  :p
 11  2   "b"  false  :q
 12  3   "b"  false  :p

all_values, all_pairs and all_triples are unchanged: covering at strength 1, 2 and 3.

seeds

# 0.4
must_test = [[1, "mid", 3.7, :relax], [1, "mid", 4.9, :relax]]
all_pairs([1, 2, 3], ["low", "mid", "high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim];
          seeds = must_test)

must_test = [(1, "mid", 3.7, :relax), (1, "mid", 4.9, :relax)]
all_pairs([1, 2, 3], ["low", "mid", "high"], [1.0, 3.7, 4.9], [:greedy, :relax, :optim];
          must_include = must_test)

10 cases (2 must-include, minimal) · strength 2 · Auto: Construction() · 4 parameters · 81 combinations
     p1  p2      p3   p4
  1  1   "mid"   3.7  :relax
  2  1   "mid"   4.9  :relax
  3  1   "low"   1.0  :greedy
  4  2   "mid"   3.7  :greedy
  5  3   "high"  4.9  :greedy
  6  2   "high"  1.0  :relax
  7  3   "low"   3.7  :relax
  8  1   "high"  3.7  :optim
  9  2   "low"   4.9  :optim
 10  3   "mid"   1.0  :optim

A positional call takes tuples or vectors. A named call takes NamedTuples, which may be partial and are then completed, or a previous TestCases, which is kept and topped up with the rows it misses.

wayness

wayness was a Dict from strength to lists of argument positions. stronger is a vector of group => strength pairs:

# 0.4
all_pairs([1, 2], [1, 2], [1, 2], [1, 2]; wayness = Dict(3 => [[2, 3, 4]]))

all_pairs([1, 2], [1, 2], [1, 2], [1, 2]; stronger = [(2, 3, 4) => 3])

8 cases (minimal) · strength 2, 3 within (p2, p3, p4) · Auto: IPOG() · 4 parameters · 16 combinations
    p1  p2  p3  p4
 1  1   1   1   1
 2  2   1   2   2
 3  1   2   1   2
 4  2   2   2   1
 5  2   1   1   2
 6  1   1   2   1
 7  1   2   1   1
 8  2   2   2   2

With named parameters, a group lists names: stronger = [(:level, :value, :kind) => 3].

Excursions

# 0.4
values_excursion([:a, :b, :c], [1, 2, 3])
pairs_excursion([:a, :b], [1, 2], [1, 2], ["a", "b"])
all_tuples([:a, :b], [1, 2], [1, 2]; n_way = 2, engine = Excursion())

excursions([:a, :b, :c], [1, 2, 3]; distance = 1)

5 cases · excursion, distance 1 from (:a, 1) · 2 parameters · 9 combinations
    p1  p2
 1  :a  1
 2  :b  1
 3  :c  1
 4  :a  2
 5  :a  3

excursions([:a, :b], [1, 2], [1, 2], ["a", "b"]; distance = 2)

11 cases · excursion, distance 2 from (:a, …) · 4 parameters · 16 combinations
     p1  p2  p3  p4
  1  :a  1   1   "a"
  2  :b  1   1   "a"
  3  :a  2   1   "a"
  4  :a  1   2   "a"
  5  :a  1   1   "b"
  6  :b  2   1   "a"
  7  :b  1   2   "a"
  8  :b  1   1   "b"
  9  :a  2   2   "a"
 10  :a  2   1   "b"
 11  :a  1   2   "b"

The base is still the first value of each parameter unless you pass one as from. An excursion is not a covering design: it guarantees only that every row is within distance changes of the base.

GND

# 0.4
all_pairs([1, 2, 3], ["a", "b"], [true, false]; engine = GND(M = 100))
all_pairs([1, 2, 3], ["a", "b"], [true, false]; engine = GND(rng = MersenneTwister(1)))

all_pairs([1, 2, 3], ["a", "b"], [true, false]; engine = GND(candidates = 100, seed = 1))

6 cases (minimal) · strength 2 · GND seed 1 · 3 parameters · 12 combinations
    p1  p2   p3
 1  3   "a"  true
 2  2   "b"  true
 3  1   "b"  false
 4  2   "a"  false
 5  3   "b"  false
 6  1   "a"  true

In 0.4 GND() drew from a fresh, unseeded generator, so each call gave different cases. In 0.5 GND() means GND(seed = 0) and each call gives the same cases. Pass seed to choose another design, or rng to draw from your own generator, which is copied, not advanced.

Counter, Excursion and generate_tuples

# 0.4
all_pairs(params...; Counter = Int8)
all_tuples(params...; engine = Excursion(), n_way = 1)
generate_tuples(IPOG(), 2, params, nothing, nothing, nothing, Int)

Drop Counter. Call excursions in place of the Excursion() engine, and covering in place of generate_tuples:

params = ([1, 2, 3], ["a", "b"], [true, false])
covering(params...; strength = 2, engine = IPOG())

6 cases (minimal) · strength 2 · IPOG · 3 parameters · 12 combinations
    p1  p2   p3
 1  1   "a"  true
 2  1   "b"  false
 3  2   "a"  false
 4  2   "b"  true
 5  3   "a"  true
 6  3   "b"  false

The return type of positional calls

A positional call returned a Vector{Vector{Any}} in 0.4:

# 0.4
cases = all_pairs([1, 2, 3], ["a", "b"], [true, false])   # Vector{Vector{Any}}
cases[1][1] = 10                  # rows were mutable vectors
push!(cases, [4, "c", true])      # and so was the result

In 0.5 it returns a TestCases, a read-only vector of tuples that keep each value's type:

cases = all_pairs([1, 2, 3], ["a", "b"], [true, false])
typeof(cases)

TestCases{Tuple{Int64, String, Bool}}

What still works: iteration with destructuring, indexing, length, eachindex, first, last, and splatting a row into a call.

for (x, label, flag) in cases
    # run one test with x, label and flag
end
(length(cases), cases[2], cases[2][1], first(cases))

(6, (2, "a", false), 2, (1, "a", true))

What changes: a row is an immutable tuple, so cases[1][1] = 10 is an error, and the result is not a Vector, so push!(cases, row) is an error and cases isa Vector is false. collect(cases) is a plain, mutable vector of the rows, and [collect(Any, row) for row in cases] rebuilds the 0.4 shape exactly:

rows = collect(cases)
push!(rows, (4, "c", true))
(typeof(rows), length(rows))

(Vector{Tuple{Int64, String, Bool}}, 7)

A positional result's parameters are named p1, p2, …. Pass the names to build a table:

using DataFrames
DataFrame(cases, parameters(cases.space))

or name the parameters in the call, and each row is a NamedTuple:

all_pairs((x = [1, 2, 3], label = ["a", "b"], flag = [true, false]))

6 cases (minimal) · strength 2 · Auto: Construction() · 3 parameters · 12 combinations
    x  label  flag
 1  1  "a"    true
 2  2  "a"    false
 3  3  "a"    true
 4  1  "b"    false
 5  2  "b"    true
 6  3  "b"    false

Values: nothing, types, and wrappers

In 0.4 nothing meant "not chosen yet" inside disallow. In 0.5 nothing and missing are ordinary values, and every value keeps its identity and type: Any[1, 1.0] is two values, and no value is converted.

all_pairs((n = [nothing, 1], s = [missing, "x"]))

4 cases (minimal) · strength 2 · Auto: Construction() · 2 parameters · 4 combinations, 4 valid
    n        s
 1  nothing  missing
 2  1        "x"
 3  1        missing
 4  nothing  "x"

Two wrapper values are new. Invalid(x) marks a value the code should reject: generation adds negative cases, each holding one invalid value beside valid ones, and hasinvalid tells a test body which kind of case it has.

negative = all_pairs(TestSpace((n = [1, 2, Invalid(-1)], mode = [:a, :b])))

6 cases (minimal) · strength 2 · Auto: Construction() · 2 parameters · 6 combinations, 6 valid · 2 negative targets
     n            mode
 1   1            :a
 2   2            :b
 3   2            :a
 4   1            :b
 5!  Invalid(-1)  :a
 6!  Invalid(-1)  :b

count(hasinvalid, negative)

2

Partition(name, draw) stands for a class of values; realize draws a concrete value for each partition at run time. Returned rows keep both wrappers. See Test invalid inputs and Combine with property-based testing.