Run a simulation campaign

A simulation model has several parameters, each run takes minutes or hours, and the full product runs to hundreds of runs. Compare the sizes of the designs before choosing one, raise the strength for the parameters that interact most, and write the cases to a file that drives the campaign.

1. Describe the model's parameters

using UnitTestDesign

space = TestSpace((
        R0          = [0.9, 1.5, 3.0],
        population  = [1_000, 100_000, 10_000_000],
        contact     = [:homogeneous, :household, :network],
        vaccination = [0.0, 0.3, 0.7],
        stepper     = [:euler, :rk4, :adaptive],
        seasonal    = [false, true],
    );
    constraints = [
        forbid((contact = :network, population = 10_000_000);
               reason = "the network model is too slow at this size"),
    ])

2. Compare the designs

design_sizes runs each strategy and measures what it covers, before you commit to one:

design_sizes(space)

strategy        cases   share    pairs  triples
full_factorial    432  100.0%  119/119  439/439  valid 432 of 486
covering(1)         4    0.9%   53/119   76/439
covering(2)        13    3.0%  119/119  231/439
covering(3)        41    9.5%  119/119  439/439
excursions(1)      12    2.8%   70/119  130/439
excursions(2)      61   14.1%  119/119  326/439
case counts are the rows each strategy produced with Auto, not lower bounds

Each row is one design: how many runs it takes, its share of the 432 valid runs, and how many of the feasible pairs and triples it holds. Pairs take 13 runs; triples take 41. The counts are the rows the engine produced for this space, not lower bounds, and another engine may give a different count.

3. Raise the strength where it matters

Transmission, contact structure, and vaccination are the parameters most likely to interact, so ask for every triple of those three and every pair of the rest:

cases = all_pairs(space; stronger = [(:R0, :contact, :vaccination) => 3])
report(cases)

27 cases cover all 146 feasible combinations at strength 2, 3 within (R0, contact, vaccination) of a 486-combination space (1 pair forbidden)
excluded:
  (population = 10000000, contact = :network): forbidden by rule 1 (the network model is too slow at this size)
size: 27 cases, minimal: the 3 × 3 × 3 = 27 combinations of R0, contact and vaccination need a case each
bonus: 353 of 439 feasible triples covered
prefix curve:
  first 6 of 27 cover 57% (84 of 146)
  first 11 of 27 cover 85% (125 of 146)
  first 17 of 27 cover 92% (135 of 146)
  first 22 of 27 cover 96% (141 of 146)
  first 27 of 27 cover 100% (146 of 146)
seed: none (Auto uses no randomness)

The guarantee line states the combined claim: every feasible pair, and every feasible triple within the group, 146 combinations in all. The bonus: line says how many of the space's triples came along anyway.

4. Make the design reproducible

The default engine, Auto(), uses no randomness, so the same space gives the same cases. GND draws candidate rows at random; give it a seed, and the same seed gives the same cases, and report prints it:

gnd = all_pairs(space; stronger = [(:R0, :contact, :vaccination) => 3],
                engine = GND(seed = 20260927))
gnd == all_pairs(space; stronger = [(:R0, :contact, :vaccination) => 3],
                 engine = GND(seed = 20260927))

true

5. Write the rows

A named result is a table, one column per parameter:

using DataFrames
first(DataFrame(cases), 5)

5×6 DataFrame

Row	R0	population	contact	vaccination	stepper	seasonal
	Float64	Int64	Symbol	Float64	Symbol	Bool
1	0.9	1000	homogeneous	0.0	euler	false
2	0.9	100000	household	0.3	rk4	true
3	0.9	1000	network	0.7	adaptive	true
4	1.5	10000000	homogeneous	0.3	adaptive	false
5	1.5	1000	household	0.0	rk4	false

To drive the campaign from a file, write it with CSV.jl, which is not part of this documentation's build:

using CSV
CSV.write("campaign.csv", cases)

CSV is text: a Symbol such as :network is written as network and reads back as a string. If the space holds Partition values, write the labeled rows for the record and realize them in the job that runs each case.

Pitfall: the seed is not the record

Identical inputs give identical cases for the same package version and the same Julia version. A later release may change the rows, their order, or their count while keeping every guarantee, and so may any edit to the space. Keep the file of rows, not the seed, as the record of what the campaign ran, and see Commit a design as data for keeping those rows when the space grows.