InstanceSpace

Syntax

Description

An InstanceSpace object holds the options, the trained model, and the evaluation results of one instance space analysis. It runs the pipeline stages PRELIM, SIFTED, PILOT, CLOISTER, PYTHIA and TRACE in order, lets you change options and re-run individual stages, evaluates the trained model on new instances, and saves and loads the model.

obj = InstanceSpace(rootdir) creates an object for the metadata in rootdir/metadata.csv. Options are read from rootdir/options.json if the file exists; missing options take their defaults. No computation runs until you call build.

obj = InstanceSpace(rootdir,opts) uses the options structure opts instead of options.json.

During build, algorithms with no good training instances are removed. Preprocessing is then fitted to the retained portfolio. Best-algorithm indices, best performance, beta-easy flags, and saved transforms all refer to that portfolio. With an absolute threshold, removing an algorithm can change the best performance on an instance where no algorithm is good.

InstanceSpace is a value class: a method that changes the object returns the changed copy, so assign the result, as in obj = obj.build().

Examples

Expand all

rootdir = 'test/data/example/';
if ~isfolder(rootdir), mkdir(rootdir); end
copyfile('test/data/metadata.csv', rootdir);
copyfile('test/data/metadata_test.csv', rootdir);

opts.perf = struct('MaxPerf', false, 'AbsPerf', true, 'epsilon', 0.20);
obj = InstanceSpace(rootdir, opts);
obj = obj.build();              % every stage; writes model.mat, CSV and PNG files
obj = obj.explore(rootdir);     % evaluates rootdir/metadata_test.csv

model = obj.getResults();       % training results
test  = obj.getResults(1);      % first explore() result

obj = InstanceSpace(rootdir, opts);
obj = obj.build('stages', {'prelim', 'sifted', 'pilot'});
obj.plot('portfolio')

obj.opts.pilot.analytic = true;
obj = obj.build('stages', {'pilot'});
obj = obj.build('stages', {'cloister', 'pythia', 'trace'});

report = @(stage, model) fprintf('%s finished\n', stage);
obj = obj.build('onStage', report);

obj.save();                              % rootdir/model.mat
obj2 = InstanceSpace.load(rootdir);
obj2 = obj2.explore(rootdir);

Input Arguments

Expand all

Properties

Expand all

Object Functions

build

obj = build(obj)
obj = build(obj,'stages',stages)
obj = build(obj,'onStage',callback)

Runs every stage, or only the named stages from {'prelim','sifted','pilot','cloister','pythia','trace'}, always in pipeline order. A stage whose prerequisites have not completed raises ISA:InstanceSpace:missingPrereq. callback(stageName, model) is called after each stage.

When all stages have completed, build saves model.mat and writes the outputs selected in opts.outputs with scriptcsv, scriptweb and scriptpng.

explore

obj = explore(obj,testRootDir)
obj = explore(obj,testRootDir,'onStage',callback)

Evaluates testRootDir/metadata_test.csv with the trained model: the trained preprocessing, feature selection, projection, classifiers and footprints are applied, not refitted. Appends the result to testResults and writes the outputs to testRootDir. Needs a fully built model.

getResults

results = getResults(obj)
results = getResults(obj,idx)

Returns obj.model, or the result of the idx-th explore call.

plot

plot(obj,view)
plot(obj,view,algoIdx)

Plots into the current figure. view is one of:

save

save(obj)

Writes obj.model to rootdir/model.mat in MAT-file version 7.3 format, one variable per field.

load

obj = InstanceSpace.load(rootdir)

Reads rootdir/model.mat, migrates a legacy model with ISAmigrateModel, fills missing options with ISAdefaults, and sets completedStages from the stages present. metadata.csv is not needed.

Version History

References

• Smith-Miles, K. & Muñoz, M.A. (2023). Instance Space Analysis for Algorithm Testing. ACM Computing Surveys, 55(12), Article 255. https://doi.org/10.1145/3572895

Unreleased review fixes

Each completed stage stores the options used to fit it. A partial rebuild rejects changed options for retained stages. Rebuild the affected stage and its dependents first. Partial builds can be saved, loaded, and resumed. Changing general.seed requires rebuilding preprocessing. Stage seeds and verbosity are copied from general defaults at construction and remain explicit stage options afterwards.

SIFTED reruns start from the saved pre-selection data, including its original row subset. Models saved without this artifact must rebuild preprocessing before rerunning SIFTED.

Build reuses an existing pool without replacing or deleting it. A pool opened by build is closed on success or error. general.parallel=false makes core stage execution serial even if a pool exists.