PYTHIA

Syntax

Description

out = PYTHIA(Z,Y,Ybin,Ybest,algolabels,opts) trains one binary classifier per algorithm that predicts, from an instance's coordinates Z, whether the algorithm is good on it (Performance Yielding Heuristic Identification of Algorithms). Hyperparameters are tuned by k-fold cross-validation. PYTHIA then combines the predictions into an algorithm selector: for each instance it picks the algorithm predicted to be good that has the highest cross-validated precision.

out = PYTHIA(Z,Y,Ybin,Ybest,algolabels,opts,trainedModel) applies the classifiers of an earlier training call to new instances and reports how accurate they are there. Classifiers are not retrained.

The classifier type is set by opts.classifier; see ISAgetClassifierFcn for the registry and the tuned hyperparameters.

Examples

Expand all

rootdir = 'test/data/example/';
if ~isfolder(rootdir), mkdir(rootdir); end
copyfile('test/data/metadata.csv', rootdir);
opts.perf = struct('MaxPerf', false, 'AbsPerf', true, 'epsilon', 0.20);
obj = InstanceSpace(rootdir, opts);
obj = obj.build('stages', {'prelim', 'sifted', 'pilot'});
d = obj.model.data;

out = PYTHIA(obj.model.pilot.Z, d.Yraw, d.Ybin, d.Ybest, d.algolabels, obj.opts.pythia);
disp(out.summary)

pythiaOpts = obj.opts.pythia;
pythiaOpts.classifier = 'svm';
pythiaOpts.tuning = 'bayes';
pythiaOpts.nTuningIter = 30;
out = PYTHIA(obj.model.pilot.Z, d.Yraw, d.Ybin, d.Ybest, d.algolabels, pythiaOpts);

copyfile('test/data/metadata_test.csv', rootdir);
obj = obj.build();                  % train every stage
obj = obj.explore(rootdir);         % reads rootdir/metadata_test.csv
res = obj.getResults(1);
res.pythia.summary

Input Arguments

Expand all

Output Arguments

Expand all

Version History

References

• Smith-Miles, K. & Muñoz, M.A. (2023). Instance Space Analysis for Algorithm Testing. ACM Computing Surveys, 55(12), Article 255. https://doi.org/10.1145/3572895

Unreleased review fixes

Evaluation uses the saved training fallback algorithm and precision weights. Legacy models without these fields use the first trained algorithm as fallback and equal voting weights. Test outcomes never determine recommendations.

Training summaries use out-of-fold predictions for both algorithm and selector rows. selection0CV and selection1CV retain these selections. Yhat, selection0, and selection1 remain fitted-data outputs for footprints and training plots. These CV metrics condition on the fitted preprocessing, projection, and selected hyperparameters. They are not an unbiased cross-validation estimate of the complete ISA pipeline. Exploration summaries label their metrics as Test_model_*.

Selector recall is the fraction of instances with an observed good algorithm on which the non-fallback selection is good. Successful selections are not also counted as missed opportunities when other algorithms are good.

Cost-sensitive weights are abs(Y-Ybest) per instance. Zero regrets use the smallest positive regret in the training matrix to keep every observed example trainable. If all regrets are zero, weights are uniform.

scoreType describes each Pr0hat column and scoreTypeCV describes Pr0sub. Values are probability, decision-score, class-score, or unavailable (unknown for old classifiers without metadata). Scores are mapped through classifier class names. SVM folds and the final model retain posterior calibration. A failed calibration is labelled as decision scores. Pr0subIsProbability identifies calibrated or probabilistic CV scores per prediction, and scoreTypeCV is mixed when folds use different score types. Failed CV candidates cannot produce a successful model: all failed candidates or an invalid selected CV result raise an error. A single-class training fold predicts its observed class.