PRELIM

Syntax

Description

[X,Y,out] = PRELIM(X,Y,opts) fits the preprocessing to the training data and applies it. PRELIM labels each algorithm as good or not good on each instance, finds the best algorithm per instance, bounds feature outliers, and applies Box-Cox and Z-score transforms to the features and the performance. The fitted bounds and transform parameters are returned in out.

[X,Y,out] = PRELIM(X,Y,opts,trainedPrelim) applies the bounds and transforms of an earlier training call, trainedPrelim, to new data instead of fitting new ones. The good/best labelling is the same code in both modes, so training and evaluation label ties the same way.

Examples

Expand all

T = readtable('test/data/metadata.csv');
X = T{:, startsWith(T.Properties.VariableNames, 'feature_')};
Y = T{:, startsWith(T.Properties.VariableNames, 'algo_')};

opts = struct('MaxPerf', false, 'AbsPerf', true, 'epsilon', 0.20, ...
              'betaThreshold', 0.55, 'auto', true, 'bound', true, ...
              'norm', true, 'iqrMultiplier', 5);
[Xn, Yn, out] = PRELIM(X, Y, opts);

fprintf('%.0f%% of instances are beta-easy\n', 100*mean(out.beta))
histogram(out.numGoodAlgos)

T2 = readtable('test/data/metadata_test.csv');
X2 = T2{:, startsWith(T2.Properties.VariableNames, 'feature_')};
Y2 = T2{:, startsWith(T2.Properties.VariableNames, 'algo_')};
[X2n, Y2n, out2] = PRELIM(X2, Y2, opts, out);

Input Arguments

Expand all

Output Arguments

Expand all

Algorithms

1. Labelling. Missing performance values are treated as the worst possible value. With MaxPerf and AbsPerf both false, algorithm j is good on instance i when Y(i,j)/Ybest(i) - 1 <= epsilon.
2. Bounding. Each feature is clipped to median ± iqrMultiplier*IQR.
3. Normalisation. Each feature is shifted to a minimum of 1, and the whole performance matrix to a small positive minimum. Each column is then Box-Cox transformed with its own fitted λ and standardised to zero mean and unit variance.

In evaluation mode, PRELIM warns (ISA:InstanceSpace:outOfDistribution) when more than 5% of the new instances fall outside the training bounds.

Version History

References

• Smith-Miles, K. & Muñoz, M.A. (2023). Instance Space Analysis for Algorithm Testing. ACM Computing Surveys, 55(12), Article 255. https://doi.org/10.1145/3572895

Unreleased review fixes

Constant columns use a unit divisor for Z-score normalisation. Evaluation applies the same policy to zero scales saved by older models.

Relative performance retains the original expressions: Y/Ybest - 1 for minimisation and 1 - Y/Ybest for maximisation, applied per instance. An algorithm is good when this value is at most epsilon. For positive scores these are equivalent to Y <= (1+epsilon)*Ybest and Y >= (1-epsilon)*Ybest, respectively. The code retains the ratio expressions to preserve floating-point behavior at the threshold.

For relative calculations only, exact zeros in the numerator and denominator are replaced by machine epsilon, as in the original method. Positive values, including values smaller than machine epsilon, are not clipped. Both transformed performance and binary labels use the same calculation, so all-zero ties are good in either direction. Ybest retains the raw best performance. A warning is issued when more than 5% of best scores are zero. This numerical convention is not a meaningful percentage of zero and is sensitive to measurement units; use an absolute threshold when zero-reference comparisons matter. Negative raw scores are rejected rather than assigned a new relative-performance definition.

Evaluation rows with no observed algorithm outcomes have P=0 and Ybest=NaN. They are excluded from Oracle probability estimates.