PILOT

Syntax

Description

out = PILOT(X,Y,featlabels,opts) finds a linear projection Z = X*A' of the feature matrix X into opts.dims dimensions, chosen so that both the features and the algorithm performance Y can be reconstructed from Z as well as possible (Prediction-based Linear Dimensionality Reduction). Instances with similar features and similar performance land close together, so trends in performance become visible as directions in the space.

PILOT has three solvers, selected by opts.method and opts.analytic:

• Numerical (default): BFGS minimisation of the reconstruction error of X and Y from Z (see Algorithms), restarted opts.ntries times. The trial whose projection best preserves the pairwise distances of the original data is kept.
• Analytic: the closed-form eigenvector solution. Fast and deterministic. If X is rank deficient, PILOT falls back to the numerical solver.
• PLS: Partial Least Squares regression of Y on X with plsregress. opts.alpha does not apply.

Examples

Expand all

rootdir = 'test/data/example/';
if ~isfolder(rootdir), mkdir(rootdir); end
copyfile('test/data/metadata.csv', rootdir);
opts.perf = struct('MaxPerf', false, 'AbsPerf', true, 'epsilon', 0.20);
obj = InstanceSpace(rootdir, opts);
obj = obj.build('stages', {'prelim', 'sifted'});

d = obj.model.data;
out = PILOT(d.X, d.Y, d.featlabels, obj.opts.pilot);
scatter(out.Z(:,1), out.Z(:,2), 12, 'filled')
xlabel('z_1'), ylabel('z_2')

pilotOpts = obj.opts.pilot;
pilotOpts.analytic = true;
outA = PILOT(d.X, d.Y, d.featlabels, pilotOpts);

pilotOpts.analytic = false;
pilotOpts.method = 'pls';
outP = PILOT(d.X, d.Y, d.featlabels, pilotOpts);

fprintf('Analytic error: %.3f   PLS error: %.3f\n', outA.error, outP.error)

obj.opts.pilot.dims = 3;
obj = obj.build('stages', {'pilot'});
Z = obj.model.pilot.Z;
scatter3(Z(:,1), Z(:,2), Z(:,3), 12, 'filled')

Input Arguments

Expand all

Output Arguments

Expand all

Algorithms

For the standard method, PILOT solves

min  ||X - Z*B'||^2 + alpha*||Y - Z*C||^2    subject to  Z = X*A'

where X and Y are the preprocessed feature and performance matrices. The analytic solution takes the leading dims eigenvectors of [X Y]'*[X Y], with the Y block weighted by sqrt(alpha) (Muñoz et al., 2018). The numerical solution runs BFGS (fminunc) from opts.ntries random starts and keeps the trial with the highest correlation between the pairwise distances of the instances before and after projection, so the chosen space also preserves the topology of the data.

Version History

References

• Muñoz, M.A., Villanova, L., Baatar, D. & Smith-Miles, K. (2018). Instance spaces for machine learning classification. Machine Learning, 107(1), 109–147. https://doi.org/10.1007/s10994-017-5629-5
• Smith-Miles, K. & Muñoz, M.A. (2023). Instance Space Analysis for Algorithm Testing. ACM Computing Surveys, 55(12), Article 255. https://doi.org/10.1145/3572895
• Simpson, C., Muñoz, M.A., Kandanaarachchi, S. & Campello, R.J.G.B. (2025). ISA3: A 3-dimensional expansion of Instance Space Analysis. Machine Learning, 114, 240. https://doi.org/10.1007/s10994-025-06871-5

Unreleased review fixes

PLS stores out.Xmean. Apply its projection as (X-out.Xmean)*out.A' for new instances. Standard PILOT projections do not subtract a mean.

Standalone calls fill numerical fallback defaults (ntries=10, seed=42). Analytic projection solves least squares directly. Pairwise feature distances are computed only when numerical restarts need ranking.