instancespace
Contains modules for instance space analysis.
The package builds an instance space from instance meta-data. The meta-data are the instance features and the algorithm performance. The instance space predicts algorithm performance and shows the algorithm footprints. The stages run in this order:
- preprocessing: Filters the meta-data. Removes the instances and features that have too many missing values.
- prelim: Preparation for Learning of Instance Meta-Data. Sets a binary measure of "good" performance. Then it bounds and scales the meta-data.
- sifted: Selection of Instance Features to Explain Difficulty. Selects the features that correlate with algorithm performance and are not redundant.
- pilot: Projecting Instances with Linearly Observable Trends. Projects the instances from the feature space to a 2D or 3D instance space. The projection shows linear trends in the features and in the algorithm performance.
- pythia: Trains one classifier for each algorithm on the instance space. Each classifier predicts good performance. PYTHIA then recommends an algorithm for each instance.
- cloister: Correlated Limits of the Instance Space's Theoretical or Experimental Regions. Estimates the boundary of the instance space from the feature bounds and the correlations between the features.
- trace: Triangulation with Removal of Areas with Contradicting Evidence. Finds the regions of the instance space where each algorithm performs well. These regions are the footprints.
InstanceSpace.build() trains all the stages on a data set. InstanceSpace.explore()
projects new instances into a trained instance space.
Reference: K. Smith-Miles and M. A. Muñoz, "Instance Space Analysis for Algorithm Testing: Methodology and Software Tools", ACM Comput. Surv. 55(12), 2023.
Combines multiple progress reporters.
Useful for reporting to both HTTP and file simultaneously.
Report that a stage has started, to every configured reporter.
Report that a stage has completed, to every configured reporter.
Report that a stage has failed, to every configured reporter.
Report that the job has completed, to every configured reporter.
Report that the job has failed, to every configured reporter.
Inherited Members
Results of applying trained model to test data.
This class encapsulates all outputs from the explore() pipeline, which applies a trained instance space model to new test data.
Attributes
- dataset_id (str): Identifier for this test dataset.
- timestamp (datetime): When the explore operation was performed.
- x (NDArray[np.double]): Processed features after PRELIM/SIFTED transformations. Shape: (n_instances, n_selected_features).
- z (NDArray[np.double]): Projected coordinates from PILOT. Shape: (n_instances, n_projection_dimensions), where the projection has two or three coordinates. TRACE3 supports full 3D exploration.
- y_hat (NDArray[np.bool_] | None):
Binary algorithm predictions from PYTHIA SVMs.
Shape: (n_instances, n_algorithms). None if PYTHIA not applied.
n_algorithmsincludes a trailing column (alwaysFalse) per test-set-only algorithm absent from training when ground truth is present - seealgo_labels. - pr0_hat (NDArray[np.double] | None):
Probability predictions from PYTHIA SVMs.
Shape: (n_instances, n_algorithms). None if PYTHIA not applied.
Same new-algorithm-column convention as
y_hat(trailing0.0). - selection0 (NDArray[np.int_] | None):
Recommended algorithm index for each instance.
Shape: (n_instances,). None if PYTHIA not applied. Never points at a
new-algorithm column (see
y_hat) - those are given zero selection precision, so they can never be the recommended algorithm. - in_good (NDArray[np.bool_] | None):
Whether each instance falls in "good" footprint for each algorithm.
Shape: (n_instances, n_algorithms). None if TRACE not applied. Same
new-algorithm-column convention as
y_hat(trailingFalse- no trained footprint exists to test membership against). - in_best (NDArray[np.bool_] | None):
Whether each instance falls in "best" footprint for each algorithm.
Shape: (n_instances, n_algorithms). None if TRACE not applied. Same
new-algorithm-column convention as
in_good. - inst_labels (pd.Series): Instance labels/identifiers from the test metadata.
- y_actual (NDArray[np.bool_] | None):
Ground-truth binary good/bad label per algorithm, derived from
test_metadata's ownalgo_*performance columns using the trainedPerformanceOptions. Shape: (n_instances, n_algorithms).Nonewhen the test set carries no algorithm performance columns at all (the feature-only case - existing callers see no change). - y_best_actual (NDArray[np.double] | None):
Best observed ground-truth performance per instance. Shape:
(n_instances,).
Noneunder the same condition asy_actual. - p_actual (NDArray[np.int_] | None):
1-based index of the best-performing algorithm per instance, ties
broken with MATLAB-compatible seeded random selection. Shape:
(n_instances,).
Noneunder the same condition asy_actual. - beta_actual (NDArray[np.bool_] | None):
Whether each instance clears the beta threshold's fraction of good
algorithms. Shape: (n_instances,).
Noneunder the same condition asy_actual. - accuracy_actual (NDArray[np.double] | None):
Per-algorithm accuracy of PYTHIA's trained classifiers against
y_actual. Shape: (n_algorithms,).NaNfor an algorithm absent from the test set's ground truth, or a test-set-only algorithm with no trained classifier to score (both "no CV model", matching MATLAB's convention).Noneunder the same condition asy_actual. - precision_actual (NDArray[np.double] | None):
Per-algorithm precision, same shape/NaN convention as
accuracy_actual. - recall_actual (NDArray[np.double] | None):
Per-algorithm recall, same shape/NaN convention as
accuracy_actual. - cvcmat_actual (NDArray[np.double] | None):
Per-algorithm confusion matrix
[tn, fp, fn, tp]againsty_actual. Shape: (n_algorithms, 4).NaNrow under the same conditions asaccuracy_actual.Noneunder the same condition asy_actual. - algo_labels (list[str] | None):
The full algorithm order (trained algorithms, then any test-set-only
algorithm absent from training, in first-encountered order) that
y_hat/pr0_hat/in_good/in_best/accuracy_actual/etc.'s per-algorithm columns index into (including new test-set algorithms) - needed to interpret those columns once new algorithms widen them beyondModel.data.algo_labels' own length.Noneunder the same condition asy_actual(when there's no ground truth, column order is implicitlyModel.data.algo_labelsand never widened). - trace_out (TraceOut | None):
Trained TRACE geometry rescored against this test set's ground truth.
Geometry and area remain unchanged; evidence metrics are updated. It is
Nonefor feature-only explore data.
Reports progress by writing to a JSON file.
Useful as a fallback when HTTP callbacks aren't available, or for
debugging/testing. If stages_dir is given, this also writes an
unsigned pickle snapshot of the InstanceSpace after every stage - a
convenient debugging artifact, but not the supported checkpoint/resume
mechanism (see InstanceSpace.save()/InstanceSpace.load(), which sign
the file the same way Model.save()/Model.load() do).
Initialize the file progress reporter.
Parameters
- progress_file (Path): Path to the progress.json file.
- stages_dir (Path | None): Optional directory to save intermediate stage pickles.
Report that a stage has completed successfully.
Report that the entire job has completed.
Inherited Members
Reports progress via HTTP POST to a callback URL.
Example callback URL: http://backend:8081/internal/jobs/123/stage-callback
The reporter can include different levels of detail in its callbacks:
- NONE: Just status updates
- METADATA: Status + output type/size information
- FULL: Status + metadata + base64-encoded pickle of InstanceSpace
Initialize the HTTP progress reporter.
Parameters
- callback_url (str): URL to POST progress updates to.
- job_id (int): Job ID for the callback payload.
- auth_token (str | None): Optional bearer token for authentication.
- timeout_seconds (int): HTTP request timeout.
- output_detail (OutputDetail): Level of detail to include for stage outputs.
- include_pickle_on_completion (bool): If True, the final callback includes the full pickle. For the final callback, this setting overrides output_detail.
Report that a stage has completed successfully.
Build an instance-space pipeline and apply its fitted model to new data.
Construct with validated Metadata and InstanceSpaceOptions objects.
build() returns a fitted Model; explore() returns predictions,
selections and optional evaluation metrics without refitting. Staged runs
expose intermediate outputs. save() and load() preserve a complete
checkpoint, including the runner needed for exploration.
Examples
With metadata and options prepared as in the Getting Started guide:
>>> space = InstanceSpace(metadata, options)
>>> model = space.build()
Create an output directory before calling explicit export methods. Only
load checkpoints from trusted producers. Call close() when the reusable
execution pool is no longer needed.
Initialise the InstanceSpace.
Parameters
- metadata (Metadata): The feature and algorithm performance data for the instance space.
- options (InstanceSpaceOptions): The options for the instance space.
- stages (list[StageClass], optional): The list of stages to run.
- additional_initial_inputs_type (type[NamedTuple] | None, optional): More initial inputs for plugins.
- progress_reporter (ProgressReporter | None, optional):
The reporter that records the progress of build() and run_stage().
For example, a SLURM job can use
HttpProgressReporterto send a callback to an orchestrator after each stage. The default reporter does nothing, so the behavior does not change if you omit this argument.
Get model.
Raises
- StageRunningError: If the InstanceSpace is not built.
Returns
- Model: The model that build() made.
Checkpoint this InstanceSpace to path via signed joblib.
Captures the entire pipeline state - not just a finished Model -
so a partially-built InstanceSpace (e.g. one stage run per SLURM
job, with the next stage triggered by a separate, later job) can be
reconstructed by load() and continue from exactly where it left
off via run_stage(). Signing follows the same scheme as
Model.save()/Model.load(): see that method's docstring for the
exact secret_key semantics and the production invariant (always
sign, never accept a user-supplied path server-side).
Restore an InstanceSpace checkpoint previously written by save().
Same four verification cases as Model.load() (matched/mismatched/
missing/unexpected signature) - see that method's docstring.
Release resources held across staged calls (currently: the TRACE pool).
Safe to call even if nothing has been built yet. A subsequent staged call recreates the pool lazily when parallel execution is enabled.
Scatter training instances in the 2D instance space, coloured by source.
Scatter instances coloured by their best-performing algorithm.
Scatter instances coloured by PYTHIA's good/bad prediction for one algorithm.
Draw one algorithm's trained footprint polygon(s) over training instances.
Build the instance space, in one call, start to finish.
Options will be broken down to sub fields to be passed to stages. You can
override inputs to stages. Progress is reported via the configured
progress_reporter (a no-op by default - see __init__), the same as
run_stage().
Returns
- Model: The output of all stages.
Run all stages, yielding between so the data can be examined.
Yields
- AnnotatedStageOutput: The output of each stage, with the name of the stage that ran. The name is necessary because the stages in one schedule can run in any order.
Run a single stage.
All inputs to the stage must either be present from previously ran stages, or be given as arguments to this function. Arguments to this function have priority over outputs from previous stages.
Progress is reported via the configured progress_reporter (a no-op
by default - see __init__). This is the entry point a SLURM job
that runs one stage per invocation should call: load a checkpoint
with InstanceSpace.load() (or construct fresh for the first
stage), call run_stage() for the stage this invocation was told to
run, then save() the result so a later, separately-triggered job
can resume. self.model becomes available once the last stage in
the schedule has been run this way, exactly as after build().
Parameters
- stage (StageClass): The stage to run.
- **arguments (Any): More inputs for the stage. If an input is missing, the stage uses the output of a previous stage.
Returns
- OUT: The output of the stage.
Run all stages until the specified stage, as well as the specified stage.
Parameters
- stage (StageClass): A stage in the last wave to execute.
- **arguments (Any): Input overrides for this run. After the stage runs, all downstream stages can use these overrides.
Returns
- dict[str, Any]: The raw output dict of all the stages that ran.
Get list of explore results from previous explore() calls.
Returns
- list[ExploreResult]: List of explore results, in order of execution.
Apply trained instance space model to new test data.
This method applies the transformations and models learned during build() to new test instances. It performs:
- Feature preprocessing (PRELIM normalization)
- Feature selection (SIFTED)
- Dimensionality reduction to the trained 2D/3D PILOT projection
- Algorithm performance prediction (PYTHIA SVMs)
- Footprint membership analysis (TRACE)
Parameters
- test_metadata (Metadata):
New instances that have the same feature columns as the training data.
The method matches the feature columns by name, not by position. Thus,
the columns can be in any order. This behavior is intentional. It is not
a strict order check like MATLAB's
featureOrderMismatch. - dataset_id (str | None, optional): The identifier for this test data set. If you do not supply an ID, the method makes an ID from a timestamp.
Returns
- ExploreResult: The projected coordinates, the algorithm predictions, and the footprint membership for the test instances.
Raises
- RuntimeError: If build() has not been called before explore().
- ValueError: If the test_metadata features are not the same as the training features. Also if the trained and explored projection dimensions are different.
Run explore() one stage at a time, yielding each stage's output.
Same computation and trained model as explore(), but instead of
returning a single ExploreResult it yields an AnnotatedExploreOutput
(an (ExploreStage, output) pair - the explore-time counterpart of
run_iter()'s AnnotatedStageOutput) after each inference stage, so the
intermediate result of every stage can be inspected or plotted before the next
one runs. In order: ExploreStage.PRELIM/SIFTED/PILOT yield the
transformed feature or coordinate array, PYTHIA yields (y_hat, pr0_hat,
selection0), TRACE yields (in_good, in_best), and - only when
test_metadata carries algorithm performance columns -
EVALUATION yields the ground-truth-vs-prediction evaluation fields
as an _EvaluationResult. EVALUATION is omitted entirely (not
yielded at all) when the test set has no ground truth, matching
explore()'s own conditional field population.
When the test set introduces algorithms absent from training (full
MATLAB alignment), PYTHIA's and TRACE's yielded arrays are
widened to include a column per new algorithm (False/0
placeholders - no trained classifier or footprint exists for them),
matching MATLAB's own evaluateTestSet reconciliation happening
before PYTHIA/TRACE run, not after.
Parameters
- test_metadata (Metadata): New instances that have the same feature columns as the training data.
Yields
- AnnotatedExploreOutput: The stage that just ran and its output.
Raises
- RuntimeError: If build() has not been called before explore().
- ValueError: If the test_metadata features are not the same as the training features.
TraceStage.predict()checks for a TRACE/projection dimension mismatch. This check occurs only when the lazy iterator gets to TRACE, after the iterator yields PYTHIA. This order is intentional.
Configure the full pipeline with validated, immutable option groups.
Attributes
- parallel (ParallelOptions): Configure supported parallel execution.
- perf (PerformanceOptions): Define good algorithm performance and instance difficulty.
- auto (AutoOptions): Control automatic preprocessing.
- bound (BoundOptions): Control feature outlier clipping.
- norm (NormOptions): Control fitted normalization.
- selvars (SelvarsOptions): Select input columns and training-instance subsets.
- sifted (SiftedOptions): Configure correlation screening and genetic feature selection.
- pilot (PilotOptions): Configure 2D or 3D projection and viewpoints.
- cloister (CloisterOptions): Configure empirical instance-space boundaries.
- pythia (PythiaOptions): Configure classifier training and hyperparameter search.
- trace (TraceOptions): Configure footprint construction and boundary membership.
- outputs (OutputOptions): Supply output preferences to callers and orchestrators.
- general (GeneralOptions): Configure logging and shared random seeds.
- prelim (PrelimConfigOptions): Configure feature bounds and missing-feature filtering.
Load configuration options from a JSON file into an object.
This function reads a JSON file from filepath, checks for expected
top-level fields as defined in InstanceSpaceOptions, initializes each part of
the InstanceSpaceOptions with data from the file, and sets missing optional
fields using their default values.
Parameters
- file_contents: Content of the dict with configuration options.
Returns
- InstanceSpaceOptions: InstanceSpaceOptions object populated with data from the file.
Raises
- ValueError: If the JSON file contains undefined sub options.
Instantiate with default values.
Metadata for problem instances.
The output of running InstanceSpace.
Initialize a Model object from the output of an InstanceSpace StageRunner.
Parameters
- cls (type[T]): The class.
- stage_runner_output (dict[str, Any]): The output of the StageRunner for an InstanceSpace.
- options (InstanceSpaceOptions): The options that built the instance space.
Returns
- Model: The Model object.
Serialise this Model to path via joblib, mirroring MATLAB's persistence.
Trained scikit-learn/shapely objects (PYTHIA's classifiers, TRACE's footprint polygons) round-trip natively - no flattening step.
If secret_key is given, an HMAC-SHA256 signature of the serialised
bytes is written alongside the model at <path>.sig, and load()
will refuse to deserialise unless the same key verifies it. If
secret_key is None (the default - the local/desktop-development
case with no server-managed secret), no signature is written and
load() performs no verification, the same trust caveat any other
unsigned pickle/joblib file already carries.
On the production web platform, every server code path that calls
load() must always supply secret_key and must never accept a
user-supplied path - that invariant lives in the deployment, not
in this method.
Deserialise a Model previously written by save().
Four cases, matching save()'s two modes:
secret_keygiven, signature present: verified before deserialising - raisesModelSignatureErroron mismatch.secret_keygiven, signature absent: raises - a caller expecting a verified load must never silently fall through to an unverified one.secret_keyisNone, signature absent: deserialises directly (the desktop/dev path).secret_keyisNone, signature present: raises. This is the downgrade-attack guard - a signed file must not become loadable-unverified just because the caller omitted the key.
Save csv outputs to a directory.
Save csv outputs used for the web frontend to a directory.
Save instance-space plots as PNG files in the output directory.
Save the algorithm-label-only MAT compatibility file for the dashboard.
Use save() for complete model persistence, including PILOT's
numeric projection and optimized viewpoint.
A no-op reporter for when progress reporting is disabled (the default).
Do nothing.
Inherited Members
Abstract base class for progress reporters.
Progress reporters are responsible for communicating stage execution status to external systems. They can include varying levels of detail about stage outputs in their reports.
Report that a stage has started.
Report that a stage has completed successfully.
Parameters
- stage_name (str): Name of the completed stage.
- duration_seconds (float | None): How long the stage took.
- instance_space (Any): The InstanceSpace object. The reporter uses it to save the intermediate state and to add the output data to the report.
Report that a stage has failed.