instancespace.stages.preprocessing
Preprocessing Stage Module.
This module defines the classes and methods for the preprocessing stage of a machine learning pipeline. It filters data rows based on provided options, and removes instances or features with too many missing values.
The preprocessing stage outputs a cleaned and filtered dataset that can be used for further modeling or analysis.
Classes
PreprocessingInput : NamedTuple Defines the input data structure for the preprocessing stage. PreprocessingOutput : NamedTuple Defines the output data structure for the preprocessing stage. PreprocessingStage : Stage Class that executes the preprocessing stage.
Validate matrix shape and the minimum viable instance-space dimensions.
Inputs for the Preprocessing stage.
Attributes
- feature_names (list[str]): List of feature names in the dataset.
- algorithm_names (list[str]): List of algorithm names in the dataset.
- instance_labels (pd.Series): Labels for each instance (row) in the dataset.
- instance_sources (pd.Series | None): Sources for each instance, optional.
- features (NDArray[np.double]): Feature matrix (instances x features) as a 2D numpy array.
- algorithms (NDArray[np.double]): Algorithm matrix (instances y algorithms) as a 2D numpy array.
- selvars_options (SelvarsOptions): Options for selecting variables (features and algorithms).
- prelim_options (PrelimOptions): Composed preliminary-processing options. Legacy direct construction uses the same defaults as the aggregate configuration.
Create new instance of PreprocessingInput(feature_names, algorithm_names, instance_labels, instance_sources, features, algorithms, selvars_options, prelim_options)
Outputs for the Preprocessing stage.
Attributes
- inst_labels (pd.Series): Series containing labels for each instance after preprocessing.
- feat_labels (list[str]): List of labels corresponding to the selected features.
- algo_labels (list[str]): List of labels corresponding to the selected algorithms.
- x (NDArray[np.double]): Preprocessed feature matrix (instances x selected features).
- y (NDArray[np.double]): Preprocessed algorithm matrix (instances y selected algorithms).
- s (pd.Series | None): Optional series containing the source of instances after preprocessing.
- x_raw (NDArray[np.double]): Original feature matrix before any modifications.
- y_raw (NDArray[np.double]): Original algorithm matrix before any modifications.
- preprocessing_features (tuple[str, ...] | None): Retained feature names in the order consumed by PRELIM, before SIFTED.
Create new instance of PreprocessingOutput(inst_labels, feat_labels, algo_labels, x, y, s, x_raw, y_raw, preprocessing_features)
Class for handling the preprocessing stage of the pipeline.
This stage includes tasks such as feature selection, algorithm selection, and removing instances or features with too many missing values.
Methods
select_features_and_algorithms(x, y, feat_labels, algo_labels, selvars) Selects features and algorithms from the dataset based on user options. remove_instances_with_many_missing_values(x, y, s, feat_labels, inst_labels) Removes instances (rows) and features (columns) with excessive missing values.
Initialize the Preprocessing stage.
Select features and algorithms from the dataset.
Based on the user's configuration, this method filters the features and algorithms that should be used in subsequent stages.
Parameters
- x (NDArray[np.double]): 2D numpy array representing the feature matrix (instances x features).
- y (NDArray[np.double]): 2D numpy array representing the algorithm matrix (instances y algorithms).
- feat_labels (list[str]): List of labels corresponding to the features in 'x'.
- algo_labels (list[str]): List of labels corresponding to the algorithms in 'y'.
- selvars (SelvarsOptions): An instance of SelvarsOptions that contains settings of the prefered algorithms and instances.
Returns
- tuple[NDArray[np.double], NDArray[np.double], list[str], list[str]]: A tuple containing:
- Modified feature matrix after feature selection and instance removal.
- Modified algorithm matrix after algorithm selection and instance removal.
- List of selected feature labels.
- List of selected algorithm labels.
Remove instances and features with excessive missing values.
Instances (rows) with too many missing values are removed. Additionally, features (columns) that exceed a missing value threshold are also removed. Washing criterion: 1. For any row, if that row in both X and Y are NaN, remove 2. For X columns, if that column's 20% grids are filled with NaN, remove
Parameters
- x (NDArray[np.double]): 2D numpy array representing the feature matrix (instances x features).
- y (NDArray[np.double]): 2D numpy array representing the algorithm matrix (instances y algorithms).
- s (pd.Series | None): Optional series containing the source of instances.
- feat_labels (list[str]): List of labels corresponding to the features in 'x'.
- inst_labels (pd.Series): Series containing labels for each instance.
- nan_threshold (float): Fraction of missing values at which a feature is removed.
Returns
- tuple[NDArray[np.double], NDArray[np.double],
- pd.Series, list[str], pd.Series | None]: A tuple containing the modified feature matrix 'x', the modified algorithm matrix 'y',updated instance labels, list of feature labels that remain after removal, and optionally modified series 's' if provided.