Reference

A container listing all available methods for conformal prediction.

A container listing all atomic MLJ models that have been tested for use with this package.

conformal_model(model::Supervised; method::Union{Nothing, Symbol}=nothing, kwargs...)

A simple wrapper function that turns a model::Supervised into a conformal model. It accepts an optional key argument that can be used to specify the desired method for conformal prediction as well as additinal kwargs... specific to the method.

emp_coverage(ŷ, y)

Computes the empirical coverage for conformal predictions ŷ.

ineff(ŷ)

Computes the inefficiency (average set size) for conformal predictions ŷ.

partial_fit(conf_model::TimeSeriesRegressorEnsembleBatch, fitresult, X, y, shift_size)

For the TimeSeriesRegressorEnsembleBatch Non-conformity scores are updated by the most recent data (X,y). shift_size determines how many points in Non-conformity scores will be discarded.

set_size(ŷ)

Helper function that computes the set size for conformal predictions.

size_stratified_coverage(ŷ, y)

Computes the size-stratified coverage for conformal predictions ŷ.

MMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)

For the AdaptiveInductiveClassifier nonconformity scores are computed by cumulatively summing the ranked scores of each label in descending order until reaching the true label $Y_i$:

$S_i^{\text{CAL}} = s(X_i,Y_i) = \sum_{j=1}^k \hat\mu(X_i)_{\pi_j} \ \text{where } \ Y_i=\pi_k, i \in \mathcal{D}_{\text{calibration}}$

MMI.fit(conf_model::CVMinMaxRegressor, verbosity, X, y)

For the CVMinMaxRegressor nonconformity scores are computed in the same way as for the CVPlusRegressor. Specifically, we have,

$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::CVPlusRegressor, verbosity, X, y)

For the CVPlusRegressor nonconformity scores are computed though cross-validation (CV) as follows,

$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifeMinMaxRegressor, verbosity, X, y)

For the JackknifeMinMaxRegressor nonconformity scores are computed in the same way as for the JackknifeRegressor. Specifically, we have,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusMinMaxAbRegressor, verbosity, X, y)

For the JackknifePlusABMinMaxRegressor nonconformity scores are as,

$S_i^{\text{J+MinMax}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusAbRegressor, verbosity, X, y)

For the JackknifePlusAbRegressor nonconformity scores are computed as

$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifePlusRegressor, verbosity, X, y)

For the JackknifePlusRegressor nonconformity scores are computed in the same way as for the JackknifeRegressor. Specifically, we have,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::JackknifeRegressor, verbosity, X, y)

For the JackknifeRegressor nonconformity scores are computed through a leave-one-out (LOO) procedure as follows,

$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.

MMI.fit(conf_model::NaiveClassifier, verbosity, X, y)

For the NaiveClassifier nonconformity scores are computed in-sample as follows:

$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::NaiveRegressor, verbosity, X, y)

For the NaiveRegressor nonconformity scores are computed in-sample as follows:

$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y)

For the SimpleInductiveClassifier nonconformity scores are computed as follows:

$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$. The simple approach only takes the softmax probability of the true label into account.

MMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y)

For the SimpleInductiveRegressor nonconformity scores are computed as follows:

$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$

A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.

MMI.fit(conf_model::TimeSeriesRegressorEnsembleBatch, verbosity, X, y)

For the TimeSeriesRegressorEnsembleBatch nonconformity scores are computed as

$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$

where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.

MMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)

For the AdaptiveInductiveClassifier prediction sets are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::CVMinMaxRegressor, fitresult, Xnew)

For the CVMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CV}} \}, \max_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{ S_i^{\text{CV}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with subset $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.

MMI.predict(conf_model::CVPlusRegressor, fitresult, Xnew)

For the CVPlusRegressor prediction intervals are computed in much same way as for the JackknifePlusRegressor. Specifically, we have,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - S_i^{\text{CV}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + S_i^{\text{CV}}\} \right] , \ i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with fold $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.

The JackknifePlusRegressor is a special case of the CVPlusRegressor for which $K=n$.

MMI.predict(conf_model::JackknifeMinMaxRegressor, fitresult, Xnew)

For the JackknifeMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife-minmax procedure is more conservative than the JackknifePlusRegressor.

MMI.predict(conf_model::JackknifePlusAbMinMaxRegressor, fitresult, Xnew)

For the JackknifePlusAbMinMaxRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}^{J+MinMax}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife+ab-minmax procedure is more conservative than the JackknifePlusAbRegressor.

MMI.predict(conf_model::JackknifePlusAbRegressor, fitresult, Xnew)

For the JackknifePlusAbRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

MMI.predict(conf_model::JackknifePlusRegressor, fitresult, Xnew)

For the JackknifePlusRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-i}(X_{n+1}) - S_i^{\text{LOO}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-i}(X_{n+1}) + S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

MMI.predict(conf_model::JackknifeRegressor, fitresult, Xnew)

For the JackknifeRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\}, \ i \in \mathcal{D}_{\text{train}}$

where $S_i^{\text{LOO}}$ denotes the nonconformity that is generated as explained in fit(conf_model::JackknifeRegressor, verbosity, X, y). The jackknife procedure addresses the overfitting issue associated with the NaiveRegressor.

MMI.predict(conf_model::NaiveClassifier, fitresult, Xnew)

For the NaiveClassifier prediction sets are computed as follows:

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \} \right\}, \ i \in \mathcal{D}_{\text{train}}$

The naive approach typically produces prediction regions that undercover due to overfitting.

MMI.predict(conf_model::NaiveRegressor, fitresult, Xnew)

For the NaiveRegressor prediction intervals are computed as follows:

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \}, \ i \in \mathcal{D}_{\text{train}}$

The naive approach typically produces prediction regions that undercover due to overfitting.

MMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew)

For the SimpleInductiveClassifier prediction sets are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, \ i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::SimpleInductiveRegressor, fitresult, Xnew)

For the SimpleInductiveRegressor prediction intervals are computed as follows,

$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}} \}, \ i \in \mathcal{D}_{\text{calibration}}$

where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.

MMI.predict(conf_model::TimeSeriesRegressorEnsemble, fitresult, Xnew)

For the TimeSeriesRegressorEnsemble prediction intervals are computed as follows,

$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$

where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor.

The AdaptiveInductiveClassifier is an improvement to the SimpleInductiveClassifier and the NaiveClassifier. Contrary to the NaiveClassifier it computes nonconformity scores using a designated calibration dataset like the SimpleInductiveClassifier. Contrary to the SimpleInductiveClassifier it utilizes the softmax output of all classes.

Constructor for CVMinMaxRegressor.

Constructor for CVPlusRegressor.

An abstract base type for conformal models that produce interval-valued predictions. This includes most conformal regression models.

An abstract base type for conformal models that produce probabilistic predictions. This includes some conformal classifier like Venn-ABERS.

An abstract base type for conformal models that produce set-valued probabilistic predictions. This includes most conformal classification models.

Constructor for JackknifeMinMaxRegressor.

Constructor for JackknifePlusAbPlusMinMaxRegressor.

Constructor for JackknifePlusAbPlusRegressor.

Constructor for JackknifePlusRegressor.

Constructor for JackknifeRegressor.

The NaiveClassifier is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier it computes nonconformity scores using a designated training dataset.

The NaiveRegressor for conformal prediction is the simplest approach to conformal regression.

The SimpleInductiveClassifier is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier it computes nonconformity scores using a designated calibration dataset.

The SimpleInductiveRegressor is the simplest approach to Inductive Conformal Regression. Contrary to the NaiveRegressor it computes nonconformity scores using a designated calibration dataset.

Constructor for TimeSeriesRegressorEnsemble.

_aggregate(y, aggregate::Union{Symbol,String})

Helper function that performs aggregation across vector of predictions.

absolute_error(y,ŷ)

Computes abs(y - ŷ) where ŷ is the predicted value.

blockbootstrap(time_series_data, block_szie)

Generate a sampling method, that block bootstraps the given data

is_classification(ŷ)

Helper function that checks if conformal prediction ŷ comes from a conformal classification model.

is_covered(ŷ, y)

Helper function to check if y is contained in conformal region. Based on whether conformal predictions ŷ are set- or interval-valued, different checks are executed.

is_covered_interval(ŷ, y)

Helper function to check if y is contained in conformal interval.

is_covered_set(ŷ, y)

Helper function to check if y is contained in conformal set.

is_regression(ŷ)

Helper function that checks if conformal prediction ŷ comes from a conformal regression model.

minus_softmax(y,ŷ)

Computes 1.0 - ŷ where ŷ is the softmax output for a given class.

qminus(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{-}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.

qplus(v::AbstractArray, coverage::AbstractFloat=0.9)

Implements the $\hat{q}_{n,\alpha}^{+}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.

reformat_interval(ŷ)

Reformats conformal interval predictions.

reformat_mlj_prediction(ŷ)

A helper function that extracts only the output (predicted values) for whatever is returned from MMI.predict(model, fitresult, Xnew). This is currently used to avoid issues when calling MMI.predict(model, fitresult, Xnew) in pipelines.

score(conf_model::ConformalProbabilisticSet, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Generic score method for the ConformalProbabilisticSet. It computes nonconformity scores using the heuristic function h and the softmax probabilities of the true class. Method is dispatched for different Conformal Probabilistic Sets and atomic models.

score(conf_model::AdaptiveInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Score method for the AdaptiveInductiveClassifier dispatched for any <:Supervised model.

score(conf_model::SimpleInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)

Score method for the SimpleInductiveClassifier dispatched for any <:Supervised model.

split_data(conf_model::ConformalProbabilisticSet, indices::Base.OneTo{Int})

Splits the data into a proper training and calibration set.