henrikbostrom / crepes Goto Github PK
View Code? Open in Web Editor NEWConformal classifiers, regressors and predictive systems
License: BSD 3-Clause "New" or "Revised" License
Conformal classifiers, regressors and predictive systems
License: BSD 3-Clause "New" or "Revised" License
Functions outside classes should also be documented.
Currently calling the evaluate()
function of the ConformalPredictiveSystem
class with argument metrics=["CRPS"]
fails with the following error:
UnboundLocalError: local variable 'lower_percentile' referenced before assignment
Example call:
score = model.evaluate(
y_hat=t[predict_col].values,
y=t[target_col].values,
bins=bins_t,
sigmas=sigma_t,
y_min=0,
metrics={"CRPS"})
Investigating the code shows there might be an issue in the implementation of evaluate()
function.
tic = time.time()
if metrics is None:
metrics = ["error","eff_mean","eff_med","CRPS","time_fit",
"time_evaluate"]
lower_percentile = (1-confidence)/2*100
higher_percentile = (confidence+(1-confidence)/2)*100
test_results = {}
if "CRPS" in metrics:
results, cpds = self.predict(y_hat, sigmas=sigmas, bins=bins, y=y,
lower_percentiles=lower_percentile,
higher_percentiles=higher_percentile,
y_min=y_min, y_max=y_max,
return_cpds=True, cpds_by_bins=True)
In this case when a call is made with metrics={"CRPS"}
, lower_percentile
is never initiated so it fails.
When using python 3.11, I got an error on row 1023 (and row 1049) in base.py when making the cps.predict
call below:
calY_pred = model.predict(calX)
testY_pred = model.predict(testX)
cps = ConformalPredictiveSystem()
cps.fit(residuals=calY-calY_pred)
interval = cps.predict(y_hat=testY_pred, lower_percentiles=[5,50], higher_percentiles=[95,50])
I do not get the error when calling any of the calls below:
interval = cps.predict(y_hat=testY_pred, lower_percentiles=[5,50])
interval = cps.predict(y_hat=testY_pred, higher_percentiles=[95,50])
The error can be solved by changing the if-clause(s) to if len(percentile_indexes) == 0
.
Will this paper be implemented in this package?
https://link.springer.com/content/pdf/10.1007/s10472-023-09847-0.pdf?pdf=button
The format of the optional output cpds of the method ConformalPredictiveSystem.predict need to be described.
I have some relatively beginer questions after trying the intro code:
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
for the code to work out of the box.
A few minimal examples should be provided in the README-file.
Running the following:
de = DifficultyEstimator()
display(de)
results in an error due to that the fitted
attribute is missing.
--> 144 if self.fitted and self.estimator_type == "knn":
145 return (f"DifficultyEstimator(fitted={self.fitted}, "
146 f"type={self.estimator_type}, "
AttributeError: 'DifficultyEstimator' object has no attribute 'fitted'.
The fitted attribute (or an is_fitted function) would be nice to have even for not yet fitted estimators.
Hi,
in a Kaggle competition I try to epxlore the Crepes
library. In the ReadMe a DifficultyEstimator
is used, which calls KNN under the hood.
As I used LGBM as the main regressor, I did not scale the data. However I wonder if this is required once I make use of the DifficultyEstimator
. Or is this done by Crepes
already?
The question came up, because this notebook performs much worse than another one where I only used the WrapRegressor
. So I assume I have some issue in the code, may it be some missing preprocessing or something else.
For reference the notebooks:
A memory issue (#9) was fixed in version 0.5.0; instead of always generating the cpds array within the predict method of ConformalPredictiveSystem, this is now done only when requested (by return_cpds=True) or when including "CRPS" among the metrics to evaluate (which currently is the default). The latter should preferably be calculated without generating the full cpds array (which may consume a lot of memory for very large calibration and test sets).
ConformalPredictiveSystem.evaluate may return the result of a test of uniformity of p-values
Hello, thank you very much for your great package. I have a quick question, where does the equation for calculate_p come from? It seems to be roughly just comparing the input predicted probabilities (i.e., softmax values) to the rank-ordering of the calibration non-conformity scores. This kind of makes sense, but does this really correspond to a p-value necessarily? Is there a reference for this?
Hi,
In order to ensure repdoducability of experiments, it would be useful to be able to set a random seed for the computation of smoothed p-values. In particular when working with small datasets. One solution might be to pass the seed as an optional argument to WrapClassifyer/WrapRegressor.
Hinge loss does not seem to work unless y_cal is re-indexed from zero.
alphas_cal = hinge(model.predict_proba(X_cal), model.classes_, y_cal)
KeyError: 0
The above exception was the direct cause of the following exception:
KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 alphas_cal = hinge(model.predict_proba(X_cal), model.classes_, y_cal)
/usr/local/lib/python3.10/dist-packages/crepes/extras.py in hinge(X_prob, classes, y)
69 if y is not None:
70 class_indexes = np.array(
---> 71 [np.argwhere(classes == y[i])[0][0] for i in range(len(y))])
72 result = 1-X_prob[np.arange(len(y)),class_indexes]
73 else:
/usr/local/lib/python3.10/dist-packages/crepes/extras.py in (.0)
69 if y is not None:
70 class_indexes = np.array(
---> 71 [np.argwhere(classes == y[i])[0][0] for i in range(len(y))])
72 result = 1-X_prob[np.arange(len(y)),class_indexes]
73 else:
/usr/local/lib/python3.10/dist-packages/pandas/core/series.py in getitem(self, key)
1005
1006 elif key_is_scalar:
-> 1007 return self._get_value(key)
1008
1009 if is_hashable(key):
/usr/local/lib/python3.10/dist-packages/pandas/core/series.py in _get_value(self, label, takeable)
1114
1115 # Similar to Index.get_value, but we do not fall back to positional
-> 1116 loc = self.index.get_loc(label)
1117
1118 if is_integer(loc):
/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py in get_loc(self, key)
3653 return self._engine.get_loc(casted_key)
3654 except KeyError as err:
-> 3655 raise KeyError(key) from err
3656 except TypeError:
3657 # If we have a listlike key, _check_indexing_error will raise
KeyError: 0
I have been running into an error when using the conformal predictive system. When I try to obtain the prediction intervals using an higher percentile >90, I get an index error (refer to traceback below). Since the intervals compute with a maximum value of 90, could the error be an issue caused by a small dataset size of 144 train, 48 cal and 49 test samples (I have tried this with a larger dataset and the intervals compute with an higher_percentile>90 as expected)? Assuming dataset size is the issue, is there a way to make this work for small datasets?
File C:\WINDOWS\System32\erty\lib\site-packages\crepes\base.py:457, in ConformalPredictiveSystem.predict(self, y_hat, sigmas, bins, y, lower_percentiles, higher_percentiles, y_min, y_max, return_cpds, cpds_by_bins)
450 for b in range(len(bin_values)):
451 higher_indexes = [
452 int(np.ceil(higher_percentile/100
453 * (len(bin_alphas[b])+1)))-1
454 for higher_percentile in higher_percentiles]
455 result[bin_indexes[b],
456 no_prec_result_cols:no_prec_result_cols
--> 457 + len(higher_indexes)] = cpds[b]
458 [:,higher_indexes]
459 if y_max < np.inf:
460 result[:,no_prec_result_cols:no_prec_result_cols
461 + len(higher_percentiles)]
462 [result[:,no_prec_result_cols:no_prec_result_cols
463 + len(higher_percentiles)]>y_max] = y_max
IndexError: index 10 is out of bounds for axis 1 with size 10
If I submit values for higher_percentile
or lower_percentile
from a numpy array, these values are not seen as float
(as they are indeed numpy.float64
) -- hence they are never transformed into the 1-valued list (see line 361 in base.py).
A working solution could be to replace each line of the kind
if type(lower_percentiles) in [int, float]:
with:
if isinstance(lower_percentiles, (int, float)):
which would return True
for both float
and numpy.float64
types.
Hi,
Would be very usefull to be able to predict prediction interval for the predict_proba scores returned by a binary classifier. Is it on the road map?
Thanks!
Hello, I am trying to use the crepes library at scale in a production environment (appx 100 million rows and 75 columns) but I am struggling with the amount of time required to fit the DifficultyEstimator and calibrating the CP wrapper. See below for the python code.
`
from crepes import WrapRegressor
from crepes.extras import DifficultyEstimator, binning
import pandas as pd
def calibrate_cp(
rf: WrapRegressor, x_tr: pd.DataFrame, y_tr: pd.DataFrame, x_ca: pd.DataFrame, y_ca: pd.DataFrame
) -> Union[WrapRegressor, DifficultyEstimator, list]:
"""Calibrates the conformal predictive system wrapper.
Args:
rf (WrapRegressor): A wrapped regression model
x_tr_n (pd.DataFrame): X train
y_tr_n (pd.DataFrame): Y train
x_ca_n (pd.DataFrame): X calibration
y_ca_n (pd.DataFrame): Y calibration
Returns:
Union[WrapRegressor, DifficultyEstimator, list]: Returns the calibrated wrapper, DifficultyEstimator (NearestNeighbors), and thersholds of the bins.
"""
de_kkn = DifficultyEstimator()
logging.info("Fitting the DifficultyEstimator")
de_kkn.fit(x_tr.to_numpy(), y=y_tr.to_numpy(), scaler=True)
logging.info("Calculate sigmas")
sigmas_cal = de_kkn.apply(x_ca.to_numpy())
logging.info("Calculate bins")
bins_cal, bin_thresholds = binning(rf.predict(x_ca), bins=5)
logging.info("Calibrate CP wrapper")
rf.calibrate(x_ca, y_ca.to_numpy().reshape((len(y_ca.to_numpy()),)), sigmas=sigmas_cal, bins=bins_cal, cps=True)
return rf, de_kkn, bin_thresholds
`
I therefore want to improve the scalability of crepes by adding GPU support (Any other suggestions for performance enhancements are also welcome).
This could for example be achieved by allowing for alternative implementations of kkn such as faiss:
https://towardsdatascience.com/make-knn-300-times-faster-than-scikit-learns-in-20-lines-5e29d74e76bb
Where the user would then set an optional parameter when initializing the class or a global config variable deciding which backend to use.
I thought I'd see if there is any interest in having this in the repository. If not I'll just make it in my own repo.
The methods and functions do not currently provide informative error messages when given incorrect input.
Concerning the wrappedclassifier with mondrian conformal prediction would it be safe to use the p-value of the predicted class as a calibrated probability or a good estimate of uncertainty. If not, any advice on could work as an estimate of uncertainty seeing that the predict_proba outputs uncalibrated probabilities based on the wrapped classifier. Note, I am working on a multiclass problem so Venn-ABERS does not seem to be applicable. I have already considered set size per instance but I was hoping for something more informative. Thanks in advance for any response.
Hello, has anyone run into an issue with using the predict method with CPS? I am just using this on a simple linear regression toy model that has about 480,000 observations, since this will need to be scaled to a way larger dataset.
I fit the normalized Conformal Predictive System following the same steps as the notebook, but when I go to run the predict method I keep getting this error:
MemoryError: Unable to allocate 939. KiB for an array with shape (120249,) and data type float64
The error was traced back to this portion of the base.py code:
--> 344 cpds = np.array([y_hat[i]+sigmas[i]*self.alphas 345 for i in range(len(y_hat))])
These were fit with the residuals for the calibration set so I am wondering if this is the issue? I have plenty of memory available to run something like this locally... like gigs worth, and the inability to allocate such an insignificant amount of memory is strange to me.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.