SLEP026: Deprecate and Remove score Method#101
Conversation
| >>> from sklearn.metrics import super_metric | ||
| >>> super_metric(y_true=y, y_pred=estimator.predict(X)) |
There was a problem hiding this comment.
I think most of the pain comes from having to know which form of predict a metric needs: (predict, predict_proba, or decision_boundary). For some metrics, you may need to slice predict_proba: est.predict_proba(X)[:, 1].
Two benefits of estimator.score(X, y, scoring="r2_score"):
- Do not have to worry about the shape of the prediction.
- If one is passing in
scoringto aGridSearchCV(est, scoring="..."), they can runest.score(..., scoring="...")to see how the metrics behave before doing the full grid search. The API is a little more consistent.
There was a problem hiding this comment.
Do not have to worry about the shape of the prediction
If that’s the case, shouldn’t we fix those metrics/scores? Do you have an example?
I don’t understand you 2nd bullet.
There was a problem hiding this comment.
If that’s the case, shouldn’t we fix those metrics/scores? Do you have an example?
An example is roc_auc_score, which can run with:
roc_auc_score(y_true, est.predict_proba(X)[:, 1])or
roc_auc_score(y_true, est.decision_function(X))Users need to understand the shape of the prediction to properly handle the metric. I guess the workaround is to:
scorer = get_scorer("roc_auc")
result = scorer(est, X, y)For the grid search example, it's the same concern. I'm thinking of someone that is using GridSearchCV(..., scoring="roc_auc") and wonder "how do I manually compute the score?". If they use get_scorer, they they do not need to learn about the shape of the input to the underlying metric.
There was a problem hiding this comment.
Yes, get_scorer was also on my mind.
I guess we should just fix the roc_auc_score with predict_proba.
|
|
||
| 4.3. Downstream packages | ||
| ------------------------ | ||
| Many packages inherit from ``ClassifierMixin`` and ``RegressorMixin``, some examples include |
There was a problem hiding this comment.
This is a good point. We can go through a transition for scikit-learn estimators, but impacting third party estimators feels a bit more painful.
There was a problem hiding this comment.
XGBoost overrides the score method, so no problem there.
LightGBM just inherits them. They would also inherit the deprecation and removal. They don’t need to do much if anything.
There was a problem hiding this comment.
I'm mostly thinking about if LightGBM wants to keep score, they would need to implement it themselves. But they will find out that does not work anymore for first party scikit-learn meta-estimators.
There could also be third party meta-estimators (Like a SuperCustomSearchCV) that depend on .score, but also need to push users to pass in a scoring="some-metric-string".
2 participants
No description provided.