Is your feature request related to a problem? Please describe.
I want there to be a way to have a measure of the QUALITY of confidence estimation metrics, so I can pick the best method to approximate where errors may be when my models make predictions.
I want to be able to measure common best metrics, like AUROC, Brier Score, ECE or other metrics.
I want these metrics to be available when I do infernce with confidence estimations enabled.
Describe the solution you'd like
The Test studio should display metrics showing the quality of the confidence estimations. The test studio should tell me the coverage of these metrics. The test studio should break these metrics out by field.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
I want there to be a way to have a measure of the QUALITY of confidence estimation metrics, so I can pick the best method to approximate where errors may be when my models make predictions.
I want to be able to measure common best metrics, like AUROC, Brier Score, ECE or other metrics.
I want these metrics to be available when I do infernce with confidence estimations enabled.
Describe the solution you'd like
The Test studio should display metrics showing the quality of the confidence estimations. The test studio should tell me the coverage of these metrics. The test studio should break these metrics out by field.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.