Skip to content

Commit a2083c6

Browse files
yeshsuryayeshsurya
authored
[Update]: recover and update compute metrics for notebook runs fix (#4210)
1 parent 2ab2c34 commit a2083c6

3 files changed

Lines changed: 134 additions & 0 deletions

File tree

‎assets/training/model_evaluation/components/compute_metrics/README.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
## Compute Metrics Component
2+
3+
### Name
4+
5+
compute_metrics
6+
7+
### Version
8+
9+
0.0.36
10+
11+
### Type
12+
13+
command
14+
15+
### Description
16+
17+
This component enables user to evaluate a model by providing generated predictions and true values to return generated metrics. (Scores the predictions provided by user. No model is required in this case).
18+
19+
## Inputs
20+
21+
22+
| Name | Description | Type | Optional |
23+
| ------------------ | ----------------------------------------------------------------------------------- | ------- | ------- |
24+
| task | Task type for which model is trained | string | True |
25+
| ground_truth | Actual ground truth to evaluate predictions against. The file should be of JSON lines format containing only one key. | uri_file | True |
26+
| ground_truth_column_name | Column name which contains ground truths in provided uri file for ground_truths. | string | True | |
27+
| prediction | Actual predictions which are to be evaluated. They should be in json lines too with only one key. | uri_file | True | |
28+
| prediction_column_name | Column name which contains predictions in provided uri file for predictions. | string | True | |
29+
| prediction_probabilites | Prediction probabilities in order to calculate better set of metrics for classification tasks. This file should be in JSON lines format as well with number of keys equals to number of unique labels. | uri_file | True | |
30+
| evaluation_config | Additional config file required by metrics package. This data asset should contain a JSON Config file. | uri_file | True | |
31+
| evaluation_config_params | JSON Serielized string of evaluation_config | string | True |
32+
33+
## Outputs
34+
35+
| Name | Description | Type |
36+
| -------------------- | -------------------------------------------------------- | ------------ |
37+
| evaluationResult | Output dir to save the finetune model and other metadata | uri_folder |

‎assets/training/model_evaluation/components/compute_metrics/asset.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
type: component
2+
spec: spec.yaml
3+
categories: ["Model Evaluation"]

‎assets/training/model_evaluation/components/compute_metrics/spec.yaml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
2+
name: compute_metrics
3+
display_name: Compute Metrics
4+
description: Calculate model performance metrics, given ground truth and prediction data.
5+
6+
version: 0.0.36
7+
type: command
8+
tags:
9+
type: evaluation
10+
sub_type: compute_metrics
11+
12+
inputs:
13+
task:
14+
type: string
15+
optional: false
16+
default: tabular-classification
17+
enum: [
18+
tabular-classification,
19+
tabular-classification-multilabel,
20+
tabular-regression,
21+
tabular-forecasting,
22+
text-classification,
23+
text-classification-multilabel,
24+
text-named-entity-recognition,
25+
text-summarization,
26+
question-answering,
27+
text-translation,
28+
text-generation,
29+
fill-mask,
30+
image-classification,
31+
image-classification-multilabel,
32+
chat-completion,
33+
image-object-detection,
34+
image-instance-segmentation,
35+
]
36+
description: "Task type"
37+
ground_truth:
38+
type: uri_folder
39+
optional: true
40+
mode: ro_mount
41+
description: "Ground Truths of Test Data as a 1-column JSON Lines file"
42+
ground_truth_column_name:
43+
type: string
44+
optional: true
45+
description: "Column name which contains ground truths in provided uri file for ground_truth. (Optional if we have one column name.)"
46+
prediction:
47+
type: uri_folder
48+
optional: false
49+
mode: ro_mount
50+
description: "Model Predictions as a 1-column JSON Lines file"
51+
prediction_column_name:
52+
type: string
53+
optional: true
54+
description: "Column name which contains ground truths in provided uri file for prediction. (Optional if we have one column name.)"
55+
prediction_probabilities:
56+
type: uri_folder
57+
optional: true
58+
mode: ro_mount
59+
description: "Predictions Probabilities as 1-column JSON Lines file"
60+
evaluation_config:
61+
type: uri_file
62+
optional: true
63+
mode: ro_mount
64+
description: "Additional parameters required for evaluation."
65+
evaluation_config_params:
66+
type: string
67+
optional: true
68+
description: "JSON Serialized string of evaluation_config"
69+
openai_config_params:
70+
type: string
71+
optional: true
72+
description: "Required OpenAI Params for calculating GPT Based metrics for QnA task"
73+
74+
outputs:
75+
evaluation_result:
76+
type: uri_folder
77+
78+
is_deterministic: True
79+
code: ../../src
80+
environment: azureml://registries/azureml/environments/model-evaluation/labels/latest
81+
82+
command: >-
83+
python download_metrics_dependencies.py &&
84+
python compute_metrics.py
85+
--task '${{inputs.task}}'
86+
$[[--ground_truths '${{inputs.ground_truth}}']]
87+
--predictions '${{inputs.prediction}}'
88+
--output '${{outputs.evaluation_result}}'
89+
$[[--prediction_probabilities '${{inputs.prediction_probabilities}}']]
90+
$[[--config-file-name '${{inputs.evaluation_config}}']]
91+
$[[--ground_truths_column_name '${{inputs.ground_truth_column_name}}']]
92+
$[[--predictions_column_name '${{inputs.prediction_column_name}}']]
93+
$[[--config_str '${{inputs.evaluation_config_params}}']]
94+
$[[--openai-config-params '${{inputs.openai_config_params}}']]

0 commit comments

Comments
 (0)