Skip to content

Commit 964ab55

Browse files
yeshsuryayeshsurya
authored
[BugFix]: Framework selector for image task fails with azure.ai impor… (#4190)
* [BugFix]: Framework selector for image task fails with azure.ai import error * [Update]: No change in env since we will not use ai-ml sdk * [BugFix]: Remove ai-ml dependency and reinstall ins seg pipeline
1 parent a18e2fb commit 964ab55

23 files changed

Lines changed: 1639 additions & 3 deletions

File tree

‎assets/training/finetune_acft_common/components/validation/asset.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
type: component
2+
spec: spec.yaml
3+
categories: ["Foundational Models", "Finetune"]

‎assets/training/finetune_acft_common/components/validation/spec.yaml‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
2+
type: command
3+
4+
version: 0.0.9
5+
name: finetune_common_validation
6+
display_name: Common Validation Component
7+
description: Component to validate the finetune job against Validation Service
8+
9+
is_deterministic: True
10+
11+
environment: azureml://registries/azureml/environments/acpt-pytorch-2.2-cuda12.1/labels/latest
12+
13+
code: ../../src/validation
14+
15+
inputs:
16+
17+
# component input: mlflow model path
18+
mlflow_model_path:
19+
type: mlflow_model
20+
optional: true
21+
description: MLflow model asset path. Special characters like \ and ' are invalid in the parameter value.
22+
23+
# ###################################### Data validation ###################################### #
24+
# component input: training mltable
25+
train_mltable_path:
26+
type: mltable
27+
optional: false
28+
description: Path to the mltable of the training dataset.
29+
30+
# optional component input: validation mltable
31+
validation_mltable_path:
32+
type: mltable
33+
optional: true
34+
description: Path to the mltable of the validation dataset.
35+
36+
# component input: test mltable
37+
test_mltable_path:
38+
type: mltable
39+
optional: true
40+
description: Path to the mltable of the test dataset.
41+
42+
user_column_names:
43+
type: string
44+
optional: true
45+
description: Comma separated list of column names to be used for training.
46+
47+
# ###################################### Compute validation ###################################### #
48+
compute_preprocess:
49+
type: string
50+
optional: true
51+
description: Compute to be used for preprocess eg. provide 'FT-Cluster' if your compute is named 'FT-Cluster'. Special characters like \ and ' are invalid in the parameter value. If compute cluster name is provided, instance_type field will be ignored and the respective cluster will be used.
52+
53+
instance_type_preprocess:
54+
type: string
55+
optional: true
56+
description: Instance type to be used for preprocess component in case of serverless compute, eg. standard_d12_v2. The parameter compute_preprocess must be set to 'serverless' for instance_type to be used
57+
58+
compute_model_import:
59+
type: string
60+
optional: true
61+
description: Compute to be used for model_import eg. provide 'FT-Cluster' if
62+
your compute is named 'FT-Cluster'
63+
64+
instance_type_model_import:
65+
type: string
66+
optional: true
67+
description: Instance type to be used for model_import component in case of serverless compute, eg. standard_d12_v2. The parameter compute_model_import must be set to 'serverless' for instance_type to be used
68+
69+
compute_finetune:
70+
type: string
71+
optional: true
72+
description: Compute to be used for finetuning eg. provide 'FT-Cluster' if your compute is named 'FT-Cluster'. Special characters like \ and ' are invalid in the parameter value. If compute cluster name is provided, instance_type field will be ignored and the respective cluster will be used
73+
74+
instance_type_finetune:
75+
type: string
76+
optional: true
77+
description: Instance type to be used for finetune component in case of serverless compute, eg. standard_nc24rs_v3. The parameter compute_finetune must be set to 'serverless' for instance_type to be used
78+
79+
instance_count:
80+
type: integer
81+
default: 1
82+
optional: true
83+
description: Number of nodes to be used for finetuning (used for distributed training)
84+
85+
process_count_per_instance:
86+
type: integer
87+
default: 1
88+
optional: true
89+
description: Number of gpus to be used per node for finetuning, should be equal
90+
to number of gpu per node in the compute SKU used for finetune
91+
92+
compute_model_evaluation:
93+
type: string
94+
optional: true
95+
description: Compute to be used for model evaluation eg. provide 'FT-Cluster' if your
96+
compute is named 'FT-Cluster'
97+
98+
instance_type_model_evaluation:
99+
type: string
100+
optional: true
101+
description: Instance type to be used for model_evaluation components in case of serverless compute, eg. standard_nc24rs_v3. The parameter compute_model_evaluation must be set to 'serverless' for instance_type to be used
102+
103+
104+
task_name:
105+
type: string
106+
enum:
107+
- tabular-classification
108+
- tabular-classification-multilabel
109+
- tabular-regression
110+
- text-classification
111+
- text-classification-multilabel
112+
- text-named-entity-recognition
113+
- text-summarization
114+
- question-answering
115+
- text-translation
116+
- text-generation
117+
- fill-mask
118+
- image-classification
119+
- image-classification-multilabel
120+
- image-object-detection
121+
- image-instance-segmentation
122+
- video-multi-object-tracking
123+
description: Which task the model is solving.
124+
125+
# ###################################### ME validation ###################################### #
126+
test_batch_size:
127+
type: integer
128+
default: 1
129+
optional: true
130+
description: Test batch size.
131+
132+
label_column_name:
133+
type: string
134+
default: label
135+
optional: true
136+
description: Label column name in provided test dataset, for example "label".
137+
138+
device:
139+
type: string
140+
optional: False
141+
default: auto
142+
enum:
143+
- auto
144+
- cpu
145+
- gpu
146+
147+
evaluation_config:
148+
type: uri_file
149+
optional: true
150+
description: Additional parameters for Computing Metrics.
151+
152+
evaluation_config_params:
153+
type: string
154+
optional: true
155+
description: Additional parameters as JSON serialized string.
156+
157+
# ############################### Task Speciffic params validation ################################### #
158+
task_specific_extra_params:
159+
type: string
160+
optional: true
161+
description: All extra params. The values should be key values pairs separated by semi-colon. For example "param1=value1;param2=value2"
162+
163+
outputs:
164+
validation_info:
165+
type: uri_file
166+
description: Validation status.
167+
168+
command: >-
169+
python validation.py
170+
--validation-info '${{outputs.validation_info}}'

‎assets/training/finetune_acft_common/src/validation/validation.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT License.
3+
4+
"""script to update validation info."""
5+
6+
7+
def main():
8+
"""Script which runs as part of validation component to update output."""
9+
import argparse
10+
11+
parser = argparse.ArgumentParser()
12+
parser.add_argument("--validation-info", required=True, help="Model source ")
13+
14+
args = parser.parse_args()
15+
16+
print("Validation info: ", args.validation_info)
17+
with open(args.validation_info, "w") as f:
18+
f.write("Validation Completed")
19+
20+
21+
if __name__ == "__main__":
22+
main()

‎assets/training/finetune_acft_image/components/finetune/mmd_od_is/asset.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
type: component
2+
spec: spec.yaml
3+
categories: ["Foundational Models", "Finetune"]

0 commit comments

Comments
 (0)