Skip to content

Llama 4 component upgrade #4150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
visahan-24 merged 8 commits into from
May 7, 2025
Merged

Llama 4 component upgrade #4150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
97cf80d
Llama 4 component upgrade
visahan-24 May 7, 2025
96763e2
Adding new line at eof
visahan-24 May 7, 2025
92525d2
Llama 4 component upgrade
visahan-24 May 7, 2025
bb82995
Adding new line at eof
visahan-24 May 7, 2025
930c132
Failing gate fixes
visahan-24 May 7, 2025
321deb6
Merge branch 'llama4-update' of https://github.com/Azure/azureml-asse…
visahan-24 May 7, 2025
e6c31b2
Failing gate fix
visahan-24 May 7, 2025
008bf68
Failing gate fixes
visahan-24 May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion assets/training/finetune_acft_hf_nlp/components/finetune/chat_completion/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Training parameters
| per_device_train_batch_size | Per gpu batch size used for training. The effective training batch size is _per_device_train_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| per_device_eval_batch_size | Per gpu batch size used for validation. The default value is 1. The effective validation batch size is _per_device_eval_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| auto_find_batch_size | If set to "true" and if the provided 'per_device_train_batch_size' goes into Out Of Memory (OOM) auto_find_batch_size will find the correct batch size by iteratively reducing batch size by a factor of 2 till the OOM is fixed | string | false | True | ['true', 'false'] |
| optim | Optimizer to be used while training | string | adamw_hf | True | ['adamw_hf', 'adamw_torch', 'adafactor'] |
| optim | Optimizer to be used while training | string | adamw_torch | True | ['adamw_torch', 'adafactor'] |
| learning_rate | Start learning rate used for training. | number | 2e-05 | True | NA |
| warmup_steps | Number of steps for the learning rate scheduler warmup phase. | integer | 0 | True | NA |
| weight_decay | Weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW optimizer | number | 0.0 | True | NA |
Expand Down
10 changes: 4 additions & 6 deletions assets/training/finetune_acft_hf_nlp/components/finetune/chat_completion/spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: chat_completion_finetune
version: 0.0.73
version: 0.0.74
type: command

is_deterministic: true

display_name: Chat Completion Finetune
description: Component to finetune Hugging Face pretrained models for chat completion task. The component supports optimizations such as LoRA, Deepspeed and ONNXRuntime for performance enhancement. See [docs](https://aka.ms/azureml/components/chat_completion_finetune) to learn more.

environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/87
environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/94

code: ../../../src/finetune

Expand Down Expand Up @@ -93,12 +93,10 @@ inputs:

optim:
type: string
default: adamw_hf
default: adamw_torch
optional: true
enum:
- adamw_hf
- adamw_torch
# - adamw_apex_fused
- adamw_torch # - adamw_apex_fused
- adafactor
description: Optimizer to be used while training

Expand Down
2 changes: 1 addition & 1 deletion .../training/finetune_acft_hf_nlp/components/finetune/question_answering/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Training parameters
| per_device_train_batch_size | Per gpu batch size used for training. The effective training batch size is _per_device_train_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| per_device_eval_batch_size | Per gpu batch size used for validation. The default value is 1. The effective validation batch size is _per_device_eval_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| auto_find_batch_size | If set to "true" and if the provided 'per_device_train_batch_size' goes into Out Of Memory (OOM) auto_find_batch_size will find the correct batch size by iteratively reducing batch size by a factor of 2 till the OOM is fixed | string | false | True | ['true', 'false'] |
| optim | Optimizer to be used while training | string | adamw_hf | True | ['adamw_hf', 'adamw_torch', 'adafactor'] |
| optim | Optimizer to be used while training | string | adamw_torch | True | ['adamw_torch', 'adafactor'] |
| learning_rate | Start learning rate used for training. | number | 2e-05 | True | NA |
| warmup_steps | Number of steps for the learning rate scheduler warmup phase. | integer | 0 | True | NA |
| weight_decay | Weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW optimizer | number | 0.0 | True | NA |
Expand Down
10 changes: 4 additions & 6 deletions assets/training/finetune_acft_hf_nlp/components/finetune/question_answering/spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: question_answering_finetune
version: 0.0.73
version: 0.0.74
type: command

is_deterministic: true

display_name: Question Answering Finetune
description: Component to finetune Hugging Face pretrained models for extractive question answering task. The component supports optimizations such as LoRA, Deepspeed and ONNXRuntime for performance enhancement. See [docs](https://aka.ms/azureml/components/question_answering_finetune) to learn more.

environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/87
environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/94

code: ../../../src/finetune

Expand Down Expand Up @@ -93,12 +93,10 @@ inputs:

optim:
type: string
default: adamw_hf
default: adamw_torch
optional: true
enum:
- adamw_hf
- adamw_torch
# - adamw_apex_fused
- adamw_torch # - adamw_apex_fused
- adafactor
description: Optimizer to be used while training

Expand Down
2 changes: 1 addition & 1 deletion assets/training/finetune_acft_hf_nlp/components/finetune/summarization/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Training parameters
| per_device_train_batch_size | Per gpu batch size used for training. The effective training batch size is _per_device_train_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| per_device_eval_batch_size | Per gpu batch size used for validation. The default value is 1. The effective validation batch size is _per_device_eval_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| auto_find_batch_size | If set to "true" and if the provided 'per_device_train_batch_size' goes into Out Of Memory (OOM) auto_find_batch_size will find the correct batch size by iteratively reducing batch size by a factor of 2 till the OOM is fixed | string | false | True | ['true', 'false'] |
| optim | Optimizer to be used while training | string | adamw_hf | True | ['adamw_hf', 'adamw_torch', 'adafactor'] |
| optim | Optimizer to be used while training | string | adamw_torch | True | ['adamw_torch', 'adafactor'] |
| learning_rate | Start learning rate used for training. | number | 2e-05 | True | NA |
| warmup_steps | Number of steps for the learning rate scheduler warmup phase. | integer | 0 | True | NA |
| weight_decay | Weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW optimizer | number | 0.0 | True | NA |
Expand Down
10 changes: 4 additions & 6 deletions assets/training/finetune_acft_hf_nlp/components/finetune/summarization/spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: summarization_finetune
version: 0.0.73
version: 0.0.74
type: command

is_deterministic: true

display_name: Summarization Finetune
description: Component to finetune Hugging Face pretrained models for summarization task. The component supports optimizations such as LoRA, Deepspeed and ONNXRuntime for performance enhancement. See [docs](https://aka.ms/azureml/components/summarization_finetune) to learn more.

environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/87
environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/94

code: ../../../src/finetune

Expand Down Expand Up @@ -93,12 +93,10 @@ inputs:

optim:
type: string
default: adamw_hf
default: adamw_torch
optional: true
enum:
- adamw_hf
- adamw_torch
# - adamw_apex_fused
- adamw_torch # - adamw_apex_fused
- adafactor
description: Optimizer to be used while training

Expand Down
2 changes: 1 addition & 1 deletion ...training/finetune_acft_hf_nlp/components/finetune/text_classification/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ Training parameters
| per_device_train_batch_size | Per gpu batch size used for training. The effective training batch size is _per_device_train_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| per_device_eval_batch_size | Per gpu batch size used for validation. The default value is 1. The effective validation batch size is _per_device_eval_batch_size_ * _num_gpus_ * _num_nodes_. | integer | 1 | True | NA |
| auto_find_batch_size | If set to "true" and if the provided 'per_device_train_batch_size' goes into Out Of Memory (OOM) auto_find_batch_size will find the correct batch size by iteratively reducing batch size by a factor of 2 till the OOM is fixed | string | false | True | ['true', 'false'] |
| optim | Optimizer to be used while training | string | adamw_hf | True | ['adamw_hf', 'adamw_torch', 'adafactor'] |
| optim | Optimizer to be used while training | string | adamw_torch | True | ['adamw_torch', 'adafactor'] |
| learning_rate | Start learning rate used for training. | number | 2e-05 | True | NA |
| warmup_steps | Number of steps for the learning rate scheduler warmup phase | integer | 0 | True | NA |
| weight_decay | Weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW optimizer | number | 0.0 | True | NA |
Expand Down
10 changes: 4 additions & 6 deletions assets/training/finetune_acft_hf_nlp/components/finetune/text_classification/spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
name: text_classification_finetune
version: 0.0.73
version: 0.0.74
type: command

is_deterministic: false

display_name: Text Classification Finetune
description: Component to finetune Hugging Face pretrained models for text classification task. The component supports optimizations such as LoRA, Deepspeed and ONNXRuntime for performance enhancement. See [docs](https://aka.ms/azureml/components/text_classification_finetune) to learn more.

environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/87
environment: azureml://registries/azureml/environments/acft-hf-nlp-gpu/versions/94

code: ../../../src/finetune

Expand Down Expand Up @@ -93,12 +93,10 @@ inputs:

optim:
type: string
default: adamw_hf
default: adamw_torch
optional: true
enum:
- adamw_hf
- adamw_torch
# - adamw_apex_fused
- adamw_torch # - adamw_apex_fused
- adafactor
description: Optimizer to be used while training

Expand Down
Loading
Loading