Skip to content

Commit ee420cc

Browse files
authored
fix(evaluate): Remove ModelPackageConfig from EvaluateBaseModel steps (#5635)
When evaluate_base_model=True, the EvaluateBaseModel step in both DETERMINISTIC_TEMPLATE and CUSTOM_SCORER_TEMPLATE incorrectly included ModelPackageConfig with SourceModelPackageArn, causing the base model evaluation to load fine-tuned model weights instead of using only the base model from the public hub. This made both evaluations identical, leading users to believe fine-tuning had no effect. Remove ModelPackageConfig from the EvaluateBaseModel step in both templates so it only uses BaseModelArn from ServerlessJobConfig. The EvaluateCustomModel step retains ModelPackageConfig to correctly load fine-tuned weights. This is consistent with the fix already applied to the LLMAJ_TEMPLATE. --- X-AI-Prompt: Fix BenchMarkEvaluator evaluate_base_model bug from D406780217 X-AI-Tool: Kiro sim: https://t.corp.amazon.com/D406780217
1 parent 73e10c1 commit ee420cc

File tree

1 file changed

+0
-8
lines changed

1 file changed

+0
-8
lines changed

sagemaker-train/src/sagemaker/train/evaluate/pipeline_templates.py

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -93,10 +93,6 @@
9393
"Type": "Training",
9494
"Arguments": {
9595
"RoleArn": "{{ role_arn }}",
96-
"ModelPackageConfig": {
97-
"ModelPackageGroupArn": "{{ model_package_group_arn }}",
98-
"SourceModelPackageArn": "{{ source_model_package_arn }}"
99-
},
10096
"ServerlessJobConfig": {
10197
"BaseModelArn": "{{ base_model_arn }}",
10298
"AcceptEula": true,
@@ -612,10 +608,6 @@
612608
"Type": "Training",
613609
"Arguments": {
614610
"RoleArn": "{{ role_arn }}",
615-
"ModelPackageConfig": {
616-
"ModelPackageGroupArn": "{{ model_package_group_arn }}",
617-
"SourceModelPackageArn": "{{ source_model_package_arn }}"
618-
},
619611
"ServerlessJobConfig": {
620612
"BaseModelArn": "{{ base_model_arn }}",
621613
"AcceptEula": true,

0 commit comments

Comments
 (0)