Commit 4f765c2
committed
fix(evaluate): Remove ModelPackageConfig from EvaluateBaseModel steps
When evaluate_base_model=True, the EvaluateBaseModel step in both
DETERMINISTIC_TEMPLATE and CUSTOM_SCORER_TEMPLATE incorrectly included
ModelPackageConfig with SourceModelPackageArn, causing the base model
evaluation to load fine-tuned model weights instead of using only the
base model from the public hub. This made both evaluations identical,
leading users to believe fine-tuning had no effect.
Remove ModelPackageConfig from the EvaluateBaseModel step in both
templates so it only uses BaseModelArn from ServerlessJobConfig. The
EvaluateCustomModel step retains ModelPackageConfig to correctly load
fine-tuned weights. This is consistent with the fix already applied to
the LLMAJ_TEMPLATE.
---
X-AI-Prompt: Fix BenchMarkEvaluator evaluate_base_model bug from D406780217
X-AI-Tool: Kiro
sim: https://t.corp.amazon.com/D4067802171 parent 6a1ba54 commit 4f765c2
File tree
1 file changed
+0
-8
lines changed- sagemaker-train/src/sagemaker/train/evaluate
1 file changed
+0
-8
lines changedLines changed: 0 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | 97 | | |
102 | 98 | | |
103 | 99 | | |
| |||
614 | 610 | | |
615 | 611 | | |
616 | 612 | | |
617 | | - | |
618 | | - | |
619 | | - | |
620 | | - | |
621 | 613 | | |
622 | 614 | | |
623 | 615 | | |
| |||
0 commit comments