Skip to content

feat(kermt): add static eval definitions for the open-model skills #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
evasnow1992 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(kermt): add static eval definitions for the open-model skills #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions open-models-skills/kermt/evals/kermt-add-cmim-pretrain/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
{
"skill_name": "kermt-add-cmim-pretrain",
"evals": [
{
"id": 7,
"prompt": "Produce a single self-contained shell or Python script that runs the complete add-cMIM-pretrain workflow end-to-end (validation, data prep, ckpt upgrade, then pretraining) — do not request inspection or interactive back-and-forth, write the whole pipeline in one solution file. I want to convert the vocab-only grover_base pretrain checkpoint at <data_dir>/model/grover_base/grover_base.pt into a hybrid (vocab + cMIM) checkpoint by adding a fresh contrastive decoder, then continue pretraining the hybrid model. Pretrain data: <kermt_repo>/tests/data/pretrain/. Run for 3 epochs only. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image and admet_dev_py311 conda env are available; one NVIDIA L4 GPU.",
"expected_output": "A script that exports KERMT_REPO, creates $KERMT_REPO/runs/add-cmim-pretrain_<ts>/, validates the grover_base ckpt as upgradeable (check_checkpoint.py --mode upgrade_to_hybrid), validates the pretrain corpus (check_data.py --mode pretrain), prepares the data without a separate vocab-dir argument (prepare_data.py --mode pretrain), runs upgrade_to_hybrid.py inside the kermt container to add the fresh cMIM decoder, then launches run_pretrain_local.py inside the kermt container with --epochs 3 against the upgraded ckpt. Hybrid pretrain target mode is implied.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-upgrade-helper",
"description": "Calls upgrade_to_hybrid.py to add the cMIM decoder",
"contains": [
"upgrade_to_hybrid.py"
],
"check": "Script contains 'upgrade_to_hybrid.py'"
},
{
"id": "invokes-pretrain-helper",
"description": "Calls run_pretrain_local.py after the upgrade",
"contains": [
"run_pretrain_local.py"
],
"check": "Script contains 'run_pretrain_local.py'"
},
{
"id": "upgrade-mode-check",
"description": "Validates the input ckpt under upgrade_to_hybrid mode",
"contains": [
"--mode upgrade_to_hybrid"
],
"check": "Script contains '--mode upgrade_to_hybrid'"
},
{
"id": "right-source-checkpoint",
"description": "Sources the grover_base ckpt as the upgrade input",
"contains": [
"grover_base",
"grover_base.pt"
],
"check": "Script contains 'grover_base' and 'grover_base.pt'"
},
{
"id": "right-pretrain-corpus",
"description": "Reads the KERMT tests/data/pretrain/ smoke corpus",
"contains": [
"tests/data/pretrain"
],
"check": "Script contains 'tests/data/pretrain'"
},
{
"id": "epochs-three",
"description": "Caps training at 3 epochs",
"contains": [
"--epochs 3"
],
"check": "Script contains '--epochs 3'"
}
]
}
]
}
122 changes: 122 additions & 0 deletions open-models-skills/kermt/evals/kermt-continue-pretrain/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
{
"skill_name": "kermt-continue-pretrain",
"evals": [
{
"id": 8,
"prompt": "Continue pretraining from the hybrid checkpoint at <data_dir>/model/hybrid_biogen_openadmet_chemblmt_checkpoint/last_checkpoint.pt on the small pretrain corpus at <kermt_repo>/tests/data/pretrain/. The checkpoint directory bundles the vocab files. Run for 3 epochs only. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image and admet_dev_py311 conda env are available; one NVIDIA L4 GPU.",
"expected_output": "A script that exports KERMT_REPO, creates $KERMT_REPO/runs/continue-pretrain_<ts>/, validates the hybrid ckpt as continue-pretrain-capable (check_checkpoint.py --mode continue_pretrain) and the corpus (check_data.py --mode pretrain), prepares the data with the bundled vocab (prepare_data.py --mode pretrain --vocab-dir ...), then launches run_pretrain_local.py inside the kermt container with --epochs 3. Pretrain-target-mode auto-dispatches to hybrid based on the ckpt.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-pretrain-helper",
"description": "Calls run_pretrain_local.py, not pretrain_ddp.py directly",
"contains": [
"run_pretrain_local.py"
],
"check": "Script contains 'run_pretrain_local.py'"
},
{
"id": "continue-pretrain-mode-check",
"description": "Validates the input ckpt under continue_pretrain mode",
"contains": [
"--mode continue_pretrain"
],
"check": "Script contains '--mode continue_pretrain'"
},
{
"id": "right-checkpoint",
"description": "Sources the hybrid_biogen_openadmet_chemblmt checkpoint",
"contains": [
"hybrid_biogen_openadmet_chemblmt_checkpoint",
"last_checkpoint.pt"
],
"check": "Script contains 'hybrid_biogen_openadmet_chemblmt_checkpoint' and 'last_checkpoint.pt'"
},
{
"id": "right-pretrain-corpus",
"description": "Reads the KERMT tests/data/pretrain/ smoke corpus",
"contains": [
"tests/data/pretrain"
],
"check": "Script contains 'tests/data/pretrain'"
},
{
"id": "epochs-three",
"description": "Caps training at 3 epochs",
"contains": [
"--epochs 3"
],
"check": "Script contains '--epochs 3'"
}
]
},
{
"id": 9,
"prompt": "Continue pretraining from the publicly released KERMT v2.0 model on Hugging Face (nvidia/NV-KERMT-70M-v2) on the small pretrain corpus at <kermt_repo>/tests/data/pretrain/. Download the model from Hugging Face first \u2014 the HF repo bundles the model checkpoint plus the vocab files (atom / bond / SMILES vocab). The training-time optimizer state has been stripped from this release-candidate slim checkpoint, so resume-with-saved-schedule mode is not available \u2014 use a fresh-schedule continue-pretrain. Run for 3 epochs only. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image and admet_dev_py311 conda env are available; one NVIDIA L4 GPU.",
"expected_output": "A script that exports KERMT_REPO, uses huggingface_hub to fetch the nvidia/NV-KERMT-70M-v2 snapshot (model checkpoint + bundled vocab files) into a local cache directory, then proceeds with a normal continue-pretrain: creates $KERMT_REPO/runs/continue-pretrain_<ts>/, validates the downloaded ckpt under continue_pretrain mode (check_checkpoint.py --mode continue_pretrain), validates the corpus (check_data.py --mode pretrain), prepares the data with the bundled vocab (prepare_data.py --mode pretrain --vocab-dir ...), then launches run_pretrain_local.py inside the kermt container with --epochs 3 against the downloaded ckpt path. Does not pass --resume (fresh schedule, optimizer state is stripped). Pretrain-target-mode auto-dispatches to hybrid.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-pretrain-helper",
"description": "Calls run_pretrain_local.py",
"contains": [
"run_pretrain_local.py"
],
"check": "Script contains 'run_pretrain_local.py'"
},
{
"id": "continue-pretrain-mode-check",
"description": "Validates the input ckpt under continue_pretrain mode",
"contains": [
"--mode continue_pretrain"
],
"check": "Script contains '--mode continue_pretrain'"
},
{
"id": "fetches-hf-model",
"description": "Targets the nvidia/NV-KERMT-70M-v2 model on Hugging Face",
"contains": [
"nvidia/NV-KERMT-70M-v2"
],
"check": "Script contains 'nvidia/NV-KERMT-70M-v2'"
},
{
"id": "uses-huggingface-hub",
"description": "Downloads via the released-model skill helper (fetch_released_model.py) or huggingface_hub directly",
"contains_any": [
"fetch_released_model.py",
"pretrained-release",
"huggingface_hub",
"snapshot_download",
"hf_hub_download"
],
"check": "Script downloads via the skill helper (fetch_released_model.py / --pretrained-release) or huggingface_hub directly (snapshot_download / hf_hub_download)"
},
{
"id": "epochs-three",
"description": "Caps training at 3 epochs",
"contains": [
"--epochs 3"
],
"check": "Script contains '--epochs 3'"
}
]
}
]
}
182 changes: 182 additions & 0 deletions open-models-skills/kermt/evals/kermt-embed/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
{
"skill_name": "kermt-embed",
"evals": [
{
"id": 2,
"prompt": "Extract per-molecule KERMT embeddings (all four readout types: atom-from-atom, bond-from-atom, atom-from-bond, bond-from-bond) using the grover_base encoder-only pretrain checkpoint at <data_dir>/model/grover_base/grover_base.pt on the Biogen training CSV at <data_dir>/tests/data/Biogen_for_grover/scaffold/balance/HLM_RLM_MDCK_logS/train.csv. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image is built and the admet_dev_py311 conda env is available.",
"expected_output": "A script that exports KERMT_REPO, creates $KERMT_REPO/runs/embed_<ts>/, validates the encoder ckpt (check_checkpoint.py --mode embed) and the CSV (check_data.py --mode embed), prepares the data (prepare_data.py --mode embed), then launches run_extract_embeddings.py inside the kermt container against the grover_base ckpt + the prepared Biogen data. Outputs four readout tensors under the run directory.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-extract-embeddings",
"description": "Calls the run_extract_embeddings.py helper",
"contains": [
"run_extract_embeddings.py"
],
"check": "Script contains 'run_extract_embeddings.py'"
},
{
"id": "embed-mode-prep",
"description": "Uses embed mode for prep and validation",
"contains": [
"--mode embed"
],
"check": "Script contains '--mode embed'"
},
{
"id": "right-checkpoint",
"description": "Targets the grover_base pretrain checkpoint",
"contains": [
"grover_base",
"grover_base.pt"
],
"check": "Script contains 'grover_base' and 'grover_base.pt'"
},
{
"id": "right-input-csv",
"description": "Reads the Biogen HLM/RLM/MDCK/logS train CSV",
"contains": [
"HLM_RLM_MDCK_logS",
"train.csv"
],
"check": "Script contains 'HLM_RLM_MDCK_logS' and 'train.csv'"
},
{
"id": "kermt-repo-env",
"description": "References KERMT_REPO env var",
"contains": [
"KERMT_REPO"
],
"check": "Script contains 'KERMT_REPO'"
}
]
},
{
"id": 3,
"prompt": "Produce a single self-contained shell or Python script that runs the embedding-extraction workflow end-to-end (ckpt + CSV validation, data prep, then embedding extraction) — do not request inspection or interactive back-and-forth, write the whole pipeline in one solution file. Extract per-molecule embeddings using the hybrid checkpoint at <data_dir>/model/hybrid_biogen_openadmet_chemblmt_checkpoint/last_checkpoint.pt on the OpenADMET combined CSV at <data_dir>/tests/data/openadmet/all.csv. The hybrid checkpoint directory ships bundled vocab files. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image and admet_dev_py311 conda env are available.",
"expected_output": "A script that exports KERMT_REPO, creates a $KERMT_REPO/runs/embed_<ts>/ directory, validates the hybrid ckpt and the all.csv input under embed mode, prepares the data, then launches run_extract_embeddings.py inside the kermt container against the hybrid checkpoint and the prepared OpenADMET data. Writes the four readout tensors under the run directory.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-extract-embeddings",
"description": "Calls the run_extract_embeddings.py helper",
"contains": [
"run_extract_embeddings.py"
],
"check": "Script contains 'run_extract_embeddings.py'"
},
{
"id": "embed-mode-prep",
"description": "Uses embed mode for prep and validation",
"contains": [
"--mode embed"
],
"check": "Script contains '--mode embed'"
},
{
"id": "right-checkpoint",
"description": "Targets the hybrid_biogen_openadmet_chemblmt checkpoint",
"contains": [
"hybrid_biogen_openadmet_chemblmt_checkpoint",
"last_checkpoint.pt"
],
"check": "Script contains 'hybrid_biogen_openadmet_chemblmt_checkpoint' and 'last_checkpoint.pt'"
},
{
"id": "right-input-csv",
"description": "Reads the OpenADMET combined CSV",
"contains": [
"openadmet",
"all.csv"
],
"check": "Script contains 'openadmet' and 'all.csv'"
},
{
"id": "runs-output-dir",
"description": "Writes outputs to $KERMT_REPO/runs/embed_ run directory",
"contains": [
"runs/embed"
],
"check": "Script contains 'runs/embed'"
}
]
},
{
"id": 12,
"prompt": "Extract per-molecule KERMT embeddings (all four readout types) using the publicly released KERMT v2.0 model on Hugging Face (nvidia/NV-KERMT-70M-v2) on the Biogen training CSV at <data_dir>/tests/data/Biogen_for_grover/scaffold/balance/HLM_RLM_MDCK_logS/train.csv. Download the model from Hugging Face first \u2014 the HF repo bundles the model checkpoint plus the vocab files. The KERMT repo is at <kermt_repo> (branch main); the kermt:latest docker image and admet_dev_py311 conda env are available.",
"expected_output": "A script that exports KERMT_REPO, uses huggingface_hub to fetch the nvidia/NV-KERMT-70M-v2 snapshot (model + vocab files) into a local cache directory, creates $KERMT_REPO/runs/embed_<ts>/, validates the downloaded ckpt (check_checkpoint.py --mode embed) and CSV (check_data.py --mode embed), prepares the data, then runs run_extract_embeddings.py inside the kermt container against the downloaded ckpt path and the prepared Biogen data.",
"files": [],
"assertions": [
{
"id": "uses-container-wrapper",
"description": "Routes work through the kermt_container.sh container wrapper",
"contains": [
"kermt_container.sh"
],
"check": "Script contains 'kermt_container.sh'"
},
{
"id": "invokes-extract-embeddings",
"description": "Calls the run_extract_embeddings.py helper",
"contains": [
"run_extract_embeddings.py"
],
"check": "Script contains 'run_extract_embeddings.py'"
},
{
"id": "embed-mode-prep",
"description": "Uses embed mode for prep and validation",
"contains": [
"--mode embed"
],
"check": "Script contains '--mode embed'"
},
{
"id": "fetches-hf-model",
"description": "Targets the nvidia/NV-KERMT-70M-v2 model on Hugging Face",
"contains": [
"nvidia/NV-KERMT-70M-v2"
],
"check": "Script contains 'nvidia/NV-KERMT-70M-v2'"
},
{
"id": "uses-huggingface-hub",
"description": "Downloads via the released-model skill helper (fetch_released_model.py) or huggingface_hub directly",
"contains_any": [
"fetch_released_model.py",
"pretrained-release",
"huggingface_hub",
"snapshot_download",
"hf_hub_download"
],
"check": "Script downloads via the skill helper (fetch_released_model.py / --pretrained-release) or huggingface_hub directly (snapshot_download / hf_hub_download)"
},
{
"id": "biogen-input-csv",
"description": "Reads the Biogen HLM/RLM/MDCK/logS train CSV",
"contains": [
"HLM_RLM_MDCK_logS",
"train.csv"
],
"check": "Script contains 'HLM_RLM_MDCK_logS' and 'train.csv'"
}
]
}
]
}
Loading