Problem
A model's supervised commands in model.yaml are formatted with a fixed set of template variables: dataset_dir, labels_dir, model_initialization_dir, output_dir, model_dir, split, demo (see model_commands / fmt_command in src/MEDS_DEV/models/__init__.py).
That set is sufficient for models that train a task-specific head and predict. It is not sufficient for zero-shot models that resolve generated trajectories against the task definition itself — those need the ACES task criteria file and the dataset predicates file:
TASKS[task]["criteria_fp"] — the ACES task config YAML
DATASETS[dataset]["predicates"] — the per-dataset predicates YAML
Both are already resolved internally by meds-dev-task, but they are never handed to model commands. A model command can't recover them either: it runs in an isolated venv that doesn't have MEDS_DEV installed.
Concrete blocker
This came up wiring MEDS-EIC-AR (PR #313). Its zero-shot supervised: predict needs to call meds-trajectory-evaluation's ZSACES_label, whose signature is:
ZSACES_label task.criteria_fp=<ACES task yaml> task.predicates_fp=<predicates yaml> \
trajectories_dir=<...> output_dir=<...>
There is no template variable for task.criteria_fp or task.predicates_fp, so the command literally cannot be written today — str.format() would KeyError on an unknown placeholder.
Proposed change
Add two template variables, available to supervised commands:
{task_criteria_fp} — absolute path to the resolved ACES task criteria YAML.
{dataset_predicates_fp} — absolute path to the resolved per-dataset predicates YAML.
Both paths point at files inside the installed MEDS_DEV package on the shared filesystem, so a model command in an isolated venv can still read them. model_commands already has cfg.task_name / cfg.dataset_name available in full mode; the resolution is a registry lookup.
Open question worth deciding here: whether to also copy these into the labels_dir produced by meds-dev-task (so the task artifact is self-describing) instead of / in addition to passing them as template vars.
Related
Problem
A model's
supervisedcommands inmodel.yamlare formatted with a fixed set of template variables:dataset_dir,labels_dir,model_initialization_dir,output_dir,model_dir,split,demo(seemodel_commands/fmt_commandinsrc/MEDS_DEV/models/__init__.py).That set is sufficient for models that train a task-specific head and predict. It is not sufficient for zero-shot models that resolve generated trajectories against the task definition itself — those need the ACES task criteria file and the dataset predicates file:
TASKS[task]["criteria_fp"]— the ACES task config YAMLDATASETS[dataset]["predicates"]— the per-dataset predicates YAMLBoth are already resolved internally by
meds-dev-task, but they are never handed to model commands. A model command can't recover them either: it runs in an isolated venv that doesn't haveMEDS_DEVinstalled.Concrete blocker
This came up wiring MEDS-EIC-AR (PR #313). Its zero-shot
supervised: predictneeds to callmeds-trajectory-evaluation'sZSACES_label, whose signature is:There is no template variable for
task.criteria_fportask.predicates_fp, so the command literally cannot be written today —str.format()wouldKeyErroron an unknown placeholder.Proposed change
Add two template variables, available to
supervisedcommands:{task_criteria_fp}— absolute path to the resolved ACES task criteria YAML.{dataset_predicates_fp}— absolute path to the resolved per-dataset predicates YAML.Both paths point at files inside the installed
MEDS_DEVpackage on the shared filesystem, so a model command in an isolated venv can still read them.model_commandsalready hascfg.task_name/cfg.dataset_nameavailable infullmode; the resolution is a registry lookup.Open question worth deciding here: whether to also copy these into the
labels_dirproduced bymeds-dev-task(so the task artifact is self-describing) instead of / in addition to passing them as template vars.Related
supervised: predictpath in Register MEDS-EIC-AR as a model (micro/small/medium/large) #313 (MEDS-EIC-AR).