Skip to content

Expose ACES task criteria + dataset predicates to model supervised commands #314

Description

@mmcdermott

Problem

A model's supervised commands in model.yaml are formatted with a fixed set of template variables: dataset_dir, labels_dir, model_initialization_dir, output_dir, model_dir, split, demo (see model_commands / fmt_command in src/MEDS_DEV/models/__init__.py).

That set is sufficient for models that train a task-specific head and predict. It is not sufficient for zero-shot models that resolve generated trajectories against the task definition itself — those need the ACES task criteria file and the dataset predicates file:

  • TASKS[task]["criteria_fp"] — the ACES task config YAML
  • DATASETS[dataset]["predicates"] — the per-dataset predicates YAML

Both are already resolved internally by meds-dev-task, but they are never handed to model commands. A model command can't recover them either: it runs in an isolated venv that doesn't have MEDS_DEV installed.

Concrete blocker

This came up wiring MEDS-EIC-AR (PR #313). Its zero-shot supervised: predict needs to call meds-trajectory-evaluation's ZSACES_label, whose signature is:

ZSACES_label task.criteria_fp=<ACES task yaml> task.predicates_fp=<predicates yaml> \
    trajectories_dir=<...> output_dir=<...>

There is no template variable for task.criteria_fp or task.predicates_fp, so the command literally cannot be written today — str.format() would KeyError on an unknown placeholder.

Proposed change

Add two template variables, available to supervised commands:

  • {task_criteria_fp} — absolute path to the resolved ACES task criteria YAML.
  • {dataset_predicates_fp} — absolute path to the resolved per-dataset predicates YAML.

Both paths point at files inside the installed MEDS_DEV package on the shared filesystem, so a model command in an isolated venv can still read them. model_commands already has cfg.task_name / cfg.dataset_name available in full mode; the resolution is a registry lookup.

Open question worth deciding here: whether to also copy these into the labels_dir produced by meds-dev-task (so the task artifact is self-describing) instead of / in addition to passing them as template vars.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    ModelsAssociated with the curated set of models available in MEDS-DEVenhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions