Skip to content

[F1] Add locked_parameters schema field to enforce campaign spec-fidelity in bundles #246

@sriumcp

Description

@sriumcp

Problem

bundle.yaml.experiment_spec.verified_parameters has no campaign-side anchor. The DESIGN agent can populate it with values that contradict the campaign.yaml's stated workload/locked parameters, and nous will not catch it because:

  • The bundle schema validator only checks structural validity (integers are integers, strings are strings).
  • The HUMAN_DESIGN_GATE (the layer where a human would notice the mismatch) is bypassed under --auto-approve.
  • EXECUTE_ANALYZE then treats the deviated verified_parameters as canonical and proceeds to build code at the wrong physics.

In other words: nous enforces self-consistency (executor matches what the bundle declares) but not spec-fidelity (bundle matches what the campaign declares).

Concrete instance from paper-memorytime-mirage

The campaign.yaml locked model: meta-llama/llama-3.1-8b-instruct, concurrency_per_tenant: 32, duration_seconds: 600, warmup_seconds: 30. The first iter-1 DESIGN agent produced a bundle with model: qwen/qwen2.5-7b-instruct, concurrency_per_tenant: 8, duration_seconds: 60, warmup_seconds: 10. nous accepted all four. Without an external watchdog this would have produced wrong-physics results presented as success.

Desired behavior

A new schema field on campaign.yaml:

locked_parameters:
  model: meta-llama/llama-3.1-8b-instruct
  concurrency_per_tenant: 32
  duration_seconds: 600
  warmup_seconds: 30
  total_kv_blocks: 24576
  # ... any parameter the campaign author wants pinned

When nous loads a bundle.yaml during DESIGN, it compares each key in locked_parameters against the corresponding entry in bundle.experiment_spec.verified_parameters. Any mismatch is a hard validation failure that aborts the design phase with a diagnostic error listing each (campaign value, bundle value) deviation. This runs regardless of --auto-approve.

Suggested implementation sketch

  1. Extend the campaign.yaml schema (orchestrator/schema/campaign.yaml or wherever the canonical schema lives) to define locked_parameters as Map<String, Any>, with an explicit doc comment that each key is a hard constraint.
  2. In the bundle-validation pass that already runs after DESIGN, add a step: for each (k, v) in campaign.locked_parameters, fail if bundle.experiment_spec.verified_parameters.get(k) != v. The error must list every mismatch, not just the first.
  3. Add a unit test exercising both the pass case and a multi-deviation fail case.
  4. Update nous schema campaign rendering to surface the new field.

Acceptance criteria

  • locked_parameters is a documented field in the campaign schema and nous schema campaign displays it.
  • A bundle whose verified_parameters deviates from any locked parameter fails validation with an error message listing every (campaign, bundle) deviation pair.
  • Validation runs even under --auto-approve.
  • Unit test covers (a) all-locked-match → pass, (b) one-deviation → fail with that deviation listed, (c) multi-deviation → fail with all deviations listed.
  • Friction report F1 row in the tracking issue checks off.

Severity

HIGH — would have produced wrong-physics results in iter-1 without an external watchdog.

Source

friction-report.md F1, paper-memorytime-mirage campaign (2026-05).


Part of friction-report tracking issue #245.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or requestfriction-reportFrom external campaign-author friction reports

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions