Skip to content

system_invariants.md

Latest commit

 

History

History
113 lines (75 loc) · 4.61 KB

system_invariants.md

File metadata and controls

113 lines (75 loc) · 4.61 KB
Raw

System Invariants

Identity & Uniqueness

Run Identity

  • Identifies a pipeline execution.
  • Each pipeline call is a new run with a unique id (e.g. calling search.py starts a new search run and produces a new experiment_id; train.py produces train_run_id, etc.).
  • Each id must be unique within that pipeline (e.g. each experiment_id must differ from all of the other experiment_ids).
  • Format: {timestamp}_{uuid8} (created dynamically at runtime).
  • Only search and training can have multiple calls for the same id (failure management - e.g. if training fails at iteration 1500 out of 2000, call train.py with the same train_run_id and continue from the last iteration persisted in the failure_management folder).

Snapshot identity

  • Identifies a frozen artifact.
  • Snapshot identity must remain stable once written.
  • Snapshots are immutable after creation.

Config Determinism

Config Resolution Order

  • Final resolved config for search and training must follow the following order:
    1. Global defaults
    2. Algorithm-specific defaults
    3. Model specs
    4. Search-or-train-specific configs
    5. Environment configs
    6. best params (training only)

Config Hash

  • Config hash is saved for each run that uses configs.
  • Pipelines that use multiple separate configs store hash for all of them (e.g. train runs store config_hash for the final resolved configs, but also pipeline_cfg_hash for sklearn pipeline-specific configs).
  • Config hash must uniquely identify the fully resolved config.
  • Identical configs must produce identical hashes.
  • Any modification to resolved config must change the hash.

Validation Status

  • Config validation must be implemented wherever configs are used.
  • Pipeline execution must be halted and raise an error if config validation fails.
  • Config resolution should fail unless
    1. _meta.validation_status == "ok"
    2. _meta.validation_errors == null

Data Integrity

Data Hash Consistency

  • SHA-256 hash generated by hash_streaming.py at runtime (implemented in multiple pipelines) should match the one stored in metadata.

Data Type Integrity

  • The actual data types found in loaded datasets' columns must match those defined in the feature registry.

Feature Set Integrity

Entity Key Presence

  • Each frozen feature set must contain a Entity Key column.

Schema Integrity

  • Schema hash generated at runtime (when loading features) must equal the schema hash saved in the metadata of a given feature set.

  • Note: only validates the input schema.

In-Memory Consistency (Strict Mode)

  • In-memory hash generated at runtime (when loading features) should equal the in-memory hash saved in the metadata of a given feature set.
  • Hash mismatch does not block execution, since it does not violate system integrity. System integrity is guaranteed by file hash consistency. In-memory hash mismatch gets logged for informational purposes, but it is only possible to get in-memory hash mismatch along with a file hash match if the in-memory mismatch is harmless (pandas-related quirks). Mismatch logs a warning.

File Consistency (Strict Mode)

  • File hash generated at runtime (when loading features) must equal the file hash saved in the metadata of a given feature set.

Operator Integrity

  • Operator hash generated at runtime (during feature freezing or loading) must equal the one found in the feature registry or metadata.

Search Determinism

  • random_state must be explicitly stored.
  • Param distributions and configurations must be stored in metadata.
  • CV strategy must be serialized or describable.
  • Best parameters must originate from recorded search results.

Atomic Persistence

  • All metadata writes must be atomic.
  • Partial writes must not be observable.
  • Registry updates must occur under lock.

Promotion Safety

  • Registry mutation must occur under exclusive lock.
  • Promotion must be idempotent.
  • Failed promotions must not corrupt registry state.

Reproducibility Contract

  • Reproducibility is strongly guaranteed when the following match:

    1. config_hash matches
    2. python version matches
    3. conda environment hash matches
    4. git_commit matches
    5. OS matches
    6. hardware matches

  • If any of the above differ, reproducibility is not fully guaranteed.

  • Config hash match is very important for reproducibility; python version, conda environment hash and git commit matches are moderately important; os and hardware matches are the least important.

  • It is technically possible to get the same results with config hash match alone, but the user assumes responsibility for any unexpected results in that case.