overview.md

Architecture Overview

This document provides a high-level overview of the system architecture, including artifact lineage, orchestration flow, config resolution, and each of the pipeline scripts (from registering raw data to promotion).

Main Components

• Pipelines: Orchestration and CLI logic
• ML: Business/domain logic
• Configs: Declarative YAML configuration
• Scripts: Useful cli scripts that don't belong to ml or pipelines (quality assessment, hash generation, etc.)

Diagrams

Artifact Lineage (high-level overview)

Orchestration (high-level overview)

Config Resolution

• Applied in search.py and train.py

Data Preprocessing

Raw Snapshot Registration

Interim Dataset Building

Processed Dataset Building

Feature Freezing

Feature Set Freezing

Search

Hyperparameter Search

Runners

Training

Evaluation

Explainability

Promotion

Model Staging, Promotion and Archiving

Post-promotion

Inference

Monitoring

Design Decisions

• See decisions.md for rationale behind architectural choices.

Boundaries

• See boundaries.md for boundaries-specific information.

System Invariants

• See system_invariants.md for details on system invariants.

Validation Guarantees

• See validation_guarantees.md for details on validation guarantees.