Skip to content

Commit c827922

Browse files
dougdalydougdaly
authored
Merge pull request #13 from dougdaly/mlops-poc-terraform-github
updated documentation in PROJECT STATUS.md
2 parents 214e060 + 5d1d539 commit c827922

15 files changed

Lines changed: 701 additions & 107 deletions

File tree

‎.DS_Store‎

2 KB
Binary file not shown.

‎.github/.DS_Store‎

6 KB
Binary file not shown.

‎mlops/.DS_Store‎

-4 KB
Binary file not shown.

‎mlops/PROJECT_STATUS.md‎

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# PROJECT_STATUS.md
2+
3+
## 1. Project snapshot
4+
5+
- **Project name:** Digital Twin Resilience Model
6+
- **Project goal:** Develop a digital twin of a major streaming platform to simulate system failure, impact radius, and response time.
7+
- **Current objective:** Build toward simulation of the entitlements service using a graph-oriented approach on AWS.
8+
- **Current phase:** Repo and pipeline mechanics clarified. GitHub Actions and Terraform deploy the SageMaker pipeline definition and related infrastructure; pipeline execution is started separately and currently runs a stub synthetic-data workflow.
9+
10+
- **Current status:**
11+
12+
### Deployment
13+
Based on GitHub Actions, `terraform-plan.yml` and `terraform-apply.yml`:
14+
- generate the SageMaker pipeline definition
15+
- provision or update the SageMaker Pipeline resource
16+
- validate Terraform / infra changes
17+
18+
`digital_twin_resilience/pipeline.py`
19+
- defines the SageMaker pipeline
20+
- generates `pipeline_definition.json`
21+
22+
### Execution
23+
`start_pipeline.py`
24+
- starts a specific SageMaker pipeline execution
25+
- allows parameter overrides
26+
- triggers the registered pipeline in AWS
27+
28+
The generated pipeline definition shows three steps:
29+
- `processor.py`
30+
- `train.py`
31+
- `evaluate.py`
32+
33+
`processor.py`
34+
- generates synthetic data
35+
- populates train / validation / test outputs in S3
36+
37+
`train.py`
38+
- builds a trivial baseline model from synthetic data
39+
40+
`evaluate.py`
41+
- computes an evaluation output / trivial metric from the model
42+
43+
### Verification
44+
`check_pipeline_execution.py`
45+
- asks SageMaker for overall pipeline execution status
46+
- lists step-level statuses and related job metadata
47+
48+
### Important distinction
49+
- Deploying the pipeline is separate from executing it.
50+
- Current GitHub Actions deploy and update the pipeline definition and infrastructure.
51+
- Pipeline execution is started deliberately via `start_pipeline.py`.
52+
53+
- **Immediate next step:** Define the minimum set of starter docs and begin filling them in, starting with continuity and framing docs.
54+
- **Biggest current blockers / gaps:**
55+
- Input data contract is not yet defined
56+
- Service graph schema is not yet defined
57+
- Prediction target is not yet defined
58+
- Definition of "good" model output is not yet defined
59+
- It is not yet decided whether the first baseline should be graph ML or something simpler
60+
61+
---
62+
63+
## 2. Working understanding of the repo
64+
65+
This section is not a replacement for `repo_skeleton.yml`. It is a quick orientation note describing how the repo is currently understood.
66+
67+
### Repo orientation
68+
69+
- `.github/workflows/`
70+
- GitHub Actions workflows for Terraform plan/apply and deployment-oriented automation
71+
- current understanding: deploys pipeline definition and infra, but does not execute the pipeline or run Python tests
72+
73+
- `terraform/`
74+
- infrastructure code for AWS resources and SageMaker pipeline registration
75+
- `envs/dev/` contains environment-specific wiring
76+
- `modules/` contains reusable pieces such as S3, IAM, and SageMaker pipeline setup
77+
78+
- `mlops/pipelines/digital_twin_resilience/`
79+
- core pipeline orchestration area
80+
- `pipeline.py` defines the SageMaker pipeline and generates `pipeline_definition.json`
81+
- `start_pipeline.py` starts a pipeline execution
82+
- `check_pipeline_execution.py` checks execution status
83+
- `steps/processing/`, `steps/training/`, and `steps/evaluation/` contain the step logic executed by SageMaker
84+
85+
- `data/synthetic/`
86+
- synthetic data support for the current stub workflow
87+
88+
- `tests/`
89+
- test area exists, but CI usage has not yet been confirmed in this document
90+
91+
- `README.md`
92+
- high-level explanation of repo purpose and structure
93+
94+
### Current understanding
95+
- Deployment and execution are separate concerns
96+
- GitHub Actions currently appear focused on deployment and Terraform validation
97+
- Pipeline execution is started deliberately, not automatically from Terraform apply
98+
- The current pipeline appears to be a stub synthetic processing/training/evaluation flow
99+
100+
### Key files for current understanding
101+
102+
The following files are currently the most relevant for understanding pipeline definition, execution, and verification:
103+
104+
- `pipeline.py`
105+
- `start_pipeline.py`
106+
- `check_pipeline_execution.py`
107+
- `steps/processing/processor.py`
108+
- `steps/training/train.py`
109+
- `steps/evaluation/evaluate.py`
110+
111+
Additional files such as `parse_request.py`, `request_schema.py`, and `create_pipeline.py` are likely important next, but have not yet been examined in detail in this document.
112+
113+
---
114+
115+
## 3. Current working decisions
116+
117+
- Deployment and execution are separate concerns.
118+
- GitHub Actions currently handle pipeline-definition generation and Terraform plan/apply.
119+
- Current GitHub Actions do not appear to start pipeline execution or run Python tests.
120+
- `pipeline.py` generates the SageMaker pipeline definition and writes `pipeline_definition.json`.
121+
- `start_pipeline.py` deliberately starts a SageMaker pipeline execution.
122+
- `check_pipeline_execution.py` checks overall execution status and step-level status through SageMaker APIs.
123+
- The current registered pipeline executes three step scripts: `processor.py`, `train.py`, and `evaluate.py`.
124+
- Early work should focus on framing, contracts, scope, and evaluation before sophisticated model choices.
125+
126+
---
127+
128+
## 4. Open questions
129+
130+
### Core problem / model questions
131+
- What exact decision is the system supposed to support first?
132+
- What is the narrow REV1 scope?
133+
- What is the first prediction target?
134+
- What would count as a useful model output?
135+
- What is the simplest credible baseline for REV1: graph-based, heuristic, tabular, or other?
136+
137+
### Data / entity questions
138+
- What are the core entities?
139+
- What node and edge types belong in the first service graph?
140+
- What data sources are expected to be available?
141+
- What minimum fields are required to support the first end-to-end run?
142+
- What synthetic substitutes are acceptable early on?
143+
144+
### Evaluation questions
145+
- How will success be measured for REV1?
146+
- What does "decision-useful" mean in practice?
147+
- What outputs should `evaluate.py` emit?
148+
- What evidence would justify continuing to the next phase?
149+
150+
### Repo / process questions
151+
- Which starter doc should be written next?
152+
- What should be treated as current truth vs placeholder?
153+
- What is the first code file that should be tightened?
154+
155+
---
156+
157+
## 5. Recommended starter docs from this session
158+
159+
These were identified as the most useful starter docs.
160+
161+
### A. Problem framing doc
162+
Should answer:
163+
- What problem are we solving?
164+
- Who is the decision-maker?
165+
- What is REV1 trying to prove?
166+
- What is explicitly out of scope?
167+
168+
### B. Feasibility questions / hypotheses doc
169+
Should answer:
170+
- What are the major unknowns?
171+
- What do we believe right now?
172+
- What evidence would support or weaken each hypothesis?
173+
174+
### C. REV1 scope and success criteria doc
175+
Should answer:
176+
- What are we building now?
177+
- What are we not building?
178+
- What must be demonstrated?
179+
- What would count as failure or a stop condition?
180+
181+
### D. Data and entity contract doc
182+
Should answer:
183+
- What are the main entities?
184+
- How do they relate?
185+
- What data do we expect?
186+
- What quality risks exist?
187+
188+
### E. Repo/runbook doc
189+
Should answer:
190+
- How is the repo organized?
191+
- How does the flow run?
192+
- What is implemented vs placeholder?
193+
- How should someone orient themselves quickly?
194+
195+
### Note
196+
This `PROJECT_STATUS.md` is not a replacement for those docs. It is the continuity layer that points to them and tracks what is missing.
197+
198+
---
199+
200+
## 6. Guidance agreed in this session
201+
202+
### What not to do
203+
- Do not begin by locking in sophisticated model architecture
204+
- Do not let the repo skeleton create false confidence
205+
- Do not use a polished solution architecture doc as the first anchor
206+
- Do not hide unresolved questions under implementation detail
207+
208+
### What to do first
209+
- Clarify the project/problem framing
210+
- Make the major unknowns explicit
211+
- Define REV1 scope and success criteria
212+
- Build continuity documentation that preserves momentum
213+
- Use this file to keep current status, decisions, open questions, and next actions visible
214+
215+
---
216+
217+
## 7. Next actions
218+
219+
- [ ] Create a first draft of the problem framing doc
220+
- [ ] Create a first draft of the feasibility questions / hypotheses doc
221+
- [ ] Create a first draft of the REV1 scope and success criteria doc
222+
- [ ] Identify the most important data/entity questions for the first pass
223+
- [ ] Decide which current repo file should be examined first for concrete changes
224+
225+
---
226+
227+
## 8. Change log
228+
229+
### Session-created initial version
230+
- Created the first session-only continuity draft of `PROJECT_STATUS.md`
231+
- Purpose: establish a resumable project memory file and expose missing information clearly
232+
- Constraint: uses only information discussed in this session

‎mlops/README.md‎

Lines changed: 41 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,50 @@
11
# SageMaker Pipeline Feasibility PoC
2+
23
## Description of directory tree elements
34

4-
<b> .github/workflows/ </b></br>
5+
**.github/workflows/**
6+
57
This is CI/CD only. It is not ML logic. GitHub Actions can authenticate to AWS via OIDC instead of long-lived secrets, which is the cleaner enterprise pattern.
6-
<ul>
7-
<li><b>terraform-plan.yml</b>: runs fmt/validate/plan on PRs</li>
8-
<li><b>terraform-apply.yml</b>: applies approved infra changes to dev, maybe later prod</li>
9-
</ul>
108

11-
<b>infra/terraform/</b></br>
9+
- **terraform-plan.yml**: runs fmt/validate/plan on PRs
10+
- **terraform-apply.yml**: applies approved infra changes to dev, maybe later prod
11+
12+
**infra/terraform/**
13+
1214
This is infrastructure only.
13-
<ul>
14-
<li><b>envs/dev/</b>: environment-specific wiring</li>
15-
<li><b>modules/s3/</b>: buckets for raw, processed, model artifacts, evaluation outputs</li>
16-
<li><b>modules/iam/</b>: execution roles and policies</li>
17-
<li><b>modules/sagemaker_pipeline/</b>: Terraform resource for the SageMaker Pipeline</li>
18-
</ul>
19-
Terraform has an aws_sagemaker_pipeline resource, so using Terraform for the pipeline object itself is a legitimate pattern, not a workaround.
20-
21-
<b>pipelines/digital_twin_resilience/</b></br>
15+
16+
- **envs/dev/**: environment-specific wiring
17+
- **modules/s3/**: buckets for raw, processed, model artifacts, evaluation outputs
18+
- **modules/iam/**: execution roles and policies
19+
- **modules/sagemaker_pipeline/**: Terraform resource for the SageMaker Pipeline
20+
21+
Terraform has an aws_sagemaker_pipeline resource, so using Terraform for the pipeline object itself is a legitimate pattern, not a workaround.
22+
23+
24+
25+
**pipelines/digital_twin_resilience/**
26+
2227
This is the ML workflow definition.
23-
<ul>
24-
<li><b>pipeline.py</b>: defines the SageMaker Pipeline DAG</li>
25-
<li><b>config.py</b>: pipeline parameters and defaults</li>
26-
<li><b>steps/processing/processor.py</b>: builds datasets or synthetic inputs</li>
27-
<li><b>steps/training/train.py</b>: trains a trivial baseline model first</li>
28-
<li><b>steps/evaluation/evaluate.py</b>: computes metrics and emits a JSON report</li>
29-
<li><b>utils/</b>: shared helpers</li>
30-
</ul>
31-
SageMaker Pipelines is a DAG of interconnected steps, and AWS explicitly supports Processing and Training steps in the pipeline definition.
32-
33-
<b>data/synthetic/</b>
28+
29+
- **pipeline.py**: defines the SageMaker Pipeline DAG
30+
- **config.py**: pipeline parameters and defaults
31+
- **steps/processing/processor.py**: builds datasets or synthetic inputs
32+
- **steps/training/train.py**: trains a trivial baseline model first
33+
- **steps/evaluation/evaluate.py**: computes metrics and emits a JSON report
34+
- **utils/**: shared helpers
35+
36+
SageMaker Pipelines is a DAG of interconnected steps, and AWS explicitly supports Processing and Training steps in the pipeline definition.
37+
38+
**data/synthetic/**
3439

3540
This is discovery-sprint fuel.
36-
<ul>
37-
<li>generate fake telemetry</li>
38-
<li>define a graph-ish structure if needed</li>
39-
<li>keep it tiny and boring</li>
40-
</ul>
41-
42-
<b>tests/</b>
43-
<ul>
44-
<li><b>test_pipeline_compile.py</b>: proves the pipeline definition compiles</li>
45-
<li><b>test_smoke_synthetic.py</b>: one tiny end-to-end synthetic run</li>
46-
</ul>
41+
42+
- generate fake telemetry
43+
- define a graph-ish structure if needed
44+
- keep it tiny and boring
45+
46+
**tests/**
47+
48+
- **test_pipeline_compile.py**: proves the pipeline definition compiles
49+
- **test_smoke_synthetic.py**: one tiny end-to-end synthetic run
50+

‎mlops/pipelines/.DS_Store‎

0 Bytes
Binary file not shown.

‎mlops/pipelines/digital_twin_resilience/pipeline.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import json
12
import os
23
from pathlib import Path
34

@@ -299,6 +300,8 @@ def get_pipeline(
299300

300301
definition = pipeline.definition()
301302
out_path = Path(__file__).resolve().parent / "pipeline_definition.json"
302-
out_path.write_text(definition)
303+
with out_path.open("w", encoding="utf-8") as f:
304+
json.dump(json.loads(definition), f, indent=2, sort_keys=False)
305+
f.write("\n")
303306

304307
print(f"Wrote pipeline definition to {out_path}")

0 commit comments

Comments
 (0)