Merge pull request #21 from GregoryKogan/dev

Dev

Author: GregoryKogan

.cursor/rules/conda-environment.mdc

@@ -0,0 +1,30 @@
+---
+description: Use the project Conda env yt-framework for all shell/Python commands; no ad-hoc venvs
+alwaysApply: true
+---
+
+# Python environment (Conda)
+
+Human-readable setup: [CONTRIBUTING.md](CONTRIBUTING.md) (Development Setup). Environment name: **`yt-framework`**.
+
+## Default for agents
+
+- Run verification and tooling through that env, for example:
+  - `conda run -n yt-framework -- <command>`
+- Prefer `conda run` so activation hooks are not required. For compound commands from the repo root:
+  - `conda run -n yt-framework -- bash -lc '...'`
+
+## Do not
+
+- Create `.venv` / `venv` or run `python -m venv` for routine tests, docs, or formatting unless the user explicitly asks.
+
+## If Conda or the env is missing
+
+- Do not silently fall back to a disposable venv; tell the user to follow [CONTRIBUTING.md](CONTRIBUTING.md).
+- If Conda is available but `yt-framework` does not exist, a one-time setup is acceptable (keep commands aligned with CONTRIBUTING):
+  - `conda create -n yt-framework python=3.11 -y`
+  - `conda run -n yt-framework -- python -m pip install -e ".[dev,docs]"` (from the repository root; use `python -m pip` so installs target the env, not a system/Homebrew `pip`)
+
+## If Conda is not installed
+
+- State that clearly; do not invent a replacement workflow without user direction.

.cursor/rules/documentation.mdc

@@ -0,0 +1,31 @@
+---
+description: Keep user-facing docs aligned with code; verify coverage while working
+alwaysApply: true
+---
+
+# Documentation maintenance
+
+## Surfaces
+
+Treat as documentation: `docs/**`, root `README.md`, `CONTRIBUTING.md`, and `examples/**/README.md` when the change touches behavior, CLI, config, or workflows those files describe. Public API docstrings follow [standards.mdc](standards.mdc); keep them consistent with any narrative docs that mention the same API.
+
+## When to update docs
+
+Update docs in the same change set when behavior could confuse a reader: new/changed CLI commands or flags, defaults, config keys, env vars, user- or operator-visible errors, pipeline/stage semantics, YT/S3 integration, Docker or code upload flows, troubleshooting symptoms, or anything else someone would look up under `docs/` or READMEs.
+
+## Obligation
+
+Update every affected surface (or add a focused new section for new behavior). Do not leave stale commands, options, or explanations that contradict the code.
+
+## While working
+
+- **Early**: Map edited modules/features to the relevant `docs/` pages and READMEs; read them and note gaps or drift.
+- **Before done**: Reconcile implementation with those docs; if public API changed, fix docstrings per `standards.mdc` and any prose that describes that API.
+
+## Proof
+
+- **Touches `docs/` or example/root README/CONTRIBUTING**: run `make -C docs html` after installing doc dependencies (e.g. `pip install -e ".[docs]"` using the `docs` optional extra in repo-root `pyproject.toml`); cite success or fix warnings/errors Sphinx reports.
+- **Docstrings only, no prose under `docs/` or those READMEs**: state that Sphinx was not required; still ensure docstrings match behavior.
+- **Pure prose doc edits**: `make -C docs html` is the primary verification.
+
+Done = proof, same spirit as Standards: do not claim completion without evidence appropriate to what changed.

.cursor/rules/standards.mdc

@@ -0,0 +1,33 @@
+---
+description: Plan-first, minimal-impact, verifiable changes; SOLID/DRY/KISS/YAGNI; types & style
+alwaysApply: true
+---
+
+# Standards
+
+## Plan & approval
+
+- Non-trivial (3+ steps or architecture): Research → Plan → implement only after human approval. Artifacts: `./agent-artifacts/tasks/<task>/research.md`, `plan.md`, todo list. No implementation before approval.
+- Never deviate from an approved plan without asking; one approval does not cover later deviations.
+
+## Impact & verification
+
+- Touch only required files. Root-cause fixes over rewrites. No destructive git without request. No dead code.
+- Done = proof (test/log/command output). No "done" without evidence. Divergence → re-plan, don’t patch.
+
+## Code
+
+- SOLID, DRY, KISS, YAGNI. Explicit intent; no dead code. Clarify ambiguity before implementing.
+- Types: prefer dataclasses over `tuple[str,str,int]`, `Union[...]`, or big `dict[str, Any]`; use `Optional`, `List`, etc.; `TYPE_CHECKING` for type-only imports.
+- Public API: type hints + docstrings (Args/Returns); Literal for constrained params. Small, single-purpose functions; early returns; compact style (single-line calls, chained transforms, 88-char line). Prefer `assert` with message; comments only as `TODO:` for team decisions.
+- Quality: run `black --check --diff` on changed files; after changes, align diff with style and update .cursor/rules if needed.
+
+## Artifacts & files
+
+- Artifacts under `./cursor/artifacts/`: `tasks/<task>/`, `user-preferences/`, `project-details/`.
+- Ensure .cursor/rules exists and is up to date.
+
+## Shell
+
+- Use `cp`/`mv` for new/renamed files.
+- Avoid `| head`, `| tail`, `less`, `more` for monitoring; use command-native limits (e.g. `git log -n 10`) or read files directly.

.cursor/rules/test-authoring.mdc

@@ -0,0 +1,57 @@
+---
+description: Angry-Tests-aligned pytest style, naming, and mandatory debug loop for agents
+alwaysApply: true
+---
+
+# Test authoring and debugging (Angry Tests)
+
+Philosophy is aligned with Yegor Bugayenko’s [Angry Tests](https://www.yegor256.com/angry-tests.html) and related posts: [On the Layout of Tests](https://www.yegor256.com/2023/01/19/layout-of-tests.html), [single-statement / single-assertion tests](https://www.yegor256.com/2017/05/17/single-statement-unit-tests.html), and [unit testing anti-patterns](https://www.yegor256.com/2018/12/11/unit-testing-anti-patterns.html). Adaptations below are **pytest-specific** for this repo.
+
+Cross-links: proof obligations in [testing.mdc](testing.mdc); Conda commands in [conda-environment.mdc](conda-environment.mdc); general code style in [standards.mdc](standards.mdc).
+
+## Mindset
+
+- Tests are a **safety net**: a test that turns red after a change did its job—it localized a mistake. Fix **production code** or **update the test deliberately** when the contract changed; do not weaken assertions to “make green” without a stated reason.
+- Prefer tests that **pinpoint** the broken unit or scenario so the failure message and test name narrow the search space (layout-of-tests).
+
+## Naming and file placement
+
+- Prefer **`tests/test_<module_basename>.py`** mapping to a single primary **system under test** (SUT), e.g. `test_job_command.py` for `yt_framework.operations.job_command`.
+- For flows that **do not** map 1:1 to one module, use **`tests/integration/`** (or `tests/it/`) and **scenario-oriented** module names (IT-style), still with clear docstrings.
+- Test function names describe **behavior or rule**, not `test1` / `test_foo` (layout-of-tests).
+
+## Assertions (one logical outcome)
+
+- Aim for **one logical outcome per test**: one main `assert`, or one `pytest.raises(...)` block, or one structured equality check. If you need two checks, split the test unless they are inseparable duplicates of the same outcome.
+- Use **`match=`** on `pytest.raises` when the message is part of the contract.
+- Use **`assert expr, "short reason"`** when the default pytest output would be obscure (layout-of-tests: descriptive failures).
+
+## Structure: Arrange / Act / Assert
+
+- Keep tests **short**. Visible three phases: build inputs → call the API → assert.
+- **Shared setup:** Prefer **fake objects** or small factories that live next to production code (or clearly named helpers) over opaque shared fixtures. Avoid “god” fixtures and cross-test coupling ([anti-patterns](https://www.yegor256.com/2018/12/11/unit-testing-anti-patterns.html)).
+- If you add **`tests/support/`**, keep helpers **explicit and minimal**; document what SUT they serve.
+
+## Mocks
+
+- Use **`unittest.mock` / `patch` sparingly**. This codebase is YT/S3-heavy: prefer **real pure logic**, **fakes**, **dev-mode** boundaries, or narrow integration tests over mocking large client surfaces unless there is no cheaper option.
+
+## Mandatory fix loop (agents)
+
+When a test fails or you are debugging tests, follow this loop **in order**; do not skip straight to refactors.
+
+1. **Reproduce:** `conda run -n yt-framework -- pytest <path>::<test_name> -xvs` (or `-k` with a unique substring). Confirm the failure is stable.
+2. **Read:** Study the **assertion output**, **exception message**, and **traceback top**—identify the failing line in test and production code.
+3. **Locate SUT:** Open the production module that owns the behavior; confirm whether the test expectation or the code is wrong.
+4. **Hypothesis:** State one sentence: e.g. “Off-by-one in path normalization” or “Test encodes old API.”
+5. **Minimal change:** Apply the **smallest** edit that addresses that hypothesis only (standards: root-cause, no drive-by rewrites).
+6. **Re-run:** Same single test, then full `conda run -n yt-framework -- pytest`.
+7. **If still red:** Do **not** stack unrelated edits. Return to step 2 with fresh output. If the failure is environmental (Conda, missing env), fix the environment or document `--no-verify` only as an emergency (see CONTRIBUTING).
+
+Proof of done: cite pytest output per [testing.mdc](testing.mdc).
+
+## Project conventions
+
+- Run pytest through **`conda run -n yt-framework --`** when verifying locally ([conda-environment.mdc](conda-environment.mdc)).
+- Test code: type hints and Black like production ([standards.mdc](standards.mdc)).
+- For breadth of coverage and repo status, see [.cursor/artifacts/project-details/testing-readiness-report.md](../artifacts/project-details/testing-readiness-report.md).

.cursor/rules/testing.mdc

@@ -0,0 +1,35 @@
+---
+description: Keep automated and manual verification aligned with code; prove changes with pytest or documented alternatives
+alwaysApply: true
+---
+
+# Testing maintenance
+
+For **how** to write and debug tests (Angry Tests–aligned pytest style and mandatory fix loop), use [test-authoring.mdc](test-authoring.mdc). This file focuses on **when** to test and **proof** obligations.
+
+## Surfaces
+
+Treat as testing scope: `tests/**` for **pytest** (dev dependency in repo-root `pyproject.toml`). Complementary checks when automated coverage is thin or behavior is pipeline/integration-heavy: **dev mode** and **example pipelines** as described in `CONTRIBUTING.md` (Testing). Test code follows [standards.mdc](standards.mdc) for style and clarity (types, assertions, `black` on changed Python files).
+
+## When to add or update tests
+
+Add or extend tests in the same change set when you change behavior, fix bugs, adjust public contracts, or touch logic that already has pytest coverage. Prefer focused tests next to the modules they exercise under `tests/`. When the suite does not yet cover an area, use dev mode and/or a minimal example pipeline to verify—and add pytest when the behavior is stable enough to automate.
+
+## Obligation
+
+Do not claim a change is verified without evidence. Update or add tests so they match the new semantics; remove or rewrite assertions that encode obsolete behavior. If you rely on dev mode or examples instead of pytest, say what you ran and the outcome.
+
+## While working
+
+- **Early**: Identify which modules or flows changed and whether `tests/` already covers them; note gaps.
+- **Before done**: Run the relevant pytest scope (full suite or targeted). For integration-only or uncovered paths, run the dev-mode or example steps from `CONTRIBUTING.md` and record the result.
+
+## Proof
+
+Run commands through the project Conda env: [conda-environment.mdc](conda-environment.mdc) (e.g. `conda run -n yt-framework -- …`).
+
+- **Touches production code or existing tests**: run pytest from the repo root, e.g. `conda run -n yt-framework -- pytest`. For a narrow check: `conda run -n yt-framework -- pytest tests/<file>.py` or `pytest path::test_name`. Optional coverage: `conda run -n yt-framework -- pytest --cov=yt_framework`.
+- **Integration-heavy or uncovered areas**: cite dev-mode or example runs per `CONTRIBUTING.md` (and add pytest when practical).
+- **Test-only or rule-only edits (no `docs/` or user-facing README/CONTRIBUTING changes)**: Sphinx is not required; still run pytest if Python tests changed.
+
+Done = proof, same spirit as [standards.mdc](standards.mdc): do not claim completion without evidence appropriate to what changed.

.github/workflows/ci.yml

@@ -0,0 +1,84 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - "**"
+  pull_request:
+    branches: [main, dev]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run tests
+        run: |
+          pytest \
+            --cov=yt_framework \
+            --cov=ytjobs \
+            --cov-report=term-missing \
+            --cov-report=json:coverage.json
+
+      - name: Publish coverage badge to gist
+        if: github.event_name == 'push' && github.ref == 'refs/heads/main' && vars.COVERAGE_GIST_ID != ''
+        env:
+          COVERAGE_GIST_ID: ${{ vars.COVERAGE_GIST_ID }}
+          COVERAGE_GIST_TOKEN: ${{ secrets.COVERAGE_GIST_TOKEN }}
+        run: |
+          python <<'PY'
+          import json
+          import os
+          import urllib.error
+          import urllib.request
+
+          with open("coverage.json", encoding="utf-8") as f:
+              cov = json.load(f)
+          pct = round(float(cov["totals"]["percent_covered"]), 1)
+          if pct >= 80.0:
+              color = "brightgreen"
+          elif pct >= 60.0:
+              color = "yellow"
+          else:
+              color = "red"
+          payload = {
+              "schemaVersion": 1,
+              "label": "coverage",
+              "message": f"{pct}%",
+              "color": color,
+          }
+          body = json.dumps(
+              {"files": {"coverage-shields.json": {"content": json.dumps(payload)}}}
+          )
+          req = urllib.request.Request(
+              f"https://api.github.com/gists/{os.environ['COVERAGE_GIST_ID']}",
+              data=body.encode(),
+              method="PATCH",
+              headers={
+                  "Authorization": f"Bearer {os.environ['COVERAGE_GIST_TOKEN']}",
+                  "Accept": "application/vnd.github+json",
+                  "X-GitHub-Api-Version": "2022-11-28",
+              },
+          )
+          try:
+              with urllib.request.urlopen(req) as resp:
+                  assert 200 <= resp.status < 300, resp.status
+          except urllib.error.HTTPError as e:
+              raise SystemExit(f"gist PATCH failed: HTTP {e.code}") from e
+          PY

.gitignore

@@ -16,7 +16,9 @@ Thumbs.db
 
 # IDEs
 .vscode/
-.cursor/
+.cursor/*
+!.cursor/rules/
+!.cursor/rules/**
 .idea/
 *.swp
 *.swo
@@ -29,6 +31,10 @@ Thumbs.db
 # Checkpoints
 *.pt
 
+# Coverage
+.coverage
+coverage.json
+
 # Virtual environment
 .venv/
 venv/

.pre-commit-config.yaml

@@ -0,0 +1,17 @@
+repos:
+  - repo: https://github.com/psf/black
+    rev: 26.3.1
+    hooks:
+      - id: black
+        language_version: python3.11
+        args: [--target-version=py311]
+
+  - repo: local
+    hooks:
+      - id: pytest
+        name: pytest (pre-push)
+        entry: bash -c 'conda run -n yt-framework -- pytest'
+        language: system
+        pass_filenames: false
+        always_run: true
+        stages: [pre-push]