feat(adrs): Add ADR selecting Airflow as an orchestration tool

[martyngigg]

Add an ADR choosing Apache Airflow as an orchestration tool.

Fixes #114

Summary by CodeRabbit

• Documentation
  • Added a comprehensive Architecture Decision Record documenting the adoption of Apache Airflow as the job orchestration solution. The document outlines the current platform context and technical requirements, details the selection criteria evaluated, highlights anticipated improvements in visibility and failure notifications, and documents the associated trade-off of maintaining an additional operational tool.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0f97abd3-8258-47aa-8499-d9468ba86af5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This pull request introduces a new Architecture Decision Record that formalises the selection of Apache Airflow as the job orchestration tool, documenting the platform context, decision criteria, and anticipated consequences of moving away from the current cron-based ingestion system.

Changes

Apache Airflow Adoption Decision

Layer / File(s)

Summary

Apache Airflow adoption ADR docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md

ADR that records the decision to adopt Apache Airflow, including background on the current cron-based system with file logging and limited monitoring, decision criteria (Python-defined tasks, cron-like scheduling, REST API triggering, manual one-off runs, web UI, container deployment), and the consequences of improved observability, failure notifications, restart capabilities, and history inspection alongside the trade-off of tool maintenance.

Poem

🐰 A rabbit hops with glee so bright,
Apache Airflow takes to flight!
No more cron jobs scattered 'round,
A DAG-based path has just been found.
With REST and UI, clear and clean,
The finest orchestration seen!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding an Architecture Decision Record that selects Apache Airflow as the orchestration tool.
Linked Issues check	✅ Passed	The ADR documents Apache Airflow's selection and addresses all MVP requirements from issue #114: Python-defined tasks, cron-like scheduling, REST API, manual job execution, web UI, and container deployment.
Out of Scope Changes check	✅ Passed	The pull request contains only a new ADR file documenting the orchestration tool decision, which is directly within scope of issue #114 requirements.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md (1)

19-31: ⚡ Quick win

Add an explicit alternatives comparison to make the decision auditable.

The ADR records the choice, but it does not show why Airflow was selected over the options in Issue #114 (Dagster, Prefect, Kestra, Luigi). A short “Options considered” table mapped to the MVP criteria would make this decision easier to revisit and defend.

[recommend adding a small matrix with columns like Tool, Python tasks, Cron scheduling, REST trigger, Manual one-off, Web UI, Container deployment, Notes.]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md`
around lines 19 - 31, Add a new "Options considered" section in the ADR
(docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md)
immediately after the "Our requirements are simple" list: create a small
comparison matrix with columns Tool, Python tasks, Cron scheduling, REST
trigger, Manual one-off, Web UI, Container deployment, Notes and rows for
Airflow, Dagster, Prefect, Kestra, and Luigi; populate each cell against the MVP
criteria and add a short Notes cell summarising key trade-offs and any blockers
from Issue `#114`, then add a final line stating why Airflow best meets the
criteria to make the decision auditable.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md`:
- Around line 22-31: The ADR text uses inconsistent terminology and grammar in
several bullet points and a sentence: change "python" to "Python", "REST api" to
"REST API", and rephrase the sentence "We therefore include that Airflow is a
natural choice as an orchestration tool." to a clearer form such as "Therefore,
Airflow is a natural choice for orchestration." Update the bullets "Tasks to be
defined as python code/packages.", "Allows a cron-like timed schedule to be
created.", "A REST api to trigger jobs.", "Allows one-off jobs triggered
manually on request.", "A web-based UI.", and the concluding sentence so casing
and phrasing are consistent and grammatically correct across the ADR.

---

Nitpick comments:
In
`@docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md`:
- Around line 19-31: Add a new "Options considered" section in the ADR
(docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md)
immediately after the "Our requirements are simple" list: create a small
comparison matrix with columns Tool, Python tasks, Cron scheduling, REST
trigger, Manual one-off, Web UI, Container deployment, Notes and rows for
Airflow, Dagster, Prefect, Kestra, and Luigi; populate each cell against the MVP
criteria and add a short Notes cell summarising key trade-offs and any blockers
from Issue `#114`, then add a final line stating why Airflow best meets the
criteria to make the decision auditable.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a0a0028a-3738-4d29-acda-271b435f4b2d

📥 Commits

Reviewing files that changed from the base of the PR and between 9db2078 and 3f86e89.

📒 Files selected for processing (1)

• docs-devel/system-architecture/adrs/0009-use-apache-airflow-as-job-orchestration-tool.md