feat(docs-devel): Improve developer documentation

.markdownlint.json

@@ -0,0 +1,6 @@
+{
+  "MD013": false,
+  "MD007": {
+    "indent": 3
+  }
+}

.pre-commit-config.yaml

@@ -20,3 +20,8 @@ repos:
     rev: v0.11.0
     hooks:
       - id: shellcheck
+  - repo: https://github.com/DavidAnson/markdownlint-cli2
+    rev: v0.20.0
+    hooks:
+      - id: markdownlint-cli2-docker
+        exclude: '^(?:\.github|docs)(?:/|$)'

AGENTS.md

@@ -17,16 +17,7 @@ common developer tasks.
 
 ## Repository layout (high level)
 
-```text
-.
-├── certs/                  # Certificate request configurations used for HTTPS/SSL
-├── elt-common/             # Reusable Python package with common ETL/ELT helpers used by the warehouses
-├── docs/                   # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site.
-├── infra/
-│   ├── ansible-docker/     # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud.
-│   └── local/              # docker-compose configuration for a local development environment and end-to-end CI tests.
-└── warehouses/             # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables.
-```
+- [`docs-devel/src/index.md#repository-overview`](./docs-devel/src/index.md#repository-overview) describes the repository layout.
 
 ## Pre-commit hooks
 
@@ -37,31 +28,8 @@ common developer tasks.
 
 - There are both unit and e2e tests for the `elt-common` package under `elt-common/tests/`. They are
   written using `pytest`.
-- Install the package in editable mode + dev dependencies in a Python environment, e.g. with `uv`
-
-```bash
-cd elt-common/
-uv venv
-uv install . --editable --group dev
-```
-
-- Run unit tests with:
-
-```bash
-cd elt-common/
-uv run pytest unit_tests
-```
-
-- The e2e tests require a running data platform stack. Use the docker-based setup in `infra/local`
-  to start the stack, run the tests and bring down the stack:
-
-```bash
-pushd infra/local; docker compose up -d; popd    # start the services
-echo "127.0.0.1    adp-router" | sudo tee -a /etc/hosts   # edit /etc/hosts
-cd elt-common/
-uv run pytest e2e_tests
-pushd infra/local; docker compose down -v; popd  # stop the services
-```
+- [`elt-common/README.md`](./elt-common/README.md) explains how to run tests for the `elt-common`
+  Python package
 
 ## Cloud deployment
 
@@ -73,11 +41,12 @@ pushd infra/local; docker compose down -v; popd  # stop the services
 
 - When creating pull requests:
 
-  1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
-  2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
-     - Format: `type(scope): description`
-     - Example: `fix(warehouse/accelerator): fix join in model`
-     - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
+1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
+
+2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
+   - Format: `type(scope): description`
+   - Example: `fix(warehouse/accelerator): fix join in model`
+   - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
 
 **Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time.
 

docs-devel/data-architecture/catalogs.md

@@ -0,0 +1,9 @@
+# Catalogs
+
+This page describes the catalogs that exist within the ISIS Lakehouse.
+
+## Index
+
+- `accelerator`: Provides data for analysis of accelerator operations.
+   - **bronze layer**: All schemas beginning with the prefix `src_`.
+   - **silver layer**: All schemas beginning with the prefix `analytics_`.

docs-devel/data-architecture/index.md

@@ -0,0 +1,35 @@
+# Data Architecture
+
+These pages describe our implementation of the Data Lakehouse using a
+[Medallion-style layout](https://docs.databricks.com/aws/en/lakehouse/medallion)
+to organise data as it flows from source systems through to analytics consumers.
+
+## High-level view
+
+We use the standard layer names `bronze`/`silver`/`gold` where each layer has a clear responsibility. See diagram below
+
+### Bronze
+
+The landing zone for ingested data. Data is pushed into Iceberg tables either in its original state
+or minimally cleaned/normalised. The bronze layer is the single source of truth for all subsequent
+transformations and supports full reprocessing to regenerate the tables in the subsequent layers.
+This layer is not intended for use by end consumers.
+
+### Silver
+
+Data is cleaned and validated, including enforcing quality checks. The silver layer typically
+applies little to no aggregation leaving detailed information for end users to perform their
+own analyses.
+
+### Gold
+
+Data is curated, typically with aggregation applied, into consumption-ready datasets built for business intelligence (BI), reporting and machine learning.
+
+An example diagram of this architecture for a business operations domain is reproduced here:
+
+![Example medallion architecture](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png)
+_[Source: Databricks](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png)_
+
+For a description of the implementation of this architecture see the [system architecture](../system-architecture/index.md) documentation.
+
+For a description of the structure of the catalogs for ISIS see [catalogs](./catalogs.md).

docs-devel/deployment/index.md

@@ -0,0 +1,86 @@
+# Deployment
+
+_The system is currently deployed on the [SCD cloud](https://cloud.stfc.ac.uk/)._
+_In theory it can be deployed on any cloud infrastructure but it will require_
+_changes before this will work._
+
+_It is not yet a production-grade, HA system._
+
+There are several prerequisites steps required before deployment can begin. Please
+read [them here](./prerequisites.md).
+
+## Ansible
+
+Ansible playbooks in `<repo_root>/ansible-docker/playbooks` control the deployment
+of the system. All commands in this section assume that the current working directory
+is `infra/ansible-docker`.
+
+### Networking
+
+Create the private VM network:
+
+```sh
+ansible-playbook playbooks/cloud/private_network_create.yml
+```
+
+Create a node for Traefik (also acts as an SSH jump node):
+
+```sh
+ansible-playbook \
+  -e openstack_cloud_name=<cloud_name> \
+  -e openstack_key_name=<ssh_key> \
+  -e vm_config_file=$PWD/playbooks/traefik/group_vars/traefik.yml
+  playbooks/cloud/vm_create.yml
+```
+
+where _cloud\_name_ and _ssh\_key_ are described in the [prerequisites section](./prerequisites.md#openstack-api--vm-credentials).
+Take a note of the new node ip address and create a new inventory file:
+
+```sh
+cp inventory-sample.yml inventory.yml
+```
+
+Fill in the new Traefik ip address and deploy Traefik:
+
+```sh
+ansible-playbook -i inventory.yml playbooks/traefik/deploy.yml
+```
+
+Once deployed check the Traefik dashboard is available at `https://<domain>/traefik/dashboard/.`
+The passwords are in Keeper.
+
+### Services
+
+Now we deploy the remaining services. The deployment order is important as some
+services depend on others being available. Each service has a single VM with the
+exception of Superset that hosts multiple instances on a single node.
+
+First create the VMs:
+
+```sh
+for svc in keycloak lakekeeper trino elt; do
+  ansible-playbook \
+    -e openstack_cloud_name=<cloud_name> \
+    -e openstack_key_name=<ssh_key> \
+    -e vm_config_file=$PWD/playbooks/$svc/group_vars/$svc.yml
+    playbooks/cloud/vm_create.yml
+done
+# Now Superset
+ansible-playbook \
+  -e openstack_cloud_name=<cloud_name> \
+  -e openstack_key_name=<ssh_key> \
+  -e vm_config_file=$PWD/playbooks/superset/vm_vars.yml
+  playbooks/cloud/vm_create.yml
+```
+
+Gather the new ip addresses of each VM and fill in the appropriate section of the new `inventory.yml` created above.
+
+Now deploy the services:
+
+```sh
+for svc in keycloak lakekeeper trino elt superset; do
+  ansible-playbook -i inventory.yml playbooks/$svc/deploy.yml
+done
+```
+
+Superset should be available at `https://<domain>/workspace/accelerator`.

docs-devel/src/deployment/prerequisites.md → docs-devel/deployment/prerequisites.md

@@ -75,15 +75,11 @@ to create a new access key/secret pair. Update the Ansible vault accordingly.
 
 ## Networking
 
-Two floating IPs are required:
+A floating IP is required for the Traefik load balancer node.
 
-- One for the management/SSH jump node.
-- One for the Traefik load balancer node.
+Using the web interface create one from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring
+a description is provided.
 
-Using the web interface create them from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring
-tgat a description is provided with each.
-
-Check the values of `openstack_management_vm_fip` and
-`openstack_reverse_proxy_fip` in `<repo_root>/ansible/group_vars/all/openstack.yml`
-and update accordingly. The `openstack_reverse_proxy_fip` value must match the value configured
+Update the value of `openstack_reverse_proxy_fip` in `<repo_root>/ansible/group_vars/all/openstack.yml`.
+The `openstack_reverse_proxy_fip` value must match the value configured
 in the DNS record for the domain defined in `<repo_root>/ansible/group_vars/all/domains.yml`

docs-devel/images/.gitignore

@@ -0,0 +1,2 @@
+# Temp draw.io save files
+.$*