Skip to content

Commit 8112d6e

Browse files
martyngiggmartyngigg
authored
feat(docs-devel): Improve developer documentation (#175)
### Summary Refs #109 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Documentation** * Added comprehensive deployment documentation covering prerequisites, networking setup, and service deployment procedures. * Introduced data architecture documentation describing layered data models and catalogue structure. * Expanded system architecture documentation outlining key services and their interactions. * **Chores** * Updated configuration tooling and development infrastructure. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
1 parent a4f0e80 commit 8112d6e

42 files changed

Lines changed: 530 additions & 445 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.markdownlint.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"MD013": false,
3+
"MD007": {
4+
"indent": 3
5+
}
6+
}

‎.pre-commit-config.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,8 @@ repos:
2020
rev: v0.11.0
2121
hooks:
2222
- id: shellcheck
23+
- repo: https://github.com/DavidAnson/markdownlint-cli2
24+
rev: v0.20.0
25+
hooks:
26+
- id: markdownlint-cli2-docker
27+
exclude: '^(?:\.github|docs)(?:/|$)'

‎AGENTS.md‎

Lines changed: 9 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -17,16 +17,7 @@ common developer tasks.
1717

1818
## Repository layout (high level)
1919

20-
```text
21-
.
22-
├── certs/ # Certificate request configurations used for HTTPS/SSL
23-
├── elt-common/ # Reusable Python package with common ETL/ELT helpers used by the warehouses
24-
├── docs/ # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site.
25-
├── infra/
26-
│ ├── ansible-docker/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud.
27-
│ └── local/ # docker-compose configuration for a local development environment and end-to-end CI tests.
28-
└── warehouses/ # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables.
29-
```
20+
- [`docs-devel/src/index.md#repository-overview`](./docs-devel/src/index.md#repository-overview) describes the repository layout.
3021

3122
## Pre-commit hooks
3223

@@ -37,31 +28,8 @@ common developer tasks.
3728

3829
- There are both unit and e2e tests for the `elt-common` package under `elt-common/tests/`. They are
3930
written using `pytest`.
40-
- Install the package in editable mode + dev dependencies in a Python environment, e.g. with `uv`
41-
42-
```bash
43-
cd elt-common/
44-
uv venv
45-
uv install . --editable --group dev
46-
```
47-
48-
- Run unit tests with:
49-
50-
```bash
51-
cd elt-common/
52-
uv run pytest unit_tests
53-
```
54-
55-
- The e2e tests require a running data platform stack. Use the docker-based setup in `infra/local`
56-
to start the stack, run the tests and bring down the stack:
57-
58-
```bash
59-
pushd infra/local; docker compose up -d; popd # start the services
60-
echo "127.0.0.1 adp-router" | sudo tee -a /etc/hosts # edit /etc/hosts
61-
cd elt-common/
62-
uv run pytest e2e_tests
63-
pushd infra/local; docker compose down -v; popd # stop the services
64-
```
31+
- [`elt-common/README.md`](./elt-common/README.md) explains how to run tests for the `elt-common`
32+
Python package
6533

6634
## Cloud deployment
6735

@@ -73,11 +41,12 @@ pushd infra/local; docker compose down -v; popd # stop the services
7341

7442
- When creating pull requests:
7543

76-
1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
77-
2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
78-
- Format: `type(scope): description`
79-
- Example: `fix(warehouse/accelerator): fix join in model`
80-
- Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
44+
1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
45+
46+
2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
47+
- Format: `type(scope): description`
48+
- Example: `fix(warehouse/accelerator): fix join in model`
49+
- Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
8150

8251
**Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time.
8352

‎docs-devel/data-architecture/catalogs.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Catalogs
2+
3+
This page describes the catalogs that exist within the ISIS Lakehouse.
4+
5+
## Index
6+
7+
- `accelerator`: Provides data for analysis of accelerator operations.
8+
- **bronze layer**: All schemas beginning with the prefix `src_`.
9+
- **silver layer**: All schemas beginning with the prefix `analytics_`.

‎docs-devel/data-architecture/index.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Data Architecture
2+
3+
These pages describe our implementation of the Data Lakehouse using a
4+
[Medallion-style layout](https://docs.databricks.com/aws/en/lakehouse/medallion)
5+
to organise data as it flows from source systems through to analytics consumers.
6+
7+
## High-level view
8+
9+
We use the standard layer names `bronze`/`silver`/`gold` where each layer has a clear responsibility. See diagram below
10+
11+
### Bronze
12+
13+
The landing zone for ingested data. Data is pushed into Iceberg tables either in its original state
14+
or minimally cleaned/normalised. The bronze layer is the single source of truth for all subsequent
15+
transformations and supports full reprocessing to regenerate the tables in the subsequent layers.
16+
This layer is not intended for use by end consumers.
17+
18+
### Silver
19+
20+
Data is cleaned and validated, including enforcing quality checks. The silver layer typically
21+
applies little to no aggregation leaving detailed information for end users to perform their
22+
own analyses.
23+
24+
### Gold
25+
26+
Data is curated, typically with aggregation applied, into consumption-ready datasets built for business intelligence (BI), reporting and machine learning.
27+
28+
An example diagram of this architecture for a business operations domain is reproduced here:
29+
30+
![Example medallion architecture](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png)
31+
_[Source: Databricks](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png)_
32+
33+
For a description of the implementation of this architecture see the [system architecture](../system-architecture/index.md) documentation.
34+
35+
For a description of the structure of the catalogs for ISIS see [catalogs](./catalogs.md).

‎docs-devel/deployment/index.md‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Deployment
2+
3+
_The system is currently deployed on the [SCD cloud](https://cloud.stfc.ac.uk/)._
4+
_In theory it can be deployed on any cloud infrastructure but it will require_
5+
_changes before this will work._
6+
7+
_It is not yet a production-grade, HA system._
8+
9+
There are several prerequisites steps required before deployment can begin. Please
10+
read [them here](./prerequisites.md).
11+
12+
## Ansible
13+
14+
Ansible playbooks in `<repo_root>/ansible-docker/playbooks` control the deployment
15+
of the system. All commands in this section assume that the current working directory
16+
is `infra/ansible-docker`.
17+
18+
### Networking
19+
20+
Create the private VM network:
21+
22+
```sh
23+
ansible-playbook playbooks/cloud/private_network_create.yml
24+
```
25+
26+
Create a node for Traefik (also acts as an SSH jump node):
27+
28+
```sh
29+
ansible-playbook \
30+
-e openstack_cloud_name=<cloud_name> \
31+
-e openstack_key_name=<ssh_key> \
32+
-e vm_config_file=$PWD/playbooks/traefik/group_vars/traefik.yml
33+
playbooks/cloud/vm_create.yml
34+
```
35+
36+
where _cloud\_name_ and _ssh\_key_ are described in the [prerequisites section](./prerequisites.md#openstack-api--vm-credentials).
37+
Take a note of the new node ip address and create a new inventory file:
38+
39+
```sh
40+
cp inventory-sample.yml inventory.yml
41+
```
42+
43+
Fill in the new Traefik ip address and deploy Traefik:
44+
45+
```sh
46+
ansible-playbook -i inventory.yml playbooks/traefik/deploy.yml
47+
```
48+
49+
Once deployed check the Traefik dashboard is available at `https://<domain>/traefik/dashboard/.`
50+
The passwords are in Keeper.
51+
52+
### Services
53+
54+
Now we deploy the remaining services. The deployment order is important as some
55+
services depend on others being available. Each service has a single VM with the
56+
exception of Superset that hosts multiple instances on a single node.
57+
58+
First create the VMs:
59+
60+
```sh
61+
for svc in keycloak lakekeeper trino elt; do
62+
ansible-playbook \
63+
-e openstack_cloud_name=<cloud_name> \
64+
-e openstack_key_name=<ssh_key> \
65+
-e vm_config_file=$PWD/playbooks/$svc/group_vars/$svc.yml
66+
playbooks/cloud/vm_create.yml
67+
done
68+
# Now Superset
69+
ansible-playbook \
70+
-e openstack_cloud_name=<cloud_name> \
71+
-e openstack_key_name=<ssh_key> \
72+
-e vm_config_file=$PWD/playbooks/superset/vm_vars.yml
73+
playbooks/cloud/vm_create.yml
74+
```
75+
76+
Gather the new ip addresses of each VM and fill in the appropriate section of the new `inventory.yml` created above.
77+
78+
Now deploy the services:
79+
80+
```sh
81+
for svc in keycloak lakekeeper trino elt superset; do
82+
ansible-playbook -i inventory.yml playbooks/$svc/deploy.yml
83+
done
84+
```
85+
86+
Superset should be available at `https://<domain>/workspace/accelerator`.

docs-devel/src/deployment/prerequisites.md renamed to docs-devel/deployment/prerequisites.md

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -75,15 +75,11 @@ to create a new access key/secret pair. Update the Ansible vault accordingly.
7575

7676
## Networking
7777

78-
Two floating IPs are required:
78+
A floating IP is required for the Traefik load balancer node.
7979

80-
- One for the management/SSH jump node.
81-
- One for the Traefik load balancer node.
80+
Using the web interface create one from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring
81+
a description is provided.
8282

83-
Using the web interface create them from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring
84-
tgat a description is provided with each.
85-
86-
Check the values of `openstack_management_vm_fip` and
87-
`openstack_reverse_proxy_fip` in `<repo_root>/ansible/group_vars/all/openstack.yml`
88-
and update accordingly. The `openstack_reverse_proxy_fip` value must match the value configured
83+
Update the value of `openstack_reverse_proxy_fip` in `<repo_root>/ansible/group_vars/all/openstack.yml`.
84+
The `openstack_reverse_proxy_fip` value must match the value configured
8985
in the DNS record for the domain defined in `<repo_root>/ansible/group_vars/all/domains.yml`

docs-devel/src/deployment/troubleshooting.md renamed to docs-devel/deployment/troubleshooting.md

File renamed without changes.

‎docs-devel/images/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Temp draw.io save files
2+
.$*

0 commit comments

Comments
 (0)