diff --git a/.markdownlint.json b/.markdownlint.json new file mode 100644 index 00000000..01ec74e7 --- /dev/null +++ b/.markdownlint.json @@ -0,0 +1,6 @@ +{ + "MD013": false, + "MD007": { + "indent": 3 + } +} diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7c5ab682..dd86cfff 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -20,3 +20,8 @@ repos: rev: v0.11.0 hooks: - id: shellcheck + - repo: https://github.com/DavidAnson/markdownlint-cli2 + rev: v0.20.0 + hooks: + - id: markdownlint-cli2-docker + exclude: '^(?:\.github|docs)(?:/|$)' diff --git a/AGENTS.md b/AGENTS.md index 792b7b16..91fcf85d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,16 +17,7 @@ common developer tasks. ## Repository layout (high level) -```text -. -├── certs/ # Certificate request configurations used for HTTPS/SSL -├── elt-common/ # Reusable Python package with common ETL/ELT helpers used by the warehouses -├── docs/ # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site. -├── infra/ -│ ├── ansible-docker/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud. -│ └── local/ # docker-compose configuration for a local development environment and end-to-end CI tests. -└── warehouses/ # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables. -``` +- [`docs-devel/src/index.md#repository-overview`](./docs-devel/src/index.md#repository-overview) describes the repository layout. ## Pre-commit hooks @@ -37,31 +28,8 @@ common developer tasks. - There are both unit and e2e tests for the `elt-common` package under `elt-common/tests/`. They are written using `pytest`. -- Install the package in editable mode + dev dependencies in a Python environment, e.g. with `uv` - -```bash -cd elt-common/ -uv venv -uv install . --editable --group dev -``` - -- Run unit tests with: - -```bash -cd elt-common/ -uv run pytest unit_tests -``` - -- The e2e tests require a running data platform stack. Use the docker-based setup in `infra/local` - to start the stack, run the tests and bring down the stack: - -```bash -pushd infra/local; docker compose up -d; popd # start the services -echo "127.0.0.1 adp-router" | sudo tee -a /etc/hosts # edit /etc/hosts -cd elt-common/ -uv run pytest e2e_tests -pushd infra/local; docker compose down -v; popd # stop the services -``` +- [`elt-common/README.md`](./elt-common/README.md) explains how to run tests for the `elt-common` + Python package ## Cloud deployment @@ -73,11 +41,12 @@ pushd infra/local; docker compose down -v; popd # stop the services - When creating pull requests: - 1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format - 2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) - - Format: `type(scope): description` - - Example: `fix(warehouse/accelerator): fix join in model` - - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore` +1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format + +2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) + - Format: `type(scope): description` + - Example: `fix(warehouse/accelerator): fix join in model` + - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore` **Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time. diff --git a/docs-devel/data-architecture/catalogs.md b/docs-devel/data-architecture/catalogs.md new file mode 100644 index 00000000..fa5fa27e --- /dev/null +++ b/docs-devel/data-architecture/catalogs.md @@ -0,0 +1,9 @@ +# Catalogs + +This page describes the catalogs that exist within the ISIS Lakehouse. + +## Index + +- `accelerator`: Provides data for analysis of accelerator operations. + - **bronze layer**: All schemas beginning with the prefix `src_`. + - **silver layer**: All schemas beginning with the prefix `analytics_`. diff --git a/docs-devel/data-architecture/index.md b/docs-devel/data-architecture/index.md new file mode 100644 index 00000000..f65aae4f --- /dev/null +++ b/docs-devel/data-architecture/index.md @@ -0,0 +1,35 @@ +# Data Architecture + +These pages describe our implementation of the Data Lakehouse using a +[Medallion-style layout](https://docs.databricks.com/aws/en/lakehouse/medallion) +to organise data as it flows from source systems through to analytics consumers. + +## High-level view + +We use the standard layer names `bronze`/`silver`/`gold` where each layer has a clear responsibility. See diagram below + +### Bronze + +The landing zone for ingested data. Data is pushed into Iceberg tables either in its original state +or minimally cleaned/normalised. The bronze layer is the single source of truth for all subsequent +transformations and supports full reprocessing to regenerate the tables in the subsequent layers. +This layer is not intended for use by end consumers. + +### Silver + +Data is cleaned and validated, including enforcing quality checks. The silver layer typically +applies little to no aggregation leaving detailed information for end users to perform their +own analyses. + +### Gold + +Data is curated, typically with aggregation applied, into consumption-ready datasets built for business intelligence (BI), reporting and machine learning. + +An example diagram of this architecture for a business operations domain is reproduced here: + +![Example medallion architecture](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png) +_[Source: Databricks](https://docs.databricks.com/aws/en/assets/images/medallion-architecture-15e2d57ad70d28b1701dda4f7271d862.png)_ + +For a description of the implementation of this architecture see the [system architecture](../system-architecture/index.md) documentation. + +For a description of the structure of the catalogs for ISIS see [catalogs](./catalogs.md). diff --git a/docs-devel/deployment/index.md b/docs-devel/deployment/index.md new file mode 100644 index 00000000..9ecdfed7 --- /dev/null +++ b/docs-devel/deployment/index.md @@ -0,0 +1,86 @@ +# Deployment + +_The system is currently deployed on the [SCD cloud](https://cloud.stfc.ac.uk/)._ +_In theory it can be deployed on any cloud infrastructure but it will require_ +_changes before this will work._ + +_It is not yet a production-grade, HA system._ + +There are several prerequisites steps required before deployment can begin. Please +read [them here](./prerequisites.md). + +## Ansible + +Ansible playbooks in `/ansible-docker/playbooks` control the deployment +of the system. All commands in this section assume that the current working directory +is `infra/ansible-docker`. + +### Networking + +Create the private VM network: + +```sh +ansible-playbook playbooks/cloud/private_network_create.yml +``` + +Create a node for Traefik (also acts as an SSH jump node): + +```sh +ansible-playbook \ + -e openstack_cloud_name= \ + -e openstack_key_name= \ + -e vm_config_file=$PWD/playbooks/traefik/group_vars/traefik.yml + playbooks/cloud/vm_create.yml +``` + +where _cloud\_name_ and _ssh\_key_ are described in the [prerequisites section](./prerequisites.md#openstack-api--vm-credentials). +Take a note of the new node ip address and create a new inventory file: + +```sh +cp inventory-sample.yml inventory.yml +``` + +Fill in the new Traefik ip address and deploy Traefik: + +```sh +ansible-playbook -i inventory.yml playbooks/traefik/deploy.yml +``` + +Once deployed check the Traefik dashboard is available at `https:///traefik/dashboard/.` +The passwords are in Keeper. + +### Services + +Now we deploy the remaining services. The deployment order is important as some +services depend on others being available. Each service has a single VM with the +exception of Superset that hosts multiple instances on a single node. + +First create the VMs: + +```sh +for svc in keycloak lakekeeper trino elt; do + ansible-playbook \ + -e openstack_cloud_name= \ + -e openstack_key_name= \ + -e vm_config_file=$PWD/playbooks/$svc/group_vars/$svc.yml + playbooks/cloud/vm_create.yml +done +# Now Superset +ansible-playbook \ + -e openstack_cloud_name= \ + -e openstack_key_name= \ + -e vm_config_file=$PWD/playbooks/superset/vm_vars.yml + playbooks/cloud/vm_create.yml +``` + +Gather the new ip addresses of each VM and fill in the appropriate section of the new `inventory.yml` created above. + +Now deploy the services: + +```sh +for svc in keycloak lakekeeper trino elt superset; do + ansible-playbook -i inventory.yml playbooks/$svc/deploy.yml +done +``` + +Superset should be available at `https:///workspace/accelerator`. diff --git a/docs-devel/src/deployment/prerequisites.md b/docs-devel/deployment/prerequisites.md similarity index 87% rename from docs-devel/src/deployment/prerequisites.md rename to docs-devel/deployment/prerequisites.md index cf76745a..0298e222 100644 --- a/docs-devel/src/deployment/prerequisites.md +++ b/docs-devel/deployment/prerequisites.md @@ -75,15 +75,11 @@ to create a new access key/secret pair. Update the Ansible vault accordingly. ## Networking -Two floating IPs are required: +A floating IP is required for the Traefik load balancer node. -- One for the management/SSH jump node. -- One for the Traefik load balancer node. +Using the web interface create one from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring +a description is provided. -Using the web interface create them from _Project->Network->Floating IPs_, using _ALLOCATE IP TO PROJECT_, ensuring -tgat a description is provided with each. - -Check the values of `openstack_management_vm_fip` and -`openstack_reverse_proxy_fip` in `/ansible/group_vars/all/openstack.yml` -and update accordingly. The `openstack_reverse_proxy_fip` value must match the value configured +Update the value of `openstack_reverse_proxy_fip` in `/ansible/group_vars/all/openstack.yml`. +The `openstack_reverse_proxy_fip` value must match the value configured in the DNS record for the domain defined in `/ansible/group_vars/all/domains.yml` diff --git a/docs-devel/src/deployment/troubleshooting.md b/docs-devel/deployment/troubleshooting.md similarity index 100% rename from docs-devel/src/deployment/troubleshooting.md rename to docs-devel/deployment/troubleshooting.md diff --git a/docs-devel/images/.gitignore b/docs-devel/images/.gitignore new file mode 100644 index 00000000..a9652095 --- /dev/null +++ b/docs-devel/images/.gitignore @@ -0,0 +1,2 @@ +# Temp draw.io save files +.$* diff --git a/docs-devel/images/analytics-data-platform-arch.drawio b/docs-devel/images/analytics-data-platform-arch.drawio new file mode 100644 index 00000000..727d7793 --- /dev/null +++ b/docs-devel/images/analytics-data-platform-arch.drawio @@ -0,0 +1,155 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs-devel/images/analytics-data-platform-arch.drawio.png b/docs-devel/images/analytics-data-platform-arch.drawio.png new file mode 100644 index 00000000..b472798a Binary files /dev/null and b/docs-devel/images/analytics-data-platform-arch.drawio.png differ diff --git a/docs-devel/images/analytics-data-platform-arch.png b/docs-devel/images/analytics-data-platform-arch.png new file mode 100644 index 00000000..055a4662 Binary files /dev/null and b/docs-devel/images/analytics-data-platform-arch.png differ diff --git a/docs-devel/src/assets/images/logo.svg b/docs-devel/images/logo.svg similarity index 100% rename from docs-devel/src/assets/images/logo.svg rename to docs-devel/images/logo.svg diff --git a/docs-devel/index.md b/docs-devel/index.md new file mode 100644 index 00000000..7ef6b848 --- /dev/null +++ b/docs-devel/index.md @@ -0,0 +1,39 @@ +# Analytics Data Platform + +## Introduction + +After requirements gathering meetings and a workshop as part of the +[accelerator operational data platform project](https://stfc365.sharepoint.com/sites/ISISProject-1432) +there is appetite for trialling a system to support long-term storage, complex +queries and analysis of data from a multitude of sources across the accelerator +data space. + +This document describes the initial set of requirements along with an overview +of the design and implementation of a minimal system to prove +the [Data Lakehouse](https://stfc365-my.sharepoint.com/:w:/g/personal/martyn_gigg_stfc_ac_uk/ETTFvkI6FulFhQFWmrmfGosBRO1Syvqbiq6DhVwnqxhVbw?e=HNKUIl) +concept can be valuable to ISIS. + +## Repository overview + +This repository is a monorepo containing all of the code for the platform. It may be separated out in +the future. + +```text +. +├── certs/ # Certificate request configurations used for HTTPS/SSL +├── elt-common/ # Reusable Python package with common ETL/ELT helpers used by the warehouses +├── docs/ # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site. +├── infra/ +│ ├── ansible-docker/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud. +│ ├── container-images/ # Container definitions deployed services +│ └── local/ # docker-compose configuration for a local development environment and end-to-end CI tests. +└── warehouses/ # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables. +``` + +## Details + +Jump to: + +- [System architecture](./system-architecture/index.md) +- [Data architecture](./data-architecture/index.md) +- [Deployment](./deployment/index.md) diff --git a/docs-devel/mkdocs.yml b/docs-devel/mkdocs.yml deleted file mode 100644 index dd2b61ef..00000000 --- a/docs-devel/mkdocs.yml +++ /dev/null @@ -1,14 +0,0 @@ -docs_dir: src -site_name: ISIS Accelerator Data Platform - Developer -site_url: https://data-accelerator.isis.cclrc.ac.uk/docs-devel/ -site_author: ISIS Computing Division -site_description: Developer documentation for the ISIS Data Platform Prototype -theme: - name: readthedocs - logo: assets/images/logo.svg -plugins: - - autorefs - - search -markdown_extensions: - - admonition - - tables diff --git a/docs-devel/requirements.in b/docs-devel/requirements.in deleted file mode 100644 index de45d5b6..00000000 --- a/docs-devel/requirements.in +++ /dev/null @@ -1 +0,0 @@ -mkdocs-autorefs==1.4.1 diff --git a/docs-devel/requirements.txt b/docs-devel/requirements.txt deleted file mode 100644 index 923a64fa..00000000 --- a/docs-devel/requirements.txt +++ /dev/null @@ -1,46 +0,0 @@ -# This file was autogenerated by uv via the following command: -# uv pip compile requirements.in -o requirements.txt -click==8.2.1 - # via mkdocs -ghp-import==2.1.0 - # via mkdocs -jinja2==3.1.6 - # via mkdocs -markdown==3.8 - # via - # mkdocs - # mkdocs-autorefs -markupsafe==3.0.2 - # via - # jinja2 - # mkdocs - # mkdocs-autorefs -mergedeep==1.3.4 - # via - # mkdocs - # mkdocs-get-deps -mkdocs==1.6.1 - # via mkdocs-autorefs -mkdocs-autorefs==1.4.1 - # via -r requirements.in -mkdocs-get-deps==0.2.0 - # via mkdocs -packaging==25.0 - # via mkdocs -pathspec==0.12.1 - # via mkdocs -platformdirs==4.3.8 - # via mkdocs-get-deps -python-dateutil==2.9.0.post0 - # via ghp-import -pyyaml==6.0.2 - # via - # mkdocs - # mkdocs-get-deps - # pyyaml-env-tag -pyyaml-env-tag==1.1 - # via mkdocs -six==1.17.0 - # via python-dateutil -watchdog==6.0.0 - # via mkdocs diff --git a/docs-devel/src/assets/images/lakehouse-data-architecture-layers.jpg b/docs-devel/src/assets/images/lakehouse-data-architecture-layers.jpg deleted file mode 100644 index e076f882..00000000 Binary files a/docs-devel/src/assets/images/lakehouse-data-architecture-layers.jpg and /dev/null differ diff --git a/docs-devel/src/assets/images/lakehouse-poc.drawio b/docs-devel/src/assets/images/lakehouse-poc.drawio deleted file mode 100644 index 0b738ce9..00000000 --- a/docs-devel/src/assets/images/lakehouse-poc.drawio +++ /dev/null @@ -1,93 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs-devel/src/assets/images/lakehouse-poc.png b/docs-devel/src/assets/images/lakehouse-poc.png deleted file mode 100644 index e773f80b..00000000 Binary files a/docs-devel/src/assets/images/lakehouse-poc.png and /dev/null differ diff --git a/docs-devel/src/conf.py b/docs-devel/src/conf.py deleted file mode 100644 index 68bf3e69..00000000 --- a/docs-devel/src/conf.py +++ /dev/null @@ -1,30 +0,0 @@ -# Configuration file for the Sphinx documentation builder. -# -# For the full list of built-in configuration values, see the documentation: -# https://www.sphinx-doc.org/en/master/usage/configuration.html - -# -- Project information ----------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information - -project = "ISIS Accelerator Data" -copyright = "2025, ISIS Computing Division" -author = "ISIS Computing Division" - -# -- General configuration --------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration - -exclude_patterns = ["./common_links.rst"] -extensions = ["myst_parser"] -master_doc = "index" -source_suffix = {".rst": "restructuredtext", ".md": "markdown"} -templates_path = ["_templates"] - - -# -- Options for HTML output ------------------------------------------------- -# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output - -html_css_files = ["custom.css"] -html_theme = "sphinx_book_theme" -html_static_path = ["../../_static_shared"] -html_logo = "../../_static_shared/logo.svg" -html_theme_options = {"show_prev_next": False} diff --git a/docs-devel/src/data-architecture/index.md b/docs-devel/src/data-architecture/index.md deleted file mode 100644 index 54c6fa20..00000000 --- a/docs-devel/src/data-architecture/index.md +++ /dev/null @@ -1,12 +0,0 @@ -# Data Architecture - -The data layout follows a layered structure where each layer adds transformations -taking the data the raw source data to modeled data with business rules applied, -ready for consumption by downstream users such as BI (business intelligence) tools, -ML tools etc. - -![Lakehouse data architecture](../assets/images/lakehouse-data-architecture-layers.jpg) - -## Sources - -See [data sources](./sources.md) for an overview of the source systems. diff --git a/docs-devel/src/data-architecture/sources.md b/docs-devel/src/data-architecture/sources.md deleted file mode 100644 index bf639f5d..00000000 --- a/docs-devel/src/data-architecture/sources.md +++ /dev/null @@ -1,46 +0,0 @@ -# Data Sources - -The proof-of-concept will ingest data from the following sources: - -- Crew logs: Opralog DB -- Vista/EPICS control system 2s data: InfluxDB -- Running schedule: ISIS status display [API](https://status.isis.stfc.ac.uk/api/schedule) - -## Opralog - -The crew logs are captured by Opralog and stored in a Microsoft SQL Server database. -Full backups of the database run every 24 hours and T-Logs are run hourly. - -For the purposes of testing a full backup of the database was obtained from the -Infrastructure team in ISIS and copied to a [Manila share](https://openstack.stfc.ac.uk/project/shares/faae6094-4856-4322-bbc1-678e414e32dd/) -under the path `[mount_point]/staging/opralog/full-backups` - -The [MSSQL documentation](https://learn.microsoft.com/en-us/sql/linux/tutorial-restore-backup-in-sql-server-container?view=sql-server-2017&tabs=cli) -describes how to restore this to a server instance running in a Linux container. -The following commands will restore the database to a new container with the data stored in a docker volume `sql1data`: - -```sh -> docker run -e 'ACCEPT_EULA=Y' -e 'MSSQL_SA_PASSWORD=Strong@Passw0rd' \ - --name 'sql1' -p 1433:1433 \ - -v sql1data:/var/opt/mssql -v /mnt/isis_lakehouse/staging:/staging:ro \ - -d mcr.microsoft.com/mssql/server:2019-latest - -> docker exec -it sql1 /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P 'Strong@Passw0rd' \ - -Q 'RESTORE DATABASE LOGBOOK FROM DISK = "/staging/opralog/full-backups/LOGBOOK_backup_2024_06_24_183655_2079164.bak" WITH MOVE "LOGBOOK_Data" TO "/var/opt/mssql/LOGBOOK_Data.mdf", MOVE "LOGBOOK_Log" TO "/var/opt/mssql/LOGBOOK_Log.ldf"' -``` - -## Control system 2s data - -The 2s control system data is stored in an InfluxDB instance managed by the accelerator controls team. - -For the purposes of testing a full backup of the database was obtained from the accelerator -controls team copied to a [Manila share](https://openstack.stfc.ac.uk/project/shares/faae6094-4856-4322-bbc1-678e414e32dd/) -under the path `[mount_point]/staging/influxdb/influxdbv2/data` - -## Running schedule - -The running schedule is available from the system supporting the physical -status display screens around ISIS buildings. A JSON document with the schedule -can be retrieved from . - -It currently contains dates from 2019/1 onwards. diff --git a/docs-devel/src/deployment/index.md b/docs-devel/src/deployment/index.md deleted file mode 100644 index adc7c878..00000000 --- a/docs-devel/src/deployment/index.md +++ /dev/null @@ -1,29 +0,0 @@ -# Deployment - -_The system is currently deployed on the [SCD cloud](https://cloud.stfc.ac.uk/)._ -_In theory it can be deployed on any cloud infrastructure but it will require_ -_changes before this will work._ - -There are several prerequisites required before deployment can being. Please -read [them here](./prerequisites.md). - -## Ansible - -Ansible playbooks in `/ansible/playbooks` control the deployment -of the prototype. - -To create all VM resources run the amalgamated playbook: - -```sh -ansible-playbook -i localhost -e inventory_filename=$PWD/dataplatform.ini \ - -e openstack_cloud_name= \ - -e openstack_key_name= \ playbooks/cloud/vms_create.yml -``` - -where _cloud\_name_ and _ssh\_key_ are described in the [prerequisites section](./prerequisites.md#openstack-api--vm-credentials). - -```sh -ansible-playbook -i dataplatform.ini playbooks/cloud/services_create.yml -``` - -`ANSIBLE_HOST_KEY_CHECKING=False` diff --git a/docs-devel/src/index.md b/docs-devel/src/index.md deleted file mode 100644 index fdb967de..00000000 --- a/docs-devel/src/index.md +++ /dev/null @@ -1,20 +0,0 @@ -# Data Platform Proof-of-Concept - -## Introduction - -After requirements gathering meetings and a workshop as part of the -[accelerator operational data platform project](https://stfc365.sharepoint.com/sites/ISISProject-1432) -there is appetite for trialling a system to support long-term storage, complex -queries and analysis of data from a multitude of sources across the accelerator -data space. - -This document describes the initial set of requirements along with an overview -of the design and implementation of a minimal system to prove -the [Data Lakehouse](https://stfc365-my.sharepoint.com/:w:/g/personal/martyn_gigg_stfc_ac_uk/ETTFvkI6FulFhQFWmrmfGosBRO1Syvqbiq6DhVwnqxhVbw?e=HNKUIl) -concept can be valuable to ISIS. - -Jump to: - -- [System architecture](./system-architecture/index.md) -- [Data architecture](./data-architecture/index.md) -- [Deployment](./deployment/index.md) diff --git a/docs-devel/src/system-architecture/index.md b/docs-devel/src/system-architecture/index.md deleted file mode 100644 index 89df4a3a..00000000 --- a/docs-devel/src/system-architecture/index.md +++ /dev/null @@ -1,3 +0,0 @@ -# Architecture - -![SCD cloud implementation](../assets/images/lakehouse-poc.png) diff --git a/docs-devel/src/system-architecture/adrs/0001-record-architecture-decisions.md b/docs-devel/system-architecture/adrs/0001-record-architecture-decisions.md similarity index 97% rename from docs-devel/src/system-architecture/adrs/0001-record-architecture-decisions.md rename to docs-devel/system-architecture/adrs/0001-record-architecture-decisions.md index fdd772cb..25e74736 100644 --- a/docs-devel/src/system-architecture/adrs/0001-record-architecture-decisions.md +++ b/docs-devel/system-architecture/adrs/0001-record-architecture-decisions.md @@ -1,7 +1,5 @@ # 1. Record architecture decisions -Date: 2024-07-03 - ## Status Accepted diff --git a/docs-devel/src/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md b/docs-devel/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md similarity index 96% rename from docs-devel/src/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md rename to docs-devel/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md index ed76bd93..3aeb87aa 100644 --- a/docs-devel/src/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md +++ b/docs-devel/system-architecture/adrs/0002-use-ansible-for-vm-provisioning-and-configuration.md @@ -1,7 +1,5 @@ # 2. Use Ansible for VM provisioning and configuration -Date: 2024-07-03 - ## Status Accepted diff --git a/docs-devel/src/system-architecture/adrs/0003-use-scd-cloud-vms.md b/docs-devel/system-architecture/adrs/0003-use-scd-cloud-vms.md similarity index 97% rename from docs-devel/src/system-architecture/adrs/0003-use-scd-cloud-vms.md rename to docs-devel/system-architecture/adrs/0003-use-scd-cloud-vms.md index e07bc3c2..e75891d0 100644 --- a/docs-devel/src/system-architecture/adrs/0003-use-scd-cloud-vms.md +++ b/docs-devel/system-architecture/adrs/0003-use-scd-cloud-vms.md @@ -1,7 +1,5 @@ # 3. Use SCD Cloud VMs -Date: 2024-07-04 - ## Status Accepted diff --git a/docs-devel/src/system-architecture/adrs/0004-use-minio-as-object-store.md b/docs-devel/system-architecture/adrs/0004-use-minio-as-object-store.md similarity index 97% rename from docs-devel/src/system-architecture/adrs/0004-use-minio-as-object-store.md rename to docs-devel/system-architecture/adrs/0004-use-minio-as-object-store.md index db338e3a..f04c6435 100644 --- a/docs-devel/src/system-architecture/adrs/0004-use-minio-as-object-store.md +++ b/docs-devel/system-architecture/adrs/0004-use-minio-as-object-store.md @@ -1,7 +1,5 @@ # 4. Use MinIO as object store -Date: 2024-07-04 - ## Status Accepted diff --git a/docs-devel/src/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md b/docs-devel/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md similarity index 97% rename from docs-devel/src/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md rename to docs-devel/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md index 3e9fe8ca..11f33629 100644 --- a/docs-devel/src/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md +++ b/docs-devel/system-architecture/adrs/0005-use-apache-iceberg-as-lakehouse-table-format.md @@ -1,7 +1,5 @@ # 5. Use Apache Iceberg as Lakehouse Table Format -Date: 2024-07-04 - ## Status Accepted diff --git a/docs-devel/system-architecture/adrs/0006-use-lakekeeper-as-iceberg-catalog.md b/docs-devel/system-architecture/adrs/0006-use-lakekeeper-as-iceberg-catalog.md new file mode 100644 index 00000000..930c43fa --- /dev/null +++ b/docs-devel/system-architecture/adrs/0006-use-lakekeeper-as-iceberg-catalog.md @@ -0,0 +1,53 @@ +# 6. Use Lakekeeper as the Iceberg Catalog + +## Status + +Accepted + +## Context + +[Apache Iceberg](./0005-use-apache-iceberg-as-lakehouse-table-format.md) requires a +[catalog](https://iceberg.apache.org/terms/#catalog) to track table metadata. +Several implementations exist within the Iceberg libraries including: + +- REST: a server-side catalog that’s exposed through a REST API +- Hive Metastore: tracks namespaces and tables using a Hive metastore +- JDBC: tracks namespaces and tables in a simple JDBC database +- Nessie: a transactional catalog that tracks namespaces and tables in a database with git-like version control + +The REST catalog requires a server to implement the specification but offers more flexibility without +having to modify the Iceberg libraries, e.g a query engine using REST doesn't need to be modified to +switch REST catalog implementations. + +## Decision + +The REST specification offers greater flexibility in how clients interact with the catalog and several +implementations exist that support the Iceberg REST spec: + +- [Apache Gravitino](https://gravitino.apache.org/): Java based, offering federation across multiple data catalogs, + including support for authentication and access management. +- [Apache Polaris](https://polaris.apache.org/): Java based incubating Apache project to provide an alternative + implementation of the Iceberg REST catalog implementation. Dremio [announced](https://projectnessie.org/blog/2024/08/02/open-source-polaris-announcement/) + they would look to add Nessie-like features to Polaris. +- [Lakekeeper](https://docs.lakekeeper.io/): Newer, Rust-based catalog providing multiple warehouse support, access control, a basic UI for catalog management. + Good documentation and examples, simple to deploy. +- [Project Nessie](https://projectnessie.org/): Java-based catalog supporting aiming to bring version-control features like branching, merging to the catalog as a whole. + Interesting idea but through a basic trial the [catalog maintenance](https://iceberg.apache.org/docs/nightly/nessie/#further-use-cases) was found to + be complicated using Nessie. What will the future of Nessie be? + +We will choose Lakekeeper as a REST-based catalog server due to: + +- Ease of deployment: a single binary. +- Excellent documentation and examples of different configurations. +- Manage data access centrally in the catalog through emerging open standards such as OpenFGA and Open Policy Agent. +- Built-in support for external secret stores such as Hashicorp-Vault. +- Event publishing to NATS, Kafka. +- Lower resource footprint by using Rust. + +## Consequences + +Lakekeeper will be the backbone of the data lake and is a relatively new project so there is some +risk that it does not survive. +The design of Apache Iceberg mitigates the risk of migrating to a new catalog if that becomes +necessary as the catalog only stores metadata and would not require migrating the data on the object +store. diff --git a/docs-devel/src/system-architecture/adrs/index.md b/docs-devel/system-architecture/adrs/index.md similarity index 100% rename from docs-devel/src/system-architecture/adrs/index.md rename to docs-devel/system-architecture/adrs/index.md diff --git a/docs-devel/system-architecture/index.md b/docs-devel/system-architecture/index.md new file mode 100644 index 00000000..fdd919cc --- /dev/null +++ b/docs-devel/system-architecture/index.md @@ -0,0 +1,65 @@ +# System Architecture + +The following sections describe the high-level services and how they connect with each other. +See [deployment docs](../deployment/index.md) on how this is deployed on the STFC cloud. + +## Overview + +![SCD cloud implementation](../images/analytics-data-platform-arch.drawio.png) + +Externally managed Postgres instances are provided by the SCD Database Services team. S3-compatible +object storage is provided by the SCD Ceph team. + +## Services + +Here we provide an overview of the key services. For details on their deployment configuration +see [deployment](../deployment/index.md). + +### Keycloak + +[Keycloak](https://keycloak.org) provides identity for the platform. It acts as an OIDC/OAuth2 provider +for single-sign-on, providing a consistent view of a user across multiple services, for example +Lakekeeper & Superset. + +### Lakekeeper + +[Lakekeeper](https://docs.lakekeeper.io) implements the catalog and table management layer used by +Iceberg tables. It provides metadata APIs, coordinates table lifecycles, stores catalog metadata in a persistent metadata database and uses the object store for table data. +Other services (Trino, batch jobs) use the Lakekeeper catalog to discover and +manipulate Iceberg tables. + +### Trino + +[Trino](https://trino.io) is the distributed SQL query engine used querying Iceberg tables. +A Trino coordinator and workers execute queries against the Lakehouse using the Iceberg connector +and the Lakekeeper catalog. Trino is the primary compute engine for BI workloads and SQL-based +exploration within Superset. + +### Superset + +[Superset](https://superset.apache.org) is the business intelligence and visualization layer. +It connects to Trino, provides dashboards, charts and exploration tools, and relies +on central authentication (Keycloak) for SSO. + +## Tools + +The platform uses a small set of data engineering tools to build, orchestrate +and test ELT pipelines. Two of the primary packages are described below. + +### dlt + +[dlt](https://dlthub.com/docs/intro) provides an extraction and loading framework for the +ELT pipelines. It provides connectors for common sources, schema management, and +incremental loading semantics. + +### dbt + +[dbt](https://docs.getdbt.com) is used for transforming and modelling tabular data in +the `silver` and `gold` layers. `dbt` enables SQL-based transformations, version-controlled models, +testing, documentation, and dependency graphs for data products. + +### elt-common + +[elt-common](../../elt-common) is our own Python package that provides additional functionality +built on top of [dlt](#dlt). It is not published to PyPI and must be installed using the git source +url. diff --git a/elt-common/README.md b/elt-common/README.md index 656378d0..09c74cc6 100644 --- a/elt-common/README.md +++ b/elt-common/README.md @@ -36,7 +36,7 @@ rest catalog to test complete functionality. The local, docker-compose-based configuration provided by [infra/local/docker-compose.yml](../infra/local/docker-compose.yml) is the easiest way to spin up a set of services compatible with running the tests. -_Note the requirement to edit `/etc/hosts` described in [here](../infra/local/README.md)._ +_Note the requirement to edit `/etc/hosts` described in [infra/local/README](../infra/local/README.md)._ Once the compose services are running, execute the e2e tests using `pytest`: diff --git a/infra/ansible-docker/.gitignore b/infra/ansible-docker/.gitignore index 63b08936..9cf16516 100644 --- a/infra/ansible-docker/.gitignore +++ b/infra/ansible-docker/.gitignore @@ -3,4 +3,4 @@ condaenv/ .venv/ galaxy_roles *.ini -/*.yml +inventory.yml diff --git a/infra/ansible-docker/group_vars/all/openstack.yml b/infra/ansible-docker/group_vars/all/openstack.yml index c5784f45..5d698b87 100644 --- a/infra/ansible-docker/group_vars/all/openstack.yml +++ b/infra/ansible-docker/group_vars/all/openstack.yml @@ -12,8 +12,6 @@ openstack_virtual_network: gateway_ip: 192.168.42.1 openstack_reverse_proxy_fip: 130.246.214.124 -openstack_management_vm_fip: 130.246.81.245 -openstack_jump_host: "{{ openstack_reverse_proxy_fip }}" openstack_apt_keys: - url: https://dl.igtf.net/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-4 diff --git a/infra/ansible-docker/inventory-sample.yml b/infra/ansible-docker/inventory-sample.yml new file mode 100644 index 00000000..5ae98cf5 --- /dev/null +++ b/infra/ansible-docker/inventory-sample.yml @@ -0,0 +1,39 @@ +all: + vars: + ansible_user: + ansible_python_interpreter: /usr/bin/python3 + ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q ubuntu@"' + +traefik: + hosts: + traefik-1: + ansible_host: + +keycloak: + hosts: + keycloak-1: + ansible_host: + +lakekeeper: + hosts: + lakekeeper-1: + ansible_host: + +trino: + hosts: + trino-1: + ansible_host: + +elt: + hosts: + elt-1: + ansible_host: + +superset_accelerator: + hosts: + superset-accelerator-1: + ansible_host: + +superset_farm: + children: + superset_accelerator: diff --git a/infra/ansible-docker/playbooks/cloud/private_network_create.yml b/infra/ansible-docker/playbooks/cloud/private_network_create.yml index 6f422462..776fca95 100644 --- a/infra/ansible-docker/playbooks/cloud/private_network_create.yml +++ b/infra/ansible-docker/playbooks/cloud/private_network_create.yml @@ -1,4 +1,11 @@ --- +- name: Read global configuration + hosts: localhost + gather_facts: false + tasks: + - ansible.builtin.include_vars: + dir: "../../group_vars/all" + - name: Provision private network hosts: localhost gather_facts: False diff --git a/infra/ansible-docker/readme.md b/infra/ansible-docker/readme.md index e42c348f..542d5850 100644 --- a/infra/ansible-docker/readme.md +++ b/infra/ansible-docker/readme.md @@ -1,81 +1,5 @@ -# Ansible Scripts +# Ansible Docker -The Ansible configuration here is used to deploy the Lakehouse platform. It has -currently only been tested using the STFC cloud but in theory it will work with -other cloud providers or Linux VMs. +The playbooks defined here define the deployment of the system. -## Setup Ansible - -It is recommended to use a virtual environment to set up Ansible but you can use -any Python installation that has `pip` installed. The version of Python required -is recorded in [./python-version](./python-version). - -[uv](https://docs.astral.sh/uv/pip/environments/) is a popular and fast tool -for managing Python environments and is the recommended tool to use. - -Create a virtual environment - -```sh -uv venv -``` - -Install the requirements using `requirements-python.txt` - -```sh -uv pip install -r ./requirements-python.txt -``` - -Install the additional Ansible Galaxy roles with: - -```sh -uv run ansible-galaxy role install --roles-path galaxy_roles -r requirements-ansible-galaxy.yml -``` - -The `galaxy_roles` path has been added to `.gitignore`. - -## Openstack preparation - -TODO: Tidy up - -- Two floating IPs are required (defined in [./group_vars/all/openstack.yml](./group_vars/all/openstack.yml)): - - One for the management/SSH jump node. Check `openstack_management_vm_fip` value. - - One for the Traefik load balancer node. Check the `openstack_reverse_proxy_fip` value. - -## Creating the stack - -TODO: Tidy up - -In the following set of instructions the following variables are required: - -- `[CLOUDS_YAML_NAME]`: the name of the target Openstack cloud within the `clouds` block of the `~/.config/clouds.yaml`. -- `[SSH_KEY_NAME]`: the name of the SSH key already stored in [Openstack](https://openstack.stfc.ac.uk/project/key_pairs) - that will be added to newly created servers. - -### Create management VM - -_If a previous VM was deleted then ensure you remove the host keys for the IP address from `~/.ssh_known_hosts` to avoid connection issues._: - -```sh -sed -i "" -E -e '/^INSERT MANAGEMENT FLOATING IP HERE/d' ~/.ssh/known_hosts` -``` - -Create the management VM first: - -```sh -ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i localhost -e openstack_cloud_name=[CLOUDS_YAML_NAME] -e openstack_key_name=[SSH_KEY_NAME] playbooks/management/create.yml` -``` - -This is assigned a floating IP and acts as an SSH proxy into the private network that the service nodes live on. -Without this SSH connections to the service nodes would not be possible. - -Create the service node VMs: - -```sh -ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -i localhost -e openstack_cloud_name=[CLOUDS_YAML_NAME] -e openstack_key_name=[SSH_KEY_NAME] -e inventory_filename=$PWD/inventory-virtnet.ini playbooks/cloud/vms_create.yml` -``` - -### Deploy services - -- Remove any old VM host keys from SSH known_hosts: `sed -i "" -E -e '/^192/d' ~/.ssh/known_hosts` (this will remove all 192 IPs ) -- Run `ANSIBLE_STDOUT_CALLBACK=ansible.posix.debug ansible-playbook -i inventory-virtnet.ini playbooks/cloud/services_deploy.yml` -- You'll need to accept each key as it runs through the setup or run with `ANSIBLE_HOST_KEY_CHECKING=False` prefixed to the command. +See [deployment docs](../../docs-devel/src/deployment/index.md) for further information. diff --git a/infra/local/README.md b/infra/local/README.md index 65fcb6d3..fbe32d0a 100644 --- a/infra/local/README.md +++ b/infra/local/README.md @@ -48,11 +48,11 @@ docker compose up -d Services: -- Keycloak identity provider: http://localhost:50080/auth - - `master` realm credentials: _admin/admin_ - - `analytics-data-platform` realm credentials: _adpsuperuser/adppassword_ -- Lakekeeper iceberg catalog UI: http://localhost:50080/iceberg/ui. - - Use the same credentials as the `analytics-data-platform` realm -- Superset BI tool: http://localhost:50080/workspace/playground. - - Use the same credentials as the `analytics-data-platform` realm -- Trino endpoint: https://localhost:58443 (note the https as this is required by Trino. You may need to use an "--insecure" flag) +- Keycloak identity provider: + - `master` realm credentials: *admin/admin* + - `analytics-data-platform` realm credentials: *adpsuperuser/adppassword* +- Lakekeeper iceberg catalog UI: . + - Use the same credentials as the `analytics-data-platform` realm +- Superset BI tool: . + - Use the same credentials as the `analytics-data-platform` realm +- Trino endpoint: (note the https as this is required by Trino. You may need to use an "--insecure" flag) diff --git a/readme.md b/readme.md index 46389c36..9f4270be 100644 --- a/readme.md +++ b/readme.md @@ -7,7 +7,8 @@ aspects of the project. ## Components -See [AGENTS.md](./AGENTS.md) for an overview of the repository contents. +See [`docs-devel/src/index.md#repository-overview`](./docs-devel/src/index.md#repository-overview) +for an overview of the repository contents. ## License