|
| 1 | +# LLM Context Guide for Analytics Data Platform |
| 2 | + |
| 3 | +This repository contains ELT code for an Iceberg-based data warehouse, together with |
| 4 | +infrastructure code used to run the warehouse both locally (docker-compose) and in an |
| 5 | +OpenStack cloud (Ansible). |
| 6 | + |
| 7 | +Use this document as a concise, developer-friendly reference when working with the |
| 8 | +repository or providing context to an assistant/agent about the project structure and |
| 9 | +common developer tasks. |
| 10 | + |
| 11 | +## Purpose |
| 12 | + |
| 13 | +- Give human developers and LLM-based agents the essential repository layout and |
| 14 | + pointers for local development, testing and deployment. |
| 15 | +- Point to the configuration locations used for certificates, local environments, |
| 16 | + and deployable Ansible playbooks/roles. |
| 17 | + |
| 18 | +## Repository layout (high level) |
| 19 | + |
| 20 | +```text |
| 21 | +. |
| 22 | +├── certs/ # Certificate request configurations used for HTTPS/SSL |
| 23 | +├── elt-common/ # Reusable Python package with common ETL/ELT helpers used by the warehouses |
| 24 | +├── docs/ # User and developer documentation using MkDocs. See `docs/src` for content used in the published docs site. |
| 25 | +├── infra/ |
| 26 | +│ ├── ansible-docker/ # Ansible playbooks/roles to deploy the system to the STFC (OpenStack) cloud. |
| 27 | +│ └── local/ # docker-compose configuration for a local development environment and end-to-end CI tests. |
| 28 | +└── warehouses/ # One subdirectory per (Lakekeeper) warehouse. Each contains ELT code to extract, transform and load data from external sources into Iceberg tables. |
| 29 | +``` |
| 30 | + |
| 31 | +## Pre-commit hooks |
| 32 | + |
| 33 | +- Static code checks are enforced using `prek`. Install with your environment's tooling and run checks |
| 34 | + on all files with `prek run --all-files`. Install the pre-commit hook with `prek install`. |
| 35 | + |
| 36 | +## Running tests |
| 37 | + |
| 38 | +- There are both unit and e2e tests for the `elt-common` package under `elt-common/tests/`. They are |
| 39 | + written using `pytest`. |
| 40 | +- Install the package in editable mode + dev dependencies in a Python environment, e.g. with `uv` |
| 41 | + |
| 42 | +```bash |
| 43 | +cd elt-common/ |
| 44 | +uv venv |
| 45 | +uv install . --editable --group dev |
| 46 | +``` |
| 47 | + |
| 48 | +- Run unit tests with: |
| 49 | + |
| 50 | +```bash |
| 51 | +cd elt-common/ |
| 52 | +uv run pytest unit_tests |
| 53 | +``` |
| 54 | + |
| 55 | +- The e2e tests require a running data platform stack. Use the docker-based setup in `infra/local` |
| 56 | + to start the stack, run the tests and bring down the stack: |
| 57 | + |
| 58 | +```bash |
| 59 | +pushd infra/local; docker compose up -d; popd # start the services |
| 60 | +cd elt-common/ |
| 61 | +uv run pytest e2e_tests |
| 62 | +pushd infra/local; docker compose down -v; popd # stop the services |
| 63 | +``` |
| 64 | + |
| 65 | +## Cloud deployment |
| 66 | + |
| 67 | +- Use the Ansible playbooks in `infra/ansible-docker/` together with the |
| 68 | + `inventory*.yml` files. See the `infra/ansible-docker/readme.md` for role and |
| 69 | + variable guidance. |
| 70 | + |
| 71 | +## Pull Request Guidelines |
| 72 | + |
| 73 | +- When creating pull requests: |
| 74 | + |
| 75 | + 1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format |
| 76 | + 2. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) |
| 77 | + - Format: `type(scope): description` |
| 78 | + - Example: `fix(warehouse/accelerator): fix join in model` |
| 79 | + - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore` |
| 80 | + |
| 81 | +**Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time. |
| 82 | + |
| 83 | +## Troubleshooting & tips |
| 84 | + |
| 85 | +- Docker resource issues: the local compose stack can be resource heavy. Ensure |
| 86 | + Docker Desktop has enough CPU/memory. |
| 87 | +- Ansible role errors: ensure you have required galaxy roles (see |
| 88 | + `infra/ansible-docker/ansible-galaxy-requirements.yaml`) and the correct Python |
| 89 | + and Ansible versions installed. |
| 90 | + |
| 91 | +## Where to go next |
| 92 | + |
| 93 | +- Read `docs/` and `docs-devel/` for high-level architecture and deployment |
| 94 | + instructions. |
| 95 | +- Inspect `warehouses/` for per-warehouse ELT implementations and examples. |
| 96 | + |
| 97 | +--- |
| 98 | + |
| 99 | +This file was created to provide a concise, shareable context document for humans |
| 100 | +and LLM-based agents working with the analytics-data-platform monorepo. |
0 commit comments