Skip to content

Commit 782e6be

Browse files
avillaravillar
committed
Fix stale CLAUDE.md references and rewrite README
CLAUDE.md still described a bootstrap.py entry point and a standalone transform-plugins.yml loading mechanism, neither of which exist anymore (the entrypoint is ogc.bblocks.entrypoint, and plugins load from the plugins.transforms/plugins.validators keys in bblocks-config.yaml). Also brought the CI/CD workflow list up to date. README.md was a 13-line blurb with no usage instructions. Added how to consume this as a GitHub Action, how to run it locally/via Docker, and a repository layout overview.
1 parent 6232ab6 commit 782e6be

2 files changed

Lines changed: 114 additions & 29 deletions

File tree

‎CLAUDE.md‎

Lines changed: 25 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,22 +10,25 @@ A GitHub Action and standalone Python tool that postprocesses OGC Building Block
1010

1111
```bash
1212
# Directly via Python module
13-
python -m ogc.bblocks.bootstrap [options]
13+
python -m ogc.bblocks.entrypoint [options]
1414

1515
# Key options:
16-
# --register-file PATH Path to register.json output
17-
# --items-dir DIR Directory to scan for building blocks
18-
# --base-url URL Base URL for generated output
19-
# --clean Delete old build directories first
20-
# --steps STEPS Comma-separated list: annotate,jsonld,tests,transforms,doc,register
21-
# --filter FILTER Process only matching building block or file
22-
# --fail-on-errors Exit non-zero if validation errors found
16+
# --register-file PATH Path to register.json output
17+
# --items-dir DIR Directory to scan for building blocks
18+
# --base-url URL Base URL for generated output
19+
# --clean true|false Delete old build directories first
20+
# --steps STEPS Comma-separated list: annotate,jsonld,tests,transforms,doc,register
21+
# --filter FILTER Process only matching building block or file
22+
# --fail-on-error true|false Exit non-zero if validation errors found
23+
# --skip-permissions true|false Skip interactive prompts for transform/validator plugins (set true in CI)
2324

2425
# Via Docker
2526
docker build -t bblocks-postprocess .
2627
docker run -v /path/to/repo:/workspace bblocks-postprocess [options]
2728
```
2829

30+
All flags are string-valued (`true`/`false`, not boolean switches) — see [Adding new CLI flags](#adding-new-cli-flags) for why.
31+
2932
## Local Testing with URL Mappings
3033

3134
Create `bblocks-config-local.yml` to map remote URLs to local files:
@@ -41,9 +44,8 @@ The HTTP interceptor (`http_interceptor.py`) monkey-patches urllib/requests to r
4144
### Entry & Flow
4245

4346
```
44-
bootstrap.py Loads plugins from transform-plugins.yml, then delegates
45-
→ entrypoint.py Parses CLI args, loads bblocks-config.yaml, calls postprocess()
46-
→ postprocess.py Core orchestration: discover → annotate → validate → generate docs → register
47+
entrypoint.py Parses CLI args, loads bblocks-config.yaml, calls postprocess()
48+
→ postprocess.py Core orchestration: discover → annotate → validate → generate docs → register
4749
```
4850
4951
### Core Components
@@ -54,7 +56,7 @@ bootstrap.py Loads plugins from transform-plugins.yml, then delegates
5456
5557
- **`validate.py`** — Test validation and HTML/JSON/text report generation. Validators (JSON Schema, RDF/SHACL, semantic uplift) live in `validation/`.
5658
57-
- **`transform.py` + `transformers/`** — Applies pluggable transformers to examples. Built-in transformers: RDF (SHACL-AF, SPARQL), jq, XSLT, JSON-LD Frame, semantic uplift. External transformers load via `transform-plugins.yml`.
59+
- **`transform.py` + `transformers/`** — Applies pluggable transformers to examples. Built-in transformers: RDF (SHACL-AF, SPARQL), jq, XSLT, JSON-LD Frame, semantic uplift. External transform/validator plugins load from the `plugins.transforms` / `plugins.validators` keys in `bblocks-config.yaml` (see below).
5860
5961
- **`generate_docs.py`** — Mako-based documentation generation from templates in `templates/*/`.
6062
@@ -76,17 +78,18 @@ For each `bblock.json` found:
7678
7779
After all blocks: generate `register.json`, perform semantic uplift to JSON-LD + Turtle, optionally push to SPARQL triplestore.
7880
79-
### Plugin System (WIP on `transform-plugins` branch)
81+
### Plugin System
8082
81-
`transform-plugins.yml` allows loading external transformer modules:
83+
External transform/validator plugins are declared under `plugins.transforms` / `plugins.validators` in `bblocks-config.yaml`:
8284
```yaml
8385
plugins:
84-
- modules: [my.custom.Transformer]
85-
install:
86-
pip: my-custom-package
86+
transforms:
87+
- modules: [my.custom.Transformer]
88+
pip: [my-custom-package]
8789
```
90+
`transform.py`/`validate.py` install each plugin's pip/npm dependencies into a per-plugin sandbox (`sandbox.py`) and register it (`transformers/plugin.py`, `validation/plugin.py`). Unless `--skip-permissions` is set, the user is prompted interactively before installing/running plugin code (`permissions.py`).
8891

89-
`bootstrap.py` loads these before delegating to `entrypoint.py`.
92+
The legacy standalone `transform-plugins.yml` file is still read as a fallback for `plugins.transforms` (with a deprecation warning) if `bblocks-config.yaml` doesn't declare that key.
9093

9194
## Key Configuration Files
9295

@@ -96,7 +99,7 @@ plugins:
9699
| `bblock.json` | Per-block metadata: identifier, name, schema path, examples, SHACL, extension points |
97100
| `examples.yaml` | Example snippets with test cases |
98101
| `transforms.yaml` | Transform definitions (type, inputs, outputs, code) |
99-
| `transform-plugins.yml` | External transformer plugin loading |
102+
| `transform-plugins.yml` | Legacy external transformer plugin loading (deprecated — use `plugins.transforms` in `bblocks-config.yaml`) |
100103

101104
## Dependencies
102105

@@ -106,9 +109,11 @@ plugins:
106109

107110
## CI/CD
108111

109-
- `build-docker.yml` — builds and pushes Docker image to `ghcr.io/opengeospatial/bblocks-postprocess` on push to master
112+
- `build-docker.yml` — builds and pushes Docker image to `ghcr.io/opengeospatial/bblocks-postprocess`, triggered on push to `develop` or on `v1.*.*` tags (tag pushes also float the `master` tag)
110113
- `test-postprocess.yml` — regression tests against live bblocks repos (triggered after Docker build)
111114
- `validate-and-process.yml` — reusable workflow called by downstream repos to postprocess, commit, and deploy their building blocks
115+
- `test.yml` — exercises this action's own composite/Docker actions (`full/action.yml`, `postprocess/action.yml`)
116+
- `upload-to-triplestore.yml` — pushes semantic uplift output to a SPARQL triplestore
112117

113118
### Adding new CLI flags
114119

‎README.md‎

Lines changed: 89 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,93 @@
11
# bblocks-postprocess
22

3-
This repository contains reusable code to integrate components of a reusable data model (and any reused models that are combined to construct it):
4-
* JSON schema
5-
* JSON-LD
6-
* SHACL
7-
* test cases
8-
* examples
9-
* validation outputs
3+
A GitHub Action and standalone Python tool that postprocesses [OGC Building Blocks](https://github.com/opengeospatial/bblocks-template) — reusable data models combining:
4+
5+
* JSON Schema
6+
* JSON-LD contexts
7+
* SHACL shapes
8+
* test cases and examples
9+
* validation reports
1010
* profile declarations
11-
* associated OAS APIs
11+
* associated OpenAPI (OAS 3.0) definitions
12+
13+
For a given source repository it discovers building blocks, annotates and validates them, generates documentation, performs semantic uplift to RDF, and produces a `register.json` describing the whole collection.
14+
15+
## Using this as a GitHub Action
16+
17+
Two composite/Docker actions are published from this repository:
18+
19+
* **`opengeospatial/bblocks-postprocess/postprocess@v1`** — runs the postprocessor itself (Docker action, wraps `python -m ogc.bblocks.entrypoint`). Use this if you just want the annotate/validate/document/register steps.
20+
* **`opengeospatial/bblocks-postprocess/full@v1`** — thin wrapper around `postprocess` with the same inputs; intended to be composed into a larger workflow.
21+
22+
Most downstream repos don't call either directly — they instead invoke the reusable workflow:
23+
24+
* **`.github/workflows/validate-and-process.yml`** — checks out the repo, restores/saves a plugin sandbox cache, runs `full`, commits the generated build output back to the repo, deploys the viewer/docs to GitHub Pages, and (optionally) pushes the semantic uplift to a SPARQL triplestore via `upload-to-triplestore.yml`.
25+
26+
Example usage from a building blocks repository:
27+
28+
```yaml
29+
jobs:
30+
postprocess:
31+
uses: opengeospatial/bblocks-postprocess/.github/workflows/validate-and-process.yml@v1
32+
secrets: inherit
33+
```
34+
35+
See `postprocess/action.yml` / `full/action.yml` for the full list of inputs (register file location, items directory, base URL, viewer deployment, SPARQL credentials, etc.).
36+
37+
## Running locally
38+
39+
```bash
40+
pip install -r requirements.txt
41+
npm install
42+
43+
python -m ogc.bblocks.entrypoint \
44+
--items-dir _sources \
45+
--register-file build-local/register.json \
46+
--generated-docs-path build-local/generateddocs \
47+
--annotated-path build-local/annotated \
48+
--base-url https://example.org/bblocks/
49+
```
50+
51+
Run `python -m ogc.bblocks.entrypoint --help` for the full list of flags (`--clean`, `--filter`, `--steps`, `--fail-on-error`, `--skip-permissions`, `--log-level`, ...).
52+
53+
### Via Docker
54+
55+
```bash
56+
docker build -t bblocks-postprocess .
57+
docker run -v /path/to/repo:/workspace -w /workspace bblocks-postprocess [flags]
58+
```
59+
60+
### Local testing with URL mappings
61+
62+
Create a `bblocks-config-local.yml` next to your `bblocks-config.yaml` to redirect remote URLs to local files while developing:
63+
64+
```yaml
65+
url-mappings:
66+
https://example.com/path: /local/path
67+
```
68+
69+
`http_interceptor.py` monkey-patches `urllib`/`requests` to honor these mappings.
70+
71+
## Repository layout
72+
73+
| Path | Purpose |
74+
|------|---------|
75+
| `ogc/bblocks/` | Postprocessor source (entrypoint, orchestration, validation, transforms, doc generation) |
76+
| `postprocess/action.yml` | Docker-based GitHub Action running the postprocessor |
77+
| `full/action.yml` | Composite action wrapping `postprocess` |
78+
| `.github/workflows/validate-and-process.yml` | Reusable workflow: checkout → postprocess → commit → deploy → (optional) triplestore push |
79+
| `.github/workflows/build-docker.yml` | Builds/pushes the `ghcr.io/opengeospatial/bblocks-postprocess` image |
80+
| `.github/workflows/test-postprocess.yml` | Regression tests against live bblocks repos |
81+
| `docs/` | Design notes (transformer plugins, validation plugins) |
82+
| `scripts/` | Maintenance scripts (template hash manifest, release tagging) |
83+
84+
See `CLAUDE.md` for a deeper architectural walkthrough of the postprocessing pipeline.
85+
86+
## Dependencies
87+
88+
* **Python** (3.10): `ogc-na-tools` (semantic annotation + RDF), `pyshacl`, a patched `rdflib` fork (`avillar/rdflib@6.x`), `jsonschema`, `mako`, `requests`
89+
* **Node.js**: the `jsonld` package, used for JSON-LD processing
90+
91+
## License
1292

13-
using the [OGC BuildingBlock template](https://github.com/opengeospatial/bblock-template)
93+
See [LICENSE](LICENSE).

0 commit comments

Comments
 (0)