feat: Add comprehensive AI Agent Workflow documentation

- Introduced `ai_workflow.md` detailing the integration of AI coding agents, project structure, and VS Code integration.
- Documented the entry point for agents in `AGENTS.md`, emphasizing its role in providing project context.
- Explained the spec-driven development approach with a two-level layout for specs in `spec/` and `middleware/*/spec/`.
- Outlined the use of agent skills in `.agents/skills/`, including discovery, activation, and execution processes.
- Provided practical workflow examples for agents, including adding new config fields and fixing serialization bugs.
- Added new specs and design documents for API upload, ARC building, database access, and SQL-to-ARC conversion.
- Established principles for the FAIRagro SQL-to-ARC project, emphasizing correctness, memory safety, and security.
- Created demo environment specifications to facilitate local testing without external dependencies.

Author: Zalfsten

.specifica/skills/arctrl.md .agents/skills/arctrl/SKILL.md.specifica/skills/arctrl.md renamed to .agents/skills/arctrl/SKILL.md

@@ -1,6 +1,16 @@
-# ARCtrl — Usage Skill
+---
+name: arctrl
+description: >
+  Reference for using the arctrl Python library (v3.x) to build ARC (Annotated
+  Research Context) objects and serialize them to RO-Crate JSON-LD. Use when
+  working with ArcInvestigation, ArcStudy, ArcAssay, ArcTable, CompositeHeader,
+  CompositeCell, OntologyAnnotation, OntologySourceReference, Person, or
+  Publication objects, or when calling ToROCrateJsonString / WriteAsync.
+compatibility: Python 3.12+, arctrl (Fable-transpiled F# library)
+---
+
+# ARCtrl — Usage Reference
 
-Reference for using the `arctrl` Python library (v3.x) in this project.
 ARCtrl is a Fable-transpiled F# library — the Python surface is idiomatic
 but some internals are Fable runtime types.
 
@@ -37,19 +47,46 @@ from arctrl.py.fable_modules.fable_library.async_ import start_as_task  # type:
 # Empty / unknown
 oa = OntologyAnnotation()
 
-# With values: name=term, tan=URI (TermAccessionNumber), tsr=TermSourceREF
-oa = OntologyAnnotation(name="soil texture", tan="http://purl.obolibrary.org/obo/ENVO_00002001", tsr="")
-# tsr is usually left empty when only a URI is available
+# With values — all parameters optional:
+oa = OntologyAnnotation(
+    name="soil texture",  # human-readable term
+    tan="http://purl.obolibrary.org/obo/ENVO_00002001",  # TermAccessionNumber (URI)
+    tsr="ENVO",  # TermSourceREF: short name of the ontology source
+)
+# tsr is a back-reference to an OntologySourceReference registered on the
+# investigation (by its .Name). If no OntologySourceReference is registered,
+# tsr can be left empty or omitted.
+```
+
+### OntologySourceReference
+
+Registered on `ArcInvestigation.OntologySourceReferences`. Describes an
+ontology source and holds its version.
+
+```python
+from arctrl import OntologySourceReference
+
+osr = OntologySourceReference(
+    name="ENVO",  # short name — must match OntologyAnnotation.tsr
+    description="Environment Ontology",
+    file="http://purl.obolibrary.org/obo/envo.owl",
+    version="2024-01-01",  # ontology version / access date
+)
+investigation.OntologySourceReferences.append(osr)
 ```
 
+**Relationship:** `OntologyAnnotation.tsr` is a string key that references
+`OntologySourceReference.name`. ARCtrl does not enforce referential integrity
+at runtime, but the RO-Crate serialization will include both objects.
+
 ### ArcInvestigation
 
 ```python
 inv = ArcInvestigation.create(
-    identifier="inv001",            # required, must be non-empty
+    identifier="inv001",  # required, must be non-empty
     title="My Investigation",
     description="...",
-    submission_date="2024-01-15",   # ISO string or None
+    submission_date="2024-01-15",  # ISO string or None
     public_release_date="2025-01-01",
 )
 ```
@@ -73,7 +110,7 @@ assay = ArcAssay.create(
     identifier="assay001",
     measurement_type=OntologyAnnotation("soil metagenome", "http://...", ""),
     technology_type=OntologyAnnotation("nucleotide sequencing", "http://...", ""),
-    technology_platform=OntologyAnnotation("Illumina", None, None),  # platform is text; OA is allowed
+    technology_platform=OntologyAnnotation("Illumina", None, None),
     # technology_platform=None is fine if unknown
 )
 ```
@@ -121,16 +158,16 @@ arc.AddRegisteredStudy(study)
 arc.AddAssay(assay)
 
 # 4. Link assay → study
-study.RegisterAssay(assay.Identifier)   # pass the string identifier
+study.RegisterAssay(assay.Identifier)  # pass the string identifier
 
 # 5. Attach contacts
-arc.Contacts.append(person)             # investigation-level
-study.Contacts.append(person)           # study-level
-assay.Performers.append(person)         # assay-level
+arc.Contacts.append(person)  # investigation-level
+study.Contacts.append(person)  # study-level
+assay.Performers.append(person)  # assay-level
 
 # 6. Attach publications
-arc.Publications.append(pub)            # investigation-level
-study.Publications.append(pub)          # study-level
+arc.Publications.append(pub)  # investigation-level
+study.Publications.append(pub)  # study-level
 
 # 7. Serialize to RO-Crate JSON-LD string
 json_str: str = arc.ToROCrateJsonString()
@@ -140,51 +177,45 @@ json_str: str = arc.ToROCrateJsonString()
 
 ## ArcTable (Annotation Tables)
 
-Tables attach to a study or assay.
-
 ```python
 # Create table
 table = ArcTable.init("my-table-name")
 
-# Build a header
-header_input  = CompositeHeader.input(IOType.of_string("source_name"))
+# Build headers
+header_input = CompositeHeader.input(IOType.of_string("source_name"))
 header_output = CompositeHeader.output(IOType.of_string("sample_name"))
-header_char   = CompositeHeader.characteristic(OntologyAnnotation("pH", "", ""))
+header_char = CompositeHeader.characteristic(OntologyAnnotation("pH", "", ""))
 header_factor = CompositeHeader.factor(OntologyAnnotation("temperature", "", ""))
-header_param  = CompositeHeader.parameter(OntologyAnnotation("extraction", "", ""))
-header_comp   = CompositeHeader.component(OntologyAnnotation("reagent", "", ""))
-header_cmt    = CompositeHeader.comment("My comment label")
-header_perf   = CompositeHeader.performer    # property, not callable
-header_date   = CompositeHeader.date         # property, not callable
+header_param = CompositeHeader.parameter(OntologyAnnotation("extraction", "", ""))
+header_comp = CompositeHeader.component(OntologyAnnotation("reagent", "", ""))
+header_cmt = CompositeHeader.comment("My comment label")
+header_perf = CompositeHeader.performer  # property, not callable
+header_date = CompositeHeader.date  # property, not callable
 # Fallback for unknown/simple header names:
-header_any    = CompositeHeader.OfHeaderString("SomeColumnName")
+header_any = CompositeHeader.OfHeaderString("SomeColumnName")
 
 # IOType known strings (IOType.of_string):
 # "source_name", "sample_name", "raw_data_file", "derived_data_file",
 # "image_file", "material"
 
-# Build cells (one per row, same order as rows in the table)
-cell_text      = CompositeCell.free_text("some value")
-cell_term      = CompositeCell.term(OntologyAnnotation("sandy loam", "http://...", ""))
-cell_unitized  = CompositeCell.unitized("6.8", OntologyAnnotation("pH", "http://...", ""))
-cell_empty     = CompositeCell.free_text("")
+# Build cells
+cell_text = CompositeCell.free_text("some value")
+cell_term = CompositeCell.term(OntologyAnnotation("sandy loam", "http://...", ""))
+cell_unitized = CompositeCell.unitized("6.8", OntologyAnnotation("pH", "http://...", ""))
+cell_empty = CompositeCell.free_text("")
 
 # Add column (header + matching cell list)
 table.AddColumn(header_char, [cell_term, cell_term, cell_empty])
 
-# Attach table to study or assay
-study.AddTable(table)
-assay.AddTable(table)
-```
-
-### CompositeHeader.IsTermColumn
-
-```python
-# Check before building a cell: if True, wrap plain values in OntologyAnnotation
+# Check whether a header expects a term cell
 if header.IsTermColumn:
     cell = CompositeCell.term(OntologyAnnotation(str(value), "", ""))
 else:
     cell = CompositeCell.free_text(str(value))
+
+# Attach table to study or assay
+study.AddTable(table)
+assay.AddTable(table)
 ```
 
 ---
@@ -196,7 +227,6 @@ else:
 arc = ARC.from_rocrate_json_string(json_str)
 
 # Async write to directory (creates ISA file structure on disk)
-# Must be awaited via start_as_task (Fable async bridge)
 await start_as_task(arc.WriteAsync("/path/to/output/dir"))
 ```
 
@@ -213,53 +243,49 @@ await start_as_task(arc.WriteAsync("/path/to/output/dir"))
 ## Known Pitfalls
 
 **`start_as_task` is untyped** — always add `# type: ignore[import-untyped]`
-on the import. It is an internal Fable module and has no stubs.
+on the import.
 
 **`CompositeHeader.performer` and `.date` are properties, not constructors**
 — call them without `()`:
 
 ```python
-# CORRECT
-header = CompositeHeader.performer
-# WRONG
-header = CompositeHeader.performer()   # TypeError
+header = CompositeHeader.performer  # CORRECT
+header = CompositeHeader.performer()  # TypeError
 ```
 
-**`OntologyAnnotation()` without args is valid** — use it for empty/unknown
-ontology terms rather than `None` to avoid null-ref errors in the F# layer.
+**`OntologyAnnotation()` without args is valid** — use for empty/unknown terms
+instead of `None` to avoid null-ref errors in the F# layer.
 
-**ARC objects carry .NET interop state** — do not pickle them or transfer
-them across multiprocessing boundaries. Always serialize to JSON string first.
+**ARC objects carry .NET interop state** — do not pickle or transfer across
+multiprocessing boundaries. Serialize to JSON string first.
 
 **`ToROCrateJsonString()` + `gc.collect()`** — after serializing in a worker
 process, explicitly `del arc` and call `gc.collect()` to release .NET bridge
 memory promptly.
 
-**`ArcAssay.create(technology_platform=None)`** — passing `None` is safe and
-means "unknown platform". Passing an empty `OntologyAnnotation()` is also
-accepted.
+**`ArcAssay.create(technology_platform=None)`** — `None` is safe. An empty
+`OntologyAnnotation()` is also accepted.
 
 ---
 
 ## RO-Crate JSON-LD Output Shape
 
 ```json
 {
-  "@context": { ... },
+  "@context": { "...": "..." },
   "@graph": [
-    { "@id": "inv001", "@type": "Dataset", "identifier": "inv001", ... },
-    { "@id": "study001", "@type": "Dataset", ... },
-    { "@id": "assay001", "@type": "Dataset", ... },
-    { "@id": "#Doe_John", "@type": "Person", "familyName": "Doe", ... },
-    ...
+    { "@id": "inv001", "@type": "Dataset", "identifier": "inv001" },
+    { "@id": "study001", "@type": "Dataset" },
+    { "@id": "assay001", "@type": "Dataset" },
+    { "@id": "#Doe_John", "@type": "Person", "familyName": "Doe" }
   ]
 }
 ```
 
-Useful for test assertions:
+Test assertion pattern:
 
 ```python
 graph = json.loads(arc.ToROCrateJsonString()).get("@graph", [])
 inv_node = next(item for item in graph if item.get("identifier") == "inv001")
-person   = next(item for item in graph if item.get("familyName") == "Doe")
+person = next(item for item in graph if item.get("familyName") == "Doe")
 ```

.agents/skills/config-wrapper/SKILL.md

@@ -0,0 +1,130 @@
+---
+name: config-wrapper
+description: >
+  Reference for the ConfigWrapper / ConfigBase pattern from middleware.shared.
+  Use when adding config fields, reading config values, overriding via
+  environment variables or Docker secrets, or extending ConfigBase in a new
+  component. ConfigWrapper is the single source of truth for all configuration.
+compatibility: Python 3.12+, pydantic v2, middleware.shared
+---
+
+# ConfigWrapper — Usage Reference
+
+`ConfigWrapper` (from `middleware.shared.config`) wraps a YAML file and adds
+environment variable and Docker secret overrides. A component's `Config` class
+extends `ConfigBase` and is populated via `Config.from_config_wrapper(wrapper)`.
+
+---
+
+## Loading Configuration
+
+```python
+from middleware.shared.config.config_wrapper import ConfigWrapper
+from mycomponent.config import Config  # extends ConfigBase
+
+wrapper = ConfigWrapper.from_yaml_file(path, prefix="MY_PREFIX")
+config = Config.from_config_wrapper(wrapper)
+```
+
+---
+
+## Override Resolution Order
+
+For every config field, the wrapper resolves values in this order:
+
+1. **Environment variable**: `{PREFIX}_{FIELD_PATH}` (uppercase, `_` as separator)
+2. **Docker secret file**: `/run/secrets/{prefix}_{field_path}` (lowercase)
+3. **YAML file value**
+4. **Pydantic field default**
+
+Nested fields use `_` as path separator:
+- `api_client.api_url` with prefix `MY_APP` → `MY_APP_API_CLIENT_API_URL`
+
+---
+
+## Type Coercion (env / secret values are always strings)
+
+| String value | Parsed as |
+|---|---|
+| `"true"` / `"True"` / `"TRUE"` | `True` (bool) |
+| `"false"` / `"False"` / `"FALSE"` | `False` (bool) |
+| `"123"` | `123` (int) |
+| `"3.14"` | `3.14` (float) |
+| `""` (empty) | `None` |
+| anything else | `str` |
+
+---
+
+## Extending ConfigBase
+
+`ConfigBase` is an optional convenience base class from `middleware.shared`
+that bundles config options shared across FAIRagro middleware components. You
+can subclass it to inherit those fields, or use plain `pydantic.BaseModel` if
+your component doesn't need them.
+
+```python
+from typing import Annotated
+from pydantic import Field, SecretStr
+from middleware.shared.config.config_base import ConfigBase  # optional
+
+
+class Config(ConfigBase):  # or BaseModel if ConfigBase fields aren't needed
+    # Required field (no default)
+    connection_string: Annotated[SecretStr, Field(description="DB connection URI")]
+
+    # Optional field with default
+    batch_size: Annotated[
+        int,
+        Field(description="Records to fetch per batch.", ge=1),
+    ] = 100
+```
+
+---
+
+## ConfigBase (optional convenience base)
+
+`ConfigBase` from `middleware.shared` is a FAIRagro-specific convenience class.
+Use it when your component should share the standard logging and OpenTelemetry
+fields; skip it for components that don't need them.
+
+Inherited fields:
+
+```python
+log_level: LogLevel = "INFO"  # "DEBUG" | "INFO" | "WARNING" | "ERROR" | "CRITICAL"
+otel: OtelConfig  # OpenTelemetry settings
+```
+
+`OtelConfig` fields:
+- `endpoint: str | None` — OTLP collector URL
+- `log_console_spans: bool` — print spans to stdout
+- `log_level: LogLevel` — OTLP log export level
+
+---
+
+## Secrets Handling
+
+- `SecretStr` fields: access the value as `.get_secret_value()` only at the
+  point of use (e.g., when creating a DB engine). Never pass them to `str()`
+  or log them directly.
+- Docker secrets: mount files to `/run/secrets/`; the wrapper resolves them
+  automatically using the full key name (lowercase).
+
+---
+
+## Testing
+
+In unit tests, instantiate `Config` directly without the wrapper:
+
+```python
+config = Config(
+    connection_string=SecretStr("postgresql+asyncpg://user:pass@localhost/db"),
+    # ... other required fields
+)
+```
+
+In integration tests, mock at the wrapper boundary:
+
+```python
+mocker.patch("mycomponent.main.ConfigWrapper.from_yaml_file")
+mocker.patch("mycomponent.main.Config.from_config_wrapper", return_value=mock_config)
+```