Enhance README and configuration files with quick start instructions, example configurations, and improved environment setup for better usability

Author: Zalfsten

README.md

@@ -12,6 +12,17 @@ This repository contains the **SQL-to-ARC Converter**, a core component of the F
 | [scripts/](scripts/) | Tooling for quality checks, environment setup, and Git LFS. |
 | [docker/](docker/) | Dockerfiles and container structure tests. |
 
+## 🌟 Quick Start (Full Local Demo)
+
+For the best **out-of-the-box experience**, you can run a complete local demonstration. This setup starts a PostgreSQL database with demo data, a local Mock Middleware API, and the SQL-to-ARC converter to process and save results locally:
+
+```bash
+# Start the full demo stack (requires Docker)
+./dev_environment/start-demo.sh --build
+```
+
+> **Note:** This demo does not require any secrets or mTLS keys. Generated ARCs will be saved to `dev_environment/demo_output/`.
+
 ## 🚀 Getting Started (Development)
 
 The preferred method for working with this repository is using the **Dev Container** (VS Code).

middleware/sql_to_arc/README.md

@@ -102,6 +102,8 @@ In containerized environments, sensitive values like the `connection_string` or
 
 ## Usage
 
+The following examples assume you are in the root of the repository.
+
 ### 1. From Source (Development)
 
 Requires [uv](https://github.com/astral-sh/uv) installed.
@@ -110,26 +112,22 @@ Requires [uv](https://github.com/astral-sh/uv) installed.
 # Install dependencies for all workspace members
 uv sync --all-packages
 
-# Run the converter with a specific config file
-uv run python -m middleware.sql_to_arc.main -c my_config.yaml
+# Run the converter using the example config directly
+uv run python -m middleware.sql_to_arc.main -c middleware/sql_to_arc/config.example.yaml
 ```
 
 ### 2. Local Docker Image
 
 Build the image from the repository root:
 
 ```bash
+# Build the converter image
 docker build -f docker/Dockerfile.sql_to_arc -t sql-to-arc:local .
-```
-
-Run with environment variables:
 
-```bash
+# Run using the example config via a volume mount
 docker run --rm \
-  -e SQL_TO_ARC_CONNECTION_STRING="postgresql://..." \
-  -e SQL_TO_ARC_RDI="my-rdi" \
-  -v $(pwd)/certs:/certs \
-  -e SQL_TO_ARC_API_CLIENT__CLIENT_CERT_PATH="/certs/client.crt" \
+  --env-file .env \
+  -v $(pwd)/middleware/sql_to_arc/config.example.yaml:/etc/sql_to_arc/config.yaml:ro \
   sql-to-arc:local
 ```
 
@@ -138,12 +136,10 @@ docker run --rm \
 Pull the latest official image from Docker Hub (once available):
 
 ```bash
-docker pull fairagro/sql-to-arc:latest
-
 docker run --rm \
   --env-file .env \
-  -v $(pwd)/config.yaml:/etc/sql_to_arc/config.yaml:ro \
-  fairagro/sql-to-arc:latest -c /etc/sql_to_arc/config.yaml
+  -v $(pwd)/middleware/sql_to_arc/config.example.yaml:/etc/sql_to_arc/config.yaml:ro \
+  zalf/fairagro-advanced-middleware-sql_to_arc/sql-to-arc:latest
 ```
 
 ---

middleware/sql_to_arc/config.example.yaml

@@ -1,49 +1,96 @@
 # SQL to ARC Converter Configuration Example
-# Copy this file to config.yaml and adjust the values
+# This file contains ALL available configuration options with their default or example values.
 
-# Database Connection Settings
-# ---------------------------
-# Name of the source database (e.g., PostgreSQL database name)
-db_name: "edaphobase"
+# ------------------------------------------------------------------------------
+# 1. CORE SETTINGS
+# ------------------------------------------------------------------------------
 
-# Database user with read access
-db_user: "reader"
+# Full SQLAlchemy connection string for the source database.
+# Supported: postgresql+psycopg
+connection_string: "postgresql+psycopg://postgres:postgres@localhost:5432/rdi"
 
-# Database password (will be handled securely)
-db_password: "secure_password_here"
+# Unique identifier for the Research Data Infrastructure (RDI).
+rdi: "edaphobase"
 
-# Database host address (hostname or IP)
-db_host: "localhost"
+# Public URL of the RDI portal (used for provenance metadata).
+rdi_url: "https://edaphobase.org"
 
-# Database port (default: 5432 for PostgreSQL)
-db_port: 5432
+# Console logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL).
+log_level: "INFO"
 
-# RDI Identifier
-# -------------
-# Unique identifier for the Research Data Infrastructure (RDI)
-# This is used to tag or namespace the converted ARCs
-rdi: "edaphobase"
 
+# ------------------------------------------------------------------------------
+# 2. PROCESSING & PERFORMANCE
+# ------------------------------------------------------------------------------
+
+# Number of parallel worker processes for ARC generation (CPU-bound).
+# Recommended: Number of CPU cores available.
+max_concurrent_arc_builds: 4
+
+# Maximum concurrent tasks (IO + CPU). Defaults to 4 * max_concurrent_arc_builds.
+max_concurrent_tasks: ~
+
+# Number of investigations to fetch from the database per batch.
+db_batch_size: 100
+
+# Safety limit: Maximum number of studies per investigation.
+max_studies: 5000
+
+# Safety limit: Maximum number of assays per investigation.
+max_assays: 10000
+
+# Timeout in minutes for generating a single ARC.
+arc_generation_timeout_minutes: 30
+
+# (Optional) Limit processing to the first N investigations for debugging.
+debug_limit: ~
 
-# API Client Configuration
-# -----------------------
-# Settings for connecting to the Middleware API to upload ARCs
-api_client:
-  # Base URL of the Middleware API
-  api_url: "http://localhost:8000"
 
-  # Path to the client certificate file (PEM format) for mTLS authentication
-  client_cert_path: "/path/to/client.crt"
+# ------------------------------------------------------------------------------
+# 3. MIDDLEWARE API CLIENT (mTLS)
+# ------------------------------------------------------------------------------
 
-  # Path to the client private key file (PEM format)
-  client_key_path: "/path/to/client.key"
+api_client:
+  # Base URL of the FAIRagro Middleware API.
+  api_url: "http://localhost:8000"
 
-  # Path to the CA certificate file (optional, for self-signed server certs)
-  # ca_cert_path: "/path/to/ca.crt"
+  # Mutual TLS (mTLS) Credentials
+  client_cert_path: "dev_environment/client.crt"
+  client_key_path: "dev_environment/client.key"
+  
+  # (Optional) Path to a custom CA certificate to verify the API server.
+  ca_cert_path: ~
 
-  # Request timeout in seconds (default: 30.0)
+  # Request timeout in seconds.
   timeout: 30.0
 
-  # Verify SSL certificates (default: true)
-  # Set to false only for testing with self-signed certs without CA
+  # Whether to verify the API server's SSL certificate.
   verify_ssl: true
+
+  # Whether to follow HTTP redirects.
+  follow_redirects: true
+
+  # Maximum concurrent HTTP requests to the API.
+  max_concurrency: 10
+
+  # Maximum retries for transient HTTP errors (5xx, timeouts).
+  max_retries: 3
+
+  # Exponential backoff factor for retries.
+  retry_backoff_factor: 2.0
+
+
+# ------------------------------------------------------------------------------
+# 4. OPENTELEMETRY TRACING
+# ------------------------------------------------------------------------------
+
+otel:
+  # OTel collector endpoint (e.g., http://localhost:4318).
+  # If null (~), tracing is disabled or uses default env vars.
+  endpoint: ~
+
+  # Whether to print OpenTelemetry spans to the console in a readable format.
+  log_console_spans: false
+
+  # Logging level for OTLP log export.
+  log_level: "INFO"

scripts/load-env.sh

@@ -126,13 +126,15 @@ fi
 # Decrypt the encrypted file and write to .env
 if grep -q '"sops"' "$ENCRYPTED_FILE" 2>/dev/null; then
     # Decrypt encrypted file and write to .env
-    sops -d "$ENCRYPTED_FILE" > "$DECRYPTED_FILE" 2>/dev/null
+    # We remove the CLIENT_KEY for the .env file because it breaks Docker's --env-file parser.
+    # We use a perl regex to find the CLIENT_KEY="..." multiline block and delete it entirely.
+    sops -d "$ENCRYPTED_FILE" 2>/dev/null | perl -0777 -pe 's/CLIENT_KEY=".*?"\n?//gs' > "$DECRYPTED_FILE"
     if [ $? -eq 0 ]; then
-        echo "✅ Encrypted secrets decrypted to $DECRYPTED_FILE"
+        echo "✅ Encrypted secrets decrypted to $DECRYPTED_FILE (CLIENT_KEY omitted for Docker compatibility)"
 
-        # Also load for current shell
+        # Also load for current shell (the shell CAN handle the full file, so we re-decrypt for memory)
         set -a
-        source "$DECRYPTED_FILE"
+        source <(sops -d "$ENCRYPTED_FILE" 2>/dev/null)
         set +a
     else
         echo "❌ Error decrypting $ENCRYPTED_FILE"