docs: add demo-guide, review README and copilot-instructions

- docs/demo-guide.md: step-by-step usage guide with 4 scenarios
  (Streamlit UI, YAML editor, Python SDK, Airflow DAG), troubleshooting
  table, and observability callouts
- README.md: fix container count (17→18), remove outdated manual admin
  creation block (now automatic), add cAdvisor to UIs table, add
  Documentation section linking demo-guide/architecture/access-credentials
- .github/copilot-instructions.md: rename to ArrowFlow, update project
  structure (new docs, prometheus/grafana/ provisioning paths), Docker
  Compose table +cAdvisor (18 services), Grafana auto-provisioning note,
  PromQL metric naming convention, Airflow idempotent user creation,
  Common Tasks cheat sheet updated with new doc refs

Author: VTvito

.github/copilot-instructions.md

@@ -1,11 +1,13 @@
-# Copilot Instructions — ETL Microservices Platform
+# Copilot Instructions — ArrowFlow (ETL Microservices Platform)
 
 > These instructions provide context for AI agents (GitHub Copilot, Copilot Chat, agentic workflows) working on this codebase. They describe architecture, conventions, patterns, constraints, and lessons learned so that an AI agent can make correct decisions without re-discovering project structure.
 
 ---
 
 ## 1. Project Identity
 
+**Project name:** ArrowFlow (GitHub repo: `VTvito/arrowflow`).
+
 **What this is:** An AI-assisted, modular ETL (Extract, Transform, Load) platform where each data operation is an independent Flask microservice. Pipelines are orchestrated via Apache Airflow DAGs or via the AI agent (natural language → YAML → execution).
 
 **Primary use case:** HR / People Analytics and E-commerce — the platform ships with production-ready pipelines for the IBM HR Attrition dataset and e-commerce order analytics, plus a weather API demo. Bundled demo datasets in `data/demo/` allow out-of-the-box testing.
@@ -93,8 +95,8 @@ All services propagate `X-Correlation-ID` header for end-to-end request tracing.
 
 ```
 etl_microservices/
-├── docker-compose.yml              # Full stack: 11 services + postgres + airflow + prometheus + grafana + streamlit
-├── Makefile                        # Common commands: up, down, build, test, lint, benchmark
+├── docker-compose.yml              # Full stack: 11 ETL services + cAdvisor + postgres + airflow + prometheus + grafana + streamlit (18 containers total)
+├── Makefile                        # Common commands: up, down, build, test, lint, benchmark, demo-data, quickstart
 ├── pyproject.toml                  # Project metadata, pytest/ruff/coverage config
 ├── README.md                       # Project overview and quickstart
 ├── .env                            # Environment variables (not committed)
@@ -159,7 +161,10 @@ etl_microservices/
 │       └── ecommerce_orders.csv    # E-commerce orders sample (501 rows)
 │
 ├── docs/
-│   └── extending.md               # Developer guide: add new services & create pipelines
+│   ├── extending.md               # Developer guide: add new services & create pipelines
+│   ├── demo-guide.md              # Step-by-step usage guide (UI, YAML editor, SDK, Airflow)
+│   ├── access-credentials.md      # All service URLs, ports, credentials, env vars
+│   └── architecture.md            # Technical design: Arrow IPC, parallelism, Gunicorn, security
 │
 ├── examples/
 │   └── pipelines/                  # Ready-to-use YAML pipeline definitions
@@ -186,7 +191,13 @@ etl_microservices/
 │   └── workflows/ci.yml           # GitHub Actions: lint → test-unit → test-integration → docker build
 │
 └── prometheus/
-    └── prometheus.yml             # Scrape targets for all 11 services
+    ├── prometheus.yml              # Scrape targets for all 11 ETL services + cAdvisor + prometheus self-scrape
+    └── grafana/
+        ├── provisioning/
+        │   ├── datasources/prometheus.yml  # Auto-configures Prometheus datasource (uid: prometheus-etl)
+        │   └── dashboards/dashboards.yml   # Dashboard provider pointing to provisioned-dashboards/
+        └── dashboards/
+            └── etl_services_overview.json  # Pre-built 15-panel monitoring dashboard (uid: etl-monitoring-v1)
 ```
 
 ---
@@ -406,15 +417,23 @@ Files stored at `/app/data/<dataset_name>/xcom/<step>_<timestamp>_<uuid>.arrow`.
 - **Base image:** `python:3.9-slim` for all services
 - **Shared volume:** `etl-containers-shared-data` mounted at `/app/data` across all containers
 
-### Docker Compose Services
+### Docker Compose Services (18 total)
 
 | Category | Services |
 |---|---|
-| **Infrastructure** | postgres, statsd-exporter, prometheus, grafana |
+| **Infrastructure** | postgres, statsd-exporter, prometheus, grafana, cadvisor |
 | **Orchestration** | airflow (webserver + scheduler) |
 | **ETL Services** | 11 microservices (ports 5001–5012) |
 | **UI** | streamlit-app (port 8501) |
 
+**cAdvisor**: `gcr.io/cadvisor/cadvisor:latest`, port 8088→8080, `--docker_only=true`. Provides per-container CPU/memory metrics scraped by Prometheus.
+
+**Grafana provisioning**: datasource and dashboard are auto-loaded at startup from `prometheus/grafana/provisioning/`. No manual configuration needed. Dashboard uid: `etl-monitoring-v1`.
+
+**Prometheus metric naming**: counters follow the pattern `{slug}_requests_total` / `{slug}_success_total` / `{slug}_error_total` where slug is the service key (e.g., `extract_csv_requests_total`). PromQL aggregation pattern: `{__name__=~".*_requests_total", job=~".+-service"}`.
+
+**Airflow admin user**: created automatically at first boot by the Dockerfile CMD (idempotent — skipped if already exists). Credentials: `admin` / `admin`.
+
 ### Network
 
 Single bridge network `etl-network`. Services reference each other by container name (e.g., `http://clean-nan-service:5002`).
@@ -641,13 +660,18 @@ These are hard-won insights from building and debugging the platform. They shoul
 | Task | Command / File |
 |---|---|
 | Start all services | `make up` or `docker compose up -d` |
+| Load demo datasets | `make demo-data` |
 | Run tests | `make test` |
 | Lint | `make lint` |
 | Add new service | Follow section 10 checklist |
 | Generate benchmark data | `make benchmark-data` |
 | Run benchmark | `make benchmark-all` |
 | Trigger HR pipeline | Airflow UI → `hr_analytics_pipeline` → Trigger with config |
 | Access Streamlit | http://localhost:8501 |
+| Grafana dashboard | http://localhost:3000 (admin / GF_SECURITY_ADMIN_PASSWORD) |
+| Service credentials | `docs/access-credentials.md` |
+| Architecture doc | `docs/architecture.md` |
+| Demo walkthrough | `docs/demo-guide.md` |
 
 ---
 

README.md

@@ -39,24 +39,18 @@ cd arrowflow
 make quickstart
 ```
 
-This will build all images, start 17 containers, and load the demo datasets.
-
-### Create Airflow Admin (first time only)
-
-```bash
-docker exec -it airflow airflow users create \
-  --username admin --firstname Admin --lastname User \
-  --role Admin --email admin@example.com --password admin
-```
+This will build all images, start 18 containers, and load the demo datasets.
+The Airflow admin user (`admin`/`admin`) is created automatically on first boot.
 
 ### Open the UIs
 
 | Interface | URL | Credentials |
 |---|---|---|
 | **Streamlit** (AI Pipeline Builder) | http://localhost:8501 | — |
 | **Airflow** | http://localhost:8080 | admin / admin |
-| **Grafana** | http://localhost:3000 | admin / *GF_SECURITY_ADMIN_PASSWORD from .env* |
+| **Grafana** (pre-provisioned dashboard) | http://localhost:3000 | admin / *GF_SECURITY_ADMIN_PASSWORD from .env* |
 | **Prometheus** | http://localhost:9090 | — |
+| **cAdvisor** (container resources) | http://localhost:8088 | — |
 
 ### Try a Demo Pipeline
 
@@ -235,6 +229,14 @@ cp -r templates/new_service services/my-service
 
 Full walkthrough: [docs/extending.md](docs/extending.md)
 
+### Documentation
+
+| Doc | Contents |
+|---|---|
+| [docs/demo-guide.md](docs/demo-guide.md) | Step-by-step demo: UI, YAML editor, SDK, Airflow |
+| [docs/architecture.md](docs/architecture.md) | System design, Arrow IPC, parallelism, Gunicorn, security |
+| [docs/access-credentials.md](docs/access-credentials.md) | All service URLs, credentials, env vars |
+
 ### Project Structure
 
 <details>