inmemory store is now modularized, modularizing backends complete

Author: AlexMercedCoder

.gitignore

@@ -31,3 +31,13 @@ ui_*.log
 # Test output files
 test_output*.txt
 *_output*.txt
+
+# Databases
+*.db
+*.sqlite
+*.sqlite3
+
+# Temporary/Debug files
+temp_*
+tmp_*
+debug_*

docs/api/README.md

@@ -71,7 +71,9 @@ Authentication methods:
 - `GET/PUT /api/v1/access-requests/{id}` - Get/update access request
 
 **Audit Logs**:
-- `GET /api/v1/audit` - List audit events
+- `GET /api/v1/audit` - List audit events (with filtering)
+- `GET /api/v1/audit/count` - Get audit event counts
+- `GET /api/v1/audit/{id}` - Get specific audit event
 
 **Warehouse Management**:
 - `GET /api/v1/warehouses` - List warehouses
@@ -82,40 +84,19 @@ Authentication methods:
 - `GET /api/v1/catalogs` - List catalogs
 - `POST /api/v1/catalogs` - Create catalog
 - `GET /api/v1/catalogs/{name}` - Get catalog
-- `PUT /api/v1/catalogs/{name}` - Update catalog
-- `DELETE /api/v1/catalogs/{name}` - Delete catalog
-
-**Federated Catalog Management**:
-- `GET /api/v1/federated-catalogs` - List federated catalogs
-- `POST /api/v1/federated-catalogs` - Create federated catalog
-- `GET /api/v1/federated-catalogs/{name}` - Get federated catalog
-- `DELETE /api/v1/federated-catalogs/{name}` - Delete federated catalog
-- `POST /api/v1/federated-catalogs/{name}/test` - Test federated connection
-
-**Permission & Role Management**:
-- `GET /api/v1/roles` - List roles
-- `POST /api/v1/roles` - Create role
-- `GET /api/v1/permissions` - List permissions
-- `POST /api/v1/permissions` - Grant permission
-- `DELETE /api/v1/permissions/{id}` - Revoke permission
-
-**Token Management**:
-- `POST /api/v1/tokens` - Generate JWT token
-- `POST /api/v1/auth/revoke` - Revoke current token
-- `POST /api/v1/auth/revoke/{token_id}` - Revoke specific token
-- `GET /api/v1/users/{user_id}/tokens` - List user tokens (Admin)
-- `DELETE /api/v1/tokens/{token_id}` - Delete specific token (Admin)
-
-**System Configuration**:
-- `GET /api/v1/config/settings` - Get system settings (Admin)
-- `PUT /api/v1/config/settings` - Update system settings (Admin)
-
-**Federated Catalog Operations**:
-- `POST /api/v1/federated-catalogs/{name}/sync` - Trigger sync
-- `GET /api/v1/federated-catalogs/{name}/stats` - Get sync stats
-
-**Data Explorer**:
-- `GET /api/v1/catalogs/{prefix}/namespaces/tree` - Get namespace tree structure
+- `GET /api/v1/catalogs/{prefix}/namespaces/tree` - Get tree structure
+
+**Service Users & API Keys**:
+- `POST /api/v1/service-users` - Create service user
+- `GET /api/v1/service-users` - List service users
+- `POST /api/v1/service-users/{id}/rotate` - Rotate API key
+
+## Interactive Documentation
+
+Pangolin provides an interactive Swagger UI for live API exploration:
+
+- **Swagger UI**: `http://localhost:8080/swagger-ui`
+- **OpenAPI JSON**: `http://localhost:8080/api-docs/openapi.json`
 
 ## Contents
 

docs/api/api_overview.md

@@ -13,6 +13,7 @@ Pangolin exposes a multi-tenant REST API split into three core functional areas:
 | `tokens` | POST | Generate a long-lived JWT token for a specific user. |
 | `auth/revoke` | POST | Invalidate current user token. |
 | `service-users` | GET/POST | Manage programmatic API identities. |
+| `service-users/{id}` | GET/PUT/DELETE | Manage specific service user. |
 | `service-users/{id}/rotate` | POST | Rotate API key for a service user. |
 
 ---
@@ -62,10 +63,12 @@ Pangolin is 100% compliant with the [Apache Iceberg REST Specification](https://
 ### 4. Search & Optimization
 | Endpoint | Method | Use Case |
 | :--- | :--- | :--- |
-| `search` | GET | Unified search across Catalogs, Namespaces, and Tables. |
-| `search/assets` | GET | Optimized lookup for specific tables/views by name. |
-| `validate/names` | POST | Check if an asset name is valid and available. |
-| `bulk/assets/delete` | POST | Bulk deletion of assets. |
+| `search` | GET | Unified search across Catalogs, Namespaces, Tables, and Branches. |
+| `search/assets` | GET | Optimized lookup for specific tables/views by name with permission filtering. |
+| `validate/names` | POST | Check if a catalog or warehouse name is available. |
+| `bulk/assets/delete` | POST | Bulk deletion of assets (up to 100). |
+| `dashboard/stats` | GET | Global platform statistics. |
+| `catalogs/{name}/summary` | GET | Catalog-specific statistics. |
 
 
 ---

docs/api/authentication.md

@@ -8,9 +8,9 @@ Pangolin implements a secure authentication system based on JSON Web Tokens (JWT
     - **Root Login**: Omit `tenant-id` or set to `null`
     - **Tenant-Scoped Login**: Include `tenant-id` with tenant UUID
 2.  **Token Issuance**: Upon successful authentication, the server returns a signed JWT.
-3.  **Authenticated Requests**: Clients must include this JWT in the `Authorization` header of subsequent requests:
+3.  **API Key Authentication (Service Users)**: Machine accounts use a static API key passed in the `X-API-Key` header.
     ```
-    Authorization: Bearer <token>
+    X-API-Key: <your-api-key>
     ```
 
 ### Login Examples
@@ -47,20 +47,23 @@ Pangolin supports the following roles:
 -   **TenantAdmin**: Tenant-level administration. Can manage warehouses, catalogs, and users within a tenant.
 -   **TenantUser**: Standard access. Can read/write data based on catalog permissions.
 
-## Token Generation
+## API Key Authentication (Service Users)
 
-For automation and scripts, users can generate long-lived JWT tokens:
+Service users are intended for machine-to-machine communication (CI/CD, automated scripts). They do not use JWT tokens; instead, they use a persistent API key.
 
 ```bash
-curl -X POST http://localhost:8080/api/v1/tokens \
-  -H "Content-Type: application/json" \
-  -d '{
-    "tenant_id": "your-tenant-id",
-    "username": "your-username",
-    "expires_in_hours": 720
-  }'
+curl http://localhost:8080/api/v1/catalogs \
+  -H "X-API-Key: pgl_key_abc123..." \
+  -H "X-Pangolin-Tenant: <tenant-uuid>"
 ```
 
+> [!TIP]
+> API keys are only displayed once upon creation. If lost, the key must be rotated via the `/api/v1/service-users/{id}/rotate` endpoint.
+
+## Token Generation (Users)
+
+For temporary programmatic access by human users, you can generate long-lived JWT tokens:
+
 ## Token Revocation
 
 Tokens can be revoked for security:

docs/api/curl_examples.md

@@ -460,7 +460,8 @@ curl -X POST http://localhost:8080/api/v1/service-users \
   -d '{
     "name": "ci-cd-pipeline",
     "description": "Service user for CI/CD automation",
-    "expires_at": "2026-12-31T23:59:59Z"
+    "role": "TenantUser",
+    "expires_in_days": 30
   }'
 ```
 
@@ -523,9 +524,12 @@ curl -X POST http://localhost:8080/api/v1/permissions \
   -H "Authorization: Bearer $TOKEN" \
   -H "Content-Type: application/json" \
   -d '{
-    "user_id": "USER_ID",
-    "scope": "catalog:production",
-    "actions": ["read", "write"]
+    "user-id": "USER_ID",
+    "scope": {
+      "type": "catalog",
+      "catalog-id": "550e8400-e29b-41d4-a716-446655440000"
+    },
+    "actions": ["Read", "Write"]
   }'
 ```
 
@@ -775,7 +779,7 @@ curl -X POST http://localhost:8080/api/v1/conflicts/CONFLICT_ID/resolve \
   -H "Authorization: Bearer $TOKEN" \
   -H "Content-Type: application/json" \
   -d '{
-    "strategy": "use_source"
+    "strategy": "TakeSource"
   }'
 ```
 
@@ -1082,8 +1086,18 @@ curl -X POST http://localhost:8080/api/v1/validate/names \
   -H "Authorization: Bearer $TOKEN" \
   -H "Content-Type: application/json" \
   -d '{
-    "name": "new_table_v2",
-    "type": "asset"
+    "resource_type": "catalog",
+    "names": ["new_catalog_v2"]
+  }'
+
+### Bulk Asset Deletion
+```bash
+curl -X POST http://localhost:8080/api/v1/bulk/assets/delete \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "asset_ids": ["uuid-1", "uuid-2"]
   }'
 ```
+```
 

docs/architecture/README.md

@@ -2,17 +2,23 @@
 
 This directory contains detailed technical documentation for the Pangolin architecture.
 
-## Core Components
-- **[Models](./models.md)**: Details the core data structures (Structs), including Tenant, Warehouse, and Asset definitions.
-- **[Enums](./enums.md)**: Exhaustive list of system enumerations like `AssetType`, `CatalogType`, and `VendingStrategy`.
-- **[Traits](./traits.md)**: Deep dive into the `CatalogStore` and `Signer` traits which define the storage and security interfaces.
-- **[Handlers](./handlers.md)**: Breakdown of the API handler modules and their functional domains.
+## 🏗️ Core Structure
+- **[High-Level Architecture](./architecture.md)**: Overall system design, component interaction, and multi-tenant isolation.
+- **[API Handlers](./handlers.md)**: Map of API endpoints categorized by functional domain (Iceberg, Versioning, Admin).
+- **[Models](./models.md)**: Comprehensive guide to core system structs (Tenant, Asset, Merge, User).
+- **[Enums](./enums.md)**: Exhaustive list of system enumerations and their serialized values.
 
-## Logic & Patterns
-- **[Branching](./branching.md)**: Explanation of the "Git-for-Data" model, including branching, committing, and 3-way merge logic.
-- **[Caching](./caching.md)**: Details on the multi-layered caching strategy using `moka` (Metadata) and `DashMap` (ObjectStore).
-- **[Dependencies](./dependencies.md)**: Comprehensive list of libraries and frameworks used in Backend and Frontend.
+## 🔧 Interfaces & Logic
+- **[System Traits](./traits.md)**: In-depth look at `CatalogStore` and `Signer` interfaces.
+- **[Branching & Merging](./branching.md)**: Operational details of the "Git-for-Data" versioning model.
+- **[Caching Strategy](./caching.md)**: multi-layered performance optimizations for metadata and cloud backends.
 
-## Legacy Docs
-- `catalog-store-trait.md`: (Ref `traits.md`)
-- `signer-trait.md`: (Ref `traits.md`)
+## 🔐 Security & Operations
+- **[Authentication](./authentication.md)**: Deep dive into JWT, Service User API Keys, and RBAC.
+- **[Storage & Connectivity](./storage_and_connectivity.md)**: Cloud connectivity, modular store structure, and credential vending.
+- **[Dependencies](./dependencies.md)**: Final list of technology stack and library versions.
+
+---
+
+## 📅 Status
+All documents in this directory were audited and updated in **December 2025** to reflect the modularized backend and enhanced security features.

docs/architecture/architecture.md

@@ -27,8 +27,9 @@ Pangolin is a Rust-based, multi-tenant, branch-aware lakehouse catalog. It is fu
 
 ### 3. Storage Layer (`pangolin_store`)
 - **Metadata Persistence**: Abstracted via the `CatalogStore` trait.
+    - **Modular Backends**: All backends are refactored into focused submodules (e.g., `tenants.rs`, `warehouses.rs`, `assets.rs`) for better maintainability.
     - `MemoryStore`: Concurrent in-memory store for rapid development/testing.
-    - `PostgresStore`: SQL backend using `sqlx` for relational scale.
+    - `PostgresStore`: SQL backend using `sqlx` for production scale.
     - `MongoStore`: Document backend for high-availability deployments.
     - `SqliteStore`: Embedded backend for local dev and edge use cases.
 - **Performance**: Direct `assets_by_id` lookup for O(1) authorization checks.
@@ -37,13 +38,14 @@ Pangolin is a Rust-based, multi-tenant, branch-aware lakehouse catalog. It is fu
 
 ### 4. Security & Isolation
 - **Authentication**:
-    - **JWT**: Standard for UI and human CLI access.
-    - **API Keys**: Distributed via **Service Users** for programmatic/CI-CD access.
-    - **OAuth 2.0**: OIDC integration with Google, Microsoft, GitHub, and Okta.
+    - **JWT**: Standard for UI and corporate identity access.
+    - **API Keys**: Managed via **Service Users** for machine-to-machine/CI-CD access. Includes automatic rotation and usage tracking.
+    - **OAuth 2.0 / OIDC**: Native integration with Google, Microsoft, GitHub, and custom providers.
 - **Authorization**:
     - **RBAC**: Role-based access control with 3 default tiers (Root, TenantAdmin, TenantUser).
-    - **TBAC**: Tag-based access control allowing permissions to flow to assets with specific labels.
-- **Tenant Isolation**: Strictly enforced at the middleware layer; storage queries always include `tenant_id`.
+    - **TBAC**: Tag-based access control allowing permissions to flow to assets with specific business labels.
+    - **Access Requests**: Integrated workflow for users to request access to restricted assets via the UI.
+- **Tenant Isolation**: Strictly enforced at the middleware layer; all store queries are scoped by `tenant_id`.
 
 ### 5. Git-like Data Lifecycle
 - **Branching Engine**: Supports full and partial catalog branching.

docs/architecture/authentication.md

@@ -119,26 +119,41 @@ Tokens are signed JWTs containing:
 4. Extract claims and create session
 5. Insert session into request extensions
 
-### 3. API Key Authentication
+### 3. API Key Authentication (Service Users Only)
 
 **Location**: [auth_middleware.rs:L135-154](file:///home/alexmerced/development/personal/Personal/2026/pangolin/pangolin/pangolin_api/src/auth_middleware.rs#L135-154)
 
-API keys provide long-lived authentication for programmatic access.
+API keys provide long-lived authentication specifically for **Service Users** (machine accounts). Regular human users cannot use API keys; they use Bearer tokens.
 
 #### Header Format
 ```
 X-API-Key: <api_key_value>
 ```
 
 #### How It Works
-1. Extract API key from `X-API-Key` header
-2. Look up key in store via `get_api_key_by_value`
-3. Verify key is not expired
-4. Create session from key's user context
-5. Insert session into request extensions
+1. Extract API key from `X-API-Key` header.
+2. Hash the key and look up the matching record in the store via `get_service_user_by_api_key_hash`.
+3. Verify the service account is active and not expired.
+4. Create a session with the service user's ID, tenant context, and role.
+5. Track usage via `update_service_user_last_used`.
+
+> [!TIP]
+> Use Service Users for CI/CD pipelines, Spark clusters, or any non-interactive automation.
+
+---
+
+## Granular Permissions (RBAC)
+
+**Location**: [permission.rs](file:///home/alexmerced/development/personal/Personal/2026/pangolin/pangolin/pangolin_core/src/permission.rs)
+
+Beyond roles, Pangolin supports fine-grained permission grants at the Asset, Namespace, or Catalog level.
+
+### Permission Structure
+- **Scope**: Where the permission applies (Tenant, Catalog, Namespace, Asset, or Tag).
+- **Actions**: What is allowed (`Read`, `Write`, `Create`, `Delete`, `ManageDiscovery`, etc.).
 
-\u003e [!NOTE]
-\u003e API keys are managed through the `/api/v1/users/{user_id}/tokens` endpoints.
+### Multi-Tenant Isolation
+The auth middleware automatically extracts the tenant context from the user's session or the `X-Pangolin-Tenant` header (for Root users). All subsequent storage operations are strictly scoped to this tenant ID.
 
 ## Authentication Flow
 

docs/architecture/branching.md

@@ -20,35 +20,24 @@ A **Commit** is an immutable record of a state change.
 ### 3. Tags
 A **Tag** is an immutable reference to a specific commit. Useful for marking releases (e.g., `Q1_REPORT_FINAL`).
 
-## Workflows
-
-### Isolation (Experimentation)
-1. User creates branch `dev/experiment` from `main`.
-2. Changes made in `dev/experiment` are isolated. `main` users continue to see the old data.
-3. If the experiment fails, the branch is deleted. No impact on production.
-
-### Zero-Copy Ingestion
-1. ETL job writes data to `etl-job-123` branch.
-2. Data quality checks run against this branch.
-3. If checks pass, `etl-job-123` is merged into `main` atomically.
-4. If checks fail, the branch is discarded.
-
-## Merge Logic
-
-Merging involves taking changes from a `source` branch and applying them to a `target` branch.
-
-### 3-Way Merge Strategy
-Pangolin uses a 3-way merge algorithm to detect conflicts:
-1. Identify the **Base Commit** (common ancestor).
-2. Compare **Source** vs **Base** to find changes.
-3. Compare **Target** vs **Base** to find changes.
-4. Detect conflicts where both Source and Target modified the same asset.
-
-### Conflict Types
-- **Schema Conflict**: Table schema evolved incompatibly in both branches.
-- **Data Conflict**: Both branches wrote to the same partitions.
-- **Metadata Conflict**: Table properties changed incompatibly.
-
-### Resolution
-- **Auto-Merge**: If changes are orthogonal (e.g., Branch A touched Table X, Branch B touched Table Y), they are merged automatically.
-- **Manual**: If a conflict is detected, the merge operation pauses (`Conflicted` status) and requires API intervention to choose a winner (`TakeSource` or `TakeTarget`).
+---
+
+## Merge Lifecycle
+
+Merging is managed via the `MergeOperation` model, which tracks the transition from initiation to completion.
+
+### 1. Initiation
+A merge is started between a `source` and `target` branch. Pangolin identifies the **Base Commit** (common ancestor) to perform a 3-way analysis.
+
+### 2. Conflict Detection
+Pangolin automatically detects:
+- **Schema Conflict**: Incompatible evolution on both branches.
+- **Data Conflict**: Concurrent writes to overlapping partitions.
+- **Metadata Conflict**: Conflicting table property changes.
+
+### 3. Resolution
+- **Auto-Merge**: Orthogonal changes are applied automatically, and the operation moves to `Completed`.
+- **Manual Intervention**: If conflicts are found, the operation enters `Conflicted` status. Users must use the API to resolve each conflict using a `ResolutionStrategy` (`TakeSource`, `TakeTarget`, or `ThreeWayMerge`).
+
+### 4. Completion
+Once all conflicts are resolved, the merge is finalized, creating a new commit on the target branch.

docs/architecture/caching.md

@@ -15,7 +15,8 @@ Reading Iceberg metadata files (snapshots, manifests) from S3/GCS is a high-late
 - **Value**: Byte vector (`Vec<u8>`).
 
 ### Usage
-- Used by all CatalogStore implementations (`MemoryStore`, `SqliteStore`, `PostgresStore`, `MongoStore`).
+- Used by all `CatalogStore` implementations (`MemoryStore`, `SqliteStore`, `PostgresStore`, `MongoStore`).
+- Shared across all modular sub-backends (e.g., `postgres/assets.rs` and `postgres/tenants.rs` both leverage the same cache instance).
 - Implements a `get_or_fetch` pattern to handle cache misses transparently.
 
 ```rust