ARCHITECTURE.md

Design Document: Trade Compliance Platform

Overview

The Trade Compliance Platform is a SaaS Trade Intelligence system serving Importers, Exporters, and Custom Brokers. It automates document readiness assessment, regulatory compliance verification, sanctions screening, HS code classification, and tariff rate lookup for international trade transactions.

The platform follows a Zero-Trust Compliance philosophy: every compliance claim is traceable to a source document or regulation, with a tamper-evident audit trail for every decision.

MVP Scope

• Phase 1 — Document Readiness: Shipment creation, supplier invitation via magic links, AI-powered document scanning/extraction/scoring, completeness checking, and document versioning.
• Phase 2 — Document Compliance Check: Regulatory compliance verification, sanctions screening (OFAC, EU, UN), HS code classification, tariff rate lookup, and trade route intelligence.

Target Markets (MVP)

India, Canada, USA — with a country-agnostic data model designed for global expansion.

Key Design Decisions

Decision	Choice	Rationale
Frontend	Next.js 16 (App Router)	Best-in-class React framework, SSR/SSG, App Router for layouts
API Gateway	NestJS	TypeScript, decorators for guards/interceptors, modular architecture
AI/ML Services	Python FastAPI	Python ML ecosystem, async support, LangGraph compatibility
Inter-service Comms	BullMQ (Redis) + Direct API	Async jobs for heavy processing, direct calls for simple lookups
Agent Orchestration	LangGraph (Hierarchical Supervisor)	Stateful agent workflows, tool calling, human-in-the-loop
Primary DB	PostgreSQL + RLS	ACID compliance, Row-Level Security for multi-tenancy
Knowledge Graph	Neo4j	Native graph traversal for regulatory relationships
Vector Store	pgvector	Collocated with PostgreSQL, no extra infra for MVP
Cache/Queue	Redis	BullMQ backing, rate limiting, tariff cache
Auth	Clerk	SOC 2 compliant, org management, RBAC, B2B multi-tenant
Deployment	Serverful (ECS Fargate)	Persistent connections for agents, GPU for doc processing
IaC	Terraform	AWS infrastructure provisioning
Local Dev	Docker Compose + Ollama	Full stack local with local LLM inference

---

Architecture

High-Level System Architecture

graph TB
    subgraph "Client Layer"
        WEB[Next.js 16 Frontend<br/>App Router + RSC]
        MAGIC[Magic Link Portal<br/>Exporter Upload UI]
    end

    subgraph "API Layer"
        GW[NestJS API Gateway<br/>Auth · RBAC · Rate Limiting<br/>Input Validation · CORS]
    end

    subgraph "Processing Layer"
        AI[Python FastAPI<br/>AI/ML Service]
        BULL[BullMQ Workers<br/>Job Processing]
        LG[LangGraph Supervisor<br/>Agent Orchestration]
    end

    subgraph "Agent Layer"
        SUP[Supervisor Agent]
        DP[Document Parser Agent]
        CC[Compliance Checker Agent]
        SS[Sanctions Screener Agent]
        HSC[HS Classifier Agent]
        TL[Tariff Lookup Agent]
    end

    subgraph "Data Layer"
        PG[(PostgreSQL + pgvector<br/>RLS Multi-Tenancy)]
        NEO[(Neo4j<br/>Knowledge Graph)]
        REDIS[(Redis<br/>Cache + Queues)]
        S3[(AWS S3<br/>Document Storage)]
    end

    subgraph "External Services"
        CLERK[Clerk Auth]
        SES[AWS SES<br/>Email]
        BEDROCK[AWS Bedrock<br/>LLM Inference]
        TEXTRACT[AWS Textract<br/>OCR]
    end

    WEB --> GW
    MAGIC --> GW
    GW --> AI
    GW --> PG
    GW --> REDIS
    GW --> BULL
    BULL --> AI
    AI --> LG
    LG --> SUP
    SUP --> DP
    SUP --> CC
    SUP --> SS
    SUP --> HSC
    SUP --> TL
    AI --> PG
    AI --> NEO
    AI --> REDIS
    DP --> TEXTRACT
    DP --> BEDROCK
    CC --> NEO
    SS --> NEO
    HSC --> NEO
    TL --> NEO
    GW --> CLERK
    GW --> S3
    AI --> S3
    GW --> SES

Loading

Deployment Architecture

graph TB
    subgraph "AWS Cloud"
        subgraph "Public Subnet"
            ALB[Application Load Balancer<br/>TLS Termination]
            CF[CloudFront CDN<br/>Static Assets]
        end

        subgraph "Private Subnet - Application"
            ECS_WEB[ECS Fargate<br/>Next.js Frontend]
            ECS_GW[ECS Fargate<br/>NestJS API Gateway]
            ECS_AI[ECS Fargate<br/>Python AI Service<br/>GPU-enabled]
            ECS_WORKER[ECS Fargate<br/>BullMQ Workers]
        end

        subgraph "Private Subnet - Data"
            RDS[(RDS PostgreSQL<br/>Multi-AZ + pgvector)]
            NEO4J[(Neo4j Aura<br/>or EC2 Neo4j)]
            ELASTICACHE[(ElastiCache Redis<br/>Cluster Mode)]
            S3_DOCS[(S3 Bucket<br/>Document Storage<br/>SSE-KMS)]
        end

        subgraph "Security & Monitoring"
            KMS[AWS KMS<br/>Tenant-Specific Keys]
            WAF[AWS WAF<br/>Web Application Firewall]
            CW[CloudWatch<br/>Logs + Metrics + Alarms]
            CT[CloudTrail<br/>API Audit]
            GD[GuardDuty<br/>Threat Detection]
        end

        subgraph "AI/ML Services"
            BEDROCK_SVC[AWS Bedrock<br/>Claude Sonnet 4.5 · Nova Pro<br/>Nova Lite · Nova Micro<br/>Titan Embeddings]
            TEXTRACT_SVC[AWS Textract<br/>OCR Preprocessing]
        end
    end

    subgraph "External"
        CLERK_EXT[Clerk Auth Service]
        OFAC[OFAC SDN Feed]
        EU_SANC[EU Sanctions Feed]
        UN_SANC[UN Sanctions Feed]
    end

    CF --> ALB
    ALB --> ECS_WEB
    ALB --> ECS_GW
    ECS_GW --> ECS_AI
    ECS_GW --> ELASTICACHE
    ECS_GW --> RDS
    ECS_WORKER --> ELASTICACHE
    ECS_WORKER --> ECS_AI
    ECS_AI --> RDS
    ECS_AI --> NEO4J
    ECS_AI --> BEDROCK_SVC
    ECS_AI --> TEXTRACT_SVC
    ECS_AI --> S3_DOCS
    ECS_GW --> CLERK_EXT
    ECS_GW --> S3_DOCS
    WAF --> ALB
    KMS --> S3_DOCS
    KMS --> RDS

Loading

Local Development Architecture

graph TB
    subgraph "Docker Compose"
        NEXT[Next.js 16<br/>Port 3000]
        NEST[NestJS Gateway<br/>Port 3001]
        FASTAPI[FastAPI AI Service<br/>Port 8000]
        WORKER[BullMQ Worker<br/>Background]
        PG_LOCAL[(PostgreSQL 16<br/>Port 5432)]
        NEO_LOCAL[(Neo4j<br/>Port 7474/7687)]
        REDIS_LOCAL[(Redis<br/>Port 6379)]
        OLLAMA[Ollama<br/>Port 11434<br/>Llama 3.1 8B<br/>Mistral 7B<br/>Gemma 3 4B<br/>nomic-embed-text]
        MINIO[MinIO<br/>S3-Compatible<br/>Port 9000]
    end

    NEXT --> NEST
    NEST --> FASTAPI
    NEST --> PG_LOCAL
    NEST --> REDIS_LOCAL
    NEST --> MINIO
    WORKER --> REDIS_LOCAL
    WORKER --> FASTAPI
    FASTAPI --> PG_LOCAL
    FASTAPI --> NEO_LOCAL
    FASTAPI --> OLLAMA
    FASTAPI --> MINIO

Loading

Request Flow Architecture

sequenceDiagram
    participant C as Client (Next.js)
    participant CK as Clerk
    participant GW as NestJS Gateway
    participant R as Redis
    participant Q as BullMQ Queue
    participant AI as FastAPI AI Service
    participant LG as LangGraph Supervisor
    participant A as Specialized Agent
    participant PG as PostgreSQL
    participant NG as Neo4j
    participant S3 as S3/MinIO

    C->>CK: Authenticate
    CK-->>C: JWT Token
    C->>GW: API Request + JWT
    GW->>CK: Verify JWT
    CK-->>GW: User + Org Claims
    GW->>R: Check Rate Limit
    R-->>GW: OK / 429
    GW->>GW: RBAC Check (role + tenant)
    GW->>GW: Input Validation + Sanitization

    alt Synchronous (Simple Lookup)
        GW->>PG: Query with tenant_id RLS
        PG-->>GW: Result
        GW-->>C: JSON Response
    end

    alt Asynchronous (AI Processing)
        GW->>Q: Enqueue Job
        GW-->>C: 202 Accepted + Job ID
        Q->>AI: Dequeue Job
        AI->>LG: Dispatch to Supervisor
        LG->>A: Delegate to Agent
        A->>NG: Query Knowledge Graph
        A->>S3: Fetch Document
        A-->>LG: Agent Result
        LG-->>AI: Aggregated Result
        AI->>PG: Store Results
        AI->>R: Publish Notification
        C->>GW: Poll Job Status
        GW->>PG: Get Job Result
        GW-->>C: Job Result
    end

Loading

---

Components and Interfaces

1. Next.js 16 Frontend

Responsibility: Server-side rendered UI, client-side interactivity, Clerk auth integration.

Key Routes (App Router):

Route	Description	Auth Required
/	Landing page	No
/dashboard	Readiness Dashboard	Yes
/shipments	Shipment list	Yes
/shipments/[id]	Shipment detail + documents	Yes
/shipments/new	Create shipment wizard	Yes
/compliance/[shipmentId]	Compliance report	Yes
/tariffs/[shipmentId]	Tariff lookup results	Yes
/suppliers	Supplier Network directory	Yes
/admin	Platform admin panel	Yes (Admin)
/admin/demo	Demo management	Yes (Admin)
/portal/[token]	Magic Link exporter portal	Magic Link Token
/portal/[token]/signup	Exporter sign-up	Magic Link Token

Security Headers (Next.js Middleware):

Content-Security-Policy: default-src 'self'; script-src 'self' https://clerk.com; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; connect-src 'self' https://api.clerk.com https://api.tradecomplianceplatform.com;
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Referrer-Policy: strict-origin-when-cross-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains

Frontend Security:

• DOMPurify for all user-generated content rendering
• CSRF tokens for all state-changing form submissions (SameSite=Strict cookies + double-submit pattern)
• No dangerouslySetInnerHTML without sanitization
• All API calls via server actions or API routes (no direct client-to-backend)

2. NestJS API Gateway

Responsibility: Authentication, authorization, rate limiting, input validation, request routing, audit logging.

Module Structure:

src/
├── main.ts
├── app.module.ts
├── common/
│   ├── guards/
│   │   ├── clerk-auth.guard.ts        # JWT verification via Clerk
│   │   ├── rbac.guard.ts              # Role-based access control
│   │   ├── tenant.guard.ts            # Tenant isolation enforcement
│   │   └── magic-link.guard.ts        # Magic link token validation
│   ├── interceptors/
│   │   ├── audit-log.interceptor.ts   # Automatic audit trail logging
│   │   ├── tenant-context.interceptor.ts  # Inject tenant_id into request
│   │   └── response-transform.interceptor.ts
│   ├── decorators/
│   │   ├── roles.decorator.ts         # @Roles('Importer', 'Custom_Broker')
│   │   ├── permissions.decorator.ts   # @Permissions('shipment:write')
│   │   └── current-user.decorator.ts  # @CurrentUser() user extraction
│   ├── filters/
│   │   └── global-exception.filter.ts
│   ├── pipes/
│   │   └── validation.pipe.ts         # class-validator integration
│   └── middleware/
│       ├── cors.middleware.ts          # Strict origin whitelist
│       ├── helmet.middleware.ts        # Security headers
│       ├── csrf.middleware.ts          # CSRF protection
│       └── rate-limit.middleware.ts    # Per-user/per-IP rate limiting
├── modules/
│   ├── auth/
│   │   ├── auth.module.ts
│   │   ├── auth.controller.ts
│   │   ├── auth.service.ts
│   │   └── strategies/
│   │       ├── clerk.strategy.ts
│   │       └── magic-link.strategy.ts
│   ├── shipments/
│   │   ├── shipments.module.ts
│   │   ├── shipments.controller.ts
│   │   ├── shipments.service.ts
│   │   └── dto/
│   │       ├── create-shipment.dto.ts
│   │       └── update-shipment.dto.ts
│   ├── documents/
│   │   ├── documents.module.ts
│   │   ├── documents.controller.ts
│   │   ├── documents.service.ts
│   │   └── dto/
│   │       └── upload-document.dto.ts
│   ├── compliance/
│   │   ├── compliance.module.ts
│   │   ├── compliance.controller.ts
│   │   └── compliance.service.ts
│   ├── tariffs/
│   │   ├── tariffs.module.ts
│   │   ├── tariffs.controller.ts
│   │   └── tariffs.service.ts
│   ├── magic-links/
│   │   ├── magic-links.module.ts
│   │   ├── magic-links.controller.ts
│   │   └── magic-links.service.ts
│   ├── notifications/
│   │   ├── notifications.module.ts
│   │   └── notifications.service.ts
│   ├── jobs/
│   │   ├── jobs.module.ts
│   │   ├── jobs.controller.ts
│   │   └── jobs.service.ts
│   ├── audit/
│   │   ├── audit.module.ts
│   │   └── audit.service.ts
│   ├── suppliers/
│   │   ├── suppliers.module.ts
│   │   ├── suppliers.controller.ts
│   │   └── suppliers.service.ts
│   ├── admin/
│       ├── admin.module.ts
│       ├── admin.controller.ts
│       ├── demo.controller.ts
│       └── demo.service.ts
│   └── billing/
│       ├── billing.module.ts
│       ├── billing.controller.ts
│       ├── billing.service.ts
│       ├── stripe-webhook.controller.ts
│       ├── plan-limits.middleware.ts
│       └── dto/
│           ├── create-subscription.dto.ts
│           └── update-subscription.dto.ts
└── config/
    ├── database.config.ts
    ├── redis.config.ts
    ├── clerk.config.ts
    ├── cors.config.ts
    └── rate-limit.config.ts

API Endpoints:

Method	Endpoint	Auth	Description
POST	/api/v1/auth/webhook	Clerk Webhook	Clerk user sync webhook
GET	/api/v1/shipments	JWT + Tenant	List shipments
POST	/api/v1/shipments	JWT + Tenant	Create shipment
GET	/api/v1/shipments/:id	JWT + Tenant	Get shipment detail
PUT	/api/v1/shipments/:id	JWT + Tenant	Update shipment
POST	/api/v1/shipments/:id/documents	JWT + Tenant	Upload document
GET	/api/v1/shipments/:id/documents	JWT + Tenant	List documents
GET	/api/v1/shipments/:id/documents/:docId/versions	JWT + Tenant	Document version history
POST	/api/v1/shipments/:id/compliance/check	JWT + Tenant	Initiate compliance check
GET	/api/v1/shipments/:id/compliance/report	JWT + Tenant	Get compliance report
GET	/api/v1/shipments/:id/compliance/report/pdf	JWT + Tenant	Download compliance PDF
GET	/api/v1/shipments/:id/tariffs	JWT + Tenant	Tariff lookup
POST	/api/v1/shipments/:id/magic-links	JWT + Tenant	Generate magic link
GET	/api/v1/portal/:token	Magic Link	Get portal data
POST	/api/v1/portal/:token/documents	Magic Link	Upload via portal
POST	/api/v1/portal/:token/signup	Magic Link	Exporter sign-up
GET	/api/v1/jobs/:jobId	JWT + Tenant	Job status
GET	/api/v1/suppliers/search	JWT + Tenant	Search supplier network
GET	/api/v1/trade-corridors/:corridor/requirements	JWT + Tenant	Trade route requirements
POST	/api/v1/sanctions/:shipmentId/review	JWT + Tenant	Review sanctions hit
GET	/api/v1/dashboard/readiness	JWT + Tenant	Dashboard data
POST	/api/v1/admin/demo/reset	JWT + Admin	Reset demo tenant
GET	/api/v1/health	None	Health check
POST	/api/v1/billing/webhook	Stripe Signature	Stripe webhook endpoint
GET	/api/v1/billing/subscription	JWT + Tenant	Get current subscription
POST	/api/v1/billing/subscription	JWT + Tenant	Create/update subscription
GET	/api/v1/billing/usage	JWT + Tenant	Get usage records
POST	/api/v1/billing/portal-session	JWT + Tenant	Create Stripe billing portal session

CORS Configuration:

// cors.config.ts
const ALLOWED_ORIGINS = {
  production: ['https://app.tradecomplianceplatform.com'],
  staging: ['https://staging.tradecomplianceplatform.com'],
  development: ['http://localhost:3000'],
};
// No wildcards in production. Origin validated per-request.

Rate Limiting Configuration:

// rate-limit.config.ts
{
  authenticated: { ttl: 60_000, limit: 100 },      // 100 req/min per user
  unauthenticated: { ttl: 60_000, limit: 20 },      // 20 req/min per IP
  magicLink: { ttl: 86_400_000, limit: 10 },         // 10 links/shipment/day
  authAttempts: { ttl: 900_000, limit: 5 },           // 5 attempts/15 min
}

3. Python FastAPI AI/ML Service

Responsibility: Agent orchestration, document processing, compliance engine, ML inference.

Project Structure:

ai_service/
├── main.py
├── config/
│   ├── settings.py              # Environment-based config
│   ├── models.py                # LLM/SLM model registry
│   └── ollama.py                # Local Ollama config
├── api/
│   ├── routes/
│   │   ├── documents.py         # Document processing endpoints
│   │   ├── compliance.py        # Compliance check endpoints
│   │   ├── sanctions.py         # Sanctions screening endpoints
│   │   ├── classification.py    # HS code classification endpoints
│   │   └── health.py
│   └── dependencies.py          # FastAPI dependencies
├── agents/
│   ├── supervisor.py            # LangGraph Supervisor Agent
│   ├── document_parser.py       # Document Parser Agent
│   ├── compliance_checker.py    # Compliance Checker Agent
│   ├── sanctions_screener.py    # Sanctions Screener Agent
│   ├── hs_classifier.py         # HS Classifier Agent
│   ├── tariff_lookup.py         # Tariff Lookup Agent
│   ├── tools/
│   │   ├── ocr_tool.py          # Textract / Ollama Vision wrapper
│   │   ├── neo4j_tool.py        # Knowledge Graph query tool
│   │   ├── vector_search.py     # pgvector RAG search tool
│   │   ├── sanctions_db.py      # Sanctions list lookup tool
│   │   └── tariff_db.py         # Tariff rate lookup tool
│   └── graphs/
│       ├── document_graph.py    # Document processing workflow
│       ├── compliance_graph.py  # Compliance check workflow
│       └── screening_graph.py   # Sanctions screening workflow
├── models/
│   ├── router.py                # Model Router (complexity-based)
│   ├── ollama_client.py         # Ollama local inference
│   └── bedrock_client.py        # AWS Bedrock inference
├── workers/
│   ├── bullmq_consumer.py       # BullMQ job consumer
│   ├── document_worker.py       # Document processing worker
│   ├── compliance_worker.py     # Compliance check worker
│   └── sanctions_worker.py      # Sanctions screening worker
├── ingestion/
│   ├── scheduler.py             # Data refresh scheduler
│   ├── adapters/
│   │   ├── base.py              # Base adapter interface
│   │   ├── ofac_adapter.py      # OFAC SDN ingestion
│   │   ├── eu_sanctions.py      # EU sanctions ingestion
│   │   ├── un_sanctions.py      # UN sanctions ingestion
│   │   ├── usitc_hts.py         # US HTS data ingestion
│   │   ├── canada_tariff.py     # Canadian tariff ingestion
│   │   └── india_dgft.py        # India DGFT ingestion
│   └── validators/
│       └── schema_validator.py  # Data schema validation
├── knowledge_graph/
│   ├── client.py                # Neo4j driver wrapper
│   ├── queries.py               # Cypher query templates
│   └── versioning.py            # KG version management
├── vector_store/
│   ├── client.py                # pgvector client
│   └── embeddings.py            # Embedding generation
├── services/
│   ├── scoring.py               # Document scoring logic
│   ├── checklist.py             # Document checklist generation
│   ├── fuzzy_match.py           # Sanctions fuzzy matching
│   └── po_parser.py             # PO parsing and extraction
└── db/
    ├── session.py               # SQLAlchemy async session
    └── models.py                # SQLAlchemy ORM models

LLM/SLM Tiered Model Strategy:

The platform uses a tiered model approach: SLMs (small language models) for fast/cheap tasks, LLMs only for accuracy-critical compliance work. This optimizes cost while maintaining compliance accuracy.

Production Model Tiers (AWS Bedrock):

Tier	Model	Use Cases	Cost (per M tokens)	Context	Notes
Tier 1 — Heavy Reasoning	Claude Sonnet 4.5	Compliance analysis, cross-reference verification, complex regulatory interpretation	$3 / $15 (in/out)	200K	SWE-bench 77.2%, best reasoning
Tier 2 — HS Classification & Doc Understanding	Amazon Nova Pro	HS code classification, document understanding, trade corridor analysis	~75% cheaper than comparable	300K	Multimodal, excellent for document analysis
Tier 3 — Fast Extraction & Simple Tasks	Amazon Nova Lite	Field extraction from documents, simple data parsing, template matching	Very low	300K	Multimodal (text+images), fast
Tier 4 — Text-only Quick Tasks	Amazon Nova Micro	Agent routing decisions, classification, summarization, intent detection	Lowest	128K	Text-only, 200+ tokens/sec
Tier 5 — Vision/OCR	Amazon Nova Lite + AWS Textract	OCR preprocessing, image-based document scanning	Low + Textract pricing	—	Textract for layout, Nova Lite for understanding
Tier 6 — Embeddings	Amazon Titan Embed Text v2	RAG embeddings, semantic search	Very low	—	768-dim vectors

Local Dev Model Tiers (Ollama):

Tier	Model	Use Cases	RAM Requirement	Notes
Heavy Reasoning	Llama 3.1 8B	Compliance analysis, complex reasoning	~8GB (fits 32GB + RTX 3060)	Quantized for local dev
Fast Extraction	Mistral 7B	Field extraction, simple parsing	~6GB	Fast inference
Vision/OCR	Gemma 3 4B	Document OCR, image understanding	~4GB	Multimodal, lightweight, runs well on RTX 3060
Embeddings	nomic-embed-text	RAG embeddings	~1GB	768-dim vectors

Model Router:

# models/router.py
from enum import Enum
from typing import Optional

class TaskType(Enum):
    COMPLIANCE_ANALYSIS = "compliance_analysis"       # Tier 1: Heavy reasoning
    CROSS_REFERENCE = "cross_reference"               # Tier 1: Heavy reasoning
    HS_CLASSIFICATION = "hs_classification"           # Tier 2: Document understanding
    DOCUMENT_UNDERSTANDING = "document_understanding" # Tier 2: Document understanding
    FIELD_EXTRACTION = "field_extraction"              # Tier 3: Fast extraction
    TEMPLATE_MATCHING = "template_matching"            # Tier 3: Fast extraction
    AGENT_ROUTING = "agent_routing"                    # Tier 4: Quick text tasks
    INTENT_CLASSIFICATION = "intent_classification"   # Tier 4: Quick text tasks
    SUMMARIZATION = "summarization"                    # Tier 4: Quick text tasks
    VISION_OCR = "vision_ocr"                          # Tier 5: Vision/OCR
    EMBEDDINGS = "embeddings"                          # Tier 6: Embeddings

MODEL_ROUTING = {
    "local": {
        TaskType.COMPLIANCE_ANALYSIS: "llama3.1:8b",
        TaskType.CROSS_REFERENCE: "llama3.1:8b",
        TaskType.HS_CLASSIFICATION: "llama3.1:8b",
        TaskType.DOCUMENT_UNDERSTANDING: "llama3.1:8b",
        TaskType.FIELD_EXTRACTION: "mistral:7b",
        TaskType.TEMPLATE_MATCHING: "mistral:7b",
        TaskType.AGENT_ROUTING: "mistral:7b",
        TaskType.INTENT_CLASSIFICATION: "mistral:7b",
        TaskType.SUMMARIZATION: "mistral:7b",
        TaskType.VISION_OCR: "gemma3:4b",
        TaskType.EMBEDDINGS: "nomic-embed-text",
    },
    "production": {
        # Tier 1 — Heavy Reasoning (Claude Sonnet 4.5)
        TaskType.COMPLIANCE_ANALYSIS: "anthropic.claude-sonnet-4-5-20250514-v1:0",
        TaskType.CROSS_REFERENCE: "anthropic.claude-sonnet-4-5-20250514-v1:0",
        # Tier 2 — HS Classification & Document Understanding (Nova Pro)
        TaskType.HS_CLASSIFICATION: "amazon.nova-pro-v1:0",
        TaskType.DOCUMENT_UNDERSTANDING: "amazon.nova-pro-v1:0",
        # Tier 3 — Fast Extraction (Nova Lite)
        TaskType.FIELD_EXTRACTION: "amazon.nova-lite-v1:0",
        TaskType.TEMPLATE_MATCHING: "amazon.nova-lite-v1:0",
        # Tier 4 — Quick Text Tasks (Nova Micro)
        TaskType.AGENT_ROUTING: "amazon.nova-micro-v1:0",
        TaskType.INTENT_CLASSIFICATION: "amazon.nova-micro-v1:0",
        TaskType.SUMMARIZATION: "amazon.nova-micro-v1:0",
        # Tier 5 — Vision/OCR (Nova Lite + Textract)
        TaskType.VISION_OCR: "amazon.nova-lite-v1:0",  # + aws.textract for preprocessing
        # Tier 6 — Embeddings (Titan)
        TaskType.EMBEDDINGS: "amazon.titan-embed-text-v2:0",
    }
}

class ModelRouter:
    def __init__(self, environment: str = "production"):
        self.environment = environment
        self.routing = MODEL_ROUTING[environment]

    def route(self, task_type: TaskType) -> str:
        """Route to the appropriate model based on task type.
        
        Principle: Use SLMs (Nova Micro, Nova Lite, Gemma 3) for simple/fast tasks,
        LLMs (Claude 4.5, Nova Pro) only for accuracy-critical compliance work.
        """
        return self.routing[task_type]

    def needs_textract_preprocessing(self, task_type: TaskType) -> bool:
        """Whether this task needs AWS Textract OCR before LLM processing."""
        return task_type == TaskType.VISION_OCR and self.environment == "production"

4. LangGraph Agent System

Supervisor Pattern:

graph TB
    subgraph "LangGraph Supervisor"
        SUP[Supervisor Agent<br/>Task Router + State Manager]
    end

    subgraph "Specialized Agents"
        DP[Document Parser<br/>OCR + NLP Extraction<br/>Field Confidence Scoring]
        CC[Compliance Checker<br/>Regulation Verification<br/>Cross-Reference Validation]
        SS[Sanctions Screener<br/>Fuzzy Name Matching<br/>OFAC/EU/UN Lists]
        HSC[HS Classifier<br/>Product Classification<br/>KG + RAG Lookup]
        TL[Tariff Lookup<br/>Rate Retrieval<br/>Trade Agreement Check]
    end

    subgraph "Tools"
        OCR[OCR Tool<br/>Textract / Vision LLM]
        KG[Knowledge Graph Tool<br/>Neo4j Cypher Queries]
        VS[Vector Search Tool<br/>pgvector RAG]
        SDB[Sanctions DB Tool<br/>Fuzzy Match Engine]
        TDB[Tariff DB Tool<br/>Rate Cache + KG]
    end

    SUP --> DP
    SUP --> CC
    SUP --> SS
    SUP --> HSC
    SUP --> TL
    DP --> OCR
    DP --> VS
    CC --> KG
    CC --> VS
    SS --> SDB
    HSC --> KG
    HSC --> VS
    TL --> TDB
    TL --> KG

Loading

Agent Workflow — Document Processing:

sequenceDiagram
    participant W as BullMQ Worker
    participant S as Supervisor Agent
    participant DP as Document Parser
    participant OCR as OCR Tool
    participant LLM as LLM (Claude/Llama)
    participant PG as PostgreSQL
    participant R as Redis

    W->>S: Process Document Job
    S->>S: Determine document type
    S->>DP: Dispatch to Document Parser
    DP->>OCR: Extract raw text + layout
    OCR-->>DP: Raw text + bounding boxes
    DP->>LLM: Extract structured fields
    LLM-->>DP: Structured data + confidence
    DP->>DP: Validate extracted fields
    DP->>DP: Compute Document Score
    DP-->>S: Extraction Result + Score
    S->>PG: Store extracted data + score
    S->>R: Publish score update event
    S-->>W: Job Complete

Loading

Agent Workflow — Compliance Check:

sequenceDiagram
    participant W as BullMQ Worker
    participant S as Supervisor Agent
    participant CC as Compliance Checker
    participant SS as Sanctions Screener
    participant KG as Neo4j Knowledge Graph
    participant LLM as LLM
    participant PG as PostgreSQL

    W->>S: Compliance Check Job
    S->>SS: Screen all parties
    SS->>KG: Query sanctions lists
    KG-->>SS: Matching entities
    SS->>SS: Fuzzy match (threshold 85%)
    SS-->>S: Screening results

    alt Sanctions Hit Found
        S->>PG: Store hit, block shipment
        S-->>W: Job Complete (Blocked)
    end

    S->>CC: Check document compliance
    CC->>KG: Get corridor requirements
    KG-->>CC: Required docs + rules
    CC->>PG: Get shipment documents + scores
    PG-->>CC: Document data
    CC->>LLM: Verify compliance per rule
    LLM-->>CC: Compliance verdicts + citations
    CC->>CC: Cross-reference sources
    CC-->>S: Compliance Report
    S->>PG: Store compliance report
    S-->>W: Job Complete

Loading

5. Authentication System (Clerk)

Architecture:

graph LR
    subgraph "Clerk"
        ORG[Organizations<br/>= Tenants]
        USR[Users]
        ROLES[Roles<br/>Platform + System]
        PERMS[Permissions<br/>Resource-Based]
    end

    subgraph "Platform Roles"
        IMP[Importer]
        EXP[Exporter]
        CB[Custom_Broker]
    end

    subgraph "System Roles"
        PA[Platform_Admin]
        SA[Support_Agent]
        DU[Demo_User]
    end

    ORG --> USR
    USR --> ROLES
    ROLES --> PERMS
    ROLES --> IMP
    ROLES --> EXP
    ROLES --> CB
    ROLES --> PA
    ROLES --> SA
    ROLES --> DU

Loading

RBAC Permission Matrix:

Permissions are checked against business roles (trade identity context) and system roles (platform capabilities). A user with multiple business roles gets the union of permissions for all their roles.

Permission	Importer (business)	Exporter (business)	Custom_Broker (business)	Platform_Admin (system)	Support_Agent (system)	Demo_User (system)
shipment:create	✅	❌	✅	✅	❌	✅ (demo only)
shipment:read	Own	Linked	Assigned	All	All (read-only)	Demo only
shipment:write	Own	❌	Assigned	All	❌	Demo only
document:upload	Own	Via Magic Link	Assigned	All	❌	Demo only
document:read	Own	Linked	Assigned	All	All	Demo only
compliance:run	Own	❌	Assigned	All	❌	Demo only
compliance:read	Own	❌	Assigned	All	All	Demo only
sanctions:review	❌	❌	✅	✅	❌	Demo only
tariff:lookup	Own	❌	Assigned	All	❌	Demo only
magic_link:create	✅	❌	✅	✅	❌	Demo only
supplier:search	✅	❌	✅	✅	❌	Demo only
billing:manage	✅ (org owner)	❌	✅ (org owner)	✅	❌	❌
admin:manage	❌	❌	❌	✅	❌	❌
demo:reset	❌	❌	❌	✅	❌	❌
audit:read	Own	❌	Assigned	All	All	Demo only
audit:export	Own	❌	Assigned	All	❌	Demo only

Context-Based Permission Resolution:

• When a user with both importer and exporter business roles creates a shipment, the system checks the importer context.
• When the same user uploads documents via a magic link, the system checks the exporter context.
• The API Gateway resolves the active business role from the request context (endpoint, action type, or explicit header).

Clerk Organization Mapping:

• Each tenant = 1 Clerk Organization
• Users belong to organizations with assigned roles
• Custom_Broker can be a member of multiple organizations (one per assigned Importer)
• Clerk JWT includes org_id (tenant_id), org_role, and custom permissions claim
• API Gateway extracts tenant context from JWT on every request

6. Demo System for Sales

Architecture:

graph TB
    subgraph "Demo System"
        DT[Demo Tenant<br/>Isolated Clerk Org]
        DD[Demo Data Seeder<br/>Pre-configured Shipments]
        DR[Demo Reset Service<br/>Scheduled + On-Demand]
        DF[Demo Feature Flags<br/>Premium Features Enabled]
        DML[Demo Magic Links<br/>Pre-generated Exporter Portal]
    end

    subgraph "Demo Flow"
        SALES[Sales Rep<br/>Sends Demo Link]
        PROSPECT[Prospect<br/>Opens Demo]
        PORTAL[Exporter Portal Demo<br/>Pre-seeded Documents]
        DASH[Dashboard Demo<br/>Sample Readiness Data]
        COMP[Compliance Demo<br/>Sample Check Results]
    end

    SALES --> PROSPECT
    PROSPECT --> DML
    DML --> PORTAL
    PROSPECT --> DASH
    PROSPECT --> COMP
    DR -->|Reset every 24h| DT
    DD -->|Seed data| DT

Loading

Demo Data Seed:

• 5 sample shipments across India→USA, USA→Canada, Canada→India corridors
• Pre-uploaded documents with varying readiness scores (Complete, Needs_Attention, Non_Compliant)
• Pre-run compliance checks with sample results
• 1 sample sanctions "Potential Hit" (false positive) for demonstration
• Pre-classified HS codes with tariff rates
• Sample audit trail entries

Demo Reset Process:

1. Scheduled CRON job runs every 24 hours (configurable)
2. On-demand reset via POST /api/v1/admin/demo/reset (Platform_Admin only)
3. Reset steps:
   • Delete all demo tenant data (shipments, documents, compliance results)
   • Re-run demo data seeder
   • Regenerate demo magic links
   • Clear demo tenant cache in Redis
4. Demo tenant identified by reserved tenant_id in config

Demo Access Flow:

• Sales rep generates a magic link to the demo environment
• Prospect accesses demo with Demo_User role (read + limited write)
• All demo actions are sandboxed to the demo tenant
• Demo feature flags enable premium features for showcase

---

Data Models

PostgreSQL Schema (with Row-Level Security)

All tables follow these conventions:

• id: UUID primary key (generated by application, not auto-increment)
• tenant_id: UUID foreign key to tenants table, present on every tenant-scoped table
• created_at, updated_at: Timestamps with timezone
• created_by, updated_by: UUID references to the acting user
• Row-Level Security (RLS) policies enforce tenant_id filtering at the database level

RLS Policy Template (applied to every tenant-scoped table):

ALTER TABLE {table_name} ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON {table_name}
    USING (tenant_id = current_setting('app.current_tenant_id')::uuid);

The API Gateway sets app.current_tenant_id on every database connection from the JWT claims.

Entity Relationship Diagram

erDiagram
    TENANTS ||--o{ USERS : "belongs to"
    TENANTS ||--o{ SHIPMENTS : "owns"
    TENANTS ||--o{ ORGANIZATIONS : "has"
    TENANTS ||--o{ SUBSCRIPTIONS : "has"
    TENANTS ||--o{ USAGE_RECORDS : "tracks"
    USERS ||--o{ SHIPMENTS : "creates"
    USERS ||--o{ USER_BUSINESS_ROLES : "has business roles"
    ORGANIZATIONS ||--o{ ORG_MEMBERS : "has"
    USERS ||--o{ ORG_MEMBERS : "member of"
    SHIPMENTS ||--o{ DOCUMENTS : "contains"
    SHIPMENTS ||--o{ LINE_ITEMS : "contains"
    SHIPMENTS ||--o{ COMPLIANCE_CHECKS : "undergoes"
    SHIPMENTS ||--o{ SANCTIONS_SCREENINGS : "screened by"
    SHIPMENTS ||--o{ MAGIC_LINKS : "has"
    SHIPMENTS }o--|| TRADE_CORRIDORS : "uses"
    DOCUMENTS ||--o{ DOCUMENT_VERSIONS : "has versions"
    DOCUMENT_VERSIONS ||--o{ DOCUMENT_SCORES : "scored"
    DOCUMENT_VERSIONS ||--o{ EXTRACTED_FIELDS : "contains"
    LINE_ITEMS }o--o| HS_CODES : "classified as"
    LINE_ITEMS ||--o{ HS_CODE_SUGGESTIONS : "has suggestions"
    COMPLIANCE_CHECKS ||--o{ COMPLIANCE_RESULTS : "produces"
    SANCTIONS_SCREENINGS ||--o{ SANCTIONS_HITS : "finds"
    SANCTIONS_HITS ||--o| SANCTIONS_DISPOSITIONS : "resolved by"
    SHIPMENTS ||--o{ NOTIFICATIONS : "triggers"
    USERS ||--o{ NOTIFICATIONS : "receives"
    TENANTS ||--o{ AUDIT_LOGS : "recorded in"
    JOBS ||--o| SHIPMENTS : "processes"

Loading

Table Definitions

tenants

CREATE TABLE tenants (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    clerk_org_id VARCHAR(255) UNIQUE NOT NULL,
    name VARCHAR(255) NOT NULL,
    slug VARCHAR(100) UNIQUE NOT NULL,
    plan VARCHAR(50) NOT NULL DEFAULT 'starter',       -- starter, smb, professional, enterprise
    stripe_customer_id VARCHAR(255) UNIQUE,             -- Stripe Customer ID
    is_demo BOOLEAN NOT NULL DEFAULT FALSE,
    settings JSONB NOT NULL DEFAULT '{}',               -- tenant-specific config
    feature_flags JSONB NOT NULL DEFAULT '{}',          -- enterprise-specific feature toggles
    encryption_key_arn VARCHAR(512),                     -- AWS KMS key ARN
    status VARCHAR(20) NOT NULL DEFAULT 'active',       -- active, suspended, deleted
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

users

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    clerk_user_id VARCHAR(255) UNIQUE NOT NULL,
    email VARCHAR(255) NOT NULL,
    full_name VARCHAR(255),
    avatar_url VARCHAR(512),
    system_role VARCHAR(20),                             -- platform_admin, support_agent, demo_user
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    last_login_at TIMESTAMPTZ,
    failed_login_count INTEGER NOT NULL DEFAULT 0,
    locked_until TIMESTAMPTZ,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by UUID REFERENCES users(id),
    updated_by UUID REFERENCES users(id)
);
CREATE INDEX idx_users_tenant ON users(tenant_id);
CREATE INDEX idx_users_clerk ON users(clerk_user_id);
CREATE INDEX idx_users_email ON users(email);

user_business_roles

A user can hold multiple business roles simultaneously (e.g., a wholesaler is both importer AND exporter). Business roles define a user's trade identity, separate from system/platform roles which define what they can do on the platform.

CREATE TABLE user_business_roles (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id),
    business_role VARCHAR(30) NOT NULL,                  -- importer, exporter, custom_broker, manufacturer, wholesaler
    is_primary BOOLEAN DEFAULT FALSE,
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE(user_id, business_role, tenant_id)
);
CREATE INDEX idx_user_business_roles_user ON user_business_roles(user_id);
CREATE INDEX idx_user_business_roles_tenant ON user_business_roles(tenant_id);
CREATE INDEX idx_user_business_roles_role ON user_business_roles(business_role);

Role Model Design Rationale:

• Business roles (importer, exporter, custom_broker, manufacturer, wholesaler) define the user's trade identity — who they are in the trade ecosystem. A user can have multiple.
• System roles (platform_admin, support_agent, demo_user) define platform capabilities — what they can do on the platform. Stored on the users table as a single value.
• The RBAC system checks permissions based on the context of the action: when creating a shipment, check if the user has the importer business role; when uploading via magic link, check the exporter context.
• This replaces the previous single user_type column which incorrectly assumed a user could only be one type.

broker_assignments

CREATE TABLE broker_assignments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    broker_user_id UUID NOT NULL REFERENCES users(id),
    importer_user_id UUID NOT NULL REFERENCES users(id),
    status VARCHAR(20) NOT NULL DEFAULT 'active',       -- active, revoked
    assigned_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    revoked_at TIMESTAMPTZ,
    created_by UUID REFERENCES users(id),
    UNIQUE(broker_user_id, importer_user_id)
);
CREATE INDEX idx_broker_assignments_broker ON broker_assignments(broker_user_id);
CREATE INDEX idx_broker_assignments_importer ON broker_assignments(importer_user_id);

shipments

CREATE TABLE shipments (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    reference_number VARCHAR(100) NOT NULL,
    status VARCHAR(30) NOT NULL DEFAULT 'document_collection',
    -- Status: document_collection, under_review, compliance_check,
    --         sanctions_hold, cleared, archived
    importer_id UUID NOT NULL REFERENCES users(id),
    exporter_email VARCHAR(255) NOT NULL,
    exporter_name VARCHAR(255),
    exporter_user_id UUID REFERENCES users(id),         -- linked after signup
    custom_broker_id UUID REFERENCES users(id),
    origin_country CHAR(2) NOT NULL,                     -- ISO 3166-1 alpha-2
    destination_country CHAR(2) NOT NULL,
    trade_corridor_id UUID REFERENCES trade_corridors(id),
    total_value DECIMAL(15, 2),
    currency_code CHAR(3) NOT NULL DEFAULT 'USD',        -- ISO 4217
    readiness_score DECIMAL(5, 2) DEFAULT 0.00,          -- 0.00 to 100.00
    compliance_status VARCHAR(20),                        -- pass, fail, warning, pending
    sanctions_status VARCHAR(20) DEFAULT 'pending',       -- clear, potential_hit, confirmed_hit, pending
    checklist_generated_at TIMESTAMPTZ,
    checklist_kg_version VARCHAR(50),                     -- Knowledge Graph version used
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by UUID NOT NULL REFERENCES users(id),
    updated_by UUID REFERENCES users(id)
);
CREATE INDEX idx_shipments_tenant ON shipments(tenant_id);
CREATE INDEX idx_shipments_importer ON shipments(importer_id);
CREATE INDEX idx_shipments_status ON shipments(status);
CREATE INDEX idx_shipments_corridor ON shipments(origin_country, destination_country);
CREATE INDEX idx_shipments_readiness ON shipments(readiness_score);

trade_corridors

CREATE TABLE trade_corridors (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    origin_country CHAR(2) NOT NULL,
    destination_country CHAR(2) NOT NULL,
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE(origin_country, destination_country)
);

documents

CREATE TABLE documents (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    document_type VARCHAR(50) NOT NULL,
    -- Types: bill_of_lading, commercial_invoice, packing_list,
    --        certificate_of_origin, customs_declaration,
    --        letter_of_credit, purchase_order
    is_required BOOLEAN NOT NULL DEFAULT TRUE,
    condition_description TEXT,                           -- for conditionally required docs
    current_version_id UUID,                             -- points to latest version
    current_score VARCHAR(20),                           -- complete, needs_attention, non_compliant, missing
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by UUID REFERENCES users(id),
    updated_by UUID REFERENCES users(id)
);
CREATE INDEX idx_documents_shipment ON documents(shipment_id);
CREATE INDEX idx_documents_tenant ON documents(tenant_id);
CREATE INDEX idx_documents_type ON documents(document_type);

document_versions

CREATE TABLE document_versions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    document_id UUID NOT NULL REFERENCES documents(id),
    version_number INTEGER NOT NULL,
    file_path VARCHAR(512) NOT NULL,                     -- S3 key
    file_name VARCHAR(255) NOT NULL,
    file_size_bytes BIGINT NOT NULL,
    file_format VARCHAR(10) NOT NULL,                    -- pdf, png, jpg, tiff, csv, xlsx
    file_hash_sha256 VARCHAR(64) NOT NULL,
    uploaded_by UUID NOT NULL REFERENCES users(id),
    uploaded_via VARCHAR(20) NOT NULL DEFAULT 'portal',  -- portal, magic_link, api
    scan_status VARCHAR(20) NOT NULL DEFAULT 'pending',  -- pending, processing, completed, failed
    extracted_data JSONB,                                 -- structured extraction result
    extraction_confidence JSONB,                          -- per-field confidence scores
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE(document_id, version_number)
);
CREATE INDEX idx_doc_versions_document ON document_versions(document_id);
CREATE INDEX idx_doc_versions_tenant ON document_versions(tenant_id);
CREATE INDEX idx_doc_versions_hash ON document_versions(file_hash_sha256);

document_scores

CREATE TABLE document_scores (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    document_version_id UUID NOT NULL REFERENCES document_versions(id),
    score VARCHAR(20) NOT NULL,                          -- complete, needs_attention, non_compliant
    issues JSONB NOT NULL DEFAULT '[]',                  -- [{field, issue, recommendation}]
    field_scores JSONB NOT NULL DEFAULT '{}',            -- {field_name: {score, confidence, issues}}
    computed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    computed_by VARCHAR(50) NOT NULL DEFAULT 'ai_service' -- ai_service, manual_override
);
CREATE INDEX idx_doc_scores_version ON document_scores(document_version_id);
CREATE INDEX idx_doc_scores_tenant ON document_scores(tenant_id);

line_items

CREATE TABLE line_items (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    source_document_id UUID REFERENCES documents(id),    -- PO that contained this item
    description TEXT NOT NULL,
    quantity DECIMAL(15, 4),
    unit_of_measure VARCHAR(20),
    unit_price DECIMAL(15, 4),
    currency_code CHAR(3),
    total_value DECIMAL(15, 2),
    hs_code VARCHAR(12),                                 -- confirmed HS code
    hs_code_status VARCHAR(20) NOT NULL DEFAULT 'pending', -- pending, suggested, confirmed, manual
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_by UUID REFERENCES users(id),
    updated_by UUID REFERENCES users(id)
);
CREATE INDEX idx_line_items_shipment ON line_items(shipment_id);
CREATE INDEX idx_line_items_tenant ON line_items(tenant_id);
CREATE INDEX idx_line_items_hs ON line_items(hs_code);

hs_code_suggestions

CREATE TABLE hs_code_suggestions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    line_item_id UUID NOT NULL REFERENCES line_items(id),
    suggested_hs_code VARCHAR(12) NOT NULL,
    confidence_score DECIMAL(3, 2) NOT NULL,             -- 0.00 to 1.00
    rank INTEGER NOT NULL,                                -- 1, 2, 3
    explanation TEXT NOT NULL,
    source_references JSONB NOT NULL DEFAULT '[]',       -- KG + RAG sources
    status VARCHAR(20) NOT NULL DEFAULT 'pending',       -- pending, accepted, rejected
    reviewed_by UUID REFERENCES users(id),
    reviewed_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_hs_suggestions_item ON hs_code_suggestions(line_item_id);

compliance_checks

CREATE TABLE compliance_checks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    initiated_by UUID NOT NULL REFERENCES users(id),
    status VARCHAR(20) NOT NULL DEFAULT 'pending',       -- pending, processing, completed, failed
    overall_result VARCHAR(20),                           -- pass, fail, warning
    kg_version VARCHAR(50) NOT NULL,                     -- Knowledge Graph version at check time
    report_pdf_path VARCHAR(512),                        -- S3 key for PDF report
    started_at TIMESTAMPTZ,
    completed_at TIMESTAMPTZ,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_compliance_checks_shipment ON compliance_checks(shipment_id);
CREATE INDEX idx_compliance_checks_tenant ON compliance_checks(tenant_id);

compliance_results

CREATE TABLE compliance_results (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    compliance_check_id UUID NOT NULL REFERENCES compliance_checks(id),
    check_type VARCHAR(50) NOT NULL,                     -- document_completeness, field_validation, regulatory_compliance
    rule_reference VARCHAR(255) NOT NULL,                 -- regulation citation
    result VARCHAR(20) NOT NULL,                          -- pass, fail, warning
    description TEXT NOT NULL,
    source_regulation TEXT,                                -- full regulation text
    source_url VARCHAR(512),                              -- link to regulation
    remediation_action TEXT,                               -- recommended fix
    cross_reference_sources JSONB DEFAULT '[]',           -- independent verification sources
    discrepancy_notes TEXT,                                -- if sources disagree
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_compliance_results_check ON compliance_results(compliance_check_id);

sanctions_screenings

CREATE TABLE sanctions_screenings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    screened_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    sanctions_data_version VARCHAR(50) NOT NULL,          -- version of sanctions data used
    overall_result VARCHAR(20) NOT NULL,                   -- clear, potential_hit
    parties_screened JSONB NOT NULL,                       -- [{name, role, result}]
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_sanctions_screenings_shipment ON sanctions_screenings(shipment_id);

sanctions_hits

CREATE TABLE sanctions_hits (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    screening_id UUID NOT NULL REFERENCES sanctions_screenings(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    screened_name VARCHAR(255) NOT NULL,
    screened_role VARCHAR(50) NOT NULL,                    -- importer, exporter, consignee, notify_party
    matched_entity_name VARCHAR(255) NOT NULL,
    matched_entity_id VARCHAR(100),
    source_list VARCHAR(20) NOT NULL,                      -- ofac_sdn, eu_consolidated, un_consolidated
    similarity_score DECIMAL(5, 4) NOT NULL,               -- 0.0000 to 1.0000
    match_details JSONB NOT NULL DEFAULT '{}',
    status VARCHAR(20) NOT NULL DEFAULT 'pending_review',  -- pending_review, confirmed_match, false_positive
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_sanctions_hits_shipment ON sanctions_hits(shipment_id);
CREATE INDEX idx_sanctions_hits_status ON sanctions_hits(status);

sanctions_dispositions

CREATE TABLE sanctions_dispositions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    sanctions_hit_id UUID NOT NULL REFERENCES sanctions_hits(id) UNIQUE,
    disposition VARCHAR(20) NOT NULL,                      -- confirmed_match, false_positive
    justification TEXT NOT NULL,
    reviewed_by UUID NOT NULL REFERENCES users(id),
    reviewed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

magic_links

CREATE TABLE magic_links (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    shipment_id UUID NOT NULL REFERENCES shipments(id),
    token VARCHAR(128) UNIQUE NOT NULL,                   -- cryptographically secure token
    exporter_email VARCHAR(255) NOT NULL,
    expires_at TIMESTAMPTZ NOT NULL,
    is_used BOOLEAN NOT NULL DEFAULT FALSE,
    used_at TIMESTAMPTZ,
    reminder_sent_at TIMESTAMPTZ,
    created_by UUID NOT NULL REFERENCES users(id),
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_magic_links_token ON magic_links(token);
CREATE INDEX idx_magic_links_shipment ON magic_links(shipment_id);
CREATE INDEX idx_magic_links_expires ON magic_links(expires_at);

jobs

CREATE TABLE jobs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    bullmq_job_id VARCHAR(255),
    job_type VARCHAR(50) NOT NULL,                        -- document_scan, compliance_check, sanctions_screening, hs_classification
    status VARCHAR(20) NOT NULL DEFAULT 'queued',         -- queued, processing, completed, failed
    priority INTEGER NOT NULL DEFAULT 0,                   -- higher = more priority
    input_data JSONB NOT NULL,
    result_data JSONB,
    error_message TEXT,
    retry_count INTEGER NOT NULL DEFAULT 0,
    max_retries INTEGER NOT NULL DEFAULT 3,
    shipment_id UUID REFERENCES shipments(id),
    document_version_id UUID REFERENCES document_versions(id),
    started_at TIMESTAMPTZ,
    completed_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_jobs_tenant ON jobs(tenant_id);
CREATE INDEX idx_jobs_status ON jobs(status);
CREATE INDEX idx_jobs_type ON jobs(job_type);
CREATE INDEX idx_jobs_shipment ON jobs(shipment_id);

notifications

CREATE TABLE notifications (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    recipient_user_id UUID REFERENCES users(id),
    recipient_email VARCHAR(255),                          -- for non-user recipients (exporters)
    notification_type VARCHAR(50) NOT NULL,
    -- Types: document_uploaded, score_changed, sanctions_hit,
    --        magic_link_reminder, compliance_complete, checklist_updated
    channel VARCHAR(20) NOT NULL DEFAULT 'email',          -- email (MVP), in_app, sms (future)
    subject VARCHAR(255) NOT NULL,
    body TEXT NOT NULL,
    reference_type VARCHAR(50),                            -- shipment, document, compliance_check
    reference_id UUID,
    status VARCHAR(20) NOT NULL DEFAULT 'pending',         -- pending, sent, failed
    sent_at TIMESTAMPTZ,
    error_message TEXT,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_notifications_recipient ON notifications(recipient_user_id);
CREATE INDEX idx_notifications_status ON notifications(status);
CREATE INDEX idx_notifications_tenant ON notifications(tenant_id);

audit_logs (Append-Only)

CREATE TABLE audit_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL,                              -- no FK to allow tenant deletion
    actor_id UUID,                                         -- user who performed action (null for system)
    actor_type VARCHAR(20) NOT NULL,                       -- user, system, magic_link, webhook
    actor_email VARCHAR(255),
    actor_ip INET,
    actor_user_agent TEXT,
    action VARCHAR(100) NOT NULL,
    -- Actions: auth.login, auth.logout, auth.failed, auth.locked,
    --          shipment.create, shipment.update, document.upload,
    --          document.scan, compliance.check, compliance.report,
    --          sanctions.screen, sanctions.review, hs_code.classify,
    --          hs_code.confirm, magic_link.create, magic_link.use,
    --          tariff.lookup, admin.demo_reset, rate_limit.exceeded,
    --          data.access_denied
    resource_type VARCHAR(50),                             -- shipment, document, compliance_check, etc.
    resource_id UUID,
    old_value JSONB,                                       -- previous state (for updates)
    new_value JSONB,                                       -- new state (for creates/updates)
    metadata JSONB NOT NULL DEFAULT '{}',                  -- additional context
    source_regulation VARCHAR(255),                        -- for compliance decisions
    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- Append-only: no UPDATE or DELETE allowed
REVOKE UPDATE, DELETE ON audit_logs FROM app_user;

-- Indexes for audit queries
CREATE INDEX idx_audit_tenant ON audit_logs(tenant_id);
CREATE INDEX idx_audit_actor ON audit_logs(actor_id);
CREATE INDEX idx_audit_action ON audit_logs(action);
CREATE INDEX idx_audit_resource ON audit_logs(resource_type, resource_id);
CREATE INDEX idx_audit_timestamp ON audit_logs(timestamp);

-- Partition by month for performance
CREATE TABLE audit_logs_partitioned (LIKE audit_logs INCLUDING ALL)
    PARTITION BY RANGE (timestamp);

data_ingestion_logs

CREATE TABLE data_ingestion_logs (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source VARCHAR(50) NOT NULL,                           -- ofac_sdn, eu_sanctions, un_sanctions, usitc_hts, etc.
    fetch_timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    record_count INTEGER,
    status VARCHAR(20) NOT NULL,                           -- success, partial, failed
    error_message TEXT,
    retry_count INTEGER NOT NULL DEFAULT 0,
    data_version VARCHAR(50),                              -- version identifier for this fetch
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_ingestion_source ON data_ingestion_logs(source);
CREATE INDEX idx_ingestion_status ON data_ingestion_logs(status);

supplier_network

CREATE TABLE supplier_network (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id UUID NOT NULL REFERENCES users(id) UNIQUE,
    company_name VARCHAR(255),
    country CHAR(2),
    verified BOOLEAN NOT NULL DEFAULT FALSE,
    profile_data JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_supplier_company ON supplier_network(company_name);
CREATE INDEX idx_supplier_country ON supplier_network(country);

subscriptions

CREATE TABLE subscriptions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    stripe_subscription_id VARCHAR(255) UNIQUE NOT NULL,
    stripe_price_id VARCHAR(255) NOT NULL,
    plan VARCHAR(50) NOT NULL,                            -- starter, smb, professional, enterprise
    status VARCHAR(30) NOT NULL,                          -- active, past_due, canceled, trialing, incomplete
    current_period_start TIMESTAMPTZ NOT NULL,
    current_period_end TIMESTAMPTZ NOT NULL,
    cancel_at_period_end BOOLEAN NOT NULL DEFAULT FALSE,
    canceled_at TIMESTAMPTZ,
    trial_end TIMESTAMPTZ,
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_subscriptions_tenant ON subscriptions(tenant_id);
CREATE INDEX idx_subscriptions_stripe ON subscriptions(stripe_subscription_id);
CREATE INDEX idx_subscriptions_status ON subscriptions(status);

usage_records

CREATE TABLE usage_records (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID NOT NULL REFERENCES tenants(id),
    subscription_id UUID NOT NULL REFERENCES subscriptions(id),
    usage_type VARCHAR(50) NOT NULL,                      -- shipment_created, document_scan, compliance_check, sanctions_screening, hs_classification
    quantity INTEGER NOT NULL DEFAULT 1,
    stripe_usage_record_id VARCHAR(255),                  -- Stripe metered billing record ID
    reference_type VARCHAR(50),                            -- shipment, document, compliance_check
    reference_id UUID,
    recorded_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    billing_period_start TIMESTAMPTZ NOT NULL,
    billing_period_end TIMESTAMPTZ NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_usage_records_tenant ON usage_records(tenant_id);
CREATE INDEX idx_usage_records_subscription ON usage_records(subscription_id);
CREATE INDEX idx_usage_records_type ON usage_records(usage_type);
CREATE INDEX idx_usage_records_period ON usage_records(billing_period_start, billing_period_end);

Neo4j Knowledge Graph Schema

Node Types:

// Country node
(:Country {
    code: "US",                    // ISO 3166-1 alpha-2
    name: "United States",
    region: "North America",
    customs_authority: "CBP",
    data_source_url: "https://..."
})

// HS Code node
(:HSCode {
    code: "8471.30",               // HS code (2-10 digits)
    description: "Portable digital automatic data processing machines",
    level: 6,                       // 2, 4, 6, 8, 10 digit levels
    chapter: 84,
    section: "XVI",
    effective_date: date("2024-01-01"),
    source: "usitc_hts"
})

// Tariff Rate node
(:TariffRate {
    id: "uuid",
    rate_type: "general",           // general, preferential, anti_dumping, countervailing
    rate_value: 2.5,                // percentage
    rate_unit: "percent",           // percent, specific (per unit)
    specific_rate: null,            // for specific duties
    effective_date: date("2024-01-01"),
    expiry_date: null,
    source_regulation: "HTS Chapter 84, Subheading 8471.30",
    source_url: "https://...",
    version: "2024.1",
    last_updated: datetime()
})

// Trade Agreement node
(:TradeAgreement {
    id: "uuid",
    name: "USMCA",
    full_name: "United States-Mexico-Canada Agreement",
    effective_date: date("2020-07-01"),
    status: "active",
    source_url: "https://..."
})

// Regulation node
(:Regulation {
    id: "uuid",
    title: "Certificate of Origin Requirements",
    body: "...",
    regulation_code: "19 CFR 10.411",
    category: "documentation",      // documentation, product_restriction, labeling, country_specific
    effective_date: date("2024-01-01"),
    expiry_date: null,
    source_url: "https://...",
    version: "2024.1",
    last_updated: datetime()
})

// Document Type node
(:DocumentType {
    code: "certificate_of_origin",
    name: "Certificate of Origin",
    description: "...",
    required_fields: ["exporter_name", "importer_name", "goods_description", "hs_code", "origin_criteria"]
})

// Sanctioned Entity node
(:SanctionedEntity {
    id: "uuid",
    name: "...",
    aliases: ["...", "..."],
    entity_type: "individual",      // individual, organization, vessel
    source_list: "ofac_sdn",        // ofac_sdn, eu_consolidated, un_consolidated
    source_id: "...",               // ID from source list
    country: "IR",
    programs: ["IRAN", "SDGT"],
    effective_date: date("2024-01-01"),
    last_updated: datetime(),
    data_version: "2024-06-15"
})

// Product Category node (HS Code Sections/Chapters)
(:ProductCategory {
    id: "uuid",
    code: "30",                     // HS Chapter number
    section: "VI",                  // HS Section (Roman numeral)
    name: "Pharmaceutical Products",
    description: "Pharmaceutical products including drugs, medicines, and medical preparations",
    risk_level: "high",             // low, medium, high, critical
    special_requirements: ["drug_license", "gmp_certificate", "cold_chain_documentation"],
    metadata: {}
})

// Regulatory Body node
(:RegulatoryBody {
    id: "uuid",
    code: "FDA",
    name: "Food and Drug Administration",
    country: "US",
    jurisdiction: "federal",        // federal, state, international
    website: "https://www.fda.gov",
    data_source_url: "https://...",
    categories_governed: ["30", "01-24"],  // HS chapters
    metadata: {}
})

Product Category — Regulated Categories:

HS Chapter(s)	Category	Key Requirements	Regulatory Bodies
30	Pharmaceuticals	Drug licenses, GMP certificates, FDA/CDSCO approvals, cold chain docs	FDA (US), CDSCO (IN), Health Canada (CA)
1-24	Food & Agriculture	Phytosanitary certificates, FSSAI/FDA food safety, fumigation, health certs	FDA (US), FSSAI (IN), CFIA (CA)
93	Arms & Ammunition	Export licenses, end-user certificates, ITAR compliance	DDTC (US), DGFT (IN), GAC (CA)
28-29	Chemicals	MSDS, hazardous goods declaration, REACH compliance	EPA (US), CPCB (IN), ECCC (CA)
71	Precious Metals/Stones	Kimberley Process certificates, hallmarking	CBP (US), BIS (IN), CBSA (CA)
50-63	Textiles	Country of origin labeling, quota compliance	CBP (US), DGFT (IN), CBSA (CA)

Relationship Types:

// Trade corridor relationships
(:Country)-[:HAS_CORRIDOR {is_active: true}]->(:Country)

// HS Code hierarchy
(:HSCode)-[:PARENT_OF]->(:HSCode)

// Tariff rates for HS codes in corridors
(:HSCode)-[:HAS_TARIFF_RATE {
    origin: "IN", destination: "US"
}]->(:TariffRate)

// Trade agreement applicability
(:TradeAgreement)-[:APPLIES_TO_CORRIDOR {
    origin: "US", destination: "CA"
}]->(:Country)
(:TradeAgreement)-[:GRANTS_PREFERENTIAL_RATE]->(:TariffRate)

// Document requirements
(:HSCode)-[:REQUIRES_DOCUMENT {
    corridor_origin: "IN",
    corridor_destination: "US",
    is_mandatory: true,
    condition: null
}]->(:DocumentType)

// Regulation governance
(:Regulation)-[:GOVERNS_CORRIDOR {
    origin: "IN", destination: "US"
}]->(:Country)
(:Regulation)-[:APPLIES_TO_HS_CODE]->(:HSCode)
(:Regulation)-[:REQUIRES_DOCUMENT]->(:DocumentType)

// Sanctions relationships
(:SanctionedEntity)-[:ASSOCIATED_WITH]->(:Country)
(:SanctionedEntity)-[:ALIAS_OF]->(:SanctionedEntity)

// Product category relationships
(:HSCode)-[:BELONGS_TO_CATEGORY]->(:ProductCategory)
(:ProductCategory)-[:REGULATED_BY {
    country: "US",
    requirement_type: "mandatory"
}]->(:RegulatoryBody)
(:RegulatoryBody)-[:OPERATES_IN]->(:Country)
(:ProductCategory)-[:REQUIRES_CATEGORY_DOCUMENT {
    corridor_origin: "IN",
    corridor_destination: "US",
    is_mandatory: true,
    condition: "Required for all pharmaceutical imports"
}]->(:DocumentType)
(:RegulatoryBody)-[:ISSUES_DOCUMENT]->(:DocumentType)

Version Control for Knowledge Graph:

// Every data node has version metadata
// When data is updated, old node gets expiry_date, new node is created
// Compliance checks record the KG version they used

(:KGVersion {
    version: "2024.26",             // year.week
    created_at: datetime(),
    sources_updated: ["ofac_sdn", "usitc_hts"],
    change_summary: "OFAC SDN daily update + HTS 2024 revision"
})

(:KGVersion)-[:INCLUDES_UPDATE]->(:DataIngestionLog)

pgvector Schema (RAG Store)

-- Regulatory text embeddings for RAG
CREATE TABLE regulatory_embeddings (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_type VARCHAR(50) NOT NULL,          -- regulation, trade_ruling, document_template
    source_id VARCHAR(255) NOT NULL,
    source_url VARCHAR(512),
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    content_chunk TEXT NOT NULL,                 -- chunked text for embedding
    chunk_index INTEGER NOT NULL,
    embedding vector(768) NOT NULL,             -- nomic-embed-text / Titan dimension
    country_codes CHAR(2)[],                    -- applicable countries
    hs_code_prefixes VARCHAR(12)[],             -- applicable HS code prefixes
    metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_reg_embeddings_source ON regulatory_embeddings(source_type, source_id);
CREATE INDEX idx_reg_embeddings_vector ON regulatory_embeddings
    USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100);

Redis Data Structures

# Rate limiting (sliding window)
rate_limit:user:{user_id}          -> Sorted Set (timestamp scores)
rate_limit:ip:{ip_address}         -> Sorted Set (timestamp scores)
rate_limit:magic_link:{shipment_id} -> Counter with TTL

# Tariff rate cache
tariff:{hs_code}:{origin}:{dest}   -> JSON string (TTL: 24h)

# Job status (real-time)
job:status:{job_id}                -> Hash {status, progress, result_url}

# Session/notification pub-sub
notifications:{tenant_id}:{user_id} -> Pub/Sub channel

# Demo tenant tracking
demo:tenant:{tenant_id}:last_reset -> Timestamp
demo:tenant:{tenant_id}:seed_status -> String

# BullMQ queues
bull:document-processing            -> BullMQ queue
bull:compliance-check               -> BullMQ queue (higher priority)
bull:sanctions-screening             -> BullMQ queue (highest priority)
bull:hs-classification               -> BullMQ queue
bull:data-ingestion                  -> BullMQ queue
bull:notifications                   -> BullMQ queue

Regulatory Change Management Pipeline

graph TB
    subgraph "Data Sources"
        OFAC[OFAC SDN<br/>Daily CSV/XML]
        EU[EU Sanctions<br/>Regular XML]
        UN[UN Sanctions<br/>Regular XML]
        HTS[USITC HTS<br/>Weekly]
        CAN[Canada Tariff<br/>Weekly]
        IND[India DGFT<br/>Weekly]
    end

    subgraph "Ingestion Pipeline"
        SCHED[Scheduler<br/>Configurable Cron]
        ADAPT[Country Adapters<br/>Pluggable per Source]
        VALID[Schema Validator<br/>Reject Invalid Records]
        DIFF[Change Detector<br/>Diff Against Existing]
    end

    subgraph "Knowledge Graph Update"
        VER[Version Manager<br/>Create New KG Version]
        NEO_UP[Neo4j Writer<br/>Upsert Nodes/Relationships]
        SNAP[Snapshot Recorder<br/>Log Ingestion Details]
    end

    subgraph "Notification"
        IMPACT[Impact Analyzer<br/>Find Affected Shipments]
        NOTIFY[Notification Service<br/>Alert Affected Users]
    end

    OFAC --> SCHED
    EU --> SCHED
    UN --> SCHED
    HTS --> SCHED
    CAN --> SCHED
    IND --> SCHED
    SCHED --> ADAPT
    ADAPT --> VALID
    VALID --> DIFF
    DIFF --> VER
    VER --> NEO_UP
    VER --> SNAP
    DIFF --> IMPACT
    IMPACT --> NOTIFY

Loading

Adding a New Country:

1. Create a new adapter in ai_service/ingestion/adapters/ implementing the base adapter interface
2. Configure the data source URL and refresh schedule
3. Ingest initial data into the Knowledge Graph
4. Add trade corridor entries for the new country pairs
5. No application code changes required — the country-agnostic data model handles the rest

---

Billing, Plans, and Stripe Integration

Plan Tiers

Feature	Starter (Free)	SMB	Professional	Enterprise
Price	$0/month	$45/month	$99/month	Custom pricing
Shipments/month	5	25	100	Unlimited
Document scans/month	20	150	500	Unlimited
Compliance checks/month	5	25	100	Unlimited
Sanctions screening	❌	✅ (basic)	✅ (full)	✅ (full + re-screening)
Tariff lookup	Basic (cached only)	Full	Full	Full + historical
HS code classification	10/month	50/month	200/month	Unlimited
Trade corridors	2	All MVP corridors	All MVP corridors	All + custom
Magic links/shipment	3	5	10	Unlimited
Document versioning	Latest 3 versions	Latest 10 versions	Full history	Full history
Audit trail export	❌	✅ (PDF)	✅ (PDF)	✅ (PDF + CSV + API)
SSO (SAML/OIDC)	❌	❌	❌	✅
Dedicated support	Community	Email (72h SLA)	Email (48h SLA)	Dedicated CSM (4h SLA)
Custom integrations	❌	❌	❌	✅ (API + webhooks)
Data retention	90 days	1 year	2 years	Custom
Users/org	2	5	15	Unlimited
Supplier network access	❌	✅	✅	✅

Infrastructure Cost Estimate (AWS — Monthly)

Baseline (0-50 tenants, early stage):

Service	Spec	Monthly Cost
ECS Fargate — Next.js	0.5 vCPU, 1GB RAM	~$15
ECS Fargate — NestJS Gateway	1 vCPU, 2GB RAM	~$30
ECS Fargate — FastAPI AI Service	1 vCPU, 4GB RAM	~$55
ECS Fargate — BullMQ Workers	0.5 vCPU, 2GB RAM	~$25
RDS PostgreSQL (db.t4g.medium)	2 vCPU, 4GB RAM, 100GB gp3	~$70
ElastiCache Redis (cache.t4g.micro)	1 node	~$15
Neo4j (EC2 t3.medium)	2 vCPU, 4GB RAM, 50GB	~$35
S3 (document storage)	50GB + requests	~$5
ALB	1 load balancer	~$20
CloudFront	100GB transfer	~$10
AWS SES	10K emails/month	~$1
AWS Textract	~500 pages/month	~$8
AWS Bedrock (AI inference)	See breakdown below	~$50-150
CloudWatch + WAF	Logs, metrics, firewall	~$25
KMS	5 keys + requests	~$5
Total baseline		~$370-470/month

Bedrock AI Cost Breakdown (per 1000 document scans):

Model	Task	Tokens/scan	Cost/1K scans
Nova Micro ($0.035/$0.14 per M)	Agent routing, classification	~500 in + 200 out	~$0.05
Nova Lite ($0.06/$0.24 per M)	Field extraction	~2K in + 1K out	~$0.36
Nova Pro ($0.80/$3.20 per M)	HS classification, doc understanding	~3K in + 1K out	~$5.60
Claude Sonnet 4.5 ($3/$15 per M)	Compliance analysis (only when needed)	~4K in + 2K out	~$42.00
Titan Embeddings ($0.02 per M)	RAG embeddings	~1K per chunk	~$0.02

Key insight: By routing 80% of tasks to Nova Micro/Lite (pennies), and only using Claude 4.5 for actual compliance analysis (~10% of tasks), the average AI cost per document scan is approximately $0.05-0.15. At the SMB plan ($45/month, 150 scans), AI cost is ~$7.50-22.50, leaving healthy margin.

Scaling estimate (500 tenants):

• Infrastructure scales to ~$1,500-2,500/month
• AI inference scales linearly with usage
• Neo4j may need upgrade to dedicated instance (~$200/month)
• RDS may need Multi-AZ (~$140/month)

Billing Architecture

graph TB
    subgraph "NestJS API Gateway"
        BC[Billing Controller<br/>Subscription CRUD]
        BWH[Stripe Webhook Controller<br/>Event Processing]
        PLM[Plan Limits Middleware<br/>Usage Enforcement]
    end

    subgraph "Billing Service"
        BS[Billing Service<br/>Stripe SDK Integration]
        UT[Usage Tracker<br/>Metered Billing]
        PE[Plan Enforcer<br/>Limit Checks]
    end

    subgraph "Stripe"
        SC[Stripe Customers]
        SS[Stripe Subscriptions<br/>Stripe Billing]
        SP[Stripe Prices<br/>Standard + Custom]
        SU[Stripe Usage Records<br/>Metered Billing]
        SWH[Stripe Webhooks]
        SBP[Stripe Billing Portal<br/>Self-Service]
    end

    subgraph "Database"
        SUB[(subscriptions table)]
        UR[(usage_records table)]
        TEN[(tenants table<br/>stripe_customer_id)]
    end

    BC --> BS
    BWH --> BS
    PLM --> PE
    BS --> SC
    BS --> SS
    BS --> SP
    UT --> SU
    BS --> SUB
    UT --> UR
    BS --> TEN
    SWH --> BWH
    BC --> SBP

Loading

Stripe Integration Details

Subscription Lifecycle (Webhook Events):

Stripe Event	Platform Action
customer.subscription.created	Create subscriptions record, update tenant plan
customer.subscription.updated	Update plan tier, adjust limits
customer.subscription.deleted	Downgrade to Starter, disable premium features
invoice.payment_succeeded	Record payment, reset monthly usage counters
invoice.payment_failed	Send notification, grace period (7 days), then suspend
customer.subscription.trial_will_end	Send trial ending notification (3 days before)

Usage-Based Billing (Metered):

AI processing operations are tracked as metered usage and reported to Stripe:

• Document scans (per document processed by AI)
• Compliance checks (per shipment compliance run)
• HS code classifications (per item classified)
• Sanctions screenings (per shipment screened)

Usage is reported to Stripe at the end of each billing period via stripe.subscriptionItems.createUsageRecord().

Plan Limits Enforcement:

// plan-limits.middleware.ts
@Injectable()
export class PlanLimitsMiddleware implements NestMiddleware {
  async use(req: Request, res: Response, next: NextFunction) {
    const tenant = req.tenant;
    const action = this.resolveAction(req);
    
    const limits = PLAN_LIMITS[tenant.plan];
    const currentUsage = await this.usageService.getCurrentPeriodUsage(
      tenant.id, action
    );
    
    if (currentUsage >= limits[action]) {
      throw new HttpException(
        {
          error: 'Plan limit exceeded',
          limit: limits[action],
          current: currentUsage,
          upgradeUrl: `/billing/upgrade`,
        },
        HttpStatus.TOO_MANY_REQUESTS,
      );
    }
    
    next();
  }
}

const PLAN_LIMITS = {
  starter: {
    shipments_per_month: 5,
    document_scans_per_month: 20,
    compliance_checks_per_month: 5,
    hs_classifications_per_month: 10,
    magic_links_per_shipment: 3,
  },
  professional: {
    shipments_per_month: 100,
    document_scans_per_month: 500,
    compliance_checks_per_month: 100,
    hs_classifications_per_month: 200,
    magic_links_per_shipment: 10,
  },
  enterprise: {
    // Unlimited — enforced via feature flags, not hard limits
    shipments_per_month: Infinity,
    document_scans_per_month: Infinity,
    compliance_checks_per_month: Infinity,
    hs_classifications_per_month: Infinity,
    magic_links_per_shipment: Infinity,
  },
};

Enterprise Custom Plans:

• Custom pricing negotiated by sales team
• Stripe Price API creates per-tenant custom prices (stripe.prices.create() with lookup_key per tenant)
• Feature flags stored in tenants.feature_flags JSONB column for enterprise-specific capabilities (SSO, custom integrations, extended data retention)
• Custom SLA terms stored in tenants.settings JSONB

---

Product Category-Based Compliance

Overview

Compliance requirements vary significantly by product category. HS codes are organized into 21 Sections and 97 Chapters, each with distinct regulatory requirements. The Document Checklist generation must consider both the trade corridor AND the product category to produce accurate requirements.

Category-Specific Compliance Flow

graph TB
    subgraph "Input"
        HS[Confirmed HS Codes]
        TC[Trade Corridor<br/>Origin → Destination]
    end

    subgraph "Knowledge Graph Lookup"
        CAT[Resolve Product Category<br/>HS Chapter → Category]
        CORR[Corridor Requirements<br/>Country-pair rules]
        CATREQ[Category Requirements<br/>Category-specific docs]
        REG[Regulatory Bodies<br/>Applicable authorities]
    end

    subgraph "Checklist Generation"
        MERGE[Merge Requirements<br/>Corridor ∪ Category]
        DEDUP[Deduplicate & Prioritize<br/>Mandatory > Conditional]
        GEN[Generate Document Checklist<br/>With conditions & regulatory refs]
    end

    HS --> CAT
    TC --> CORR
    CAT --> CATREQ
    CAT --> REG
    CORR --> MERGE
    CATREQ --> MERGE
    REG --> MERGE
    MERGE --> DEDUP
    DEDUP --> GEN

Loading

Where Document Checklist Data Comes From

This is the most critical data question: there is no single downloadable database that maps "HS code + country → required documents." This mapping is our core intellectual property and competitive moat. Here's the hybrid approach — combining one-time LLM-assisted extraction with structured rule maintenance:

The Problem with Pure Scraping

Building scrapers for every government website is a bad strategy because:

• Government sites change layout/structure without notice, breaking scrapers
• Many sites (India CBIC, ICEGATE) use JSP/dynamic rendering that's hard to scrape reliably
• Regulatory text requires interpretation, not just extraction — "items under Chapter 30 require drug license" needs to be mapped to specific HS codes
• Scraping frequency doesn't match regulation change frequency (regulations change monthly, not daily)
• Legal risk — some government sites have terms prohibiting automated scraping

The Better Approach: LLM-Assisted Knowledge Extraction Pipeline

Instead of scraping live sites repeatedly, we use a one-time bulk extraction + ongoing change monitoring approach:

graph TB
    subgraph "Phase A — Initial Knowledge Base Build (One-Time)"
        GOV[Government Source Documents<br/>trade.gov guides, CBIC manuals,<br/>CBSA D-Memoranda, DGFT FTP]
        DOWNLOAD[Download/Save Source Documents<br/>PDF, HTML → local storage]
        CHUNK[Chunk Documents<br/>By section, chapter, regulation]
        LLM_EXTRACT[LLM Extraction Agent<br/>Claude 4.5 / Nova Pro<br/>Extract structured rules]
        SCHEMA[Map to Knowledge Graph Schema<br/>Country → HS Chapter → Documents<br/>+ conditions + weights + regulatory body]
        REVIEW[Human Expert Review<br/>Validate extracted rules<br/>Fix errors, add nuance]
        KG_WRITE[Write to Knowledge Graph<br/>Versioned, with source citations]
    end

    subgraph "Phase B — Ongoing Change Monitoring"
        RSS[RSS/Gazette Feeds<br/>CBIC notifications, CBP bulletins,<br/>DGFT public notices]
        DIFF_AGENT[Change Detection Agent<br/>Compare new doc vs existing rules]
        DRAFT[Draft KG Update<br/>LLM proposes rule changes]
        CURATOR[Curator Review Queue<br/>Human approves/rejects changes]
        UPDATE[Apply Approved Updates<br/>Version bump in KG]
    end

    GOV --> DOWNLOAD
    DOWNLOAD --> CHUNK
    CHUNK --> LLM_EXTRACT
    LLM_EXTRACT --> SCHEMA
    SCHEMA --> REVIEW
    REVIEW --> KG_WRITE

    RSS --> DIFF_AGENT
    DIFF_AGENT --> DRAFT
    DRAFT --> CURATOR
    CURATOR --> UPDATE

Loading

Phase A — Initial Knowledge Base Build

Step 1: Source Document Collection (manual, one-time)

Country	Source	What We Download	Format
USA	trade.gov Country Commercial Guides (all countries)	Import requirements per destination country	HTML → save as text
USA	CBP "Importing Into the United States" guide	General import documentation requirements	PDF (official publication)
USA	FDA Import Program (by product)	Food, drug, device, cosmetic import requirements	HTML pages per product category
USA	USDA APHIS import requirements	Plant/animal product requirements	HTML/PDF
India	DGFT Foreign Trade Policy 2023 + Handbook of Procedures	Import/export policy conditions by HS chapter	PDF (official publication)
India	CBIC Customs Manual	Clearance documentation requirements	PDF
India	ICEGATE Single Window Document Code Map	Document codes required per clearance type	PDF (available on icegate.gov.in)
India	FSSAI import regulations	Food product import requirements	PDF/HTML
Canada	CBSA D-Memoranda series	Customs documentation by topic	HTML/PDF
Canada	CFIA import requirements	Food/plant/animal import requirements	HTML
Canada	Health Canada import guidance	Drug/device/cosmetic requirements	HTML/PDF

Step 2: LLM-Assisted Extraction

We feed each source document to an extraction agent (Claude 4.5 in production, Llama 3.1 locally) with a structured prompt:

EXTRACTION_PROMPT = """
You are a trade compliance expert. Extract document requirements from the following 
regulatory text into structured rules.

For each rule, provide:
1. applicable_hs_chapters: list of HS chapters (2-digit) this applies to, or "ALL" 
2. applicable_hs_codes: specific HS codes if mentioned (4-6 digit), or null
3. origin_country: ISO 2-letter code, or "ALL"
4. destination_country: ISO 2-letter code
5. document_type: one of [commercial_invoice, bill_of_lading, packing_list, 
   certificate_of_origin, customs_declaration, letter_of_credit, purchase_order,
   phytosanitary_certificate, health_certificate, drug_license, gmp_certificate,
   fumigation_certificate, msds, hazardous_goods_declaration, fda_prior_notice,
   fssai_import_license, bis_registration, import_permit, end_user_certificate,
   test_report, inspection_certificate, OTHER:<specify>]
6. is_mandatory: true/false
7. condition: null or text describing when this document is required
8. regulatory_body: which authority requires this (e.g., FDA, FSSAI, CBP, CBSA)
9. regulation_reference: specific regulation citation
10. weight: critical (3.0) / important (2.0) / standard (1.0)
    - critical: shipment CANNOT clear customs without this
    - important: may cause delays or penalties if missing
    - standard: good practice, may be requested during audit
11. product_category: specific category if applicable (pharma, food, chemicals, etc.)
12. source_url: URL of the source document

Output as JSON array.

REGULATORY TEXT:
{document_text}
"""

Step 3: Human Expert Review

The LLM output goes into a curation queue where a trade compliance expert:

• Validates each extracted rule against their domain knowledge
• Fixes any misinterpretations (LLMs sometimes confuse "recommended" with "required")
• Adds nuance the LLM missed (e.g., "this only applies to shipments over $2,500")
• Assigns confidence scores to each rule
• Approves rules for Knowledge Graph insertion

Step 4: Knowledge Graph Population

Approved rules are written to Neo4j with full provenance:

// Example: FDA Prior Notice required for food imports to USA
CREATE (rule:DocumentRule {
    id: "rule-usa-food-fda-prior-notice",
    document_type: "fda_prior_notice",
    is_mandatory: true,
    condition: "Required for ALL food products (HS Chapters 1-24) imported into the USA",
    weight: 3.0,
    regulatory_body: "FDA",
    regulation_reference: "21 CFR Part 1, Subpart I — Prior Notice of Imported Food",
    source_url: "https://www.fda.gov/food/importing-food-products-united-states/prior-notice-imported-foods",
    extraction_method: "llm_assisted",
    reviewed_by: "expert_user_id",
    reviewed_at: datetime(),
    confidence: 0.95,
    kg_version: "2026.01"
})

// Link to applicable HS chapters
MATCH (hs:HSCode) WHERE hs.chapter IN [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24]
MATCH (dest:Country {code: "US"})
CREATE (hs)-[:REQUIRES_DOCUMENT {
    corridor_destination: "US",
    corridor_origin: "ALL",
    is_mandatory: true,
    condition: "All food products",
    weight: 3.0
}]->(rule)

// Link to product category
MATCH (cat:ProductCategory {code: "food_agriculture"})
CREATE (cat)-[:REQUIRES_CATEGORY_DOCUMENT]->(rule)

Phase B — Ongoing Change Monitoring

Change Sources (automated monitoring):

Source	Method	Frequency	What Changes
CBIC Customs Notifications	RSS feed from cbic.gov.in	Daily check	Tariff changes, new document requirements, policy amendments
DGFT Public Notices	RSS/email subscription	Daily check	Import/export policy changes, HS code restrictions
CBP Trade Bulletins	CSMS (Cargo Systems Messaging Service)	Real-time	Entry requirements, system changes
FDA Import Alerts	RSS feed from fda.gov	Daily check	New product restrictions, country-specific alerts
Canada Gazette	RSS feed	Weekly check	Tariff amendments, regulatory changes
CBSA Customs Notices	Email subscription	As published	Documentation requirement changes

Change Processing Pipeline:

1. Detect: Automated monitor picks up new notification/bulletin
2. Classify: Nova Micro classifies the change type (tariff, document requirement, restriction, informational)
3. Extract: If document-requirement-related, Claude 4.5 extracts the structured rule change
4. Diff: Compare extracted change against existing KG rules
5. Draft: Generate a proposed KG update (add/modify/remove rules)
6. Queue: Place in curator review queue with urgency level
7. Review: Human curator approves/rejects within SLA (24h for critical, 72h for standard)
8. Apply: Approved changes are applied to KG with version bump
9. Notify: Affected users are notified per the existing notification system

Document Weight & Condition System

Every document rule in the Knowledge Graph carries:

interface DocumentRule {
  document_type: string;
  
  // Weight determines how critical this document is
  weight: 3.0 | 2.0 | 1.0;
  // 3.0 = CRITICAL: Shipment WILL be held at customs without this
  //   Examples: Bill of Lading, Commercial Invoice, FDA Prior Notice for food
  // 2.0 = IMPORTANT: May cause delays, penalties, or additional inspection
  //   Examples: Certificate of Origin (for preferential rates), Packing List
  // 1.0 = STANDARD: Good practice, may be requested during audit
  //   Examples: Insurance certificate, detailed packing dimensions
  
  // Condition determines WHEN this document is required
  is_mandatory: boolean;
  condition: string | null;
  // null = always required for this HS+corridor combination
  // "Claiming preferential tariff rate under USMCA" = conditional
  // "Shipment value exceeds $2,500" = value-based condition
  // "Wood packaging materials present" = content-based condition
  
  // Category narrows applicability
  product_category: string | null;
  // null = applies to all products in the HS range
  // "pharma" = only for pharmaceutical products
  // "food_agriculture" = only for food/ag products
  
  // Provenance for zero-trust compliance
  regulatory_body: string;
  regulation_reference: string;
  source_url: string;
  confidence: number;        // 0.0-1.0, how confident we are in this rule
  reviewed_by: string;       // who validated this rule
  last_verified_at: datetime; // when was this rule last confirmed current
}

Condition Evaluation at Runtime:

When generating a checklist for a shipment, the system evaluates conditions:

def evaluate_condition(rule: DocumentRule, shipment: Shipment) -> bool:
    """Determine if a conditional document is required for this shipment."""
    if rule.is_mandatory and rule.condition is None:
        return True  # Always required
    
    # Parse condition and evaluate against shipment context
    condition = rule.condition.lower()
    
    if "preferential tariff" in condition or "trade agreement" in condition:
        return shipment.claims_preferential_rate
    
    if "value exceeds" in condition:
        threshold = extract_value(condition)
        return shipment.total_value > threshold
    
    if "wood packaging" in condition:
        return shipment.has_wood_packaging
    
    if "hazardous" in condition or "dangerous goods" in condition:
        return any(item.is_hazardous for item in shipment.line_items)
    
    if "cold chain" in condition or "temperature" in condition:
        return any(item.requires_cold_chain for item in shipment.line_items)
    
    # For complex conditions, use LLM to evaluate
    return llm_evaluate_condition(rule.condition, shipment.to_context())

Effort Estimate for Initial Build

Task	Effort	Who
Download & organize source documents (3 countries)	8 hours	Developer
Build LLM extraction pipeline	16 hours	Developer
Run extraction on all source documents	4 hours	Automated + developer oversight
Expert review of extracted rules (~500 rules for 3 countries)	30-40 hours	Trade compliance expert
Build change monitoring pipeline	16 hours	Developer
Set up RSS/notification feeds	4 hours	Developer
Total	~80 hours dev + 40 hours expert

This is significantly more robust than scraping because:

• Source documents are saved locally — no dependency on live website availability
• LLM extraction handles varied document formats (PDF, HTML, prose)
• Human review catches errors before they affect compliance decisions
• Change monitoring is event-driven (notifications/RSS), not polling-based scraping
• Every rule has full provenance (source, reviewer, confidence) for audit compliance

Checklist Generation Query (Neo4j)

// Get document requirements for a shipment based on HS codes + corridor + product category
MATCH (hs:HSCode {code: $hs_code})-[:BELONGS_TO_CATEGORY]->(cat:ProductCategory)
MATCH (cat)-[:REQUIRES_CATEGORY_DOCUMENT {
    corridor_origin: $origin,
    corridor_destination: $destination
}]->(doc:DocumentType)
OPTIONAL MATCH (cat)-[:REGULATED_BY {country: $destination}]->(reg:RegulatoryBody)
RETURN doc, cat, reg, 
       cat.risk_level AS risk_level,
       cat.special_requirements AS special_requirements
UNION
// Also get corridor-level requirements (not category-specific)
MATCH (hs:HSCode {code: $hs_code})-[:REQUIRES_DOCUMENT {
    corridor_origin: $origin,
    corridor_destination: $destination
}]->(doc:DocumentType)
RETURN doc, null AS cat, null AS reg, null AS risk_level, null AS special_requirements

Category Risk Levels

Product categories are assigned risk levels that affect processing:

Risk Level	Behavior	Example Categories
Critical	Block shipment until all category docs verified, require manual compliance officer review	Arms & Ammunition (Ch. 93)
High	Flag for priority compliance check, require all category-specific documents	Pharmaceuticals (Ch. 30), Chemicals (Ch. 28-29)
Medium	Standard compliance check, category documents required but auto-clearable	Food & Agriculture (Ch. 1-24), Textiles (Ch. 50-63)
Low	Standard processing, minimal category-specific requirements	General merchandise

---

Compliance Data Sourcing

Overview — Why NOT Scraping

The user's concern is valid: building and maintaining scrapers for every government website across multiple countries is fragile, expensive, and unsustainable. We do NOT rely on web scraping as the primary strategy. Instead, we use a tiered approach:

Tier 1 — Structured Official Downloads (Primary, ~60% of data): These are government-published files in machine-readable formats (CSV, XML, JSON) designed for programmatic consumption. No scraping needed — just download, parse, and ingest.

Source	Format	How We Get It	Reliability
OFAC SDN List	CSV, XML	Direct download from ofac.treasury.gov — official SLS service	Very high — official US Treasury
EU Consolidated Sanctions	XML	Direct download from EC DG Relex — XML with structured tags	Very high — official EU
USITC HTS	JSON (13MB), CSV (4MB), XLSX	Direct download from usitc.gov — each revision published separately	Very high — official USITC
OpenSanctions	JSON, CSV	Bulk download or API — aggregates 200+ sanctions lists (OFAC, EU, UN) into normalized format	High — best single source for sanctions

Tier 2 — Official APIs (Secondary, ~10% of data): Some governments provide APIs for real-time or near-real-time data access.

Source	API Type	How We Get It	Reliability
India ICEGATE	JSON REST API	API client — supports BE/SB filing, exchange rates	Medium-high — government API
USITC DataWeb	Query interface	Programmatic query for tariff rates and trade data	High
HTS API (htsapi.dev)	REST API	Third-party wrapping official USITC data with duty rates	Medium — third-party

Tier 3 — Curated Manual Ingestion (Tertiary, ~30% of data): For sources that only publish in PDF/HTML (India CBIC, Canada CBSA, UN sanctions HTML, CBP CROSS rulings), we use a human-in-the-loop curation process rather than brittle scrapers:

1. Initial load: Trade compliance experts manually structure the regulations into our Knowledge Graph schema (one-time effort per country)
2. Change monitoring: RSS feeds, government gazette subscriptions, and email alerts notify us of changes
3. AI-assisted updates: When a change is detected, the RAG pipeline ingests the new document, and an AI agent drafts the Knowledge Graph update for human review
4. Expert review: A compliance curator reviews and approves the update before it goes live

This is more sustainable than scraping because:

• Government websites change layout frequently, breaking scrapers
• Regulatory text requires interpretation, not just extraction
• Errors in compliance data have legal consequences — human review is essential
• The volume of changes is manageable (weekly/monthly, not real-time)

Cost of curation: For MVP (3 countries), initial setup is ~2-4 weeks of a trade compliance expert's time. Ongoing maintenance is ~5-10 hours/week across all sources.

Free Government Data Sources (MVP)

Source	Data	Format	Frequency	URL	Verified
OFAC SDN List	US sanctioned entities	CSV, XML (official direct download)	Daily	ofac.treasury.gov/ofac-sanctions-lists	✅ Confirmed — CSV and advanced XML standard
EU Consolidated Sanctions	EU sanctioned entities	XML (direct download)	Regular	ec.europa.eu (DG Relex)	✅ Confirmed — XML with tags, importable to Excel/DBMS
UN Sanctions	UN sanctioned entities	Structured HTML (not clean XML)	Regular	scsanctions.un.org/consolidated	⚠️ HTML with consistent field structure — parse or use OpenSanctions
USITC HTS	US tariff schedule	JSON (13MB), CSV (4MB), XLSX	As amended	usitc.gov (HTS revisions page)	✅ Confirmed — official JSON/CSV/XLSX downloads per revision
CBP CROSS	US customs rulings database	Searchable web DB	Ongoing	rulings.cbp.gov	⚠️ No bulk download — requires search queries or scraping
Canada Customs Tariff	Canadian tariff schedule	HTML (chapter-by-chapter)	Annual + amendments	cbsa-asfc.gc.ca	⚠️ HTML only — no CSV/JSON. Requires Tier 3 curation
India CBIC	Indian customs tariff	PDF	As amended	cbic.gov.in	⚠️ PDF only — no structured data. Requires Tier 3 curation
India ICEGATE	Indian trade facilitation	JSON API (BE/SB filing, exchange rates)	Real-time	icegate.gov.in	✅ Confirmed — JSON-based API for filings
Indian Trade Portal	Trade agreements, regulations	HTML	Ongoing	indiantradeportal.in	⚠️ HTML only — Tier 3 curation
OpenSanctions	Aggregated sanctions (200+ lists incl. OFAC, EU, UN)	JSON, CSV (bulk download + API)	Daily	opensanctions.org	✅ Confirmed — best single source for sanctions aggregation
WCO HS Nomenclature	HS code structure (base)	PDF	Every 5 years	wcoomd.org	⚠️ PDF only — one-time manual ingestion

Ingestion Strategy

graph TB
    subgraph "Source Adapters (Pluggable)"
        A1[OFAC Adapter<br/>CSV/XML Parser]
        A2[EU Sanctions Adapter<br/>XML Parser]
        A3[UN Sanctions Adapter<br/>XML Parser]
        A4[USITC HTS Adapter<br/>JSON/CSV Parser]
        A5[CBP CROSS Adapter<br/>Web Scraper]
        A6[Canada Tariff Adapter<br/>HTML/PDF Parser]
        A7[India CBIC Adapter<br/>PDF/HTML Parser]
        A8[ICEGATE Adapter<br/>API Client]
        A9[OpenSanctions Adapter<br/>JSON/CSV Parser]
    end

    subgraph "Ingestion Pipeline"
        FETCH[Fetch Raw Data<br/>HTTP/API/Scrape]
        PARSE[Parse & Normalize<br/>Source-specific logic]
        VALIDATE[Schema Validation<br/>Reject invalid records]
        DIFF[Change Detection<br/>Diff against existing KG data]
        VERSION[Version Tracking<br/>Assign data version ID]
    end

    subgraph "Knowledge Graph"
        WRITE[Neo4j Writer<br/>Upsert nodes/relationships]
        HISTORY[Version History<br/>Preserve old versions]
    end

    subgraph "Manual Curation"
        CURATOR[Curation Interface<br/>Complex regulations]
        REVIEW[Expert Review Queue<br/>Ambiguous rules]
    end

    A1 --> FETCH
    A2 --> FETCH
    A3 --> FETCH
    A4 --> FETCH
    A5 --> FETCH
    A6 --> FETCH
    A7 --> FETCH
    A8 --> FETCH
    A9 --> FETCH
    FETCH --> PARSE
    PARSE --> VALIDATE
    VALIDATE --> DIFF
    DIFF --> VERSION
    VERSION --> WRITE
    VERSION --> HISTORY
    VALIDATE -->|Complex/Ambiguous| CURATOR
    CURATOR --> REVIEW
    REVIEW --> WRITE

Loading

Adapter Pattern:

Each data source has a pluggable adapter implementing a common interface:

# ingestion/adapters/base.py
from abc import ABC, abstractmethod
from typing import List, Optional
from datetime import datetime

class BaseDataAdapter(ABC):
    """Base adapter for regulatory data source ingestion."""
    
    @abstractmethod
    async def fetch_raw_data(self) -> bytes:
        """Fetch raw data from the source."""
        pass
    
    @abstractmethod
    async def parse(self, raw_data: bytes) -> List[dict]:
        """Parse raw data into normalized records."""
        pass
    
    @abstractmethod
    async def validate(self, records: List[dict]) -> tuple[List[dict], List[dict]]:
        """Validate records against schema. Returns (valid, invalid)."""
        pass
    
    @abstractmethod
    async def detect_changes(self, new_records: List[dict]) -> dict:
        """Diff new records against existing KG data.
        Returns {added: [], modified: [], removed: []}."""
        pass
    
    @abstractmethod
    def get_source_name(self) -> str:
        """Return the source identifier (e.g., 'ofac_sdn')."""
        pass
    
    @abstractmethod
    def get_refresh_schedule(self) -> str:
        """Return cron expression for refresh schedule."""
        pass

Schema Validation:

• Every record is validated against a source-specific JSON Schema before KG write
• Invalid records are logged to data_ingestion_logs with error details
• Partial ingestion is allowed — valid records proceed, invalid records are quarantined

Change Detection:

• Each ingestion run diffs new data against existing KG nodes
• Changes are categorized: added, modified, removed
• Only changed records are written to the KG (upsert pattern)
• Every change is versioned with a data_version identifier

RAG Store for Unstructured Regulatory Text

Not all regulatory data is structured. Trade rulings, guidance documents, and regulatory interpretations are stored in the RAG store (pgvector) for retrieval-augmented generation:

• CBP CROSS rulings: Full text of customs rulings, chunked and embedded
• Trade agreement text: USMCA, India-Canada CEPA provisions
• Regulatory guidance: FDA import alerts, FSSAI advisories, CBSA D-memoranda
• Document templates: Standard forms and their field requirements

Regulatory Change Monitoring

Monitoring Approach:

Method	Sources	Frequency
Scheduled polling	OFAC, EU, UN sanctions; USITC HTS; Canada Tariff	Per source schedule (daily/weekly)
RSS/Atom feeds	CBP trade bulletins, FDA import alerts	Hourly check
API polling	ICEGATE (India)	Real-time where available
Manual monitoring	WCO HS updates, trade agreement changes	Quarterly review

Change Impact Analysis:

When a regulation changes, the platform performs impact analysis:

1. Identify affected entities: Query KG for all HS codes, corridors, and document types linked to the changed regulation
2. Find active shipments: Query PostgreSQL for all shipments in document_collection or under_review status that match affected HS codes and corridors
3. Assess impact severity: Categorize as informational (minor wording change), action_required (new document needed), or blocking (shipment cannot proceed under new rules)
4. Notify users: Send notifications to affected shipment owners and custom brokers within 48 hours of detected change
5. Offer checklist regeneration: For action_required changes, prompt users to regenerate their Document Checklist with updated requirements

---

Document Readiness & Compliance Scoring Framework

Overview

Scoring is the core value proposition — it tells users exactly what's good, what's bad, and what to fix. There are two distinct scoring systems:

1. Document Readiness Score (Phase 1) — Is the document complete and well-formed?
2. Compliance Score (Phase 2) — Does the document meet regulatory requirements?

Document Readiness Scoring — What Each Agent Does

Step 1: Document Parser Agent (OCR + Extraction)

Input: Raw uploaded document (PDF, image, CSV) Model used: Nova Lite (field extraction) + Textract (OCR preprocessing) Output: Structured extraction result with per-field confidence scores

The Document Parser extracts fields based on document type templates:

Document Type	Required Fields	Conditional Fields
Commercial Invoice	invoice_number, date, seller_name, buyer_name, item_descriptions, quantities, unit_prices, total_value, currency, incoterms	hs_codes, country_of_origin, payment_terms
Bill of Lading	bl_number, shipper, consignee, notify_party, vessel_name, port_of_loading, port_of_discharge, description_of_goods, gross_weight, number_of_packages	container_numbers, seal_numbers, freight_terms
Packing List	shipper, consignee, item_descriptions, quantities, net_weight, gross_weight, package_type, number_of_packages	dimensions, marks_and_numbers
Certificate of Origin	exporter_name, importer_name, goods_description, hs_code, origin_criteria, origin_country	certifying_authority, certificate_number
Purchase Order	po_number, buyer, seller, line_items (description, qty, unit_price), total_value	hs_codes, delivery_terms, payment_terms
Customs Declaration	declarant, importer_of_record, entry_type, port_of_entry, hs_codes, declared_value, country_of_origin	bond_number, broker_reference
Letter of Credit	lc_number, issuing_bank, beneficiary, applicant, amount, currency, expiry_date	shipment_deadline, documents_required, partial_shipment_allowed

Per-field output:

{
  "field_name": "invoice_number",
  "extracted_value": "INV-2026-00451",
  "confidence": 0.95,
  "bounding_box": {"page": 1, "x": 120, "y": 45, "w": 200, "h": 20},
  "extraction_method": "ocr_structured",
  "needs_review": false
}

Step 2: Readiness Scoring Engine (Deterministic — No LLM)

The scoring engine is deterministic, not AI-based — it applies rules to the extraction output. This ensures consistency and auditability.

Per-Field Score Calculation:

Condition	Field Score	Label
Extracted with confidence >= 0.85	1.0	✅ Confident
Extracted with confidence 0.70-0.84	0.7	⚠️ Low confidence
Extracted with confidence 0.50-0.69	0.3	⚠️ Needs review
Extracted with confidence < 0.50	0.0	❌ Unreliable
Not found / missing	0.0	❌ Missing

Document-Level Readiness Score Formula:

Document_Readiness_Score = Σ(field_score × field_weight) / Σ(field_weight)

Where field_weight is assigned per document type:

Weight	Meaning	Example Fields
3.0 (Critical)	Document is useless without this	invoice_number, bl_number, total_value, hs_code
2.0 (Important)	Needed for compliance, but document is still identifiable	date, incoterms, country_of_origin, gross_weight
1.0 (Standard)	Useful but not blocking	payment_terms, marks_and_numbers, dimensions

Document Status Mapping:

Readiness Score	Status	Visual	Meaning
>= 0.85	complete	✅	All critical fields extracted with high confidence
0.60 - 0.84	needs_attention	⚠️	Some fields missing or low confidence — human review recommended
< 0.60	non_compliant	❌	Critical fields missing or unreadable — document likely needs re-upload

Cross-Document Validation Rules (bonus checks):

Rule	Check	Impact
Invoice-PO match	Invoice total matches PO total (±5%)	Flags discrepancy if mismatch
Party consistency	Seller on invoice = shipper on B/L	Flags if names don't fuzzy-match (>80%)
HS code consistency	HS codes on invoice match PO line items	Flags mismatches
Weight consistency	Gross weight on packing list ≈ B/L weight (±10%)	Flags discrepancy
Value consistency	Declared value on customs declaration ≈ invoice total	Flags if >10% difference

Step 3: Shipment-Level Readiness Score

Shipment_Readiness_Score = (count of "complete" documents / total required documents) × 100

Displayed as a percentage on the dashboard. A shipment is "ready" when score = 100%.

Compliance Scoring — What Each Agent Does

Compliance scoring happens in Phase 2 and involves multiple specialized agents working in sequence.

Agent 1: Sanctions Screener Agent

Input: All party names from the shipment (importer, exporter, consignee, notify party) Model used: No LLM — uses fuzzy string matching (Levenshtein distance + Jaro-Winkler) against Neo4j sanctions nodes Output:

{
  "party_name": "ABC Trading Co.",
  "party_role": "exporter",
  "screening_result": "clear",  // or "potential_hit"
  "matches": [
    {
      "matched_name": "ABC Trading Company Ltd",
      "source_list": "ofac_sdn",
      "similarity_score": 0.87,
      "entity_id": "OFAC-12345",
      "programs": ["IRAN", "SDGT"]
    }
  ]
}

Scoring: Binary — clear or potential_hit. Any hit above the threshold (default 85%) blocks the shipment.

Agent 2: HS Classifier Agent

Input: Product descriptions from PO line items (where HS code is missing) Model used: Nova Pro (production) / Llama 3.1 8B (local) — needs reasoning for classification Tools: Knowledge Graph query (HS code hierarchy), RAG search (trade rulings, classification guidance) Output:

{
  "item_description": "Portable laptop computer, 15 inch, 16GB RAM",
  "candidates": [
    {"hs_code": "8471.30", "confidence": 0.92, "explanation": "Portable digital automatic data processing machines, weighing not more than 10 kg (HS Chapter 84, Section XVI). Classified under 8471.30 per GRI 1 and Note 5(A) to Chapter 84."},
    {"hs_code": "8471.41", "confidence": 0.15, "explanation": "Other data processing machines comprising a CPU and I/O unit. Less likely as item is portable."},
    {"hs_code": "8528.52", "confidence": 0.05, "explanation": "Monitors. Unlikely — item is a complete computer, not a display."}
  ]
}

Agent 3: Compliance Checker Agent

Input: Complete shipment data (documents, extracted fields, HS codes, trade corridor) Model used: Claude Sonnet 4.5 (production) — needs deep reasoning for regulatory interpretation Tools: Knowledge Graph (corridor requirements, regulations), RAG (trade rulings, regulatory guidance) Output: A compliance report with per-check results

Compliance Check Categories:

Category	What It Checks	Source
Document Completeness	Are all required documents present per the checklist?	Knowledge Graph (HS + corridor → required docs)
Field Validation	Do extracted fields meet format/value requirements? (e.g., valid HS code format, valid country codes, dates not expired)	Deterministic rules
Cross-Document Consistency	Do values match across documents? (invoice total vs customs declaration, party names, HS codes)	Deterministic comparison
Regulatory Compliance	Does the shipment meet corridor-specific regulations? (e.g., Certificate of Origin required for preferential tariff, phytosanitary cert for food products)	Knowledge Graph + RAG (Claude 4.5 interprets)
Product Category Compliance	Does the shipment meet category-specific requirements? (e.g., FDA approval for pharma, FSSAI for food)	Knowledge Graph (product category → regulatory body → requirements)
Tariff Accuracy	Are declared HS codes correct? Do tariff rates match?	Knowledge Graph (HS → tariff rates for corridor)

Per-Check Output:

{
  "check_id": "reg_compliance_001",
  "check_type": "regulatory_compliance",
  "rule_reference": "19 CFR 10.411 — USMCA Certificate of Origin",
  "result": "fail",
  "severity": "blocking",
  "description": "Certificate of Origin is required when claiming preferential tariff rate under USMCA. Document is missing from shipment.",
  "source_regulation": "19 CFR Part 10, Subpart P — United States-Mexico-Canada Agreement",
  "source_url": "https://www.ecfr.gov/current/title-19/chapter-I/part-10/subpart-P",
  "remediation": "Upload a valid USMCA Certificate of Origin (CBP Form 434) signed by the exporter or producer.",
  "cross_references": [
    {"source": "USMCA Article 5.2", "agrees": true},
    {"source": "CBP Ruling HQ H301234", "agrees": true}
  ]
}

Shipment Compliance Score:

Overall Result	Condition
pass	All checks pass
warning	All checks pass but some have informational warnings (e.g., tariff rate may change soon)
fail	One or more checks fail with blocking severity
blocked	Sanctions screening returned a potential_hit that hasn't been resolved

The compliance report is stored as a structured JSON and can be exported as a PDF with full regulatory citations for audit purposes.

Scoring Data Flow Summary

graph TB
    subgraph "Phase 1 — Document Readiness"
        UPLOAD[Document Upload] --> TEXTRACT[Textract OCR<br/>Nova Lite extraction]
        TEXTRACT --> FIELDS[Extracted Fields<br/>+ Confidence Scores]
        FIELDS --> RULES[Deterministic Scoring Engine<br/>Weighted field scores]
        RULES --> DOC_SCORE[Document Score<br/>complete / needs_attention / non_compliant]
        DOC_SCORE --> SHIP_SCORE[Shipment Readiness %<br/>complete docs / total required]
    end

    subgraph "Phase 2 — Compliance"
        SHIP_SCORE --> SANCTIONS[Sanctions Screener<br/>Fuzzy match — no LLM]
        SANCTIONS --> HS_CHECK[HS Classifier<br/>Nova Pro — reasoning]
        HS_CHECK --> COMPLIANCE[Compliance Checker<br/>Claude 4.5 — deep reasoning]
        COMPLIANCE --> REPORT[Compliance Report<br/>pass / warning / fail / blocked]
    end

Loading

---