Skip to content

project-kickoff.md

Latest commit

 

History

History
301 lines (240 loc) · 13.3 KB

project-kickoff.md

File metadata and controls

301 lines (240 loc) · 13.3 KB
Raw

Crypto Reference Data Platform - Implementation Plan Request

Project Overview

I'm building a production-quality crypto reference data platform positioned as "GoldenSource for Crypto" - a modern, streamlined alternative using industry best practices. The goal is to create a clean, maintainable reference data system that demonstrates staff-level data platform engineering excellence.

Core Value Proposition: Purpose-built reference data system for crypto markets that handles the unique challenges crypto presents (symbol chaos, high-frequency changes, cross-venue mapping, perpetual contracts) using modern data architecture patterns and industry-standard practices.

Technical Context

Technology Stack:

  • Storage: Apache Iceberg (lakehouse format with ACID transactions, time-travel)
  • Streaming: Kafka + Schema Registry
  • Object Storage: MinIO (S3-compatible)
  • Transformations: DBT (data modeling and transformations)
  • API Layer: FastAPI (modern Python web framework)
  • Orchestration: Makefile-driven workflows
  • Containerization: Docker Compose for reproducible environments

Architecture Pattern: Medallion architecture (Bronze/Silver/Gold layers)

Core Problem Statement

Crypto reference data is fundamentally harder than traditional finance reference data:

  1. No central numbering authority - Same asset has different symbols across exchanges (BTC, XBT, BTCUSDT, BTC-PERP)
  2. Perpetual contract complexity - Each exchange has proprietary specifications (funding rates, tick sizes, settlement methods, index composition)
  3. High-frequency changes - Exchanges change specifications mid-day, launch new pairs daily, delist with minimal notice
  4. Corporate actions - Token swaps, hard forks, airdrops, redenominations require accurate tracking
  5. Cross-venue symbology hell - Trading the same instrument across venues requires robust mapping

Business Impact: Incorrect reference data causes failed trades, compliance violations, and incorrect P&L calculations. A robust reference data platform is mission-critical infrastructure.

Technical Requirements

Data Model Requirements

Phase 1: Core Entities

  1. Instruments Table (Silver Layer - SCD Type 2 + Bitemporal)

    • Exchange-native instrument specifications (spot, perpetual, futures)
    • Contract specifications: tick_size, lot_size, min_notional, max_leverage
    • Funding parameters for perpetual contracts
    • Bitemporal design:
      • valid_from, valid_to → Business time (when specs were actually effective)
      • record_created_at, record_updated_at → System time (when we learned about it)
    • Complete audit trail: source_system, change_reason, changed_by
    • Support for late-arriving corrections without corrupting historical queries

  2. Symbology Master Table (Gold Layer)

    • Canonical instrument identifiers (internal standard)
    • Cross-exchange symbol mappings (binance_symbol, kraken_symbol, bybit_symbol, etc.)
    • Asset classification (base_asset, quote_asset, asset_class)
    • Hierarchical relationships (e.g., perpetuals derived from spot)

  3. Trading Calendars Table (Silver Layer)

    • Exchange maintenance windows, funding events, settlement times
    • Scheduled vs actual times (exchanges frequently run late)
    • Affected symbols and impact assessment

Architecture Requirements

Data Sources → Kafka (Bronze) → DBT Transformations → Iceberg (Silver/Gold) → FastAPI

Ingestion Layer:

  • Poll exchange REST APIs (/exchangeInfo, /AssetPairs) for contract specifications
  • WebSocket subscriptions for real-time specification updates (optional enhancement)
  • Support for manual CSV uploads (corrections, overrides, emergency fixes)
  • Idempotent ingestion (safe to replay)
  • Schema validation at ingestion boundary

Storage Layer:

  • Bronze: Append-only event log preserving full API responses (raw JSON)
  • Silver: Normalized relational schema with SCD Type 2 logic, data quality constraints
  • Gold: Symbology master, aggregations, derived analytics

API Layer (REST):

  • Point-in-time queries: GET /instruments?as_of=2024-01-15T10:00:00Z
  • Symbology lookup: GET /symbology/{canonical_id}
  • Reverse symbology: GET /symbology/resolve?exchange=binance&symbol=BTCUSDT
  • Calendar queries: GET /calendars/{exchange}?start=...&end=...
  • Manual override endpoint: POST /instruments (with authorization)
  • Audit trail endpoint: GET /instruments/{exchange}/{symbol}/history
  • Health checks and metadata endpoints

Staff-Level Implementation Standards

Code Quality & Structure

Expected Standards:

  • Type safety: Full type hints in Python (mypy strict mode)
  • Error handling: Comprehensive exception handling with proper logging
  • Testing: Unit tests, integration tests, contract tests for API
  • Code organization: Clear separation of concerns, single responsibility principle
  • Dependency injection: Avoid global state, make dependencies explicit
  • Configuration management: Environment-based config (12-factor app principles)

Project Structure Best Practices

ke-reference-data-platform/
├── src/
│   ├── ingestion/           # Data ingestion components
│   │   ├── sources/         # Exchange-specific adapters
│   │   ├── producers/       # Kafka producers
│   │   └── schemas/         # Avro/JSON schemas
│   ├── api/                 # FastAPI application
│   │   ├── routers/         # API route definitions
│   │   ├── models/          # Pydantic models
│   │   ├── dependencies/    # Dependency injection
│   │   └── middleware/      # Auth, logging, error handling
│   ├── common/              # Shared utilities
│   │   ├── config/          # Configuration management
│   │   ├── logging/         # Structured logging
│   │   └── db/              # Database connections
│   └── cli/                 # Command-line tools
├── dbt/
│   ├── models/
│   │   ├── bronze/          # Raw data models
│   │   ├── silver/          # Normalized models
│   │   └── gold/            # Analytics models
│   ├── macros/              # Reusable SQL macros
│   ├── tests/               # Data quality tests
│   └── docs/                # DBT documentation
├── tests/
│   ├── unit/                # Fast, isolated tests
│   ├── integration/         # Cross-component tests
│   └── e2e/                 # End-to-end scenarios
├── infrastructure/
│   ├── docker/              # Dockerfiles
│   ├── compose/             # Docker Compose files
│   └── scripts/             # Setup/deployment scripts
├── docs/
│   ├── architecture/        # ADRs, diagrams
│   ├── api/                 # API documentation
│   ├── runbooks/            # Operational procedures
│   └── development/         # Developer guides
├── Makefile                 # Standard workflows
├── pyproject.toml           # Python project config (Poetry/pip)
├── .pre-commit-config.yaml  # Code quality automation
└── README.md                # Project overview

Documentation Standards

First-Class Documentation Required:

  1. Architecture Decision Records (ADRs)

    • Use MADR (Markdown Any Decision Records) format
    • Document: context, decision, consequences, alternatives considered
    • Required ADRs:
      • Bitemporal vs snapshot pattern
      • Polling vs streaming for reference data
      • Manual override workflow design
      • API authentication/authorization approach
      • Schema evolution strategy
      • Caching strategy (if implemented)

  2. API Documentation

    • OpenAPI/Swagger spec (auto-generated from FastAPI)
    • Request/response examples for all endpoints
    • Error code catalog
    • Rate limiting documentation

  3. Runbooks

    • System startup/shutdown procedures
    • Troubleshooting guides
    • Manual override procedures
    • Data correction workflows
    • Monitoring and alerting

  4. Developer Documentation

    • Local development setup guide
    • Testing strategy and how to run tests
    • Code contribution guidelines
    • Architecture overview with diagrams
    • Data flow diagrams

  5. Data Documentation

    • Entity-relationship diagrams
    • Schema documentation (auto-generated from DBT)
    • Sample queries for common use cases
    • Data lineage documentation

Key Design Principles

1. Bitemporal Modeling (Critical Differentiator)

Why it matters: Reference data changes are often announced before effective, and corrections arrive late.

Requirements:

  • Business time (valid_from/valid_to) tracks when specs were actually effective
  • System time (record_created_at) tracks when we learned about changes
  • Point-in-time queries must return accurate state as-of any timestamp
  • Late-arriving corrections must not corrupt historical queries
  • Full audit trail for regulatory compliance

Example Scenario:

Jan 10, 9am:  Binance announces tick size change effective Jan 15
Jan 11, 3pm:  We ingest the announcement (record_created_at)
Jan 15, 12am: Change goes live (valid_from)
Jan 16, 8am:  Exchange corrects: "Actually Jan 14, 11pm" (new record_created_at, updated valid_from)

Query as_of=2024-01-14T22:00:00Z must return correct historical state.

2. Idempotency and Replay Safety

  • Ingestion must be safe to replay (same input → same output)
  • Use deterministic IDs (hash of content, not auto-increment)
  • Detect duplicates before writing to Bronze
  • Transformations must be pure functions

3. Data Quality as Code

  • Schema validation at Bronze ingestion boundary
  • DBT data quality tests (not null, unique, relationships)
  • Freshness checks (alert if data stops flowing)
  • Anomaly detection (e.g., tick size changes >10x require manual review)

4. API Design Excellence

  • RESTful conventions (proper use of HTTP verbs, status codes)
  • Consistent error response format
  • Pagination for large result sets
  • API versioning strategy (URL-based: /v1/instruments)
  • Rate limiting and authentication headers
  • Comprehensive request/response logging

5. Observability by Default

  • Structured logging (JSON format)
  • Distributed tracing headers (correlation IDs)
  • Metrics endpoints (Prometheus format)
  • Health checks (liveness, readiness)
  • Audit logs for all manual overrides

Implementation Plan Request

Please create a comprehensive implementation plan that:

1. Technical Design

  • Data model specifications (DDL for all Iceberg tables with partitioning strategy)
  • API contract definitions (OpenAPI spec or detailed endpoint descriptions)
  • Kafka topic schemas (Avro schemas for all topics)
  • DBT model structure (DAG dependencies, incremental vs full refresh strategy)

2. Implementation Phases

  • Phase breakdown with clear completion criteria for each phase
  • Dependency order (what must be built first, what can be parallel)
  • Testing strategy for each phase (how to validate correctness)
  • Rollback plan (how to handle breaking changes)

3. Architecture Decisions

  • Identify all ADRs needed with suggested structure
  • Tradeoff analysis for key decisions (e.g., polling interval vs API rate limits)
  • Alternatives considered for major design choices
  • Performance considerations (query patterns, indexing strategy)

4. Quality Standards

  • Testing pyramid (unit/integration/e2e coverage targets)
  • Code quality checks (linters, formatters, type checking)
  • CI/CD pipeline structure (what to automate)
  • Documentation templates (ADR template, runbook template)

5. Operational Readiness

  • Monitoring strategy (what metrics to track)
  • Alerting rules (when to notify operators)
  • Manual override workflow (approval process, audit requirements)
  • Data correction procedures (how to fix bad data)

Success Criteria

The implementation should demonstrate:

  1. Correctness: Bitemporal queries return accurate historical state
  2. Robustness: System handles late data, corrections, and edge cases gracefully
  3. Maintainability: Code is clean, well-tested, and documented
  4. Operability: Clear runbooks, comprehensive monitoring, debuggable
  5. Scalability: Design can handle 100+ exchanges, 10k+ instruments (even if MVP starts smaller)

Philosophical Approach

  • Simplicity over cleverness: Prefer straightforward solutions
  • Explicit over implicit: Make assumptions and constraints visible
  • Progressive disclosure: Start simple, add complexity only when needed
  • Documentation as code: ADRs, diagrams, and runbooks live with the code
  • Production-grade from day one: Build it right, not fast

Request

Please provide a detailed, production-quality implementation plan that prioritizes:

  1. Getting the bitemporal modeling right (this is the core technical differentiator)
  2. Industry-standard project structure (following Python/data engineering best practices)
  3. Comprehensive documentation (ADRs, API docs, runbooks, developer guides)
  4. Operational excellence (observability, data quality, manual override workflows)
  5. Clean, maintainable code (type safety, testing, separation of concerns)

The plan should be concrete and actionable, with specific technical recommendations for data models, API contracts, testing strategies, and architectural decisions.