Skip to content

Architecture.md

Latest commit

 

History

History
128 lines (97 loc) · 5.87 KB

Architecture.md

File metadata and controls

128 lines (97 loc) · 5.87 KB
Raw

Time Series Data Catalog Application Design Document

1. Overview

The Time Series Data Catalog Application is a web-based platform designed for managing, analyzing, and visualizing time-indexed datasets. It empowers users, particularly data scientists and analysts, to efficiently merge datasets, apply complex mathematical operations, filter data, and interactively explore time series data through rich visualizations and GIS-based mapping. This application addresses the challenge of fragmented time series data by providing a unified environment for data discovery, manipulation, and analysis.

2. Technology Stack

  • Backend: Python with FastAPI (for non-app services)
  • Frontend: Panel (Holoviz) for UI components and interactivity
  • Data Processing: Pandas for efficient handling of time-indexed dataframes
  • Mathematical Computation: NumPy, SciPy, and a safe math evaluation library (instead of potentially insecure eval())
  • Visualization: Holoviews, Geoviews, and Bokeh for interactive plots and maps
  • Data Storage: Azure File Share (supporting CSV, HDF5, etc.) and potentially Azure Database for metadata.
  • Deployment: Azure Apps (containerized deployment)
  • Authentication: Managed through Azure Authentication

3. Core Features

3.1 Data Management

  • Data Unification: Merge and concatenate data catalogs with user-specified join columns and concatenation options.
  • Data Storage: Support for file-based (CSV, HDF5) and remote (Blobs, HTTPS) storage.
  • Metadata Management: Unified catalog with metadata support, including data types and provenance.

3.2 Data Manipulation

  • Mathematical Operations: User-defined expressions using NumPy, SciPy, and a safe math evaluation library.
  • Data Selection & Filtering: Filtering based on time range, IDs, variables, and geospatial criteria.
  • Transformations: Mapping, reshaping, resampling, and rolling window functions for data transformation.
  • Data Quality Checks: Handling missing data, outlier detection, and data consistency checks.

3.3 Visualization

  • Time Series Plotting: Interactive visualizations using Holoviews with multi-selection support.
  • GIS-Based Map View: Spatial representation of datasets (points, lines, polygons) with region-based filtering.
  • Selection Synchronization: Data selections reflected across all views.
  • Replayable Actions: Record and replay user interactions.

3.4 User Interaction & Controls

  • UI Widgets: Time range pickers, filters, multi-selection options, and a user-friendly math expression builder.
  • Data Export: Save selections, operations, and views as YAML configurations.
  • Annotation & Flagging: Mark data points, add notes, and track edits with user roles and permissions.
  • Collaboration & Sharing: Generate permalinks for dataset views and enable real-time collaborative annotation.
  • User Feedback: Progress indicators and error messages.

4. Data Catalog Schema

Each entry in the data catalog refers to a single pandas DataFrame, typically time-indexed. Key attributes:

  • ID: Unique identifier (string)
  • Name: Human-readable name (string)
  • Time Start / End: Time range of the data (datetime)
  • Source: Storage location (CSV, HDF5, etc.) (string)
  • Variable(s): Data columns (list of strings)
  • Geometry: GIS reference (point, line, polygon) (GeoJSON)
  • Metadata: Additional attributes (JSON)
  • Data Provenance: Source, processing steps (JSON)

5. UI Layout (Panel-Based)

5.1 Sidebar (Filters & Controls)

  • Time Range Selector (pn.widgets.DatetimeRangePicker)
  • Variable & ID Selection (pn.widgets.MultiChoice)
  • Geometry Filter (pn.widgets.Checkbox, pn.widgets.Select)
  • Apply / Reset Filters (pn.widgets.Button)
  • Save / Load Views (pn.widgets.FileInput, pn.widgets.Button)

5.2 Main Panel (Tabbed Layout)

  • 📜 Catalog View:
    • Table Representation (pn.widgets.Tabulator) with filtering, sorting, and search.
    • Metadata Display for dataset attributes.
  • 📈 Time Series View:
    • Interactive Plots (hv.Curve, hv.Scatter)
    • Multi-Selection Support for datasets.
    • Options to visualize missing data and outliers.
  • 🌍 Map View:
    • Geospatial Representation (gv.Points, gv.Polygons)
    • Region-Based Filtering.
  • ⚙️ Operations (Math & Merging):
    • Math Expression Input (pn.widgets.TextAreaInput or visual builder).
    • Apply Transformations (pn.widgets.Button).

6. Data Processing & Backend

  • FastAPI Services for retrieving and processing data.
  • Catalog Adapters for different storage formats.
  • Safe expression evaluation using a dedicated library.
  • Merging & Concatenation logic to unify datasets.
  • Asynchronous processing for long-running operations.
  • Caching mechanisms for performance improvement.

7. Deployment & Hosting

  • Containerized Deployment on Azure Apps.
  • Azure File Share for persistent data storage.
  • Azure Database for metadata storage.
  • Azure Authentication for secure access.
  • Monitoring & Logging strategies.
  • CI/CD pipelines.

8. Next Steps

  1. Prototype UI Implementation with Panel: (High Priority)
  2. Develop Backend APIs for Data Retrieval & Processing: (High Priority)
  3. Implement Safe Math Expression Evaluation: (High Priority)
  4. Enhance GIS Mapping Capabilities:
  5. Implement Data Quality Checks:
  6. Implement Collaborative Features (annotations, version control):
  7. Implement Asynchronous Processing and Caching:
  8. Setup Monitoring, Logging, and CI/CD:
  9. User Testing and Feedback Collection:

9. Security Considerations

  • Replace eval() with a secure math evaluation library.
  • Implement user authentication and authorization.
  • Secure data storage and transmission.
  • Regular security audits.

10. Scalability

  • Use scalable database solutions for metadata.
  • Implement load balancing for backend services.
  • Optimize data processing and caching.