Add cloud-sync service and UI, local table picker and selection API

Expose local table management: add GET /api/local-tables and POST /api/select-local-table endpoints to list available local tables and switch the active local source (selection is blocked while a sync is running and it invalidates unsynced cache/state). Implement SyncEngine.list_local_tables to query the local DB information_schema for tables matching ^wfr[0-9].

Update the web UI to include a local table dropdown, loadLocalTables/selectLocalTable JS handlers, polling integration, and minor label tweak ("New" -> "New cloud").

Normalize the default local table name from wfr26test_base to wfr26base across config, .env.macbook and docker-compose defaults (also normalizing casing for TIMESCALE_TABLE), so the composed services and environment use the new base table name.

Introduce a new cloud-sync microservice to copy rows from local TimescaleDB to a cloud TimescaleDB. Adds a FastAPI-based API and static web UI (static/index.html), a SyncEngine (sync.py) that uses psycopg2 to upsert rows and manage dynamic signal columns, a simple config module, requirements, and a Dockerfile exposing port 8092 with a healthcheck. Integrates the service into docker-compose.macbook-base.yml (service definition, env vars, depends_on), adds deploy/pg_hba.conf and mounts it into the TimescaleDB container with a command to load the custom hba, and updates deploy/.env.macbook with example CLOUD_POSTGRES_DSN and LOCAL_TABLE overrides. Cloud DSN is optional; sync is gated on CLOUD_POSTGRES_DSN being configured.

Fix cloud-sync cursor; update deploy and status

Handle named-server cursor behavior in cloud-sync: fetch the first batch before reading cursor.description so column names and signal columns are populated correctly, and iterate fetchmany() in a proper loop to avoid missing batches. Update .env.macbook to point DBC_HOST_PATH to the secret DBC location. Clarify docker-compose.macbook-base usage and environment file path in comments, and add status and Cloud Sync access URLs. Enhance status UI to show the DBC file name (shortened), color-code it (red for example.dbc), and retain the existing warning banner behavior for the example DBC.

Use DBC_DISPLAY_NAME and don't clear frozenRef

Preserve frozenRef when clearing traces and add a configurable DBC display name.

Optional prompt-guide for CI

Author: haoruizhou

pecan/src/pages/Trace.tsx

@@ -458,7 +458,6 @@ function Trace() {
 
   const handleClear = useCallback(() => {
     clearTrace();
-    frozenRef.current = [];
   }, [clearTrace]);
 
   // Handle Easter Egg Trigger

server/installer/sandbox/Dockerfile

@@ -12,7 +12,9 @@ RUN python -c "from langchain_community.embeddings import FastEmbedEmbeddings; F
 # Copy application code
 COPY code_generator.py .
 COPY stats_report.py .
-COPY prompt-guide.txt prompt-guide.txt.example ./
+COPY prompt-guide.txt.example ./
+# prompt-guide.txt is optional (local customization); fall back to example if absent
+RUN if [ ! -f prompt-guide.txt ]; then cp prompt-guide.txt.example prompt-guide.txt; fi
 
 # Create directories for generated code, ChromaDB, and cache
 RUN mkdir -p /app/generated /app/chroma_db /app/cache

universal-telemetry-software/cloud-sync/Dockerfile

@@ -0,0 +1,19 @@
+FROM python:3.11-slim
+
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1
+
+WORKDIR /app
+
+COPY requirements.txt /tmp/requirements.txt
+RUN pip install --no-cache-dir -r /tmp/requirements.txt
+
+COPY config.py sync.py app.py ./
+COPY static/ ./static/
+
+EXPOSE 8092
+
+HEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8092/api/status')"
+
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8092"]

universal-telemetry-software/cloud-sync/app.py

@@ -0,0 +1,216 @@
+import logging
+import time
+from datetime import datetime, timezone
+from typing import Optional
+
+from fastapi import BackgroundTasks, FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import RedirectResponse
+from fastapi.staticfiles import StaticFiles
+from pydantic import BaseModel
+
+import config
+from sync import SyncEngine
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s %(name)s %(levelname)s %(message)s",
+)
+logger = logging.getLogger("cloud-sync")
+
+app = FastAPI(title="WFR Cloud Sync")
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+app.mount("/static", StaticFiles(directory="static"), name="static")
+
+engine = SyncEngine()
+
+# ── Sync state (module-level, single process) ─────────────────────────────────
+
+_sync_state: dict = {
+    "running": False,
+    "rows_done": 0,
+    "rows_total": 0,
+    "last_sync_iso": None,      # ISO timestamp of last completed sync
+    "last_sync_rows": None,     # row count of last completed sync
+    "last_sync_elapsed": None,  # seconds
+    "last_error": None,
+    # cached from last status call so /api/status is fast
+    "_cloud_cursor": None,
+    "_unsynced_count": None,
+    "_unsynced_ts": 0.0,        # monotonic time of last unsynced_count fetch
+}
+
+_UNSYNCED_CACHE_TTL = 30.0  # seconds
+
+
+@app.get("/")
+def root():
+    return RedirectResponse(url="/static/index.html")
+
+
+@app.get("/api/status")
+def status():
+    # Local count — always fresh (fast local query)
+    try:
+        local_count = engine.get_local_count()
+    except Exception as e:
+        local_count = None
+        logger.warning(f"get_local_count failed: {e}")
+
+    # Unsynced count — cached with TTL to avoid hammering cloud on every poll
+    now = time.monotonic()
+    if now - _sync_state["_unsynced_ts"] > _UNSYNCED_CACHE_TTL and not _sync_state["running"]:
+        try:
+            cursor = engine.get_cloud_cursor()
+            _sync_state["_cloud_cursor"] = cursor.isoformat() if cursor else None
+            _sync_state["_unsynced_count"] = engine.get_unsynced_count(cursor)
+            _sync_state["_unsynced_ts"] = now
+        except Exception as e:
+            logger.warning(f"unsynced_count fetch failed: {e}")
+
+    cloud_configured = bool(config.CLOUD_POSTGRES_DSN)
+
+    return {
+        "local_count": local_count,
+        "local_table": config.LOCAL_TABLE,
+        "cloud_table": engine.cloud_table,
+        "cloud_configured": cloud_configured,
+        "cloud_cursor": _sync_state["_cloud_cursor"],
+        "unsynced_count": _sync_state["_unsynced_count"],
+        "last_sync_iso": _sync_state["last_sync_iso"],
+        "last_sync_rows": _sync_state["last_sync_rows"],
+        "last_sync_elapsed": _sync_state["last_sync_elapsed"],
+        "last_error": _sync_state["last_error"],
+        "sync_running": _sync_state["running"],
+    }
+
+
+@app.post("/api/check-cloud")
+def check_cloud():
+    result = engine.check_cloud_connection()
+    return result
+
+
+@app.post("/api/sync")
+def trigger_sync(background_tasks: BackgroundTasks):
+    if _sync_state["running"]:
+        raise HTTPException(status_code=409, detail="Sync already in progress")
+
+    if not config.CLOUD_POSTGRES_DSN:
+        raise HTTPException(status_code=400, detail="CLOUD_POSTGRES_DSN not configured")
+
+    _sync_state["running"] = True
+    _sync_state["rows_done"] = 0
+    _sync_state["rows_total"] = 0
+    _sync_state["last_error"] = None
+
+    background_tasks.add_task(_run_sync)
+    return {"status": "started"}
+
+
+@app.get("/api/sync-status")
+def sync_status():
+    return {
+        "running": _sync_state["running"],
+        "rows_done": _sync_state["rows_done"],
+        "rows_total": _sync_state["rows_total"],
+        "last_sync_iso": _sync_state["last_sync_iso"],
+        "last_sync_rows": _sync_state["last_sync_rows"],
+        "last_sync_elapsed": _sync_state["last_sync_elapsed"],
+        "last_error": _sync_state["last_error"],
+    }
+
+
+def _progress_cb(rows_done: int, rows_total: int) -> None:
+    _sync_state["rows_done"] = rows_done
+    _sync_state["rows_total"] = rows_total
+
+
+class SelectTablePayload(BaseModel):
+    table: str
+
+
+class CreateTablePayload(BaseModel):
+    table: str
+
+
+@app.get("/api/local-tables")
+def list_local_tables():
+    """List existing local tables (tables matching ^wfr[0-9] on the local DB)."""
+    tables = engine.list_local_tables()
+    return {"tables": tables, "current": engine.local_table}
+
+
+@app.post("/api/select-local-table")
+def select_local_table(payload: SelectTablePayload):
+    """Switch the active local source table for the next sync."""
+    if _sync_state["running"]:
+        raise HTTPException(status_code=409, detail="Cannot change table while sync is running")
+    name = payload.table.lower().strip()
+    if not name:
+        raise HTTPException(status_code=400, detail="Table name is required")
+    engine.local_table = name
+    # Invalidate unsynced cache
+    _sync_state["_unsynced_ts"] = 0.0
+    _sync_state["_unsynced_count"] = None
+    _sync_state["_cloud_cursor"] = None
+    return {"selected": name}
+
+
+@app.get("/api/cloud-tables")
+def list_cloud_tables():
+    """List existing cloud tables (tables matching ^wfr[0-9] on the cloud DB)."""
+    tables = engine.list_cloud_tables()
+    return {"tables": tables, "current": engine.cloud_table}
+
+
+@app.post("/api/cloud-tables")
+def create_cloud_table(payload: CreateTablePayload):
+    """Create a new cloud hypertable."""
+    name = payload.table.lower().strip()
+    if not name:
+        raise HTTPException(status_code=400, detail="Table name is required")
+    try:
+        engine.create_cloud_table(name)
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+    # Invalidate unsynced cache
+    _sync_state["_unsynced_ts"] = 0.0
+    return {"created": name}
+
+
+@app.post("/api/select-table")
+def select_table(payload: SelectTablePayload):
+    """Switch the active cloud table for the next sync."""
+    if _sync_state["running"]:
+        raise HTTPException(status_code=409, detail="Cannot change table while sync is running")
+    name = payload.table.lower().strip()
+    if not name:
+        raise HTTPException(status_code=400, detail="Table name is required")
+    engine.cloud_table = name
+    # Invalidate unsynced cache so next /api/status recalculates
+    _sync_state["_unsynced_ts"] = 0.0
+    _sync_state["_unsynced_count"] = None
+    _sync_state["_cloud_cursor"] = None
+    return {"selected": name}
+
+
+def _run_sync() -> None:
+    try:
+        result = engine.sync(progress_cb=_progress_cb)
+        _sync_state["last_sync_iso"] = datetime.now(timezone.utc).isoformat()
+        _sync_state["last_sync_rows"] = result["rows_synced"]
+        _sync_state["last_sync_elapsed"] = result["elapsed_s"]
+        _sync_state["last_error"] = None
+        # Invalidate unsynced cache
+        _sync_state["_unsynced_ts"] = 0.0
+    except Exception as e:
+        logger.error(f"Sync failed: {e}")
+        _sync_state["last_error"] = str(e)
+    finally:
+        _sync_state["running"] = False

universal-telemetry-software/cloud-sync/config.py

@@ -0,0 +1,12 @@
+import os
+
+LOCAL_POSTGRES_DSN = os.getenv(
+    "LOCAL_POSTGRES_DSN",
+    "postgresql://wfr:wfr_password@timescaledb:5432/wfr",
+)
+LOCAL_TABLE = os.getenv("LOCAL_TABLE", "wfr26base").lower()
+
+CLOUD_POSTGRES_DSN = os.getenv("CLOUD_POSTGRES_DSN", "")
+CLOUD_TABLE = os.getenv("CLOUD_TABLE", "wfr26").lower()
+
+SYNC_BATCH_SIZE = int(os.getenv("SYNC_BATCH_SIZE", "5000"))

universal-telemetry-software/cloud-sync/requirements.txt

@@ -0,0 +1,3 @@
+fastapi==0.115.4
+uvicorn[standard]==0.32.1
+psycopg2-binary>=2.9.9