Merge branch 'main' into copilot/add-copilot-instructions

Author: EZoni

.github/copilot-instructions.md

@@ -21,6 +21,9 @@ synapse/
 │   ├── environment.yml     # Conda dependencies for ML
 │   └── environment-lock.yml
 ├── experiments/            # Experiment configs (cloned from private repos)
+├── tests/                  # Integration tests (ML pipeline)
+│   ├── test_ml_pipeline.py # Full train/save/load ML lifecycle test
+│   └── check_model.py      # Model checking utility
 ├── dashboard.Dockerfile    # Docker image for the GUI
 ├── ml.Dockerfile           # Docker image for ML training (CUDA 12.4)
 ├── publish_container.py    # Script to build & push Docker containers to NERSC registry
@@ -71,7 +74,20 @@ python publish_container.py --gui --ml
 
 ## Testing
 
-There is **no automated test suite** in this repository — no pytest, unittest, or similar framework is configured. There are no test files. Validation is done manually by running the dashboard or ML training scripts.
+There is no pytest/unittest framework configured, but `tests/test_ml_pipeline.py` exercises the full ML lifecycle (training → upload to MLflow → download → accuracy check). It requires a local MLflow server:
+
+```bash
+# Start a local MLflow server
+docker run -p 127.0.0.1:5000:5000 ghcr.io/mlflow/mlflow mlflow server --host 0.0.0.0
+
+# Run the test from the repository root
+python tests/test_ml_pipeline.py
+
+# Optionally restrict to a specific model type or config
+python tests/test_ml_pipeline.py --model NN --config_file experiments/synapse-bella-ip2
+```
+
+Dashboard validation is done manually by running the application.
 
 ## CI/CD
 
@@ -84,31 +100,31 @@ The only CI workflow is **CodeQL Advanced** (`.github/workflows/codeql.yml`), wh
 - Built on [Trame](https://kitware.github.io/trame/) — a Python framework for interactive web applications.
 - Uses the **manager pattern**: each feature area has a dedicated `*_manager.py` class that handles its UI components and business logic.
 - `state_manager.py` manages the global Trame server, state, and controller.
-- Data flows through MongoDB (PyMongo) for experiment and simulation data.
-- Data flows through MLflow for ML models.
+- Data flows through MongoDB (PyMongo) for experiment and simulation data.
+- Data flows through MLflow for ML models.
 - NERSC Superfacility API integration is in `sfapi_manager.py`.
 
 ### ML Training
 
 - `train_model.py` supports three model types: Gaussian Process (GP), Neural Network (NN), and Ensemble.
 - Uses PyTorch, BoTorch, and GPyTorch for model training.
 - CUDA is auto-detected for GPU acceleration.
-- Models are serialized and stored in an MLflow tracking server.
+- Models are serialized and stored in an MLflow tracking server.
 
 ### Data Storage
 
-- **MongoDB** is used for persistent data from experiments and simulations.
-- **MLflow** is used for persistent data from ML models.
+- **MongoDB** is used for persistent data from experiments and simulations.
+- **MLflow** is used for persistent data from ML models.
 - Database access requires SSH tunneling to NERSC when running locally.
-- Environment variables: `SF_DB_HOST`, `SF_DB_READONLY_PASSWORD` (dashboard), `SF_DB_ADMIN_PASSWORD` (ML training).
+- Environment variables: `SF_DB_HOST` (dashboard), `SF_DB_READONLY_PASSWORD` (dashboard and ML training), `AM_SC_API_KEY` (dashboard and ML training, required when MLflow tracking_uri is AmSC).
 
 ## Common Pitfalls and Workarounds
 
 1. **No `pyproject.toml` or `ruff.toml`**: Ruff uses default rules. Do not create these files unless the project explicitly adopts them.
 2. **Conda, not pip**: Dependencies are managed via `conda` and `conda-lock`, not `pip`. Do not add `requirements.txt` or modify `pyproject.toml` for dependencies. Update `environment.yml` in the relevant component directory and regenerate the lock file.
 3. **Separate environments**: The dashboard and ML components have independent Conda environments (`synapse-gui` and `synapse-ml`). Changes to dependencies must be made in the correct `environment.yml`.
 4. **Docker builds from root**: Dockerfiles reference paths relative to the repository root. Always run `docker build` from the repository root directory.
-5. **No test infrastructure**: Since there is no test framework, validate changes by running the linter (`ruff check .`) and verifying logic through code review.
+5. **Limited test infrastructure**: There is no pytest/unittest framework, but `tests/test_ml_pipeline.py` can validate ML changes end-to-end (requires a local MLflow server). Always run the linter (`ruff check .`) and verify logic through code review.
 6. **Experiment configs are external**: The `experiments/` directory contains cloned private repositories. These are not checked into this repository (excluded via `.gitignore`).
 7. **NERSC-specific infrastructure**: Much of the deployment depends on NERSC services (Spin, Superfacility API, Perlmutter). Code changes affecting deployment or data access should be tested against NERSC services when possible.
 

.pre-commit-config.yaml

@@ -1,7 +1,7 @@
 repos:
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version
-  rev: v0.15.2
+  rev: v0.15.7
   hooks:
     # Run the linter
     - id: ruff-check

README.md

@@ -25,7 +25,7 @@ To display ML predictions, the application requires the following:
 - **Simulation and experimental data points**: Each data point consists of values for the scalar inputs and outputs defined in the experiment configuration file.
 Data points are stored in a [MongoDB](https://www.mongodb.com/) database, where each experiment is represented by a separate collection.
 Experimental and simulation data points are stored in the same collection and are distinguished by the `experimental_flag` attribute.
-- **ML models**: Machine learning models that interpolate between data points and are stored in a separate MongoDB collection named `models`.
+- **ML models**: Machine learning models that interpolate between data points and are stored in [MLflow](https://mlflow.org/).
 - **Simulation movies** (optional): For certain experiments, users can click on simulation data points to visualize simulation movies.
 The corresponding MP4 files are stored in the Perlmutter shared file system at `/global/cfs/cdirs/m558/superfacility/simulation_data`.
 This directory is mounted on the container image running on Spin.

dashboard/README.md

@@ -54,10 +54,11 @@ conda-lock install --name synapse-gui environment-lock.yml
 
 2. Move to the [dashboard/](./) directory.
 
-3. Set up the database settings (read-only):
+3. Set up the database settings (read-only) and the AmSC MLflow API key:
    ```bash
    export SF_DB_HOST='127.0.0.1'
    export SF_DB_READONLY_PASSWORD='your_password_here'  # Use SINGLE quotes around the password!
+   export AM_SC_API_KEY='your_amsc_api_key_here'        # Required when MLflow tracking_uri is AmSC
    ```
 
 4. Activate the conda environment `synapse-gui`:
@@ -85,11 +86,11 @@ conda-lock install --name synapse-gui environment-lock.yml
 
 4. Run the Docker container:
    ```bash
-   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' synapse-gui
+   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' synapse-gui
    ```
    For debugging, you can enter the container without starting the app:
    ```bash
-   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -it synapse-gui bash
+   docker run --network=host -v /etc/localtime:/etc/localtime -v $PWD/ml:/app/ml -e SF_DB_HOST='127.0.0.1' -e SF_DB_READONLY_PASSWORD='your_password_here' -e AM_SC_API_KEY='your_amsc_api_key_here' -it synapse-gui bash
    ```
    Note that `-v /etc/localtime:/etc/localtime` is necessary to synchronize the time zone in the container with the host machine.
 

dashboard/app.py

@@ -6,7 +6,7 @@
 from trame.ui.vuetify3 import SinglePageWithDrawerLayout
 from trame.widgets import plotly, router, vuetify3 as vuetify, html
 
-from model_manager import ModelManager
+from model_manager import ModelManager, model_type_tag_dict
 from outputs_manager import OutputManager
 from optimization_manager import OptimizationManager
 from parameters_manager import ParametersManager
@@ -16,6 +16,7 @@
 from error_manager import error_panel, add_error
 from utils import (
     data_depth_panel,
+    load_config_dict,
     load_experiments,
     load_database,
     load_data,
@@ -64,14 +65,18 @@ def update(
         state.experiment
     )
     # load data
-    db = load_database(state.experiment)
-    exp_data, sim_data = load_data(db)
+    config_dict = load_config_dict(state.experiment)
+    db = load_database(config_dict)
+    exp_data, sim_data = load_data(db, state.experiment, state.experiment_date_range)
     # reset output
     if reset_output:
         out_manager = OutputManager(output_variables)
     # reset model
     if reset_model:
-        mod_manager = ModelManager(db)
+        mod_manager = ModelManager(
+            config_dict=config_dict,
+            model_type_tag=model_type_tag_dict[state.model_type],
+        )
         opt_manager = OptimizationManager(mod_manager)
     # reset parameters
     if reset_parameters:
@@ -257,7 +262,8 @@ def find_simulation(event, db):
 
 
 def open_simulation_dialog(event):
-    db = load_database(state.experiment)
+    config_dict = load_config_dict(state.experiment)
+    db = load_database(config_dict)
     try:
         data_directory, file_path = find_simulation(event, db)
         state.simulation_video = file_path.endswith(".mp4")

dashboard/error_manager.py

@@ -1,8 +1,13 @@
 from trame.widgets import vuetify3 as vuetify, html
-from state_manager import state
+from state_manager import state, server
 
 
 def add_error(title, msg):
+    if not server.running:
+        # Outside of a Trame app (e.g. check_model.py), raise a Python error
+        # to surface the error to the caller.
+        raise RuntimeError(f"{title}: {msg}")
+    # Otherwise: Inside a Trame app, add the error to the state.
     state.errors.append(
         {
             "id": state.error_counter,