Add documentation

.gitignore

@@ -6,3 +6,6 @@ experiments/*/
 
 # JetBrains IDE idea directories
 .idea/
+
+# Documentation
+docs/build/

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/docs.yml

@@ -0,0 +1,12 @@
+name: synapse-docs
+
+channels:
+  - conda-forge
+  - nodefaults
+
+dependencies:
+  - myst-parser
+  - sphinx
+  - sphinx-autobuild
+  - sphinx-book-theme
+  - sphinx-copybutton

docs/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

docs/source/conf.py

@@ -0,0 +1,30 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "Synapse"
+copyright = "BSD-3-Clause-LBNL"
+author = "Arjun Dhamrait, Andrea Diaz, Marco Garten, Axel Huebl, Revathi Jambunathan, Remi Lehe, Ethan Rodriguez, Olga Shapoval, Jean-Luc Vay, Edoardo Zoni"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ["myst_parser", "sphinx_copybutton"]
+myst_heading_anchors = 2
+
+templates_path = ["_templates"]
+exclude_patterns = []
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_book_theme"
+html_theme_options = {
+    "show_navbar_depth": 1,
+    "max_navbar_depth": 1,
+}
+html_static_path = []

docs/source/dashboard.md

@@ -0,0 +1,27 @@
+# Dashboard
+
+The dashboard is a Trame application rooted at `dashboard/app.py`.
+It discovers experiments from `experiments/synapse-*`, reads each experiment's `config.yaml`, connects to MongoDB, loads MLflow models, and builds the GUI used to inspect data and launch jobs.
+
+## Main Managers
+
+- `state_manager.py`: shared Trame server, state, controller, and startup defaults.
+- `model_manager.py`: MLflow model lookup, download, evaluation, and model training launch.
+- `parameters_manager.py`: input sliders, parameter bounds, and single-simulation launch.
+- `outputs_manager.py`: displayed output selection.
+- `optimization_manager.py`: model-based input optimization with SciPy.
+- `calibration_manager.py`: simulation-to-experiment variable conversion.
+- `sfapi_manager.py`: Superfacility API credential upload, Perlmutter status, and job monitoring.
+- `error_manager.py`: user-visible error collection.
+- `utils.py`: config loading, database access, date filters, and Plotly figures.
+
+## Views
+
+- `/`: experiment selection, plots, parameter controls, optimization, ML controls, calibration controls, and errors.
+- `/hpc`: NERSC Superfacility API credential and Perlmutter status panel.
+- `/chat`: embedded assistant route for experiment support; currently backed by [synapse-chat.lbl.gov](https://synapse-chat.lbl.gov/).
+
+## NERSC Credentials
+
+Simulation and ML training launches require a Superfacility API key file uploaded through the dashboard.
+The file must be PEM-formatted and include the Superfacility API client ID as the first line, followed by the private key.

docs/source/data-model.md

@@ -0,0 +1,29 @@
+# Data Model
+
+MongoDB stores experiment and simulation records.
+The collection name should match `experiment` in `config.yaml`.
+
+## Record Types
+
+- `experiment_flag: 1`: experimental data.
+- `experiment_flag: 0`: simulation data.
+
+## Required Fields
+
+Each record must contain fields for the configured input and output variables, named to match `config.yaml`.
+
+Simulation records may use simulation-space names when `simulation_calibration` maps them back to experimental names.
+
+## Optional Fields
+
+The dashboard uses these when present:
+
+- `date`: filtering and hover text for experimental records.
+- `scan_number`: hover text.
+- `shot_number`: hover text.
+- `_id`: hover text and lookup for linked simulation media, such as MP4 files described in [Simulation Outputs](simulations.md#simulation-outputs).
+
+## Date Filtering
+
+Dashboard date filtering applies only to experimental records.
+Simulation records are loaded without the date filter.

docs/source/deployment.md

@@ -0,0 +1,34 @@
+# Deployment
+
+Synapse is deployed with Docker images and NERSC services.
+
+## Dashboard Image
+
+From the repository root:
+
+```bash
+docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-gui -f dashboard.Dockerfile .
+```
+
+## ML Image
+
+From the repository root:
+
+```bash
+docker build --platform linux/amd64 --output type=image,oci-mediatypes=true -t synapse-ml -f ml.Dockerfile .
+```
+
+The two build commands differ only by image tag and Dockerfile.
+
+## Publish Helper
+
+```bash
+python publish_container.py --gui --ml
+```
+
+## NERSC Assumptions
+
+- Dashboard runs on Spin.
+- Training and simulations run on Perlmutter through Superfacility API.
+- Images are pushed to `registry.nersc.gov/m558/superfacility`.
+- Before publishing, validate locally and, when possible, against a staging Spin deployment.

docs/source/developer-notes.md

@@ -0,0 +1,34 @@
+# Developer Notes
+
+## Style
+
+Python code is linted and formatted with Ruff through pre-commit:
+
+```bash
+pre-commit run --files <modified files>
+```
+
+Ruff runs with its default rule set; there is no `pyproject.toml` or `ruff.toml` to override it.
+
+## Environments
+
+- Dashboard dependencies live in `dashboard/environment.yml`.
+- ML dependencies live in `ml/environment.yml`.
+- Regenerate the matching `environment-lock.yml` after dependency changes.
+
+## Testing
+
+The project does not have a full pytest suite.
+The main integration check is:
+
+```bash
+python tests/test_ml_pipeline.py
+```
+
+It requires a local MLflow server.
+
+## Patterns
+
+- Dashboard features use manager classes in `dashboard/*_manager.py`.
+- Experiment-specific behavior belongs under `experiments/synapse-*`.
+- Shared dashboard helpers live in `dashboard/utils.py`.

docs/source/experiment-configuration.md

@@ -0,0 +1,57 @@
+# Experiment Configuration
+
+An experiment is a directory named `experiments/synapse-<experiment>/`.
+The dashboard strips `synapse-` and uses the rest as the experiment identifier.
+
+Each experiment should provide:
+
+- `config.yaml`
+- optional `simulation_scripts/`
+- optional `experiment_scripts/`
+
+## Required Config Sections
+
+- `experiment`: collection and model namespace, for example `bella-ip2`.
+- `database`: MongoDB connection and credential environment variables.
+- `mlflow`: tracking URI and optional API key environment variable.
+- `execution_mode`: ML training and simulation mode hints.
+- `inputs`: scalar variables with `name`, `type`, `default`, and `value_range`.
+- `outputs`: scalar variables with `name` and `type`.
+
+## Calibration
+
+`simulation_calibration` maps simulation variable names to experimental variable names:
+
+```yaml
+simulation_calibration:
+  input1:
+    name: "simulation_variable"
+    unit: "unit"
+    depends_on: "experimental_variable"
+    alpha_guess: 1.0
+    alpha_uncertainty: 0.1
+    beta_guess: 0.0
+    beta_uncertainty: 0.0
+```
+
+Dashboard display uses:
+
+```text
+experimental = simulation / alpha + beta
+```
+
+Simulation launch uses:
+
+```text
+simulation = alpha * (experimental - beta)
+```
+
+These are inverse conversions: display maps simulation to experimental units, while launch maps dashboard parameters back to simulation units.
+
+## Add an Experiment
+
+1. Clone or create `experiments/synapse-<experiment>/`.
+2. Add `config.yaml`.
+3. Ensure MongoDB fields match the configured input and output variable names.
+4. Add `simulation_scripts/` only if dashboard launch is needed.
+5. Train and register a model if dashboard predictions are needed.

docs/source/getting-started.md

@@ -0,0 +1,40 @@
+# Getting Started
+
+For a reproducible install, use `environment-lock.yml` rather than the unpinned `environment.yml`.
+
+## Dashboard
+
+From `dashboard/`:
+
+```bash
+conda-lock install --name synapse-gui environment-lock.yml
+conda activate synapse-gui
+export SF_DB_HOST='127.0.0.1'
+export SF_DB_READONLY_PASSWORD='...'
+export AM_SC_API_KEY='...'
+python -u app.py --port 8080
+```
+
+For local MongoDB access, open a tunnel first:
+
+```bash
+ssh -L 27017:mongodb05.nersc.gov:27017 <username>@dtn03.nersc.gov -N
+```
+
+## ML Training
+
+From `ml/`:
+
+```bash
+conda-lock install --name synapse-ml environment-lock.yml
+conda activate synapse-ml
+export SF_DB_READONLY_PASSWORD='...'
+export AM_SC_API_KEY='...'
+python train_model.py --test --config_file ../experiments/synapse-bella-ip2/config.yaml --model NN
+```
+
+## Required Environment Variables
+
+- `SF_DB_HOST`: MongoDB host for the dashboard.
+- `SF_DB_READONLY_PASSWORD`: read-only MongoDB password.
+- `AM_SC_API_KEY`: American Science Cloud MLflow API key when the config uses that service.

docs/source/index.rst

@@ -0,0 +1,19 @@
+Synapse
+=======
+
+Welcome to the Synapse documentation!
+
+.. toctree::
+   :hidden:
+   :maxdepth: 1
+   :caption: Contents:
+
+   overview
+   getting-started
+   dashboard
+   data-model
+   experiment-configuration
+   simulations
+   ml-training
+   deployment
+   developer-notes