dapi v0.5.0: PyLauncher sweeps, API renames, Sphinx autodoc (#7)

* Add TMS credential management to dapi

Add check_credentials(), establish_credentials(), and revoke_credentials()
to manage Tapis Managed Secrets (TMS) SSH keys on TACC execution systems.
Includes CredentialError exception, 31 tests, docs, and example notebook.
Also fixes missing comma bug in __init__.py __all__ list.

* Switch from mkdocs to Jupyter Book v2

Remove mkdocs.yml, add myst.yml for Jupyter Book v2 / MyST-MD.
Update docs.yml workflow to use jupyter-book>=2 with Node.js.
Add jupyter-book ^2.0.0 as docs dependency in pyproject.toml.
Add TMS credentials example to _toc.yml.

* Migrate from Poetry to uv + hatchling

Replace Poetry build system with hatchling and PEP 621 metadata.
Update all GitHub Actions workflows to use astral-sh/setup-uv.
Remove poetry.lock. Add pytest step to CI workflow.

* Update README for uv, TMS credentials, and Jupyter Book v2

* Add _build to gitignore, hide MyST footer in docs

* Add ds.jobs.list() and standardize docs to use ds variable

Add list_jobs() to fetch Tapis jobs as a pandas DataFrame with
optional filtering by app_id and status. Includes 14 tests.
Rename client variable to ds across all documentation examples.

* Auto-TMS on DSClient init, configurable jobs list output

- DSClient() now auto-establishes TMS credentials on TACC systems
  (frontera, stampede3, ls6). Skips gracefully on errors/no allocation.
- ds.jobs.list() supports output="df" (default), "list", or "raw".
- Simplify auth docs: DB credentials use built-in defaults, no .env needed.
- Remove manual TMS step from quickstart (now automatic).
- 59 tests pass (8 new for setup_tms, 6 new for output formats).

* Clarify DB credentials are intentionally public read-only

* WIP: parametric sweeps and redo generate and submit

* Rename commands for brevity

* Update pylauncher notebook sweep example

* Docs

* fix formatting

* Using ruff instead of black

* Fix tests

* Fix archive system

* Version update to 0.5.0

* Ruff format pylauncher notebook

* Fixes #3 TMS support automatically on init client

* Dapi examples 0.5.0

* Documentation

Author: kks32

.github/workflows/build-test.yml

@@ -7,22 +7,18 @@ jobs:
       fail-fast: false
       matrix:
         python-version: ["3.13"]
-        poetry-version: ["2.1.2"]
         os: [ubuntu-latest]
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
-      - name: Run image
-        uses: abatilo/actions-poetry@v2
-        with:
-          poetry-version: ${{ matrix.poetry-version }}
-      - name: Update lock file
-        run: poetry lock
-      - name: Install the project dependencies
-        run: poetry install
-      - name: Lint with black
-        run: poetry run black --check .
-  
+      - name: Install dependencies
+        run: uv pip install --system ".[dev]"
+      - name: Lint and format check
+        run: ruff format --check . && ruff check .
+      - name: Run tests
+        run: pytest tests/

.github/workflows/docs.yml

@@ -1,4 +1,4 @@
-name: deploy-book
+name: deploy-docs
 
 on:
   push:
@@ -16,36 +16,45 @@ concurrency:
   cancel-in-progress: false
 
 jobs:
-  deploy-book:
+  build:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
 
-      - name: Set up Python
-        uses: actions/setup-python@v5
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
         with:
-          python-version: "3.12"
+          node-version: "22"
 
       - name: Install dependencies
-        run: |
-          pip install "jupyter-book<1"
-          pip install .
+        run: uv pip install --system ".[docs]"
+
+      # Build narrative docs (guides, examples) with Jupyter Book / MyST
+      - name: Build narrative docs
+        run: jupyter-book build --html
 
-      - name: Build the book
-        run: |
-          jupyter-book build . --path-output out
+      # Build API reference with Sphinx
+      - name: Build API docs
+        run: sphinx-build -b html docs-api _build/html/api
 
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v3
         with:
-          path: ./out/_build/html
+          path: ./_build/html
 
   deploy:
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
     runs-on: ubuntu-latest
-    needs: deploy-book
+    needs: build
     steps:
       - name: Deploy to GitHub Pages
         id: deployment

.github/workflows/pypi.yml

@@ -2,7 +2,7 @@ name: Publish to PyPI
 on:
   push:
     tags:
-      - 'v[0-9]+.[0-9]+.[0-9]+'    
+      - 'v[0-9]+.[0-9]+.[0-9]+'
 jobs:
   pypi-publish:
     name: Upload release to PyPI
@@ -11,21 +11,18 @@ jobs:
       name: pypi
       url: https://pypi.org/p/dapi
     permissions:
-      id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
-      contents: read   # Required for checkout
+      id-token: write
+      contents: read
     steps:
       - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
       - uses: actions/setup-python@v5
         with:
-          python-version: 3.9
-      - name: Poetry image
-        uses: abatilo/actions-poetry@v2
-      - name: Install the project dependencies
-        run: poetry install
+          python-version: "3.13"
       - name: Build package
-        run: poetry build
+        run: uv build
       - name: Publish package distributions to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
-          # Enable verbose logging for debugging
           verbose: true

.gitignore

@@ -141,6 +141,7 @@ venv.bak/
 .ropeproject
 
 # Jupyter Book build output
+/_build/
 /out/
 
 # mkdocs documentation (legacy)

README.md

@@ -2,29 +2,36 @@
 
 [![build and test](https://github.com/DesignSafe-CI/dapi/actions/workflows/build-test.yml/badge.svg)](https://github.com/DesignSafe-CI/dapi/actions/workflows/build-test.yml)
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.md)
-[![Docs](https://img.shields.io/badge/view-docs-8A2BE2?color=8A2BE2)](https://designsafe-ci.github.io/dapi/dapi/index.html)
+[![PyPI version](https://badge.fury.io/py/dapi.svg)](https://badge.fury.io/py/dapi)
+[![Docs](https://img.shields.io/badge/view-docs-8A2BE2?color=8A2BE2)](https://designsafe-ci.github.io/dapi/)
 
-`dapi` is a library that simplifies the process of submitting, running, and monitoring [TAPIS v3](https://tapis.readthedocs.io/en/latest/) jobs on [DesignSafe](https://designsafe-ci.org) via [Jupyter Notebooks](https://jupyter.designsafe-ci.org).
+`dapi` is a Python library for submitting, running, and monitoring [TAPIS v3](https://tapis.readthedocs.io/en/latest/) jobs on [DesignSafe](https://designsafe-ci.org) via [Jupyter Notebooks](https://jupyter.designsafe-ci.org) or the command line.
 
 <img src="https://raw.githubusercontent.com/DesignSafe-CI/dapi/main/dapi.png" alt="dapi" width="300">
 
-
 ## Features
 
 ### Jobs
+- Generate TAPIS v3 job requests with automatic app parameter mapping
+- Submit, monitor (with progress bars), and manage jobs
+- Access and download job outputs
+- Discover and explore available DesignSafe applications
 
-* Get TAPIS v3 templates for jobs: No need to fiddle with complex API requests. `dapi` abstracts away the complexities.
+### TMS Credentials
+- Establish, check, and revoke SSH keys on TACC execution systems (Frontera, Stampede3, LS6)
+- Works from any environment -- DesignSafe JupyterHub, command line, CI/CD
 
-* Seamless Integration with DesignSafe Jupyter Notebooks: Launch DesignSafe applications directly from the Jupyter environment.
+### Files
+- Translate DesignSafe paths (`/MyData`, `/CommunityData`, `/projects`) to TAPIS URIs
+- Upload, download, and list files on DesignSafe storage
 
 ### Database
-
 Connects to SQL databases on DesignSafe:
 
 | Database | dbname | env_prefix |
 |----------|--------|------------|
 | NGL | `ngl`| `NGL_` |
-| Earthake Recovery | `eq` | `EQ_` |
+| Earthquake Recovery | `eq` | `EQ_` |
 | Vp | `vp` | `VP_` |
 
 Define the following environment variables:
@@ -37,115 +44,119 @@ Define the following environment variables:
 
 For e.g., to add the environment variable `NGL_DB_USER` edit `~/.bashrc`, `~/.zshrc`, or a similar shell-specific configuration file for the current user and add `export NGL_DB_USER="dspublic"`.
 
-
 ## Installation
 
-Install `dapi` via pip
-
 ```shell
-pip3 install dapi
+pip install dapi
 ```
 
-To install the current development version of the library use:
+To install the development version:
 
 ```shell
-pip install git+https://github.com/DesignSafe-CI/dapi.git --quiet
+pip install git+https://github.com/DesignSafe-CI/dapi.git@dev --quiet
 ```
 
-## Example usage:
+In Jupyter notebooks:
 
-### Storing credentials
+```python
+%pip install dapi --quiet
+```
 
-Dapi uses the Tapis v3 SDK to authenticate with the DesignSafe API. To store your credentials, create a `.env` file in the root of your project with the following content:
+To test the latest dev branch in a notebook:
 
-```shell
-DESIGNSAFE_USERNAME=<your_designsafe_username>
-DESIGNSAFE_PASSWORD=<your_designsafe_password>
+```python
+%pip uninstall dapi --yes
+%pip install git+https://github.com/DesignSafe-CI/dapi.git@dev --quiet
 ```
 
-### Jobs
+For local development (editable install — changes to source are reflected immediately):
 
-* [Jupyter Notebook Templates](example-notebooks/template-mpm-run.ipynb) using dapi.
+```shell
+pip install -e .
+```
 
-* View [dapi API doc](https://designsafe-ci.github.io/dapi/dapi/index.html)
+## Quick Start
 
-On [DesignSafe Jupyter](https://jupyter.designsafe-ci.org/):
+### Authentication
 
-Install the latest version of `dapi` and restart the kernel (Kernel >> Restart Kernel):
+Create a `.env` file with your DesignSafe credentials:
 
-```python
-# Remove any previous installations
-!pip uninstall dapi -y
-# Install
-!pip install dapi --quiet
+```shell
+DESIGNSAFE_USERNAME=your_username
+DESIGNSAFE_PASSWORD=your_password
 ```
 
-* Import `dapi` library
-```python
-import dapi
-```
+### Setup and submit a job
 
-* To list all functions in `dapi`
 ```python
-dir(dapi)
+from dapi import DSClient
+
+# Authenticate
+client = DSClient()
+
+# Establish TMS credentials (one-time per system)
+client.systems.establish_credentials("frontera")
+
+# Submit a job
+job_request = client.jobs.generate(
+    app_id="matlab-r2023a",
+    input_dir_uri="/MyData/analysis/input/",
+    script_filename="run_analysis.m",
+    max_minutes=30,
+    allocation="your_allocation"
+)
+job = client.jobs.submit(job_request)
+final_status = job.monitor()
 ```
 
 ### Database
+
 ```python
-from dapi.db import DSDatabase
+from dapi import DSClient
 
-db = DSDatabase("ngl")
-sql = 'SELECT * FROM SITE'
-df = db.read_sql(sql)
+client = DSClient()
+df = client.db.ngl.read_sql("SELECT * FROM SITE LIMIT 10")
 print(df)
-
-# Optionally, close the database connection when done
-db.close()
 ```
 
-
 ## Support
 
-For any questions, issues, or feedback submit an [issue](https://github.com/DesignSafe-CI/dapi/issues/new)
+For any questions, issues, or feedback submit an [issue](https://github.com/DesignSafe-CI/dapi/issues/new).
 
 ## Development
 
-To develop or test the library locally. Install [Poetry](https://python-poetry.org/docs/#installation). In the current repository run the following commands
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then:
 
 ```shell
-poetry shell
-poetry install
-poetry build
+uv venv
+uv pip install -e ".[dev]"
 ```
 
-To run the unit test
+Run tests:
 ```shell
-poetry run pytest -v
+pytest tests/ -v
 ```
 
+Build the package:
+```shell
+uv build
+```
 
-## License
-
-`dapi` is licensed under the [MIT License](LICENSE.md).
-
-## Authors
-
-* Krishna Kumar, University of Texas at Austin
-* Prof. Pedro Arduino, University of Washington
-* Prof. Scott Brandenberg, University of California Los Angeles
-
+### Documentation
 
-## Documentation
+Documentation uses [Jupyter Book v2](https://mystmd.org). To build and serve locally:
 
-View [dapi API doc](https://designsafe-ci.github.io/dapi/dapi/index.html)
+```shell
+uv pip install -e ".[docs]"
+jupyter-book start
+```
 
-### Running documentation locally
+## License
 
-To serve the MkDocs documentation locally:
+`dapi` is licensed under the [MIT License](LICENSE.md).
 
-```shell
-poetry install
-poetry run mkdocs serve
-```
+## Authors
 
-This will start a local server at `http://127.0.0.1:8000/dapi/` where you can view the documentation.
+- Krishna Kumar, University of Texas at Austin
+- Prof. Pedro Arduino, University of Washington
+- Prof. Scott Brandenberg, University of California Los Angeles