Merge branch 'develop' into feat/inference-cohort-selection

Author: chapmanhk

.devcontainer/devcontainer.json

@@ -15,11 +15,11 @@
         "vscode": {
             "extensions": [
                 "hashicorp.terraform",
-                "ms-python.black-formatter"
+                "charliermarsh.ruff"
             ],
             "settings": {
                 "[python]": {
-                    "editor.defaultFormatter": "ms-python.black-formatter",
+                    "editor.defaultFormatter": "charliermarsh.ruff",
                     "editor.formatOnSave": true
                   }
             }

.vscode/launch.json

@@ -29,6 +29,21 @@
       "env": {
         "ENV_FILE_PATH": "${workspaceFolder}/src/worker/.env"
       }
+    },
+    {
+      "name": "pytest (current file)",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "pytest",
+      "args": [
+        "${file}",
+        "-v",
+        "-s"
+      ],
+      "cwd": "${workspaceFolder}",
+      "env": {
+        "ENV_FILE_PATH": "${workspaceFolder}/src/webapp/.env"
+      }
     }
   ],
 }

CONTRIBUTING.md

@@ -10,8 +10,8 @@ To get an overview of the project, please read the [README](README.md) and our [
 ## Getting started
 ### Creating Issues
 
-If you spot a problem, [search if an issue already exists](https://github.com/datakind/sst-app-api/issues). If a related issue doesn't exist,
-you can open a new issue using a relevant [issue form](https://github.com/datakind/sst-app-api/issues/new).
+If you spot a problem, [search if an issue already exists](https://github.com/datakind/edvise-api/issues). If a related issue doesn't exist,
+you can open a new issue using a relevant [issue form](https://github.com/datakind/edvise-api/issues/new).
 
 As a general rule, we don’t assign issues to anyone. If you find an issue to work on, you are welcome to open a PR with a fix.
 
@@ -28,7 +28,7 @@ poetry install --no-interaction
 As many other open source projects, we use the famous [gitflow](https://nvie.com/posts/a-successful-git-branching-model/) to manage our branches.
 
 Summary of our git branching model:
-- Get all the latest work from the upstream `datakind/sst-app-api` repository
+- Get all the latest work from the upstream `datakind/edvise-api` repository
   (`git checkout main`)
 - Create a new branch off with a descriptive name (for example:
   `feature/new-test-macro`, `bugfix/bug-when-uploading-results`). You can
@@ -107,7 +107,7 @@ You can type `pytest` to run your tests, no matter which type of test it is.
 
 ## Continuous Integration
 
-We use [GitHub Actions](https://github.com/datakind/sst-app-api/actions)
+We use [GitHub Actions](https://github.com/datakind/edvise-api/actions)
 for continuous integration.
 See [here](https://docs.github.com/en/actions) for GitHub's documentation.
 

README.md

@@ -2,14 +2,25 @@
 
 This repo contains:
 
-* [src/webapp/](https://github.com/datakind/sst-app-api/tree/develop/src/webapp): The source code for the SST API (which is called by the SST frontend and by any direct API callers)
-* [src/worker/](https://github.com/datakind/sst-app-api/tree/develop/src/worker): The source code for the SFTP Worker (which calls the SST API)
+* [src/webapp/](https://github.com/datakind/edvise-api/tree/develop/src/webapp): The source code for the SST API (which is called by the SST frontend and by any direct API callers)
+* [src/worker/](https://github.com/datakind/edvise-api/tree/develop/src/worker): The source code for the SFTP Worker (which calls the SST API)
 * [terraform/]
-(https://github.com/datakind/sst-app-api/tree/develop/terraform): The Terraform configuration for the SST API/Frontend and other GCP resources including Cloud SQL setup, networking setup, secrets setup
+(https://github.com/datakind/edvise-api/tree/develop/terraform): The Terraform configuration for the SST API/Frontend and other GCP resources including Cloud SQL setup, networking setup, secrets setup
 * .devcontainer/ and .vscode/: which allow easy setup if you are using VSCode as your IDE.
-* [devtools/](https://github.com/datakind/sst-app-api/tree/develop/devtools): is a place to put utility scripts
-* .github/: contains mostly copied over files when this directory was forked from the student-success-tool repo, so likely much of it is outdated. The only Github action we've added is the [webapp-and-worker-precommit](https://github.com/datakind/sst-app-api/blob/develop/.github/workflows/webapp-and-worker-precommit.yml) which is run on every push to develop. This action contains a python linter (we use [black](https://black.readthedocs.io/en/stable/)), and automated runs of the unit tests in the src/webapp/ and src/worker/ directories.
-* Additionally, [pyproject.toml](https://github.com/datakind/sst-app-api/blob/develop/pyproject.toml) and [uv.lock](https://github.com/datakind/sst-app-api/blob/develop/uv.lock) are important for dependency management. At time of writing, the worker is just skeleton code so there's no separate dependency management. In the long-term consider separating out the dependency management for the two programs. 
+* [devtools/](https://github.com/datakind/edvise-api/tree/develop/devtools): is a place to put utility scripts
+* .github/: contains mostly copied over files when this directory was forked from the student-success-tool repo, so likely much of it is outdated. The only Github action we've added is the [webapp-and-worker-precommit](https://github.com/datakind/edvise-api/blob/develop/.github/workflows/webapp-and-worker-precommit.yml) which is run on every push to develop. This action contains a python linter (we use [black](https://black.readthedocs.io/en/stable/)), and automated runs of the unit tests in the src/webapp/ and src/worker/ directories.
+* Additionally, [pyproject.toml](https://github.com/datakind/edvise-api/blob/develop/pyproject.toml) and [uv.lock](https://github.com/datakind/edvise-api/blob/develop/uv.lock) are important for dependency management. At time of writing, the worker is just skeleton code so there's no separate dependency management. In the long-term consider separating out the dependency management for the two programs. 
 
 
 NOTE: this repo was forked from the https://github.com/datakind/student-success-tool repo, which means some of the static files (e.g. CONTRIBUTING.md) may be outdated or may include irrelevant information from that repo. Please update those as you see fit. For information about the specific items listed above, defer to the specific readmes in the relevant directory.
+
+## Local edvise development override
+
+Production uses a pinned Git reference for `edvise`. For local development, use an
+editable install after syncing the environment.
+
+1. Clone `edvise` alongside `edvise-api` (so `../edvise` exists).
+2. Run `uv sync`.
+3. Override locally: `uv pip install -e ../edvise`
+
+To revert back to the pinned Git dependency, run `uv sync --reinstall-package edvise`.

SECURITY.md

@@ -8,4 +8,4 @@ If we verify a reported security vulnerability, our policy is:
 
 ## Reporting a Security Issue
 
-To report any security issues, please [raise an issue](https://github.com/datakind/sst-app-api/issues/new/choose) and select **Security issue*
+To report any security issues, please [raise an issue](https://github.com/datakind/edvise-api/issues/new/choose) and select **Security issue*

cloudbuild-webapp.yaml

@@ -0,0 +1,34 @@
+# Cloud Build config for webapp (dev-webapp trigger).
+# _REGION and _ENVIRONMENT are set by the trigger (Terraform).
+steps:
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - build
+      - '-f'
+      - src/webapp/Dockerfile
+      - '-t'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/webapp:$COMMIT_SHA'
+      - '-t'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/webapp:latest'
+      - .
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - push
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/webapp:$COMMIT_SHA'
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - push
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/webapp:latest'
+  - name: gcr.io/cloud-builders/gcloud
+    args:
+      - run
+      - deploy
+      - '${_ENVIRONMENT}-webapp'
+      - '--image'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/webapp:$COMMIT_SHA'
+      - '--region'
+      - '${_REGION}'
+timeout: 600s
+options:
+  logging: CLOUD_LOGGING_ONLY
+  dynamicSubstitutions: true

cloudbuild-worker.yaml

@@ -0,0 +1,34 @@
+# Cloud Build config for worker (dev-worker trigger).
+# _REGION and _ENVIRONMENT are set by the trigger (Terraform).
+steps:
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - build
+      - '-f'
+      - src/worker/Dockerfile
+      - '-t'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/worker:$COMMIT_SHA'
+      - '-t'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/worker:latest'
+      - .
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - push
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/worker:$COMMIT_SHA'
+  - name: gcr.io/cloud-builders/docker
+    args:
+      - push
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/worker:latest'
+  - name: gcr.io/cloud-builders/gcloud
+    args:
+      - run
+      - deploy
+      - '${_ENVIRONMENT}-worker'
+      - '--image'
+      - '${_REGION}-docker.pkg.dev/${PROJECT_ID}/edvise-api/worker:$COMMIT_SHA'
+      - '--region'
+      - '${_REGION}'
+timeout: 600s
+options:
+  logging: CLOUD_LOGGING_ONLY
+  dynamicSubstitutions: true

pyproject.toml

@@ -8,7 +8,7 @@ dependencies = [
     "databricks-sdk~=0.38.0",
     "pydantic~=2.10",
     "fastapi[standard]~=0.115.4",
-    "google-cloud-storage~=2.18.2",
+    "google-cloud-storage==2.19.0",
     "paramiko~=3.5.0",
     "cloud-sql-python-connector[pymysql]~=1.14.0",
     "sqlalchemy~=2.0.36",
@@ -28,13 +28,14 @@ dependencies = [
     "thefuzz[speedup]~=0.22.1",
     "databricks-sql-connector~=3.5.0",
     "pandera~=0.13",
-    "mlflow~=2.15.0",
+    "mlflow~=2.22",
     "cachetools",
     "types-cachetools",
+    "edvise",
 ]
 
 [project.urls]
-Repository = "https://github.com/datakind/sst-app-api"
+Repository = "https://github.com/datakind/edvise-api"
 
 [dependency-groups]
 dev = [
@@ -52,9 +53,17 @@ dev = [
 requires = ["hatchling"]
 build-backend = "hatchling.build"
 
+[tool.hatch.metadata]
+allow-direct-references = true
+
 [tool.uv]
 default-groups = ["dev"]
 
+[tool.uv.sources]
+# Install edvise from GitHub (branch, tag, or commit).
+# Use rev = "main" for default branch, or rev = "v1.0.0" for a release tag.
+edvise = { git = "https://github.com/datakind/edvise.git", rev = "feat/eda_summary_class" }
+
 [tool.ruff]
 line-length = 88
 indent-width = 4

src/webapp/README.md

@@ -51,7 +51,7 @@ In the long-term, look into a way to have the API key --> token conversion be ha
 
 ## Databases
 
-All data is stored in MySQL databases for dev/staging/prod, these are databases in GCP's Cloud SQL. In the local environment, the database is sqlite. The main file you'll want to look at for database table definitions is [src/webapp/database.py](https://github.com/datakind/sst-app-api/blob/develop/src/webapp/database.py).
+All data is stored in MySQL databases for dev/staging/prod, these are databases in GCP's Cloud SQL. In the local environment, the database is sqlite. The main file you'll want to look at for database table definitions is [src/webapp/database.py](https://github.com/datakind/edvise-api/blob/develop/src/webapp/database.py).
 
 At time of writing, the databases the API cares about and tracks, are as follows:
 
@@ -112,7 +112,7 @@ Enter into the root directory of the repo.
 
 You're now in your virtual env with all your dependencies added.
 
-For all of the following, the steps above are pre-requisites and you should be in the root folder of `sst-app-api/`.
+For all of the following, the steps above are pre-requisites and you should be in the root folder of `edvise-api/`.
 
 ### Spin up the app locally:
 

src/webapp/database.py

@@ -63,6 +63,9 @@ class Base(DeclarativeBase):
 # Test institution - same ID as DEV USC Beaufort for testing
 TEST_INST_UUID = uuid.UUID("942d4b0e-12e7-4d2a-9187-9508ae3cef7c")
 TEST_BATCH_UUID = uuid.UUID("3182f472-e079-4678-a0a1-9ca5ead6c49a")
+# Montana Tech test institution + batch (matches DEV data for local EDA testing)
+MONTANA_TECH_INST_UUID = uuid.UUID("1e2c628cabda4088b900cfd9ced44268")
+MONTANA_TECH_BATCH_UUID = uuid.UUID("1df31913e1fe4c458f5039527c967b13")
 
 
 @event.listens_for(Mapper, "before_insert")
@@ -224,6 +227,57 @@ def init_db(env: str) -> None:
                 session.merge(test_file_student)
                 session.merge(test_file_course)
                 session.merge(test_batch)
+
+            # Montana Tech - matches DEV data for local EDA testing
+            session.merge(
+                InstTable(
+                    id=MONTANA_TECH_INST_UUID,
+                    name="Montana Tech",
+                    state="MT",
+                    schemas=["COURSE", "STUDENT"],
+                    created_at=DATETIME_TESTING,
+                    updated_at=DATETIME_TESTING,
+                    created_by=LOCAL_USER_UUID,
+                )
+            )
+            montana_student_file = FileTable(
+                id=uuid.UUID("a2b0f7b2-0c57-4aa9-8dd0-8e1a49d1be01"),
+                inst_id=MONTANA_TECH_INST_UUID,
+                name="1764013161378_AO1600pdp_AO1600_AR_DEIDENTIFIED_STUDYID_20251028121051.csv",
+                source="MANUAL_UPLOAD",
+                uploader=LOCAL_USER_UUID,
+                sst_generated=False,
+                valid=True,
+                schemas=["STUDENT"],
+                created_at=DATETIME_TESTING,
+                updated_at=DATETIME_TESTING,
+            )
+            montana_course_file = FileTable(
+                id=uuid.UUID("d8b0f2c3-2d5a-4d7b-8e33-7e4c3b2609d2"),
+                inst_id=MONTANA_TECH_INST_UUID,
+                name="1764013161379_AO1600pdp_AO1600_COURSE_LEVEL_AR_DEIDENTIFIED_STUDYID_20251028121051.csv",
+                source="MANUAL_UPLOAD",
+                uploader=LOCAL_USER_UUID,
+                sst_generated=False,
+                valid=True,
+                schemas=["COURSE"],
+                created_at=DATETIME_TESTING,
+                updated_at=DATETIME_TESTING,
+            )
+            montana_batch = BatchTable(
+                id=MONTANA_TECH_BATCH_UUID,
+                inst_id=MONTANA_TECH_INST_UUID,
+                name="Batch_2025-10-28_1764013161000",
+                completed=True,
+                created_by=LOCAL_USER_UUID,
+                created_at=DATETIME_TESTING,
+                updated_at=DATETIME_TESTING,
+            )
+            montana_batch.files.add(montana_student_file)
+            montana_batch.files.add(montana_course_file)
+            session.merge(montana_student_file)
+            session.merge(montana_course_file)
+            session.merge(montana_batch)
             session.commit()
     except Exception as e:
         session.rollback()
@@ -781,7 +835,6 @@ def init_connection_pool_local() -> sqlalchemy.engine.base.Engine:
     """Creates a local sqlite db for local env testing."""
     return sqlalchemy.create_engine(
         "sqlite://",
-        echo=True,
         echo_pool="debug",
         connect_args={"check_same_thread": False},
         poolclass=StaticPool,