Skip to content

Commit 413da99

Browse files
prestoncabeprestoncabe
committed
library-api: refactor metadata sync to work in dev
Parameterizes the sync so that it will work with either local dev (using the firebase emulators) or in production. Defaults to dev mode (and automatically works with devbox services/ process-compose).
1 parent 75c184f commit 413da99

11 files changed

Lines changed: 310 additions & 16 deletions

File tree

‎.env.example‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,8 @@ QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE="http://127.0.0.1:9199"
44
GCS_BUCKET_NAME="demo-bdt-dev.appspot.com"
55
FIRESTORE_EMULATOR_HOST="127.0.0.1:8080"
66

7+
# Library API Configuration
8+
# Defaults to http://localhost:8083 (no config needed for dev)
9+
# For production sync, set:
10+
# LIBRARY_API_BASE_URL=https://library-api-1034049717668.us-central1.run.app
11+

‎.github/workflows/load-library-metadata.yml‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,20 +16,17 @@ jobs:
1616
with:
1717
python-version: "3.11"
1818

19-
- name: Install dependencies
20-
working-directory: scripts
21-
run: pip install -r requirements.txt
22-
2319
- name: Create GCP credentials file
2420
run: |
25-
echo '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}' > scripts/gcp-key.json
21+
echo '${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}' > bin/library/gcp-key.json
2622
27-
- name: Run script
28-
working-directory: scripts
23+
- name: Run sync
24+
working-directory: bin/library
2925
env:
3026
GOOGLE_APPLICATION_CREDENTIALS: gcp-key.json
27+
LIBRARY_API_BASE_URL: https://library-api-1034049717668.us-central1.run.app
3128
run: |
32-
python load-library-metadata.py
29+
./sync-metadata
3330
3431
- name: Cleanup credentials
35-
run: rm scripts/gcp-key.json
32+
run: rm bin/library/gcp-key.json

‎CLAUDE.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,57 @@ Admin → builder-frontend → builder-api → Firebase (Firestore + Storage)
167167
- Learn DMN basics: https://learn-dmn-in-15-minutes.com/
168168
- Access raw XML: Right-click → "Reopen with Text Editor"
169169

170+
### Library Check Metadata Sync
171+
172+
The builder-api needs metadata about available library checks (from library-api). This metadata is stored in Firebase Storage and referenced from Firestore.
173+
174+
**Automatic Sync (Development)**:
175+
- Runs automatically when you start services via `devbox services up`
176+
- Syncs from local library-api (http://localhost:8083) to Firebase emulators
177+
- Happens after library-api starts, before builder-api starts
178+
- Library checks will then be visible in the builder UI!
179+
180+
**Manual Sync (Development)**:
181+
```bash
182+
# Re-sync after making library-api changes
183+
./scripts/sync-library-metadata.sh
184+
185+
# Then restart builder-api to pick up new metadata
186+
# (In process-compose UI, restart the builder-api process)
187+
```
188+
189+
**Production Sync** (maintainers only):
190+
```bash
191+
# Set production library-api URL and unset emulator variables
192+
export LIBRARY_API_BASE_URL=https://library-api-1034049717668.us-central1.run.app
193+
unset FIRESTORE_EMULATOR_HOST
194+
unset GCS_BUCKET_NAME
195+
unset QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE
196+
197+
# Authenticate with GCP
198+
gcloud auth application-default login
199+
200+
# Run sync
201+
./scripts/sync-library-metadata.sh
202+
```
203+
204+
**How It Works**:
205+
1. Fetches OpenAPI spec from library-api
206+
2. Extracts check metadata (inputs, outputs, versions)
207+
3. Uploads JSON to Firebase Storage
208+
4. Updates Firestore `system/config` with storage path
209+
5. builder-api reads this metadata on startup
210+
211+
**Environment Configuration**:
212+
- **Default**: `http://localhost:8083` (development mode - no config needed)
213+
- **Production**: Set `LIBRARY_API_BASE_URL` to production Cloud Run URL
214+
- Environment is inferred from URL pattern (localhost = dev, else = prod)
215+
216+
**Troubleshooting**:
217+
- **"Firebase Storage emulator not responding"**: Start emulators first (`firebase emulators:start`)
218+
- **"library-api not responding"**: Start library-api (`cd library-api && quarkus dev`)
219+
- **Stale metadata in builder-api**: Restart builder-api (reads metadata on startup)
220+
170221
### Firebase Emulators
171222

172223
The project uses Firebase emulators for local development:

scripts/load-library-metadata.py renamed to bin/library/load-library-metadata.py

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,16 +4,51 @@
44
from firebase_admin import credentials, storage, firestore
55
import json
66
from datetime import datetime
7+
import os
8+
9+
# -----------------------------------
10+
# CONFIGURATION
11+
# -----------------------------------
12+
13+
# Default to localhost for developer-friendly setup
14+
DEFAULT_LIBRARY_API_URL = "http://localhost:8083"
15+
LIBRARY_API_BASE_URL = os.getenv("LIBRARY_API_BASE_URL", DEFAULT_LIBRARY_API_URL)
16+
17+
# Infer production mode from URL (for versioned URL logic)
18+
IS_PRODUCTION = not ("localhost" in LIBRARY_API_BASE_URL or "127.0.0.1" in LIBRARY_API_BASE_URL)
19+
20+
# Storage bucket defaults - use dev bucket by default
21+
DEFAULT_DEV_BUCKET = "demo-bdt-dev.appspot.com"
22+
DEFAULT_PROD_BUCKET = "benefit-decision-toolkit-play.firebasestorage.app"
23+
STORAGE_BUCKET = os.getenv("GCS_BUCKET_NAME",
24+
DEFAULT_PROD_BUCKET if IS_PRODUCTION else DEFAULT_DEV_BUCKET)
25+
26+
# Log configuration
27+
print(f"========================================")
28+
print(f"Library Metadata Sync Configuration")
29+
print(f"========================================")
30+
print(f"Mode: {'production' if IS_PRODUCTION else 'development'}")
31+
print(f"Library API URL: {LIBRARY_API_BASE_URL}")
32+
print(f"Storage Bucket: {STORAGE_BUCKET}")
33+
print(f"========================================\n")
734

835
# -----------------------------------
936
# INIT FIREBASE
1037
# -----------------------------------
1138

39+
# Point google-cloud-storage SDK at the emulator using the existing Quarkus config variable
40+
storage_host_override = os.getenv("QUARKUS_GOOGLE_CLOUD_STORAGE_HOST_OVERRIDE")
41+
if storage_host_override:
42+
os.environ["STORAGE_EMULATOR_HOST"] = storage_host_override
43+
1244
cred = credentials.ApplicationDefault()
1345

14-
firebase_admin.initialize_app(cred, {
15-
"storageBucket": "benefit-decision-toolkit-play.firebasestorage.app"
16-
})
46+
firebase_options = {"storageBucket": STORAGE_BUCKET}
47+
if not IS_PRODUCTION:
48+
# Emulators need an explicit project ID; production gets it from credentials
49+
firebase_options["projectId"] = os.getenv("QUARKUS_GOOGLE_CLOUD_PROJECT_ID", "demo-bdt-dev")
50+
51+
firebase_admin.initialize_app(cred, firebase_options)
1752

1853
db = firestore.client()
1954
bucket = storage.bucket()
@@ -292,7 +327,9 @@ def save_json_to_storage_and_update_firestore(json_string, firestore_doc_path):
292327
# --------------------------------------------
293328
if __name__ == "__main__":
294329

295-
url = "https://library-api-1034049717668.us-central1.run.app/q/openapi.json"
330+
url = f"{LIBRARY_API_BASE_URL}/q/openapi.json"
331+
332+
print(f"Fetching OpenAPI spec from: {url}")
296333

297334
# Send a GET request
298335
response = requests.get(url)

scripts/requirements.txt renamed to bin/library/requirements.txt

File renamed without changes.

‎bin/library/sync-metadata‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
5+
PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
6+
7+
# Load environment from root .env if it exists
8+
if [ -f "$PROJECT_ROOT/.env" ]; then
9+
set -a
10+
source "$PROJECT_ROOT/.env"
11+
set +a
12+
fi
13+
14+
# Determine library API URL (defaults to localhost)
15+
LIBRARY_API_URL="${LIBRARY_API_BASE_URL:-http://localhost:8083}"
16+
17+
# Infer mode from URL
18+
if [[ "$LIBRARY_API_URL" == *"localhost"* ]] || [[ "$LIBRARY_API_URL" == *"127.0.0.1"* ]]; then
19+
MODE="development"
20+
else
21+
MODE="production"
22+
fi
23+
24+
echo "=========================================="
25+
echo "Library Metadata Sync"
26+
echo "=========================================="
27+
echo "Mode: $MODE"
28+
echo "Library API: $LIBRARY_API_URL"
29+
30+
if [ "$MODE" = "development" ]; then
31+
echo "Target: Firebase Emulators (Firestore + Storage)"
32+
echo ""
33+
echo "Prerequisites:"
34+
echo " 1. Firebase emulators must be running"
35+
echo " 2. library-api must be running (quarkus dev)"
36+
echo ""
37+
38+
# Check if Firebase Storage emulator is running
39+
if ! curl -s http://localhost:9199 >/dev/null 2>&1; then
40+
echo "ERROR: Firebase Storage emulator not responding at localhost:9199"
41+
echo "Start emulators with: firebase emulators:start --project demo-bdt-dev --only auth,storage,firestore"
42+
exit 1
43+
fi
44+
45+
# Check if library-api is running
46+
if ! curl -s "${LIBRARY_API_URL}/q/health" >/dev/null 2>&1; then
47+
echo "ERROR: library-api not responding at ${LIBRARY_API_URL}"
48+
echo "Start library-api with: cd library-api && quarkus dev"
49+
exit 1
50+
fi
51+
52+
# Check if $VENV_DIR is set and activate it
53+
if [[ "$VENV_DIR" != "" ]]; then
54+
if [ -f "$VENV_DIR/bin/activate" ]; then
55+
source "$VENV_DIR/bin/activate"
56+
echo "✓ Activated virtual environment at $VENV_DIR"
57+
else
58+
echo "WARNING: Virtual environment not found at $VENV_DIR"
59+
fi
60+
fi
61+
62+
echo "✓ Firebase emulators are running"
63+
echo "✓ library-api is running"
64+
echo ""
65+
else
66+
echo "Target: Production Firebase"
67+
echo ""
68+
echo "Prerequisites:"
69+
echo " - Valid Google Cloud credentials (Application Default Credentials)"
70+
echo " - Deployed library-api at: $LIBRARY_API_URL"
71+
echo ""
72+
73+
# Check if we have GCP credentials
74+
if ! gcloud auth application-default print-access-token >/dev/null 2>&1; then
75+
echo "ERROR: No valid Google Cloud credentials found"
76+
echo "Authenticate with: gcloud auth application-default login --project benefit-decision-toolkit-play"
77+
exit 1
78+
fi
79+
80+
echo "✓ Google Cloud credentials found"
81+
echo ""
82+
fi
83+
84+
# Run the Python script
85+
echo "Running metadata sync..."
86+
pip install -q -r $SCRIPT_DIR/requirements.txt
87+
python3 "$SCRIPT_DIR/load-library-metadata.py"
88+
89+
echo ""
90+
echo "=========================================="
91+
echo "Metadata sync complete!"
92+
echo "=========================================="

‎builder-api/src/main/java/org/acme/service/LibraryApiService.java‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
import org.acme.model.domain.EligibilityCheck;
1313
import org.acme.persistence.StorageService;
1414
import org.acme.persistence.FirestoreUtils;
15+
import org.eclipse.microprofile.config.inject.ConfigProperty;
1516

1617
import java.net.URI;
1718
import java.net.http.HttpClient;
@@ -25,14 +26,36 @@
2526

2627
@ApplicationScoped
2728
public class LibraryApiService {
29+
private static final String DEFAULT_LIBRARY_API_URL = "http://localhost:8083";
30+
2831
@Inject
2932
private StorageService storageService;
3033

34+
@ConfigProperty(name = "library-api.base-url")
35+
Optional<String> libraryApiBaseUrl;
36+
3137
private List<EligibilityCheck> checks;
38+
private String effectiveBaseUrl;
39+
private boolean useVersionedUrls;
3240

3341
@PostConstruct
3442
void init() {
3543
try {
44+
// Determine effective base URL
45+
effectiveBaseUrl = libraryApiBaseUrl.orElse(DEFAULT_LIBRARY_API_URL);
46+
47+
// Infer environment from URL - localhost = development, else = production
48+
boolean isProduction = !(effectiveBaseUrl.contains("localhost") || effectiveBaseUrl.contains("127.0.0.1"));
49+
useVersionedUrls = isProduction;
50+
51+
Log.info("========================================");
52+
Log.info("Library API Configuration");
53+
Log.info("========================================");
54+
Log.info("Base URL: " + effectiveBaseUrl);
55+
Log.info("Mode: " + (isProduction ? "production" : "development"));
56+
Log.info("Versioned URLs: " + (useVersionedUrls ? "enabled" : "disabled"));
57+
Log.info("========================================");
58+
3659
// Get path of most recent library schema json document
3760
Optional<Map<String, Object>> configOpt = FirestoreUtils.getFirestoreDocById("system", "config");
3861
if (configOpt.isEmpty()){
@@ -51,6 +74,7 @@ void init() {
5174
ObjectMapper mapper = new ObjectMapper();
5275

5376
checks = mapper.readValue(apiSchemaJson, new TypeReference<List<EligibilityCheck>>() {});
77+
Log.info("Loaded " + checks.size() + " library checks");
5478
} catch (Exception e) {
5579
throw new RuntimeException("Failed to load library api metadata", e);
5680
}
@@ -87,8 +111,20 @@ public EvaluationResult evaluateCheck(CheckConfig checkConfig, Map<String, Objec
87111
String bodyJson = mapper.writeValueAsString(data);
88112

89113
HttpClient client = HttpClient.newHttpClient();
90-
String urlEncodedVersion = checkConfig.getCheckVersion().replace('.', '-');
91-
String baseUrl = String.format("https://library-api-v%s---library-api-cnsoqyluna-uc.a.run.app", urlEncodedVersion);
114+
115+
// Determine base URL based on configuration
116+
String baseUrl;
117+
if (useVersionedUrls) {
118+
// Production: Use versioned Cloud Run URLs
119+
String urlEncodedVersion = checkConfig.getCheckVersion().replace('.', '-');
120+
baseUrl = String.format("https://library-api-v%s---library-api-cnsoqyluna-uc.a.run.app", urlEncodedVersion);
121+
Log.debug("Using versioned URL: " + baseUrl);
122+
} else {
123+
// Development: Use configured base URL directly
124+
baseUrl = effectiveBaseUrl;
125+
Log.debug("Using base URL: " + baseUrl);
126+
}
127+
92128
HttpRequest request = HttpRequest.newBuilder()
93129
.uri(URI.create(baseUrl + checkConfig.getEvaluationUrl()))
94130
.header("Content-Type", "application/json")

‎builder-api/src/main/resources/application.properties‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,8 @@ quarkus.http.auth.permission.secured.policy=authenticated
1414

1515
quarkus.http.auth.permission.public.paths=/api/published/*
1616
quarkus.http.auth.permission.public.policy=permit
17+
18+
# Library API Configuration
19+
# Defaults to localhost:8083 (development mode)
20+
# For production, set LIBRARY_API_BASE_URL to production Cloud Run URL
21+
library-api.base-url=${LIBRARY_API_BASE_URL:http://localhost:8083}

‎devbox.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@
88
"google-cloud-sdk@latest",
99
"nodejs@22",
1010
"bruno-cli@latest",
11-
"process-compose@latest"
11+
"process-compose@latest",
12+
"python@latest"
1213
],
1314
"env_from": ".env",
1415
"shell": {

0 commit comments

Comments
 (0)