Full Google Cloud + Dynatrace stack: Vertex AI Gemini, Cloud Firestore, live Dynatrace

- Persistence migrated MongoDB -> Google Cloud Firestore (100% Google Cloud)
- Gemini on Vertex AI (Cloud Run service identity) + live Dynatrace REST
- Official Dynatrace MCP server integration (local + code path)
- README/INTEGRATIONS/.env docs + frontend   Powered by  updated

Author: skypank-coder

.env.example

@@ -1,20 +1,32 @@
 # ── BACKEND (copy to backend/.env) ───────────────────────────────────────────
 
-# Gemini — recommended. Without it, agents use the scenario-accurate offline mode.
-GEMINI_API_KEY=your_gemini_api_key_here
-# Optional: pin a specific model. Otherwise a resilient candidate chain is used.
-GEMINI_MODEL=
+# Google Cloud AI — Gemini.
+#  - On Cloud Run: set GOOGLE_GENAI_USE_VERTEXAI=true + GOOGLE_CLOUD_PROJECT;
+#    auth is the service's own identity (no key needed) → Gemini via Vertex AI.
+#  - Locally: a GEMINI_API_KEY (AI Studio) is the simplest fallback.
+GOOGLE_GENAI_USE_VERTEXAI=false
 GOOGLE_CLOUD_PROJECT=your_gcp_project_id
+GOOGLE_CLOUD_LOCATION=us-central1
+GEMINI_API_KEY=your_gemini_api_key_here
+GEMINI_MODEL=gemini-2.5-flash
 
 # Demo mode — true uses simulated Dynatrace telemetry (no account needed).
 DEMO_MODE=true
 
-# Optional: real Dynatrace (only used when DEMO_MODE=false)
+# Live Dynatrace via the OFFICIAL MCP server (preferred). DT_ENVIRONMENT is your
+# platform/apps URL; the agent connects to @dynatrace-oss/dynatrace-mcp-server.
+# (Local uses an interactive browser login. Leave unset on headless hosts unless
+#  you also provide OAUTH_CLIENT_ID/OAUTH_CLIENT_SECRET/OAUTH_URN.)
+DT_ENVIRONMENT=https://your-env-id.apps.dynatrace.com
+
+# Legacy fallback: classic Dynatrace REST (used only if DT_ENVIRONMENT is unset).
 DYNATRACE_URL=https://your-env-id.live.dynatrace.com
 DYNATRACE_API_TOKEN=your_dynatrace_api_token_here
 
-# Optional: MongoDB Atlas for persistent incident history (graceful if unset)
-MONGODB_URI=mongodb+srv://user:pass@cluster.mongodb.net/specterops
+# Persistent incident history — Google Cloud Firestore (graceful in-memory fallback).
+# Just set GOOGLE_CLOUD_PROJECT (above) and enable Firestore on the project; on
+# Cloud Run it authenticates as the service identity. Set to disable explicitly:
+# FIRESTORE_DISABLED=true
 
 # Optional: Slack incoming webhook to deliver post-mortems to a channel
 SLACK_WEBHOOK_URL=https://hooks.slack.com/services/your/webhook/url

INTEGRATIONS.md

@@ -66,18 +66,22 @@ Lets SpecterOps pull **real open problems** from your tenant and analyze them.
 
 ---
 
-## 3. MongoDB Atlas (persistent history) — free M0
+## 3. Google Cloud Firestore (persistent history) — free tier
 
-Without it, incidents live in memory and reset on restart. With it, history persists.
+Without it, incidents live in memory and reset on restart. With it, history persists —
+on Google Cloud, so the whole stack stays on Google + Dynatrace (fully track-compliant).
 
-1. Create a free account at **https://www.mongodb.com/atlas** and a free **M0** cluster.
-2. **Database Access** → Add a database user (username + password).
-3. **Network Access** → Add IP → allow your IP (or `0.0.0.0/0` for testing).
-4. **Connect → Drivers → Python** → copy the SRV connection string and insert your password:
+1. In the [Google Cloud Console](https://console.cloud.google.com), select your project.
+2. Enable the API and create the default database (Native mode):
+   ```bash
+   gcloud services enable firestore.googleapis.com
+   gcloud firestore databases create --location=nam5
    ```
-   MONGODB_URI=mongodb+srv://USER:PASSWORD@cluster0.xxxxx.mongodb.net/specterops
-   ```
-5. Put it in `backend/.env`, restart. The status badge flips to **DB · persisted**.
+3. Set `GOOGLE_CLOUD_PROJECT=<your-project-id>` in `backend/.env`.
+   - **Local:** run `gcloud auth application-default login` once for credentials.
+   - **Cloud Run:** nothing to do — it authenticates as the service's own identity
+     (grant the runtime service account `roles/datastore.user`).
+4. Restart. The status badge flips to **Firestore · persisted**.
 
 ---
 

README.md

@@ -23,7 +23,7 @@ evidence-backed post-mortem in **under 60 seconds** — no human in the loop.
 [![Partner: Dynatrace](https://img.shields.io/badge/Partner-Dynatrace-9b6cff)]()
 [![AI: Gemini 2.5 Flash](https://img.shields.io/badge/AI-Gemini%202.5%20Flash-4f8cff)]()
 [![Frontend: Firebase](https://img.shields.io/badge/Frontend-Firebase%20Hosting-f5a623)]()
-[![Backend: Render](https://img.shields.io/badge/Backend-Render-46e3b7)]()
+[![Backend: Cloud Run](https://img.shields.io/badge/Backend-Cloud%20Run-4f8cff)]()
 [![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-3776ab.svg)]()
 [![React 18](https://img.shields.io/badge/React-18-61dafb.svg)]()
 
@@ -57,18 +57,18 @@ cascade across three services — with **94% confidence, in ~0.3s of compute.**
 ## ✨ What's inside
 
 - 🤖 **Four autonomous agents** orchestrated end-to-end, reasoning with **Google Gemini 2.5 Flash**.
-- 🟣 **Live Dynatrace integration** — **browse all open problems** in your tenant and investigate any one you pick (or auto-grab the latest); the finished RCA can be **pushed back onto the problem as a comment** (closing the loop).
+- 🟣 **Official Dynatrace MCP server integration** — the agent is a Model Context Protocol client that calls real MCP tools (`list_problems`, `execute_dql`, `find_entity_by_name`) to **browse all open problems** and investigate any one with live telemetry.
 - 🧪 **Scenario library + custom incidents** — four real-world failure scenarios (missing index, memory leak, bad deploy, dependency outage) plus a **"describe your own"** form.
 - 📡 **Real-time SSE streaming** — every agent's start/finish streams live to the dashboard.
 - 🕸️ **D3 causal graph** + **Three.js** 3D hero + **Framer Motion** — a business-grade UI, not a toy.
 - 🔐 **Firebase auth** (Google + email) with a graceful demo fallback; investigations are saved **per user**.
-- 💾 **MongoDB Atlas persistence** — history survives restarts; graceful in-memory fallback when unset.
+- 💾 **Cloud Firestore persistence** — history survives restarts on Google Cloud; graceful in-memory fallback when unset.
 - 📤 **Export & deliver** — copy/download the post-mortem as Markdown, or **send it to Slack**.
 - 🛡️ **Resilient by design** — a Gemini model-fallback chain, a scenario-accurate offline mode, and graceful degradation everywhere mean it runs with **zero accounts** and never breaks on stage.
 - ✅ **Tested** — `pytest` suite + **GitHub Actions CI** on every push.
 
 > Every integration is optional and upgrades a capability from *demo* → *real*. See
-> **[INTEGRATIONS.md](INTEGRATIONS.md)** for exact setup (Gemini, Dynatrace, MongoDB, Firebase, Slack).
+> **[INTEGRATIONS.md](INTEGRATIONS.md)** for exact setup (Gemini, Dynatrace, Firestore, Firebase, Slack).
 
 ---
 
@@ -169,47 +169,45 @@ cd frontend && npm install && npm run dev
 | Layer | Technology |
 |:------|:-----------|
 | **Agents / Backend** | Python 3.11 · FastAPI · asyncio · Server-Sent Events |
-| **Reasoning** | Google **Gemini 2.5 Flash** via the `google-genai` SDK (resilient model-fallback chain) |
-| **Observability** | **Dynatrace** v2 API — problems · metrics · entities · logs |
+| **Reasoning** | Google **Gemini 2.5 Flash** on **Vertex AI** (Google Cloud), via the `google-genai` SDK |
+| **Observability** | **Dynatrace** via the official **MCP server** — `list_problems` · `execute_dql` · `find_entity_by_name` |
 | **Frontend** | React 18 + Vite · Tailwind CSS · Framer Motion · **Three.js** (3D hero) · **D3** force-directed graph |
 | **Auth** | Firebase Authentication (Google + email) with graceful demo fallback |
-| **Persistence** | MongoDB Atlas · graceful in-memory fallback |
-| **Deployment** | Docker · **Render** (backend) · **Firebase Hosting** (frontend) — 100% free tier |
+| **Persistence** | **Google Cloud Firestore** · graceful in-memory fallback |
+| **Deployment** | Docker · **Google Cloud Run** (backend) · **Firebase Hosting** (frontend) |
 | **Quality** | `pytest` + `pytest-asyncio` · GitHub Actions CI |
 
 </div>
 
-### Dynatrace integration
+### Dynatrace integration — via the official MCP server
 
-The `TraceArchaeologist` talks to Dynatrace through one async client
-([`dynatrace_client.py`](backend/mcp/dynatrace_client.py)). The **"Investigate latest live
-Dynatrace problem"** action auto-pulls a real open problem and runs the full pipeline on it;
-the finished RCA can be **pushed back onto the problem** as a comment (closing the loop).
+This is the heart of the Dynatrace track. The `TraceArchaeologist` agent is a **Model
+Context Protocol client** ([`dynatrace_mcp.py`](backend/connectors/dynatrace_mcp.py)) that
+connects to the **official Dynatrace MCP server** (`@dynatrace-oss/dynatrace-mcp-server`)
+and calls real MCP tools to investigate live incidents:
 
-| Capability     | Endpoint                          | Token scope        |
-|:---------------|:----------------------------------|:-------------------|
-| Open problems  | `GET  /api/v2/problems`           | `problems.read`    |
-| Problem detail | `GET  /api/v2/problems/{id}`      | `problems.read`    |
-| Metrics        | `GET  /api/v2/metrics/query`      | `metrics.read`     |
-| Topology       | `GET  /api/v2/entities`           | `entities.read`    |
-| Logs           | `GET  /api/v2/logs/search`        | `logs.read`        |
-| Push RCA back  | `POST /api/v2/problems/{id}/comments` | `problems.write` |
+| MCP tool | What SpecterOps does with it |
+|:---|:---|
+| **`list_problems`** | Pull live open problems from the tenant (the picker + the chosen incident) |
+| **`execute_dql`** | Fetch logs / events / spans / metrics via Grail DQL as investigation evidence |
+| **`find_entity_by_name`** | Resolve the affected monitored entity |
 
-`DEMO_MODE=true` uses a deterministic simulation ([`scenarios.py`](backend/mcp/scenarios.py))
-so the system runs with **zero accounts**; `DEMO_MODE=false` investigates **real** problems.
-Response time (µs) and failure rate (%) from live Dynatrace are unit-normalized; values that
-can't be resolved are hidden rather than shown wrong.
+A single MCP server process + session is spawned and reused, so the OAuth login happens once.
+`DEMO_MODE=true` uses a deterministic simulation ([`scenarios.py`](backend/connectors/scenarios.py))
+so the system also runs with **zero accounts**; a classic Dynatrace REST client remains as a
+legacy fallback. Live values are unit-normalized; anything that can't be resolved is hidden
+rather than shown wrong.
 
 ### Environment variables & going live
 
 Everything runs with no keys. Each one upgrades a capability from demo → real — see the full
-step-by-step **[INTEGRATIONS.md](INTEGRATIONS.md)** (Gemini, Dynatrace, MongoDB, Firebase, Slack).
+step-by-step **[INTEGRATIONS.md](INTEGRATIONS.md)** (Gemini, Dynatrace, Firestore, Firebase, Slack).
 
 | Variable | Unlocks |
 |:---|:---|
-| `GEMINI_API_KEY` (+ `GEMINI_MODEL=gemini-2.5-flash`) | Real AI reasoning (vs offline fallback) |
+| `GOOGLE_GENAI_USE_VERTEXAI=true` + `GOOGLE_CLOUD_PROJECT` + `GOOGLE_CLOUD_LOCATION` | Gemini reasoning on **Vertex AI** (Google Cloud) |
 | `DEMO_MODE=false` + `DYNATRACE_URL` + `DYNATRACE_API_TOKEN` | Investigate real Dynatrace problems |
-| `MONGODB_URI` | Persistent incident history |
+| `GOOGLE_CLOUD_PROJECT` (+ Firestore enabled) | Persistent incident history on **Cloud Firestore** |
 | `VITE_FIREBASE_*` (in `frontend/.env`) | Real Google / email sign-in |
 | `SLACK_WEBHOOK_URL` | Deliver post-mortems to Slack |
 
@@ -240,17 +238,17 @@ DEMO_MODE=true python -m pytest      # 7 passed
 ```
 SpecterOps/
 ├── .github/workflows/ci.yml    # backend tests + frontend build
-├── render.yaml                 # Render Blueprint (backend deploy)
 ├── firebase.json · .firebaserc # Firebase Hosting (frontend deploy)
-├── INTEGRATIONS.md             # go-live guide (Gemini · Dynatrace · Mongo · Firebase · Slack)
+├── backend/Dockerfile          # container → Google Cloud Run (Node + Python + MCP)
+├── INTEGRATIONS.md             # go-live guide (Gemini · Dynatrace · Firestore · Firebase · Slack)
 ├── LICENSE                     # MIT
 ├── backend/                    # FastAPI app
 │   ├── agents/                 # orchestrator + the 4 agents
-│   ├── mcp/                    # Dynatrace client + scenario engine
+│   ├── connectors/             # Dynatrace MCP client + scenario engine
 │   ├── models/                 # Pydantic domain models
 │   ├── routes/                 # incidents · webhook · SSE stream · config
 │   ├── utils/                  # Gemini client (model fallback chain)
-│   ├── storage.py              # MongoDB persistence (graceful fallback)
+│   ├── storage.py              # Cloud Firestore persistence (graceful fallback)
 │   └── tests/                  # pytest suite
 └── frontend/                   # React + Vite dashboard
     └── src/components/         # launcher · agents · D3 graph · evidence · post-mortem · auth
@@ -274,7 +272,7 @@ SpecterOps/
 ## 🗺️ Roadmap
 
 Already shipped: live Dynatrace problem pull + RCA push-back, scenario library + custom
-incidents, MongoDB persistence, Firebase auth, Slack delivery. Next:
+incidents, Cloud Firestore persistence, Firebase auth, Slack delivery. Next:
 
 - 🤖 Approval-gated **auto-remediation execution** (wire the proposed fix into CI/CD)
 - 👥 **Per-user Dynatrace** connect (multi-tenant, OAuth) so each user investigates their own tenant
@@ -290,5 +288,5 @@ incidents, MongoDB persistence, Firebase auth, Slack delivery. Next:
 
 <div align="center">
 <br/>
-<sub>Built with Gemini 2.5 Flash · Dynatrace · Render · Firebase — for the Google Cloud Rapid Agent Hackathon 2026.</sub>
+<sub>Built with Gemini 2.5 Flash on Vertex AI · Dynatrace MCP · Cloud Run · Firebase — for the Google Cloud Rapid Agent Hackathon 2026.</sub>
 </div>

backend/.dockerignore

@@ -0,0 +1,7 @@
+venv/
+**/__pycache__/
+*.pyc
+.env
+*.log
+.pytest_cache/
+_*.py

backend/.gcloudignore

@@ -0,0 +1,8 @@
+# Keep the Cloud Build upload small and free of secrets / local junk.
+venv/
+**/__pycache__/
+*.pyc
+.env
+*.log
+.pytest_cache/
+_*.py

backend/Dockerfile

@@ -1,8 +1,19 @@
 FROM python:3.11-slim
 WORKDIR /app
+
+# Node.js — required to run the official Dynatrace MCP server
+# (@dynatrace-oss/dynatrace-mcp-server) that the agent connects to as an MCP client.
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends curl ca-certificates gnupg \
+    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+    && apt-get install -y --no-install-recommends nodejs \
+    && npm install -g @dynatrace-oss/dynatrace-mcp-server \
+    && apt-get purge -y curl gnupg && apt-get autoremove -y \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt
 COPY . .
 EXPOSE 8080
-# Bind the platform-provided $PORT (Render, Cloud Run, etc.); default 8080 locally.
+# Bind the platform-provided $PORT (Cloud Run, Render, etc.); default 8080 locally.
 CMD ["sh", "-c", "uvicorn main:app --host 0.0.0.0 --port ${PORT:-8080} --workers 1"]

backend/agents/blame_agent.py

@@ -11,7 +11,7 @@
 from datetime import datetime, timedelta
 from typing import Any, Dict
 
-from mcp import scenarios
+from connectors import scenarios
 from models.incident import (
     AgentStep,
     CausalEdge,

backend/agents/narrator_agent.py

@@ -8,7 +8,7 @@
 
 from typing import List
 
-from mcp import scenarios
+from connectors import scenarios
 from models.incident import AgentStep, Incident
 from utils.gemini_client import gemini_reason
 

backend/agents/orchestrator.py

@@ -22,7 +22,7 @@
 from agents.trace_agent import run_trace_archaeologist
 from models.incident import Incident, IncidentStatus
 
-# In-memory stores (sufficient for the demo; swap for MongoDB in production).
+# In-memory hot cache; durably mirrored to Google Cloud Firestore (see storage.py).
 INCIDENTS: Dict[str, Incident] = {}
 SSE_QUEUES: Dict[str, List[asyncio.Queue]] = {}
 # Per-incident event log so late subscribers can replay everything they missed.

backend/agents/sentinel_agent.py

@@ -10,7 +10,7 @@
 import json
 from typing import Any, Dict
 
-from mcp import scenarios
+from connectors import scenarios
 from models.incident import AgentStep, Incident, IncidentSeverity
 from utils.gemini_client import gemini_json