Merge branch 'main' into feature/structured-extraction-b3

Author: Abhijeet2409

.gitignore

@@ -64,6 +64,7 @@ standards_cache.sqlite
 
 ### Docs
 *.md
+!AGENTS.md
 
 ### Dev DBDumps
 *.sql

AGENTS.md

@@ -0,0 +1,136 @@
+# OpenCRE Agent Instructions
+
+Cursor agents working in this repo must follow the rules in `.cursor/rules/`.
+
+## Quick start
+
+1. **Requirements gate** — If goal, success criteria, `@` file refs, or constraints are missing, stop and ask (`requirements-gate.mdc`).
+2. **Plan first** — Non-trivial or multi-file work requires a plan and user approval before coding (`plan-first-workflow.mdc`, `multi-agent-workflow.mdc`).
+3. **Verify** — After code changes, run checks and iterate until green (`verifiable-goals.mdc`):
+   - `make lint`
+   - `make mypy`
+   - `make test`
+   - `make frontend` (if frontend touched)
+4. **Review** — Substantive work needs independent judge/subagent review (`multi-agent-workflow.mdc`).
+5. **Production gap analysis** — On Heroku/opencreorg, gap analysis is cache-only: serve precomputed rows from Postgres; do not compute GA on production (cache miss → 404).
+
+## Operational scripts
+
+Prefer existing Makefile targets and `scripts/` helpers over ad-hoc Docker, GA, import, or prod-DB setup. Read script headers or `--help` for env vars and flags; do not reimplement their logic.
+
+### Local Docker services
+
+Use Makefile targets — do not hand-roll `docker run`:
+
+```bash
+make docker-neo4j      # Neo4j (7474/7687)
+make docker-redis      # Redis Stack (6379/8001)
+make docker-postgres   # local Postgres (5432, cre/password)
+make start-containers  # neo4j + redis only
+make migrate-upgrade   # after postgres is up
+```
+
+Reset volumes: `make docker-neo4j-rm` / `make docker-redis-rm`.
+
+### Imports and Neo4j populate
+
+```bash
+make import-all        # full standards import via scripts/import-all.sh
+make import-projects   # skip core CRE import (CRE_SKIP_IMPORT_CORE=1)
+make import-neo4j      # populate Neo4j from cache
+```
+
+`scripts/import-all.sh` handles parallel importers, SQLite export, and verification. For embeddings-only SQLite → Postgres push, use `scripts/sync_embeddings_table.py` (see its docstring).
+
+### Gap analysis (local compute, prod verify)
+
+**Local backfill** (starts containers, workers, migrations — see `scripts/backfill_gap_analysis.sh`):
+
+```bash
+make backfill-gap-analysis              # parallel workers + --ga_backfill_missing
+make backfill-gap-analysis-sync         # Neo4j populate + backfill without queue
+make sync-gap-analysis-table-local      # upsert material rows sqlite → local Postgres
+```
+
+**Prod / staging checks** (HTTP only — never compute GA on opencreorg):
+
+```bash
+make verify-ga-complete-prod    # scripts/verify_ga_completeness.py
+make monitor-ga-health-prod     # scripts/monitor_ga_health.py (503 / empty result alerts)
+make verify-ga-parity-local     # scripts/verify_ga_postgres_neo_parity.py
+```
+
+Table sync between databases: `scripts/sync_gap_analysis_table.py`, `scripts/sync_embeddings_table.py`.
+
+### Production DB (Heroku opencreorg)
+
+Use `scripts/db/*` — never raw destructive `psql` against prod without these wrappers. All flows capture and wait for a fresh Heroku backup first (`scripts/db/common.sh`).
+
+```bash
+scripts/db/backup-opencreorg.sh                                    # backup only
+scripts/db/surgery-opencreorg.sh --sql-file path/to/change.sql     # targeted SQL
+scripts/db/surgery-opencreorg.sh --sql-file … --destructive        # DELETE/DROP/TRUNCATE
+scripts/db/sync-local-to-opencreorg.sh [--table node]…             # local → prod sync
+```
+
+Destructive surgery requires `CONFIRM_DESTRUCTIVE=I_UNDERSTAND_OPENCREORG_PROD_DB_DESTRUCTIVE_ACTION` (exact phrase). Override app: `APP_NAME=opencreorg`. See `production-db-ops-safety.mdc`.
+
+After prod GA cache changes, verify with `make verify-ga-complete-prod` / `make monitor-ga-health-prod`.
+
+### Weekly prod GA & data completeness (Cursor Automation)
+
+Schedule a **Cursor Automation** (not GitHub Actions) to run weekly — prod is cache-only; checks are HTTP + repo scripts only.
+
+| Setting | Value |
+|---------|--------|
+| Trigger | Cron: `0 9 * * 1` (Mondays 09:00) |
+| Repo | `OWASP/OpenCRE`, branch `main` |
+| Tools | None required (cloud agent uses repo checkout) |
+
+**Agent prompt (paste into Automations editor):**
+
+```
+Weekly OpenCRE prod GA and data completeness for opencreorg.
+
+1. python3 scripts/monitor_ga_health.py --base-url https://opencre.org --output-json tmp/prod-ga-health.json
+2. python3 scripts/verify_ga_completeness.py --base-url https://opencre.org --output-json tmp/prod-ga-completeness.json
+3. Confirm /rest/v1/standards and /rest/v1/ga_standards return non-empty lists.
+4. If incomplete_pairs > 0 or non-zero exit: list failing pairs/buckets; recommend AGENTS.md Operational scripts (local backfill + scripts/sync_gap_analysis_table.py). Do not compute GA on Heroku or run destructive prod DB ops without explicit approval.
+5. If all pass: report complete/total pairs and standards counts.
+```
+
+Create via **Cursor → Automations → New** (Agents Window). Do not hand-roll docker/GA setup in the automation prompt.
+
+### Staging bootstrap
+
+`scripts/setup-heroku-staging.sh` — provisions staging from prod + local SQLite; supports `--embeddings`, `--gap_analysis`, or full sync. Requires `PROD_APP`, `STAGING_APP`, `LOCAL_SQLITE_DB`.
+
+### Deploy / migrations
+
+Before deploy or `flask db upgrade`: `make alembic-guardrail` (or `python scripts/check_alembic_revision_guardrail.py`).
+
+## Rule index
+
+| Rule | Purpose |
+|------|---------|
+| `requirements-gate.mdc` | Clarifying questions + requirements template |
+| `complete-ticket.mdc` | Ticket gate for `.md`/`.txt` files; uses `requirements-gate` template + coding standards |
+| `plan-first-workflow.mdc` | Plan Mode before non-trivial edits |
+| `multi-agent-workflow.mdc` | Big changes, approval gates, builder ≠ judge |
+| `verifiable-goals.mdc` | Lint, mypy, test, CI — show evidence |
+| `never-assume.mdc` | No guessing; complete code; minimal scope |
+| `tdd-workflow.mdc` | Test-first for new behavior and importers |
+| `autonomous-workflow.mdc` | Execute after approval; no unsolicited commits |
+| `context-management.mdc` | `/clear`, `@` refs, stale context recovery |
+| `production-db-ops-safety.mdc` | Destructive prod DB confirmation |
+| `alembic-deploy-guardrail.mdc` | Pre-deploy migration guardrail |
+
+## OpenCRE commands
+
+```bash
+make lint              # black + frontend prettier
+make mypy              # Python typecheck
+make test              # Python unittest suite
+make frontend          # yarn build (when TS/TSX changed)
+make alembic-guardrail # before deploy/migration ops
+```

Makefile

@@ -153,6 +153,12 @@ verify-ga-complete-prod:
 		--base-url "https://opencre.org" \
 		--output-json "$(CURDIR)/tmp/prod-ga-completeness.json"
 
+monitor-ga-health-prod:
+	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
+	python scripts/monitor_ga_health.py \
+		--base-url "https://opencre.org" \
+		--output-json "$(CURDIR)/tmp/prod-ga-health.json"
+
 verify-ga-parity-local:
 	@[ -d "./.venv" ] && . ./.venv/bin/activate || ([ -d "./venv" ] && . ./venv/bin/activate); \
 	export CRE_CACHE_FILE="$${CRE_CACHE_FILE:-postgresql://cre:password@127.0.0.1:5432/cre}"; \

application/database/db.py

@@ -47,6 +47,7 @@
     make_resources_key,
     make_subresources_key,
     primary_gap_analysis_payload_is_material,
+    should_persist_primary_gap_analysis_cache,
 )
 
 
@@ -2428,6 +2429,22 @@ def add_gap_analysis_result(self, cache_key: str, ga_object: str):
             .filter(GapAnalysisResults.cache_key == cache_key)
             .first()
         )
+        if gap_analysis_cache_key_is_primary(cache_key):
+            existing_payload = existing.ga_object if existing else None
+            if not should_persist_primary_gap_analysis_cache(
+                ga_object, existing_payload
+            ):
+                if existing is None:
+                    logger.info(
+                        "Skipping empty primary gap analysis cache insert for %s",
+                        cache_key,
+                    )
+                else:
+                    logger.warning(
+                        "Refusing non-material primary gap analysis update for %s",
+                        cache_key,
+                    )
+                return
         if existing:
             existing.ga_object = ga_object
             self.session.add(existing)

application/tests/db_test.py

@@ -1327,15 +1327,73 @@ def test_gap_analysis_paths_without_base_standard_nodes(self, gap_mock):
             collection.gap_analysis_exists(make_resources_key(["788-788", "788-789"])),
         )
 
+    def test_add_gap_analysis_result_skips_empty_primary_insert(self):
+        collection = db.Node_collection()
+        key = make_resources_key(["788-788", "b"])
+        collection.add_gap_analysis_result(key, '{"result": {}}')
+        self.assertIsNone(
+            collection.session.query(db.GapAnalysisResults)
+            .filter(db.GapAnalysisResults.cache_key == key)
+            .first()
+        )
+        self.assertFalse(collection.gap_analysis_exists(key))
+
+    def test_add_gap_analysis_result_does_not_clobber_material_primary(self):
+        collection = db.Node_collection()
+        key = make_resources_key(["788-788", "b"])
+        material = '{"result": {"111-111": {"paths": {}}}}'
+        collection.add_gap_analysis_result(key, material)
+        collection.add_gap_analysis_result(key, '{"result": {}}')
+        row = (
+            collection.session.query(db.GapAnalysisResults)
+            .filter(db.GapAnalysisResults.cache_key == key)
+            .first()
+        )
+        self.assertEqual(row.ga_object, material)
+        self.assertTrue(collection.gap_analysis_exists(key))
+
+    def test_add_gap_analysis_result_allows_material_over_empty_primary(self):
+        collection = db.Node_collection()
+        key = make_resources_key(["788-788", "b"])
+        collection.session.add(
+            db.GapAnalysisResults(cache_key=key, ga_object='{"result": {}}')
+        )
+        collection.session.commit()
+        material = '{"result": {"111-111": {"paths": {}}}}'
+        collection.add_gap_analysis_result(key, material)
+        row = (
+            collection.session.query(db.GapAnalysisResults)
+            .filter(db.GapAnalysisResults.cache_key == key)
+            .first()
+        )
+        self.assertEqual(row.ga_object, material)
+
+    def test_add_gap_analysis_result_allows_empty_subresource(self):
+        collection = db.Node_collection()
+        sub = make_subresources_key(["788-788", "b"], "111-111")
+        collection.add_gap_analysis_result(sub, '{"result": {}}')
+        row = (
+            collection.session.query(db.GapAnalysisResults)
+            .filter(db.GapAnalysisResults.cache_key == sub)
+            .first()
+        )
+        self.assertIsNotNone(row)
+        self.assertEqual(row.ga_object, '{"result": {}}')
+
     @patch.object(db.NEO_DB, "gap_analysis")
     def test_gap_analysis_removes_stale_empty_primary_when_neo_empty(self, gap_mock):
         """Placeholder ``{"result":{}}`` rows must not survive a recompute with no Neo paths."""
         collection = db.Node_collection()
         collection.neo_db.connected = True
         key = make_resources_key(["788-788", "b"])
-        collection.add_gap_analysis_result(key, '{"result": {}}')
+        collection.session.add(
+            db.GapAnalysisResults(cache_key=key, ga_object='{"result": {}}')
+        )
         sub = make_subresources_key(["788-788", "b"], "111-111")
-        collection.add_gap_analysis_result(sub, '{"result": {}}')
+        collection.session.add(
+            db.GapAnalysisResults(cache_key=sub, ga_object='{"result": {}}')
+        )
+        collection.session.commit()
         self.assertFalse(collection.gap_analysis_exists(key))
 
         gap_mock.return_value = ([], [])

application/tests/ga_parity_test.py

@@ -35,6 +35,16 @@ def test_empty_result_json_not_material(self):
     def test_whitespace_only_tags_not_material(self):
         self.assertFalse(gap_analysis.primary_gap_analysis_payload_is_material("  "))
 
+    def test_primary_cache_key_ignores_arrow_in_importing_name(self):
+        self.assertTrue(
+            gap_analysis.gap_analysis_cache_key_is_primary("A->B >> Compare")
+        )
+        self.assertFalse(
+            gap_analysis.gap_analysis_cache_key_is_primary(
+                "A >> Compare->node-section-id"
+            )
+        )
+
 
 if __name__ == "__main__":
     unittest.main()

application/tests/gap_analysis_pair_job_test.py

@@ -27,6 +27,25 @@ def get_gap_analysis_result(self, cache_key: str):
 
 
 class TestGapAnalysisPairJob(unittest.TestCase):
+    def test_should_persist_primary_gap_analysis_cache(self):
+        g = gap_analysis
+        self.assertFalse(
+            g.should_persist_primary_gap_analysis_cache('{"result":{}}', None)
+        )
+        self.assertFalse(
+            g.should_persist_primary_gap_analysis_cache(
+                '{"result":{}}', '{"result":{"k":1}}'
+            )
+        )
+        self.assertTrue(
+            g.should_persist_primary_gap_analysis_cache(
+                '{"result":{"k":1}}', '{"result":{}}'
+            )
+        )
+        self.assertTrue(
+            g.should_persist_primary_gap_analysis_cache('{"result":{"k":1}}', None)
+        )
+
     def test_primary_gap_analysis_payload_is_material(self):
         g = gap_analysis
         self.assertFalse(g.primary_gap_analysis_payload_is_material(None))

application/tests/monitor_ga_health_test.py

@@ -0,0 +1,33 @@
+import io
+import unittest
+import urllib.error
+from unittest import mock
+
+from scripts import monitor_ga_health as monitor
+
+
+class MonitorGaHealthTest(unittest.TestCase):
+    def test_material_result_detection(self) -> None:
+        self.assertTrue(monitor._http_gap_result_is_material({"a": 1}))
+        self.assertFalse(monitor._http_gap_result_is_material({}))
+        self.assertFalse(monitor._http_gap_result_is_material(None))
+
+    def test_check_pair_503_uses_regression_bucket(self) -> None:
+        body = b"Service Unavailable"
+
+        def _raise_503(*_args, **_kwargs):
+            raise urllib.error.HTTPError(
+                "http://example", 503, "Service Unavailable", {}, io.BytesIO(body)
+            )
+
+        with mock.patch("urllib.request.urlopen", side_effect=_raise_503):
+            result = monitor._check_pair("https://opencre.org/rest/v1", "A", "B", 10)
+
+        self.assertIsNotNone(result)
+        assert result is not None
+        self.assertEqual(result["bucket"], "http_503_regression")
+        self.assertEqual(result["status_code"], 503)
+
+
+if __name__ == "__main__":
+    unittest.main()