Add Devin for Group Audit demo kit (RACM ontology, playbooks, reporting templates)

Co-Authored-By: Alex Hammett <alex.hammett@windsurf.com>

Author: devin-ai-integration[bot]

audit/README.md

@@ -0,0 +1,95 @@
+# Devin for Group Audit — Demo Kit
+
+A self-contained demonstration of how Devin supports an **internal audit (third line of
+defence)** function: an **audit ontology** (Risk & Control Matrix), **control testing**
+(ITGC + application controls + code review as a continuous control), and **audit
+reporting** — all grounded in this banking application as the auditee.
+
+> **Thesis.** Devin is the *execution & evidence layer* of internal audit. Under the
+> auditor's direction and sign-off, it plans testing from a machine-readable control
+> catalogue, executes the tests, re-performs key calculations, reviews code changes as a
+> continuous control, and drafts workpapers & findings with full traceability. It does
+> **not** replace the auditor's judgement, independence, or sign-off.
+
+## What's in this kit
+
+| Path | What it is |
+|---|---|
+| `racm.yaml` | **The audit ontology** — Risk & Control Matrix: entities, risks, 11 controls, regulations, test procedures, expected findings |
+| `ontology.md` | The ontology model explained (entity → risk → control → test → evidence → finding → action) |
+| `playbooks/01-audit-planning-coverage.md` | Coverage mapping + fieldwork test plan (read-only scoping) |
+| `playbooks/02-itgc-testing.md` | ITGC testing over the full population (change mgmt, access, SDLC, secrets, data, logging) |
+| `playbooks/03-application-control-reperformance.md` | Re-perform transaction/transfer/authorisation controls |
+| `playbooks/04-findings-and-reporting.md` | 5-Cs findings, roll-up reporting, remediation validation |
+| `templates/workpaper-template.md` | Per-control workpaper |
+| `templates/finding-5cs-template.md` | Finding (Condition / Criteria / Cause / Consequence / Recommendation) |
+| `templates/audit-committee-onepager.md` | Audit-committee roll-up |
+| `samples/example-workpaper-APP-TXN-01.md` | Worked example workpaper (what Devin produces) |
+| `samples/example-finding-FND-01.md` | Worked example finding |
+| `ccm/continuous-controls-monitoring.md` | Scheduled continuous-controls-monitoring spec |
+
+## The demo flow (Ask Devin → Ask Devin → Devin session)
+
+Follows the progressive narrative: two lightweight **Ask Devin** discovery/scoping prompts,
+then one **Devin session** that executes and reports.
+
+**1. Ask Devin (discover):**
+> Here is our Cards platform repo and our Risk & Control Matrix at `audit/racm.yaml`. Map
+> each control to where it's implemented (or should be) in this codebase and tell me what's auditable.
+
+**2. Ask Devin (scope the fieldwork):**
+> Using `@playbook:01-audit-planning-coverage`, produce the fieldwork test plan: for each
+> control, the test procedure, the population, and the sample. Prioritise the key controls.
+
+**3. Devin session (execute + report):**
+> Run the audit fieldwork. Execute `@playbook:02-itgc-testing` and
+> `@playbook:03-application-control-reperformance`, then `@playbook:04-findings-and-reporting`.
+> Produce a workpaper per control, 5-Cs findings, and the audit-committee one-pager — every
+> result cited to `file:line`, commit/PR, or command output.
+
+*(Replace the `@playbook:` names with the `@playbook:playbook-<id>` references once the
+Devin Playbooks are created in the workspace.)*
+
+## Expected findings (so the demo is repeatable)
+
+These exist naturally in the auditee code — Devin discovers real gaps, nothing is faked:
+
+| Control | Result | The gap |
+|---|---|---|
+| APP-TXN-01 | **FAIL** | No positive-amount validation → negative transfer steals from recipient (`AccountService.java:51-135`) |
+| APP-TXN-02 | **FAIL** | `transferAmount` not `@Transactional` → partial failure destroys money (`AccountService.java:103-135`) |
+| APP-TXN-03 | **FAIL** | No limits, no maker-checker on transfers |
+| APP-ACC-04 | **FAIL** | Single hardcoded `"USER"` role, no SoD (`AccountService.java:99-101`) |
+| APP-SEC-05 | **FAIL** | CSRF disabled on state-changing endpoints (`SecurityConfig.java:30`) |
+| ITGC-SEC-06 | **FAIL** | DB credentials committed in `application.properties:4-5` |
+| ITGC-CM-07 | **FAIL** | No CODEOWNERS/branch protection; direct commits to `DevOps` |
+| ITGC-CM-08 | **FAIL** | Pipeline builds from a different upstream repo (`Jenkinsfile:26`) |
+| ITGC-SDLC-09 | **PARTIAL** | Trivy/OWASP/SonarQube present, but only test is `contextLoads()` |
+| ITGC-DATA-10 | **FAIL** | `ddl-auto=update` auto-mutates schema (`application.properties:9`) |
+| ITGC-LOG-11 | **FAIL** | No audit logging of financial events |
+
+## Optional closing beats
+
+- **Code review as a continuous control:** open a PR that subtly weakens a control (e.g.
+  remove the insufficient-funds check in `withdraw`) and let Devin Review flag the control
+  breach *before* merge. See "the weakening PR" in the demo notes below.
+- **Continuous Controls Monitoring:** schedule the CCM session (`ccm/`) to re-test nightly
+  and escalate only on change.
+- **Remediation validation:** fix APP-TXN-01, then have Devin re-test that one control and
+  close FND-01 with fresh evidence.
+
+### The "weakening PR" (seed for the code-review beat)
+In `AccountService.withdraw`, delete the guard:
+```java
+if (account.getBalance().compareTo(amount) < 0) {
+    throw new RuntimeException("Insufficient funds");
+}
+```
+Open it as a PR. Devin's review should flag that it removes a key transaction-integrity
+control (APP-TXN-01 family) and would allow overdrawn balances — caught pre-merge.
+
+## Guardrails (why this is audit-appropriate)
+- **Independence / human-in-the-loop:** Devin proposes & drafts; the auditor concludes and signs off.
+- **Read-only assurance:** in audit mode Devin never changes the audited system.
+- **The auditor is auditable:** every session is logged, reproducible, and exportable.
+- **Least privilege & synthetic data:** scoped, read-only access; no real customer data.

audit/ccm/continuous-controls-monitoring.md

@@ -0,0 +1,40 @@
+# Continuous Controls Monitoring (CCM)
+
+Point-in-time, sample-based testing is the traditional audit model. CCM uses a **scheduled
+Devin session** to re-run the automated control tests on a cadence, diff against the last
+run, and escalate **only on change** — moving toward continuous, full-population assurance.
+
+## Scope (automatable controls only)
+The controls suitable for unattended re-testing each cycle:
+
+| Control | What is re-checked each run |
+|---|---|
+| APP-TXN-01 | positive-amount validation still present on all monetary methods |
+| APP-TXN-02 | transfer still executes within a transaction boundary |
+| APP-ACC-04 | role/privilege model unchanged or improved |
+| APP-SEC-05 | CSRF protection state on state-changing endpoints |
+| ITGC-SEC-06 | no new secrets committed (secret scan over diff) |
+| ITGC-CM-07 | every new change to protected branch had an independent approval |
+| ITGC-SDLC-09 | scans + tests still gate; test coverage not regressed |
+| ITGC-DATA-10 | no uncontrolled runtime schema mutation reintroduced |
+| ITGC-LOG-11 | financial-event logging present |
+
+## Schedule spec
+- **Cadence:** nightly (or per-merge trigger on the protected branch).
+- **Trigger:** scheduled Devin session (see the `managing-schedules` capability) or a CI/webhook trigger.
+- **Prompt (pinned):**
+  > Using `audit/racm.yaml`, re-run the automated control tests for the controls flagged
+  > CCM-eligible. Compare results to the previous run. Escalate ONLY controls whose status
+  > changed (improved or regressed). Produce a short delta report and update open findings.
+- **Escalation:** post a delta report to the audit channel; open/refresh a finding for any
+  regression; auto-close (pending auditor sign-off) any finding whose control now passes.
+
+## Output each cycle
+- A **delta report**: controls changed since last run (regressions first), with citations.
+- Updated finding statuses (open / remediated / validated), each with fresh evidence.
+- The session transcript, retained as the audit trail for that monitoring cycle.
+
+## Guardrails
+- Read-only against the auditee.
+- No-change runs produce a one-line "no exceptions; no change" entry — no noise.
+- Auditor signs off any status transition to "validated".

audit/ontology.md

@@ -0,0 +1,65 @@
+# Audit Ontology
+
+The ontology is the backbone of "Devin for Group Audit". It is the machine-readable
+vocabulary that lets Devin turn *"go audit this system"* into concrete, repeatable,
+attributable tests — and lets every result roll up to a risk and a regulation.
+
+## The model
+
+```
+Auditable Entity ──has──> Risk ──mitigated by──> Control ──tested by──> Test Procedure
+       │                   │                        │                       │
+  (system/process)   (what goes wrong)     (preventive/detective/      (inquiry / observation /
+                                            corrective; manual/auto;     inspection / re-performance;
+                                            key/non-key; owner)          sample; pass/fail criteria)
+                                                 │
+                                         mapped to Regulation
+                                   (ICFR, PRA/FCA, SMCR, Consumer Duty,
+                                    Op-Resilience/DORA, GDPR)
+                                                 │
+  Test Procedure ──produces──> Evidence ──substantiates──> Result ──exception──> Finding
+                                                                                    │
+                                                                            tracked to Action
+                                                                          (remediation + re-test)
+```
+
+## Entities in the ontology
+
+| Concept | Meaning | Where it lives |
+|---|---|---|
+| **Auditable Entity** | A system or process in scope (the slice of the audit universe) | `racm.yaml: auditable_entities` |
+| **Risk** | What could go wrong if the control fails | `racm.yaml: risks` |
+| **Control** | The mechanism that mitigates the risk; typed by nature, automation, and whether it is *key* | `racm.yaml: controls` |
+| **Regulation** | The obligation the control supports | `racm.yaml: regulations` |
+| **Test Procedure** | How the control is tested (the four classic methods below) | `controls[].test_procedure` |
+| **Evidence** | Artifacts that substantiate a test result (must be a citation or re-runnable) | `controls[].evidence_required` |
+| **Finding / Issue** | An exception where the control failed, rated by severity | produced by Devin → `templates/finding-5cs-template.md` |
+| **Action** | Remediation, validated by a re-test | tracked via issue + re-test |
+
+## Control attributes (the typing that drives testing)
+
+- **Nature:** `preventive` (stops it happening) · `detective` (spots it after) · `corrective` (fixes it)
+- **Automation:** `automated` · `manual` · `hybrid`
+- **Key:** `true` if failure alone could lead to a material misstatement / significant risk
+- **Owner:** the accountable first-line owner (links to SMCR accountability)
+
+## The four test methods (auditing standard)
+
+1. **Inquiry** — ask the owner / read documentation (weakest alone).
+2. **Observation** — watch the control operate.
+3. **Inspection** — examine evidence (config, code, logs, PR metadata).
+4. **Re-performance** — independently execute the control and compare the result (strongest).
+
+Devin's leverage is greatest on **inspection** and **re-performance**, executed over the
+**full population** rather than a sample — e.g. inspecting every merge for approval, or
+re-performing a transfer calculation against crafted inputs.
+
+## Why this matters
+
+Because every control row carries a `control_id`, a `risk`, a `regulation`, a
+`test_procedure`, `pass_criteria`, and `evidence_required`, Devin's output is:
+
+- **Traceable** — each conclusion cites `file:line`, a PR/commit, or command output.
+- **Reproducible** — a reviewer or regulator can re-run the test.
+- **Roll-up-able** — results aggregate by risk, by regulation, and by key/non-key for the audit-committee view.
+- **Independent** — Devin tests and evidences; the auditor concludes and signs off.

audit/playbooks/01-audit-planning-coverage.md

@@ -0,0 +1,64 @@
+# Playbook — Audit Planning & Coverage Mapping
+
+## Overview
+Ingest the audit ontology (`audit/racm.yaml`) and the auditee codebase, then produce
+(a) a **coverage map** of where each control is implemented (or should be) in the code
+and (b) a **fieldwork test plan** with a procedure, population and sample per control.
+This is a read-only scoping activity — no findings are concluded here.
+
+## What's Needed From User
+- Repository access for the auditee (Springboot-BankApp)
+- The RACM at `audit/racm.yaml` (loaded as Knowledge or read from the repo)
+- The in-scope audit period (for change-management population)
+
+<phase name="Ontology & Scope Intake" id="1">
+## Phase 1 — Ontology & Scope Intake
+
+1. Read `audit/racm.yaml`. Enumerate every control with its `control_id`, `auditable_entity`, `risk`, `regulation`, `test_procedure`, `population`, and `pass_criteria`.
+2. Read `audit/ontology.md` to confirm the entity/risk/control/test/evidence/finding model.
+3. Confirm the in-scope period and the protected branch for change-management population.
+
+<verification>
+- Every control in `racm.yaml` is enumerated with its key attributes
+- Each control is mapped to its auditable entity and the regulation(s) it supports
+- The audit period and protected branch are recorded
+</verification>
+</phase>
+
+<phase name="Coverage Mapping" id="2">
+## Phase 2 — Coverage Mapping
+
+1. For each control, locate where it is (or should be) implemented in the codebase and record exact `file:line` references.
+2. Classify each control as: `implemented`, `partially implemented`, `not implemented`, or `not applicable` — with a one-line rationale and citation. Do NOT yet conclude pass/fail.
+3. Flag any control whose `auditable_entity.primary_source` does not exist or cannot be located.
+
+<verification>
+- Every control has at least one code citation or an explicit "no implementation found" note
+- Coverage classification recorded per control with rationale
+- Gaps in locating sources are flagged
+</verification>
+</phase>
+
+<phase name="Test Plan" id="3">
+## Phase 3 — Fieldwork Test Plan
+
+1. For each control, restate the test procedure, the population, the sampling approach (full population vs sample size), and the pass criteria.
+2. Choose the strongest feasible test method (prefer inspection / re-performance over inquiry).
+3. Produce the plan as a table ordered by `key: true` first, then by severity, and save it as a workpaper-style artifact.
+
+<verification>
+- Test plan covers 100% of controls in the RACM
+- Each control's population and sample are explicit and justified
+- Key controls are prioritised first
+- Output saved as a planning workpaper
+</verification>
+</phase>
+
+## Specifications
+- Output: a coverage map + a fieldwork test plan, both fully cited.
+- Read-only. No code changes. No pass/fail conclusions in this playbook.
+
+## Forbidden Actions
+- Do not modify the auditee codebase.
+- Do not conclude findings — that happens in the testing playbooks.
+- Do not rely on inquiry alone where inspection/re-performance is possible.

audit/playbooks/02-itgc-testing.md

@@ -0,0 +1,78 @@
+# Playbook — ITGC Testing
+
+## Overview
+Test the IT General Controls in the RACM (change management, logical access, secure
+SDLC, secrets/config, data/schema control, logging) by **inspection over the full
+population** of code, config, pipeline and repo history. Produce one workpaper per
+control with cited evidence and a pass/fail/partial result. Read-only.
+
+## What's Needed From User
+- Auditee repository access (code + full git history; un-shallow if needed)
+- `audit/racm.yaml`
+- The in-scope period and protected branch
+
+<phase name="Change Management & Pipeline Integrity" id="1">
+## Phase 1 — Change Management & Pipeline (ITGC-CM-07, ITGC-CM-08)
+
+1. Check for `CODEOWNERS` and branch-protection evidence on the protected branch (design test).
+2. For the in-scope population of changes to the protected branch, build a per-change evidence table: commit/PR, author, approver (must differ from author), linked change record, CI status. Note any direct-to-branch pushes.
+3. Trace the CI/CD checkout source (`Jenkinsfile`) against the audited repository to test build provenance (ITGC-CM-08).
+
+<verification>
+- Per-change evidence table produced for the population/sample
+- Author≠approver tested for each sampled change
+- Branch-protection / CODEOWNERS design state recorded
+- Pipeline source provenance traced and concluded
+</verification>
+</phase>
+
+<phase name="Logical Access & Secrets" id="2">
+## Phase 2 — Logical Access & Secrets (APP-ACC-04, APP-SEC-05, ITGC-SEC-06)
+
+1. Inspect the granted-authorities/role model and endpoint authorisation rules; assess least privilege and SoD (APP-ACC-04).
+2. Inspect security configuration for CSRF and protections on state-changing endpoints (APP-SEC-05).
+3. Scan all committed config and history for embedded secrets/credentials (ITGC-SEC-06). Cite exact `file:line`.
+
+<verification>
+- Role/privilege model assessed with citations
+- CSRF / state-changing-endpoint protection concluded
+- Secret scan run over source + history with results cited
+</verification>
+</phase>
+
+<phase name="SDLC, Data & Logging" id="3">
+## Phase 3 — Secure SDLC, Data & Logging (ITGC-SDLC-09, ITGC-DATA-10, ITGC-LOG-11)
+
+1. Inspect the pipeline for SAST/SCA/filesystem scanning and a test-execution gate; inventory the test suite and assess business-logic coverage (ITGC-SDLC-09).
+2. Inspect persistence config for uncontrolled runtime schema mutation (ITGC-DATA-10).
+3. Inspect transaction code paths for audit logging of financial/security events (ITGC-LOG-11).
+
+<verification>
+- Pipeline scanning + test gating concluded; test coverage inventoried
+- Schema/data change control concluded with citation
+- Financial-event logging concluded with citation
+</verification>
+</phase>
+
+<phase name="Workpapers" id="4">
+## Phase 4 — Workpapers
+
+1. For each ITGC tested, produce a workpaper using `audit/templates/workpaper-template.md`.
+2. Record result (pass/partial/fail), every piece of evidence as a citation, and any exceptions.
+3. Hand exceptions to the Findings & Reporting playbook.
+
+<verification>
+- One workpaper per ITGC control tested
+- Every result backed by a citation (file:line / commit / command output)
+- Exceptions listed and ready for findings
+</verification>
+</phase>
+
+## Specifications
+- Test the full population wherever feasible (e.g. all merges, all config).
+- Every conclusion must be reproducible from the cited evidence.
+
+## Forbidden Actions
+- Do not modify the auditee codebase or its history.
+- Do not conclude a pass without inspecting actual evidence (no inquiry-only passes).
+- Do not print secret values — cite the location only.