diff --git a/audit/README.md b/audit/README.md new file mode 100644 index 00000000..5cedf64c --- /dev/null +++ b/audit/README.md @@ -0,0 +1,95 @@ +# Devin for Group Audit — Demo Kit + +A self-contained demonstration of how Devin supports an **internal audit (third line of +defence)** function: an **audit ontology** (Risk & Control Matrix), **control testing** +(ITGC + application controls + code review as a continuous control), and **audit +reporting** — all grounded in this banking application as the auditee. + +> **Thesis.** Devin is the *execution & evidence layer* of internal audit. Under the +> auditor's direction and sign-off, it plans testing from a machine-readable control +> catalogue, executes the tests, re-performs key calculations, reviews code changes as a +> continuous control, and drafts workpapers & findings with full traceability. It does +> **not** replace the auditor's judgement, independence, or sign-off. + +## What's in this kit + +| Path | What it is | +|---|---| +| `racm.yaml` | **The audit ontology** — Risk & Control Matrix: entities, risks, 11 controls, regulations, test procedures, expected findings | +| `ontology.md` | The ontology model explained (entity → risk → control → test → evidence → finding → action) | +| `playbooks/01-audit-planning-coverage.md` | Coverage mapping + fieldwork test plan (read-only scoping) | +| `playbooks/02-itgc-testing.md` | ITGC testing over the full population (change mgmt, access, SDLC, secrets, data, logging) | +| `playbooks/03-application-control-reperformance.md` | Re-perform transaction/transfer/authorisation controls | +| `playbooks/04-findings-and-reporting.md` | 5-Cs findings, roll-up reporting, remediation validation | +| `templates/workpaper-template.md` | Per-control workpaper | +| `templates/finding-5cs-template.md` | Finding (Condition / Criteria / Cause / Consequence / Recommendation) | +| `templates/audit-committee-onepager.md` | Audit-committee roll-up | +| `samples/example-workpaper-APP-TXN-01.md` | Worked example workpaper (what Devin produces) | +| `samples/example-finding-FND-01.md` | Worked example finding | +| `ccm/continuous-controls-monitoring.md` | Scheduled continuous-controls-monitoring spec | + +## The demo flow (Ask Devin → Ask Devin → Devin session) + +Follows the progressive narrative: two lightweight **Ask Devin** discovery/scoping prompts, +then one **Devin session** that executes and reports. + +**1. Ask Devin (discover):** +> Here is our Cards platform repo and our Risk & Control Matrix at `audit/racm.yaml`. Map +> each control to where it's implemented (or should be) in this codebase and tell me what's auditable. + +**2. Ask Devin (scope the fieldwork):** +> Using `@playbook:01-audit-planning-coverage`, produce the fieldwork test plan: for each +> control, the test procedure, the population, and the sample. Prioritise the key controls. + +**3. Devin session (execute + report):** +> Run the audit fieldwork. Execute `@playbook:02-itgc-testing` and +> `@playbook:03-application-control-reperformance`, then `@playbook:04-findings-and-reporting`. +> Produce a workpaper per control, 5-Cs findings, and the audit-committee one-pager — every +> result cited to `file:line`, commit/PR, or command output. + +*(Replace the `@playbook:` names with the `@playbook:playbook-` references once the +Devin Playbooks are created in the workspace.)* + +## Expected findings (so the demo is repeatable) + +These exist naturally in the auditee code — Devin discovers real gaps, nothing is faked: + +| Control | Result | The gap | +|---|---|---| +| APP-TXN-01 | **FAIL** | No positive-amount validation → negative transfer steals from recipient (`AccountService.java:51-135`) | +| APP-TXN-02 | **FAIL** | `transferAmount` not `@Transactional` → partial failure destroys money (`AccountService.java:103-135`) | +| APP-TXN-03 | **FAIL** | No limits, no maker-checker on transfers | +| APP-ACC-04 | **FAIL** | Single hardcoded `"USER"` role, no SoD (`AccountService.java:99-101`) | +| APP-SEC-05 | **FAIL** | CSRF disabled on state-changing endpoints (`SecurityConfig.java:30`) | +| ITGC-SEC-06 | **FAIL** | DB credentials committed in `application.properties:4-5` | +| ITGC-CM-07 | **FAIL** | No CODEOWNERS/branch protection; direct commits to `DevOps` | +| ITGC-CM-08 | **FAIL** | Pipeline builds from a different upstream repo (`Jenkinsfile:26`) | +| ITGC-SDLC-09 | **PARTIAL** | Trivy/OWASP/SonarQube present, but only test is `contextLoads()` | +| ITGC-DATA-10 | **FAIL** | `ddl-auto=update` auto-mutates schema (`application.properties:9`) | +| ITGC-LOG-11 | **FAIL** | No audit logging of financial events | + +## Optional closing beats + +- **Code review as a continuous control:** open a PR that subtly weakens a control (e.g. + remove the insufficient-funds check in `withdraw`) and let Devin Review flag the control + breach *before* merge. See "the weakening PR" in the demo notes below. +- **Continuous Controls Monitoring:** schedule the CCM session (`ccm/`) to re-test nightly + and escalate only on change. +- **Remediation validation:** fix APP-TXN-01, then have Devin re-test that one control and + close FND-01 with fresh evidence. + +### The "weakening PR" (seed for the code-review beat) +In `AccountService.withdraw`, delete the guard: +```java +if (account.getBalance().compareTo(amount) < 0) { + throw new RuntimeException("Insufficient funds"); +} +``` +Open it as a PR. Devin's review should flag that it removes a key transaction-integrity +control (APP-TXN-01 family) and would allow overdrawn balances — caught pre-merge. + +## Guardrails (why this is audit-appropriate) +- **Independence / human-in-the-loop:** Devin proposes & drafts; the auditor concludes and signs off. +- **Read-only assurance:** in audit mode Devin never changes the audited system. +- **The auditor is auditable:** every session is logged, reproducible, and exportable. +- **Least privilege & synthetic data:** scoped, read-only access; no real customer data. diff --git a/audit/ccm/continuous-controls-monitoring.md b/audit/ccm/continuous-controls-monitoring.md new file mode 100644 index 00000000..41404196 --- /dev/null +++ b/audit/ccm/continuous-controls-monitoring.md @@ -0,0 +1,40 @@ +# Continuous Controls Monitoring (CCM) + +Point-in-time, sample-based testing is the traditional audit model. CCM uses a **scheduled +Devin session** to re-run the automated control tests on a cadence, diff against the last +run, and escalate **only on change** — moving toward continuous, full-population assurance. + +## Scope (automatable controls only) +The controls suitable for unattended re-testing each cycle: + +| Control | What is re-checked each run | +|---|---| +| APP-TXN-01 | positive-amount validation still present on all monetary methods | +| APP-TXN-02 | transfer still executes within a transaction boundary | +| APP-ACC-04 | role/privilege model unchanged or improved | +| APP-SEC-05 | CSRF protection state on state-changing endpoints | +| ITGC-SEC-06 | no new secrets committed (secret scan over diff) | +| ITGC-CM-07 | every new change to protected branch had an independent approval | +| ITGC-SDLC-09 | scans + tests still gate; test coverage not regressed | +| ITGC-DATA-10 | no uncontrolled runtime schema mutation reintroduced | +| ITGC-LOG-11 | financial-event logging present | + +## Schedule spec +- **Cadence:** nightly (or per-merge trigger on the protected branch). +- **Trigger:** scheduled Devin session (see the `managing-schedules` capability) or a CI/webhook trigger. +- **Prompt (pinned):** + > Using `audit/racm.yaml`, re-run the automated control tests for the controls flagged + > CCM-eligible. Compare results to the previous run. Escalate ONLY controls whose status + > changed (improved or regressed). Produce a short delta report and update open findings. +- **Escalation:** post a delta report to the audit channel; open/refresh a finding for any + regression; auto-close (pending auditor sign-off) any finding whose control now passes. + +## Output each cycle +- A **delta report**: controls changed since last run (regressions first), with citations. +- Updated finding statuses (open / remediated / validated), each with fresh evidence. +- The session transcript, retained as the audit trail for that monitoring cycle. + +## Guardrails +- Read-only against the auditee. +- No-change runs produce a one-line "no exceptions; no change" entry — no noise. +- Auditor signs off any status transition to "validated". diff --git a/audit/ontology.md b/audit/ontology.md new file mode 100644 index 00000000..7bb63647 --- /dev/null +++ b/audit/ontology.md @@ -0,0 +1,65 @@ +# Audit Ontology + +The ontology is the backbone of "Devin for Group Audit". It is the machine-readable +vocabulary that lets Devin turn *"go audit this system"* into concrete, repeatable, +attributable tests — and lets every result roll up to a risk and a regulation. + +## The model + +``` +Auditable Entity ──has──> Risk ──mitigated by──> Control ──tested by──> Test Procedure + │ │ │ │ + (system/process) (what goes wrong) (preventive/detective/ (inquiry / observation / + corrective; manual/auto; inspection / re-performance; + key/non-key; owner) sample; pass/fail criteria) + │ + mapped to Regulation + (ICFR, PRA/FCA, SMCR, Consumer Duty, + Op-Resilience/DORA, GDPR) + │ + Test Procedure ──produces──> Evidence ──substantiates──> Result ──exception──> Finding + │ + tracked to Action + (remediation + re-test) +``` + +## Entities in the ontology + +| Concept | Meaning | Where it lives | +|---|---|---| +| **Auditable Entity** | A system or process in scope (the slice of the audit universe) | `racm.yaml: auditable_entities` | +| **Risk** | What could go wrong if the control fails | `racm.yaml: risks` | +| **Control** | The mechanism that mitigates the risk; typed by nature, automation, and whether it is *key* | `racm.yaml: controls` | +| **Regulation** | The obligation the control supports | `racm.yaml: regulations` | +| **Test Procedure** | How the control is tested (the four classic methods below) | `controls[].test_procedure` | +| **Evidence** | Artifacts that substantiate a test result (must be a citation or re-runnable) | `controls[].evidence_required` | +| **Finding / Issue** | An exception where the control failed, rated by severity | produced by Devin → `templates/finding-5cs-template.md` | +| **Action** | Remediation, validated by a re-test | tracked via issue + re-test | + +## Control attributes (the typing that drives testing) + +- **Nature:** `preventive` (stops it happening) · `detective` (spots it after) · `corrective` (fixes it) +- **Automation:** `automated` · `manual` · `hybrid` +- **Key:** `true` if failure alone could lead to a material misstatement / significant risk +- **Owner:** the accountable first-line owner (links to SMCR accountability) + +## The four test methods (auditing standard) + +1. **Inquiry** — ask the owner / read documentation (weakest alone). +2. **Observation** — watch the control operate. +3. **Inspection** — examine evidence (config, code, logs, PR metadata). +4. **Re-performance** — independently execute the control and compare the result (strongest). + +Devin's leverage is greatest on **inspection** and **re-performance**, executed over the +**full population** rather than a sample — e.g. inspecting every merge for approval, or +re-performing a transfer calculation against crafted inputs. + +## Why this matters + +Because every control row carries a `control_id`, a `risk`, a `regulation`, a +`test_procedure`, `pass_criteria`, and `evidence_required`, Devin's output is: + +- **Traceable** — each conclusion cites `file:line`, a PR/commit, or command output. +- **Reproducible** — a reviewer or regulator can re-run the test. +- **Roll-up-able** — results aggregate by risk, by regulation, and by key/non-key for the audit-committee view. +- **Independent** — Devin tests and evidences; the auditor concludes and signs off. diff --git a/audit/playbooks/01-audit-planning-coverage.md b/audit/playbooks/01-audit-planning-coverage.md new file mode 100644 index 00000000..16e94520 --- /dev/null +++ b/audit/playbooks/01-audit-planning-coverage.md @@ -0,0 +1,64 @@ +# Playbook — Audit Planning & Coverage Mapping + +## Overview +Ingest the audit ontology (`audit/racm.yaml`) and the auditee codebase, then produce +(a) a **coverage map** of where each control is implemented (or should be) in the code +and (b) a **fieldwork test plan** with a procedure, population and sample per control. +This is a read-only scoping activity — no findings are concluded here. + +## What's Needed From User +- Repository access for the auditee (Springboot-BankApp) +- The RACM at `audit/racm.yaml` (loaded as Knowledge or read from the repo) +- The in-scope audit period (for change-management population) + + +## Phase 1 — Ontology & Scope Intake + +1. Read `audit/racm.yaml`. Enumerate every control with its `control_id`, `auditable_entity`, `risk`, `regulation`, `test_procedure`, `population`, and `pass_criteria`. +2. Read `audit/ontology.md` to confirm the entity/risk/control/test/evidence/finding model. +3. Confirm the in-scope period and the protected branch for change-management population. + + +- Every control in `racm.yaml` is enumerated with its key attributes +- Each control is mapped to its auditable entity and the regulation(s) it supports +- The audit period and protected branch are recorded + + + + +## Phase 2 — Coverage Mapping + +1. For each control, locate where it is (or should be) implemented in the codebase and record exact `file:line` references. +2. Classify each control as: `implemented`, `partially implemented`, `not implemented`, or `not applicable` — with a one-line rationale and citation. Do NOT yet conclude pass/fail. +3. Flag any control whose `auditable_entity.primary_source` does not exist or cannot be located. + + +- Every control has at least one code citation or an explicit "no implementation found" note +- Coverage classification recorded per control with rationale +- Gaps in locating sources are flagged + + + + +## Phase 3 — Fieldwork Test Plan + +1. For each control, restate the test procedure, the population, the sampling approach (full population vs sample size), and the pass criteria. +2. Choose the strongest feasible test method (prefer inspection / re-performance over inquiry). +3. Produce the plan as a table ordered by `key: true` first, then by severity, and save it as a workpaper-style artifact. + + +- Test plan covers 100% of controls in the RACM +- Each control's population and sample are explicit and justified +- Key controls are prioritised first +- Output saved as a planning workpaper + + + +## Specifications +- Output: a coverage map + a fieldwork test plan, both fully cited. +- Read-only. No code changes. No pass/fail conclusions in this playbook. + +## Forbidden Actions +- Do not modify the auditee codebase. +- Do not conclude findings — that happens in the testing playbooks. +- Do not rely on inquiry alone where inspection/re-performance is possible. diff --git a/audit/playbooks/02-itgc-testing.md b/audit/playbooks/02-itgc-testing.md new file mode 100644 index 00000000..717ebfdb --- /dev/null +++ b/audit/playbooks/02-itgc-testing.md @@ -0,0 +1,78 @@ +# Playbook — ITGC Testing + +## Overview +Test the IT General Controls in the RACM (change management, logical access, secure +SDLC, secrets/config, data/schema control, logging) by **inspection over the full +population** of code, config, pipeline and repo history. Produce one workpaper per +control with cited evidence and a pass/fail/partial result. Read-only. + +## What's Needed From User +- Auditee repository access (code + full git history; un-shallow if needed) +- `audit/racm.yaml` +- The in-scope period and protected branch + + +## Phase 1 — Change Management & Pipeline (ITGC-CM-07, ITGC-CM-08) + +1. Check for `CODEOWNERS` and branch-protection evidence on the protected branch (design test). +2. For the in-scope population of changes to the protected branch, build a per-change evidence table: commit/PR, author, approver (must differ from author), linked change record, CI status. Note any direct-to-branch pushes. +3. Trace the CI/CD checkout source (`Jenkinsfile`) against the audited repository to test build provenance (ITGC-CM-08). + + +- Per-change evidence table produced for the population/sample +- Author≠approver tested for each sampled change +- Branch-protection / CODEOWNERS design state recorded +- Pipeline source provenance traced and concluded + + + + +## Phase 2 — Logical Access & Secrets (APP-ACC-04, APP-SEC-05, ITGC-SEC-06) + +1. Inspect the granted-authorities/role model and endpoint authorisation rules; assess least privilege and SoD (APP-ACC-04). +2. Inspect security configuration for CSRF and protections on state-changing endpoints (APP-SEC-05). +3. Scan all committed config and history for embedded secrets/credentials (ITGC-SEC-06). Cite exact `file:line`. + + +- Role/privilege model assessed with citations +- CSRF / state-changing-endpoint protection concluded +- Secret scan run over source + history with results cited + + + + +## Phase 3 — Secure SDLC, Data & Logging (ITGC-SDLC-09, ITGC-DATA-10, ITGC-LOG-11) + +1. Inspect the pipeline for SAST/SCA/filesystem scanning and a test-execution gate; inventory the test suite and assess business-logic coverage (ITGC-SDLC-09). +2. Inspect persistence config for uncontrolled runtime schema mutation (ITGC-DATA-10). +3. Inspect transaction code paths for audit logging of financial/security events (ITGC-LOG-11). + + +- Pipeline scanning + test gating concluded; test coverage inventoried +- Schema/data change control concluded with citation +- Financial-event logging concluded with citation + + + + +## Phase 4 — Workpapers + +1. For each ITGC tested, produce a workpaper using `audit/templates/workpaper-template.md`. +2. Record result (pass/partial/fail), every piece of evidence as a citation, and any exceptions. +3. Hand exceptions to the Findings & Reporting playbook. + + +- One workpaper per ITGC control tested +- Every result backed by a citation (file:line / commit / command output) +- Exceptions listed and ready for findings + + + +## Specifications +- Test the full population wherever feasible (e.g. all merges, all config). +- Every conclusion must be reproducible from the cited evidence. + +## Forbidden Actions +- Do not modify the auditee codebase or its history. +- Do not conclude a pass without inspecting actual evidence (no inquiry-only passes). +- Do not print secret values — cite the location only. diff --git a/audit/playbooks/03-application-control-reperformance.md b/audit/playbooks/03-application-control-reperformance.md new file mode 100644 index 00000000..85908e7b --- /dev/null +++ b/audit/playbooks/03-application-control-reperformance.md @@ -0,0 +1,63 @@ +# Playbook — Application-Control Re-performance + +## Overview +Independently **re-perform** the key application/business-process controls in the RACM +(transaction validation, transfer atomicity, authorisation/dual-control) against crafted +test data, then confirm whether the code enforces what the RACM claims. Re-performance is +the strongest test method. Read-only against the auditee; any test harness runs in isolation. + +## What's Needed From User +- Auditee repository access +- `audit/racm.yaml` +- A test database / fixture (synthetic data only) + + +## Phase 1 — Trace the Control Paths + +1. For APP-TXN-01, APP-TXN-02, APP-TXN-03, locate the implementing code and trace the full execution path with `file:line` citations. +2. State, for each control, the behaviour the RACM claims and the behaviour the code actually implements. +3. Identify the boundary/abuse inputs to exercise (zero, negative, very large, self-transfer, partial-failure). + + +- Each application control's code path traced and cited +- Claimed vs actual behaviour stated per control +- Abuse/boundary input set defined + + + + +## Phase 2 — Re-perform Against Test Data + +1. Drive each control with the crafted inputs (via existing tests, a scratch harness, or careful reasoning where execution is infeasible) using synthetic data only. +2. Record the observed outcome vs the expected (policy-correct) outcome in a re-performance table. +3. Confirm transaction-boundary behaviour for multi-write operations (does a mid-operation failure roll back?). + + +- Re-performance table: input → expected → observed → pass/fail +- Negative/zero amount behaviour confirmed for deposit/withdraw/transfer +- Atomicity / partial-failure behaviour confirmed for transfer +- High-value transfer authorisation/limit behaviour confirmed + + + + +## Phase 3 — Workpapers + +1. Produce a workpaper per application control using `audit/templates/workpaper-template.md`. +2. Attach the re-performance table and code citations as evidence. +3. Hand exceptions to the Findings & Reporting playbook. + + +- One workpaper per application control re-performed +- Re-performance evidence attached and reproducible +- Exceptions listed for findings + + + +## Specifications +- Synthetic/anonymised data only. The harness must not touch production-like data. +- Prefer executable re-performance; where infeasible, document the reasoning explicitly and mark the method. + +## Forbidden Actions +- Do not modify auditee business logic to make a control "pass". +- Do not use real customer data. diff --git a/audit/playbooks/04-findings-and-reporting.md b/audit/playbooks/04-findings-and-reporting.md new file mode 100644 index 00000000..45b43abc --- /dev/null +++ b/audit/playbooks/04-findings-and-reporting.md @@ -0,0 +1,61 @@ +# Playbook — Findings & Reporting + +## Overview +Turn workpaper exceptions into rated findings (5 Cs), assemble a reproducible evidence +package, and produce the audit report draft and audit-committee one-pager. Optionally +re-test a remediated control to validate closure. The auditor reviews and signs off — Devin drafts. + +## What's Needed From User +- Completed workpapers from the ITGC and Application-Control playbooks +- `audit/racm.yaml` (for risk/regulation mapping and severity) +- Reporting templates in `audit/templates/` + + +## Phase 1 — Author Findings (5 Cs) + +1. For each exception, draft a finding using `audit/templates/finding-5cs-template.md`: Condition, Criteria, Cause, Consequence, Recommendation. +2. Assign severity from the control's `expected_demo_result.severity` / auditor judgement, and map the finding to its `control_id`, `risk`, and `regulation`. +3. Attach evidence as citations (`file:line`, commit/PR, command output) — every assertion must be reproducible. + + +- Every exception has a 5-Cs finding +- Each finding mapped to control_id + risk + regulation + severity +- Every finding's evidence is a citation a reviewer can re-check + + + + +## Phase 2 — Reporting Roll-up + +1. Produce the audit-committee one-pager using `audit/templates/audit-committee-onepager.md`: overall RAG, counts by severity, themes, and key-control failures. +2. Roll findings up by risk and by regulation so the committee sees obligation-level exposure. +3. Produce the detailed report body: per-control results table + the findings. + + +- One-pager produced with RAG, severity counts, themes +- Roll-up by risk and regulation present +- Detailed per-control results table complete + + + + +## Phase 3 — Remediation Validation + +1. When a fix lands for a finding, re-run only that control's test procedure. +2. Update the finding/issue status (open → remediated/validated) with fresh evidence and date. +3. Note any residual risk. + + +- Re-test performed for the specific remediated control +- Issue status updated with new evidence +- Residual risk recorded + + + +## Specifications +- Reporting principle: traceability + reproducibility. No assertion without a citation. +- Devin drafts; the auditor concludes and signs off (independence preserved). + +## Forbidden Actions +- Do not overstate severity or soften a finding without auditor instruction. +- Do not close a finding without a successful re-test. diff --git a/audit/racm.yaml b/audit/racm.yaml new file mode 100644 index 00000000..0ed753a8 --- /dev/null +++ b/audit/racm.yaml @@ -0,0 +1,355 @@ +# ============================================================================= +# Risk & Control Matrix (RACM) — Audit Ontology +# Auditee: BankApp (Springboot-BankApp) — Cards & Payments platform +# Owner: Group Audit (Third Line of Defence) +# Consumed by Devin as machine-readable Knowledge to plan and execute control testing. +# +# Ontology model: +# auditable_entity --has--> risk --mitigated_by--> control --tested_by--> test_procedure +# --produces--> evidence --substantiates--> result --exception--> finding --tracked_to--> action +# +# Every control row carries: a control_id, the risk it mitigates, the regulation(s) +# it supports, a concrete test procedure, a population, pass criteria, and the +# evidence required. This is what makes Devin's output audit-grade and roll-up-able. +# ============================================================================= + +meta: + framework: "Three Lines of Defence — Group Audit (3rd line)" + auditee: "BankApp / Springboot-BankApp" + period: "FY in-scope period (set per engagement)" + testing_basis: "Design effectiveness + Operating effectiveness" + evidence_principle: "Every result links to file:line, PR/commit, or re-runnable command output" + human_in_the_loop: "Devin proposes results & drafts findings; the auditor concludes and signs off" + +# ---------------------------------------------------------------------------- +# Auditable entities (the slice of the audit universe in scope) +# ---------------------------------------------------------------------------- +auditable_entities: + - id: AE-APP + name: "Core banking application logic" + description: "Account, deposit, withdrawal, transfer and transaction-history processing" + primary_source: "src/main/java/com/example/bankapp/" + - id: AE-IAM + name: "Authentication & authorisation" + description: "Login, password handling, role/privilege model, session controls" + primary_source: "src/main/java/com/example/bankapp/config/SecurityConfig.java" + - id: AE-SDLC + name: "Change management & secure SDLC" + description: "Branch protection, review/approval, CI pipeline, security scanning" + primary_source: ["Jenkinsfile", ".github/", "repo history"] + - id: AE-CONFIG + name: "Configuration & secrets" + description: "Application config, credential handling, schema/data controls" + primary_source: "src/main/resources/application.properties" + +# ---------------------------------------------------------------------------- +# Regulation / obligation taxonomy (controls map to these) +# ---------------------------------------------------------------------------- +regulations: + - id: ICFR + name: "Internal Control over Financial Reporting (SOX 302/404-equivalent)" + - id: PRA-FCA + name: "PRA/FCA prudential & conduct rulebooks (e.g. SYSC)" + - id: SMCR + name: "Senior Managers & Certification Regime — individual accountability" + - id: CONSUMER-DUTY + name: "FCA Consumer Duty — avoidance of foreseeable customer harm" + - id: OP-RESILIENCE + name: "Operational Resilience / DORA-equivalent — integrity & continuity" + - id: GDPR + name: "UK GDPR / DPA — confidentiality & integrity of personal data" + +# ---------------------------------------------------------------------------- +# Risks (what could go wrong) +# ---------------------------------------------------------------------------- +risks: + - id: R-01 + title: "Monetary value is misappropriated or fabricated through transaction processing" + - id: R-02 + title: "Account balances become inconsistent / non-reconcilable" + - id: R-03 + title: "Unauthorised movement of customer funds" + - id: R-04 + title: "Excessive or unsegregated access enables privilege misuse" + - id: R-05 + title: "Credentials are exposed, enabling unauthorised system/data access" + - id: R-06 + title: "Unauthorised or unreviewed change reaches production" + - id: R-07 + title: "Defects or vulnerabilities reach production undetected" + - id: R-08 + title: "Inability to detect, investigate or evidence fraudulent activity" + +# ---------------------------------------------------------------------------- +# Controls (the testable core) +# ---------------------------------------------------------------------------- +controls: + + - control_id: APP-TXN-01 + title: "Transaction amount validation" + auditable_entity: AE-APP + objective: "All monetary transactions are validated to be positive and within bounds before posting." + risk: R-01 + control_type: preventive + automation: automated + key: true + owner: "Head of Payments Engineering" + regulation: [ICFR, CONSUMER-DUTY] + test_procedure: > + Inspect deposit, withdraw and transfer code paths. Re-perform with boundary + inputs (zero, negative, very large). Confirm non-positive amounts are rejected. + population: "All monetary mutation methods in AccountService" + sample: "Full population (3 methods) + re-performance with crafted inputs" + pass_criteria: "Non-positive and out-of-bounds amounts are rejected on every path" + evidence_required: ["code citation per method", "re-performance output table"] + expected_demo_result: + status: FAIL + severity: high + detail: > + AccountService.deposit (L51-62), withdraw (L64-78) and transferAmount + (L103-135) never check amount > 0. A negative transfer credits the sender + and debits the recipient (fund misappropriation); a negative withdrawal + inflates the balance. + + - control_id: APP-TXN-02 + title: "Transfer atomicity (transactional integrity)" + auditable_entity: AE-APP + objective: "A funds transfer debits the sender and credits the recipient as a single atomic unit of work." + risk: R-02 + control_type: preventive + automation: automated + key: true + owner: "Head of Payments Engineering" + regulation: [ICFR, OP-RESILIENCE] + test_procedure: > + Inspect transferAmount for transaction-boundary management (@Transactional or + equivalent). Reason about partial-failure behaviour between debit and credit. + population: "transferAmount and all multi-write mutations" + sample: "Full population" + pass_criteria: "All multi-write monetary operations execute within one transaction; partial failure rolls back" + evidence_required: ["code citation", "transaction-boundary analysis"] + expected_demo_result: + status: FAIL + severity: high + detail: > + transferAmount (L103-135) performs four separate repository writes with no + @Transactional boundary. A failure after the debit save (L113) but before + the credit save (L117) destroys money and leaves balances non-reconcilable. + + - control_id: APP-TXN-03 + title: "Authorisation & dual-control on value movement" + auditable_entity: AE-APP + objective: "High-value transfers are subject to limits and independent second-person authorisation (maker-checker)." + risk: R-03 + control_type: preventive + automation: hybrid + key: true + owner: "Payments Operations / Engineering" + regulation: [SMCR, ICFR] + test_procedure: > + Determine whether per-transaction / daily limits and a second-approver step + exist for transfers above a threshold. Re-perform a large transfer. + population: "Transfer execution path" + sample: "Full population + re-performance" + pass_criteria: "Transfers above policy threshold require an independent approval and enforce limits" + evidence_required: ["code citation", "re-performance of high-value transfer"] + expected_demo_result: + status: FAIL + severity: high + detail: > + transferAmount enforces no transaction or daily limit and has no + maker-checker / second-approver step. Any authenticated user can move any + amount up to their balance in a single unreviewed action. + + - control_id: APP-ACC-04 + title: "Role-based access & least privilege" + auditable_entity: AE-IAM + objective: "Users are granted only the privileges required for their role; privileged functions are segregated." + risk: R-04 + control_type: preventive + automation: automated + key: true + owner: "IAM / Application Security" + regulation: [PRA-FCA, SMCR] + test_procedure: > + Inspect the granted-authorities / role model and endpoint authorisation rules. + Identify whether distinct roles and segregated privileged operations exist. + population: "Authority assignment + endpoint authorisation config" + sample: "Full population" + pass_criteria: "Distinct least-privilege roles exist; privileged actions are restricted and segregated" + evidence_required: ["code citation (authorities + security config)"] + expected_demo_result: + status: FAIL + severity: medium + detail: > + AccountService.authorities() (L99-101) hardcodes a single "USER" authority + for every account. There is no admin/operator segregation and no + role-restricted endpoints — no basis for least privilege or SoD. + + - control_id: APP-SEC-05 + title: "Web application protections on state-changing operations" + auditable_entity: AE-IAM + objective: "State-changing requests (deposit, withdraw, transfer) are protected against CSRF and request forgery." + risk: R-03 + control_type: preventive + automation: automated + key: false + owner: "Application Security" + regulation: [ICFR, GDPR] + test_procedure: "Inspect security configuration for CSRF protection on POST endpoints." + population: "SecurityConfig + state-changing endpoints" + sample: "Full population" + pass_criteria: "CSRF protection enabled for all state-changing endpoints" + evidence_required: ["code citation (SecurityConfig)"] + expected_demo_result: + status: FAIL + severity: medium + detail: > + SecurityConfig (L30) calls csrf.disable(). The /deposit, /withdraw and + /transfer POST endpoints are exposed to cross-site request forgery. + + - control_id: ITGC-SEC-06 + title: "Credential & secrets management" + auditable_entity: AE-CONFIG + objective: "Secrets (DB credentials, keys) are not stored in source control; they are injected from a secret store." + risk: R-05 + control_type: preventive + automation: automated + key: true + owner: "Platform Engineering" + regulation: [PRA-FCA, GDPR] + test_procedure: "Scan configuration and source for embedded credentials/secrets across the repo and history." + population: "All committed configuration + repo history" + sample: "Full population" + pass_criteria: "No plaintext secrets in source; secrets sourced from env/secret manager" + evidence_required: ["code citation", "secret-scan output"] + expected_demo_result: + status: FAIL + severity: high + detail: > + application.properties (L4-5) commits DB username 'root' and password + 'Test@123' in plaintext. Anyone with repo read access obtains DB credentials. + + - control_id: ITGC-CM-07 + title: "Change management — independent review & approval" + auditable_entity: AE-SDLC + objective: "Changes to the protected branch are independently reviewed and approved (author != approver) and traceable to a change record." + risk: R-06 + control_type: preventive + automation: hybrid + key: true + owner: "Head of Engineering" + regulation: [ICFR, OP-RESILIENCE] + test_procedure: > + For a sample of changes to the protected branch in-period, confirm an + approving review by someone other than the author, a linked change record, + and that direct-to-branch pushes are prevented. Inspect branch-protection + and CODEOWNERS configuration as the design test. + population: "All changes merged/pushed to the protected branch in-period" + sample: "25 changes (or full population where small)" + pass_criteria: "100% of sampled changes independently approved + traceable; direct push prevented" + evidence_required: ["per-change evidence table (commit/PR links)", "branch-protection config"] + expected_demo_result: + status: FAIL + severity: high + detail: > + No CODEOWNERS file and no evidence of branch protection on 'DevOps'. + History shows direct commits to the default branch (e.g. 'Update Jenkinsfile', + 'Update bankapp-deployment.yml') with no associated reviewed PR — i.e. + production pipeline & k8s manifests changed without independent approval. + + - control_id: ITGC-CM-08 + title: "Pipeline integrity & source traceability" + auditable_entity: AE-SDLC + objective: "The artifact deployed to production is built from the same reviewed source that was audited (provenance)." + risk: R-06 + control_type: preventive + automation: automated + key: true + owner: "Platform / Release Engineering" + regulation: [OP-RESILIENCE, ICFR] + test_procedure: "Trace the CI checkout source against the audited repository; confirm build provenance." + population: "CI/CD pipeline definition" + sample: "Full population" + pass_criteria: "Pipeline builds from the audited, protected source; provenance is verifiable" + evidence_required: ["pipeline code citation", "source/branch mapping"] + expected_demo_result: + status: FAIL + severity: medium + detail: > + Jenkinsfile (L26) checks out a different upstream repository + (LondheShubham153/Springboot-BankApp.git, branch DevOps) than the repo under + audit. The deployed artifact's provenance is not tied to the reviewed source. + + - control_id: ITGC-SDLC-09 + title: "Secure SDLC — scanning & automated test gating" + auditable_entity: AE-SDLC + objective: "Security scanning and automated tests run and gate every release." + risk: R-07 + control_type: detective + automation: automated + key: true + owner: "DevSecOps" + regulation: [ICFR, OP-RESILIENCE] + test_procedure: > + Inspect the pipeline for SAST/SCA/filesystem scanning and a test-execution + stage with a quality gate. Confirm tests provide meaningful coverage of + business logic. + population: "Pipeline definition + test suite" + sample: "Full population" + pass_criteria: "Scans + tests run and gate releases; business logic is covered by tests" + evidence_required: ["pipeline citation", "test-suite inventory"] + expected_demo_result: + status: PARTIAL + severity: medium + detail: > + Strengths: Jenkinsfile runs Trivy (L31), OWASP dependency-check (L39) and + SonarQube with a quality gate (L47-61). Gap: the only test is + BankappApplicationTests.contextLoads() — no tests cover deposit/withdraw/ + transfer logic, and there is no test-execution stage in the pipeline. + + - control_id: ITGC-DATA-10 + title: "Production schema/data change control" + auditable_entity: AE-CONFIG + objective: "Database schema changes are explicit, reviewed and version-controlled — not auto-applied at runtime." + risk: R-02 + control_type: preventive + automation: automated + key: false + owner: "Data Platform" + regulation: [ICFR, OP-RESILIENCE] + test_procedure: "Inspect ORM/schema configuration for uncontrolled auto-migration in non-dev environments." + population: "Persistence configuration" + sample: "Full population" + pass_criteria: "Schema changes use controlled migrations; no runtime auto-mutate in prod" + evidence_required: ["config citation"] + expected_demo_result: + status: FAIL + severity: medium + detail: > + application.properties (L9) sets spring.jpa.hibernate.ddl-auto=update, + letting Hibernate mutate the schema at runtime in every environment with no + review or migration trail. + + - control_id: ITGC-LOG-11 + title: "Logging & monitoring of financial events" + auditable_entity: AE-APP + objective: "Security-relevant and financial events are logged to support detection, investigation and audit trail." + risk: R-08 + control_type: detective + automation: automated + key: true + owner: "Security Operations" + regulation: [PRA-FCA, OP-RESILIENCE] + test_procedure: "Inspect transaction code paths for audit logging of financial events; assess log content & integrity." + population: "Transaction code paths + logging config" + sample: "Full population" + pass_criteria: "Financial/security events are logged with sufficient, protected detail" + evidence_required: ["code citation", "logging configuration"] + expected_demo_result: + status: FAIL + severity: medium + detail: > + No audit logging exists around deposit/withdraw/transfer. The only logging + signal is spring.jpa.show-sql=true (L11), which leaks SQL to stdout but is + not an audit trail and risks exposing data in logs. diff --git a/audit/samples/example-finding-FND-01.md b/audit/samples/example-finding-FND-01.md new file mode 100644 index 00000000..52da2391 --- /dev/null +++ b/audit/samples/example-finding-FND-01.md @@ -0,0 +1,52 @@ +# Finding FND-01 — Monetary transactions accept non-positive amounts + +> WORKED EXAMPLE — output of the Findings & Reporting playbook for control APP-TXN-01. + +| Field | Value | +|---|---| +| **Finding ID** | FND-01 | +| **Related control** | APP-TXN-01 (and APP-TXN-02 atomicity) | +| **Risk** | R-01 — Monetary value is misappropriated or fabricated | +| **Regulation(s)** | ICFR; Consumer Duty | +| **Severity** | High | +| **Status** | Open | +| **Owner (1st line)** | Head of Payments Engineering | +| **Target date** | {YYYY-MM-DD} | + +## Condition +`AccountService.deposit` (L51-62), `withdraw` (L64-78) and `transferAmount` (L103-135) +post monetary movements without validating that the amount is positive. Re-performance +confirmed a `-100` transfer credits the sender and debits the recipient, and a `-100` +withdrawal increases the balance. + +## Criteria +Monetary transactions must validate inputs (positive, within limits) before posting, and +account balances must remain complete and accurate (ICFR; FCA Consumer Duty — avoidance +of foreseeable customer harm). + +## Cause +Input validation was never implemented in the service layer; the controller passes the +`BigDecimal amount` straight through, and the only balance check (`balance >= amount`) +is trivially satisfied by negative values. + +## Consequence +Any authenticated user can misappropriate funds from another account via a negative +transfer, or inflate their own balance via a negative withdrawal. Direct financial loss, +mis-stated balances, and customer harm. Combined with the absence of a transaction +boundary (APP-TXN-02), partial failures can also destroy money. + +## Recommendation +1. Reject non-positive amounts on `deposit`, `withdraw` and `transferAmount` (fail closed). +2. Enforce per-transaction and daily limits; add maker-checker above a threshold (APP-TXN-03). +3. Wrap `transferAmount` in a single `@Transactional` unit of work (APP-TXN-02). +4. Add re-performance unit tests covering zero/negative/over-limit inputs (closes the ITGC-SDLC-09 test-coverage gap for this logic). + +## Evidence +| # | Evidence | Reference | +|---|---|---| +| 1 | No positive-amount check on any monetary method | `src/main/java/com/example/bankapp/service/AccountService.java:51-135` | +| 2 | Controller forwards raw amount | `src/main/java/com/example/bankapp/controller/BankController.java:50-96` | +| 3 | Re-performance table | `audit/samples/example-workpaper-APP-TXN-01.md` | + +## Management response +> {to be completed by the first line} diff --git a/audit/samples/example-workpaper-APP-TXN-01.md b/audit/samples/example-workpaper-APP-TXN-01.md new file mode 100644 index 00000000..25713079 --- /dev/null +++ b/audit/samples/example-workpaper-APP-TXN-01.md @@ -0,0 +1,55 @@ +# Workpaper — APP-TXN-01: Transaction amount validation + +> WORKED EXAMPLE — illustrates the output Devin produces from the Application-Control +> Re-performance playbook. Line numbers reference the repo at the time of writing. + +| Field | Value | +|---|---| +| **Control ID** | APP-TXN-01 | +| **Auditable entity** | AE-APP — Core banking application logic | +| **Control objective** | All monetary transactions are validated to be positive and within bounds before posting. | +| **Risk mitigated** | R-01 — Monetary value is misappropriated or fabricated through transaction processing | +| **Regulation(s)** | ICFR; Consumer Duty | +| **Control type** | preventive · automated · key | +| **Control owner** | Head of Payments Engineering | +| **Period** | FY in-scope period | +| **Tester** | Devin (drafted) · {Auditor} (reviewed) | +| **Date** | 2026-06-19 | + +## Test method +Inspection + re-performance. The control is automated and code-resident, so the +strongest test is to read the implementing code and re-perform it with boundary inputs. + +## Population & sample +- **Population:** all monetary mutation methods in `AccountService` (deposit, withdraw, transfer). +- **Sample:** full population (3 methods) + crafted boundary inputs. + +## Procedure performed +1. Located the three monetary mutation methods and read each fully. +2. Identified the validations present on each path. +3. Re-performed each with inputs: `0`, `-100`, and a value greater than balance. + +## Evidence +| # | Evidence | Reference | +|---|---|---| +| 1 | `deposit` adds `amount` with no sign/bounds check | `src/main/java/com/example/bankapp/service/AccountService.java:51-62` | +| 2 | `withdraw` checks balance ≥ amount but not amount > 0 | `src/main/java/com/example/bankapp/service/AccountService.java:64-78` | +| 3 | `transferAmount` checks balance ≥ amount but not amount > 0 | `src/main/java/com/example/bankapp/service/AccountService.java:103-135` | +| 4 | Controller passes `BigDecimal amount` straight through, no validation | `src/main/java/com/example/bankapp/controller/BankController.java:50-96` | + +## Re-performance +| Method | Input | Expected (policy) | Observed | Result | +|---|---|---|---|---| +| deposit | `-100` | rejected | balance increases by… wait, decreases (credits a negative) | FAIL | +| withdraw | `-100` | rejected | balance **increases** by 100 (subtracting a negative) | FAIL | +| transfer | `-100` to victim | rejected | sender **gains** 100, recipient **loses** 100 | FAIL | +| transfer | `0` | rejected | accepted (no-op transaction record created) | FAIL | + +## Result +- **Conclusion:** FAIL +- **Pass criteria:** Non-positive and out-of-bounds amounts are rejected on every path. +- **Exceptions:** No positive-amount validation on any monetary method; negative amounts invert the intended money movement. + +## Notes / residual risk +Highest-impact gap in the audit: a negative transfer is a direct fund-misappropriation +vector exploitable by any authenticated user. Raised as Finding FND-01. diff --git a/audit/templates/audit-committee-onepager.md b/audit/templates/audit-committee-onepager.md new file mode 100644 index 00000000..19053ad1 --- /dev/null +++ b/audit/templates/audit-committee-onepager.md @@ -0,0 +1,47 @@ +# Audit Committee One-Pager — {Auditee} {Period} + +**Audit:** {e.g. Cards & Payments platform — application & ITGC controls} +**Overall opinion:** {RAG — Satisfactory / Needs Improvement / Unsatisfactory} +**Prepared by:** Devin (drafted) · {Auditor} (concluded & signed off) · {date} + +## Headline +{2–3 sentences: scope tested, overall control health, the single most important message.} + +## Results at a glance +| Severity | Count | +|---|---| +| Critical | {n} | +| High | {n} | +| Medium | {n} | +| Low | {n} | + +| Coverage | Value | +|---|---| +| Controls tested | {n} / {total} | +| Key controls failed | {n} / {key total} | +| Tested over full population | {n} | + +## Exposure by regulation +| Regulation | Findings | Most severe | +|---|---|---| +| ICFR | {n} | {sev} | +| PRA/FCA | {n} | {sev} | +| SMCR | {n} | {sev} | +| Consumer Duty | {n} | {sev} | +| Op-Resilience | {n} | {sev} | +| GDPR | {n} | {sev} | + +## Key themes +1. {theme — e.g. "Transaction integrity controls are absent in the service layer"} +2. {theme — e.g. "Change-management evidence is missing on the production branch"} +3. {theme} + +## Top findings +| ID | Finding | Severity | Control | Status | +|---|---|---|---|---| +| {FND-01} | {short title} | {sev} | {CONTROL_ID} | {Open} | +| {FND-02} | … | … | … | … | + +## Assurance over this audit +Every result is reproducible: each finding links to `file:line`, a commit/PR, or a +re-runnable command. The full Devin session transcript is retained as the audit trail. diff --git a/audit/templates/finding-5cs-template.md b/audit/templates/finding-5cs-template.md new file mode 100644 index 00000000..8c9beb27 --- /dev/null +++ b/audit/templates/finding-5cs-template.md @@ -0,0 +1,40 @@ +# Finding {FINDING_ID} — {short title} + +| Field | Value | +|---|---| +| **Finding ID** | {FINDING_ID} | +| **Related control** | {CONTROL_ID} | +| **Risk** | {R-XX — title} | +| **Regulation(s)** | {…} | +| **Severity** | {Critical / High / Medium / Low} | +| **Status** | {Open / Remediated / Validated} | +| **Owner (1st line)** | {accountable owner} | +| **Target date** | {YYYY-MM-DD} | + +## Condition +*What we found* — the actual state observed, with citations. +> {e.g. `AccountService.transferAmount` (L103-135) performs no positive-amount check and is not `@Transactional`.} + +## Criteria +*What should be true* — the policy/standard/regulation the condition is measured against. +> {e.g. Monetary transactions must validate inputs and execute atomically (ICFR; Consumer Duty — avoidance of foreseeable harm).} + +## Cause +*Why it happened* — root cause, not just the symptom. +> {e.g. Validation and transaction-boundary management were never implemented in the service layer.} + +## Consequence +*Impact* — what the risk could lead to (quantify where possible). +> {e.g. A negative transfer misappropriates funds; a partial failure mid-transfer destroys money and breaks reconciliation.} + +## Recommendation +*What to do* — specific, actionable, testable. +> {e.g. Reject non-positive amounts on all monetary methods; wrap transfer in a single `@Transactional` unit; add limits + maker-checker above threshold; add a re-performance test.} + +## Evidence +| # | Evidence | Reference | +|---|---|---| +| 1 | {…} | `{file:line}` / {PR/commit} / {command output} | + +## Management response +> {to be completed by the first line} diff --git a/audit/templates/workpaper-template.md b/audit/templates/workpaper-template.md new file mode 100644 index 00000000..3d1547dc --- /dev/null +++ b/audit/templates/workpaper-template.md @@ -0,0 +1,40 @@ +# Workpaper — {CONTROL_ID}: {Control Title} + +| Field | Value | +|---|---| +| **Control ID** | {CONTROL_ID} | +| **Auditable entity** | {AE-XX — name} | +| **Control objective** | {objective from RACM} | +| **Risk mitigated** | {R-XX — title} | +| **Regulation(s)** | {ICFR / PRA-FCA / SMCR / Consumer Duty / Op-Resilience / GDPR} | +| **Control type** | {preventive / detective / corrective} · {automated / manual / hybrid} · {key / non-key} | +| **Control owner** | {first-line owner} | +| **Period** | {audit period} | +| **Tester** | Devin (drafted) · {Auditor} (reviewed/signed off) | +| **Date** | {YYYY-MM-DD} | + +## Test method +{inquiry / observation / inspection / re-performance — and why this method} + +## Population & sample +- **Population:** {definition + size} +- **Sample:** {full population / sample size + selection basis} + +## Procedure performed +{step-by-step description of exactly what was done} + +## Evidence +> Every row must be a citation a reviewer can independently re-check. + +| # | Evidence | Reference | +|---|---|---| +| 1 | {what it shows} | `{file:line}` / {PR or commit URL} / {command + output ref} | +| 2 | … | … | + +## Result +- **Conclusion:** {PASS / PARTIAL / FAIL} +- **Pass criteria:** {from RACM} +- **Exceptions:** {list, or "none"} + +## Notes / residual risk +{anything the auditor should weigh; limitations of the test}