Does an independent verifier make an LLM penetration-testing agent more reliable? Sometimes. It depends on the model.
This repository holds the code, data, and analysis for an MSc dissertation that measured run-to-run variance in an autonomous LLM penetration-testing agent, then tested whether an external consistency checker reduces that variance.
An LLM agent asked to solve the same penetration-testing task ten times will solve it ten different ways. Some runs finish in six steps, some in twenty-five, some fail. This project measured that inconsistency as a formal metric, which prior work had not done, then added a separate service that verifies the agent's claims against the real target after every action.
On the strongest test case, the checker cut step-count variance by 58% with no loss of success. On a weaker model, the same checker made things worse. The deciding factor was the capability of the underlying model, not the design of the checker.
How does independent verification affect the reliability of LLM-based penetration-testing agents?
"Reliability" here means run-to-run consistency on identical tasks under identical settings. The effectiveness metric is task success rate. The consistency metric is the standard deviation of step counts across repeated runs.
Task WS0 (path traversal on a PHP app), model gpt-4.1, ten runs per condition:
| Condition | Runs | Success | Mean steps | Step SD |
|---|---|---|---|---|
| Baseline (no checker) | 10 | 100% | 6.6 | 1.26 |
| With checker | 10 | 100% | 7.5 | 0.53 |
Step-count standard deviation fell from 1.26 to 0.53, a 58% reduction. Success held at 100% in both conditions. Mann-Whitney U on the step counts returned p = 0.008 with Cohen's d = 0.93.
The checker did not make the agent solve more. It made a capable agent solve the same task far more predictably. Predictability is what matters for cost and time estimation in a real engagement, where each step is an API call and a chunk of wall-clock time.
The checker was tested against two models across the same five tasks. Success rates across all four conditions:
| Task | 4o-mini base | 4o-mini + checker | gpt-4.1 base | gpt-4.1 + checker |
|---|---|---|---|---|
| AC0 (sudo brute-force) | 100% | 90% | 100% | 90% |
| AC2 (cron root) | 0% | 0% | 0% | 0% |
| AC4 (SETUID env) | 0% | 0% | 30% | 30% |
| NS0 (SSH standard port) | 80% | 20% | 90% | 70% |
| WS0 (path traversal) | 0% | 20% | 100% | 100% |
| Aggregate | 18/50 (36%) | 13/50 (26%) | 32/50 (64%) | 29/50 (58%) |
On gpt-4o-mini, adding the checker dropped aggregate success from 36% to 26%. The weaker model could not act on correction prompts. Each failed check consumed a step of the budget without producing progress. The stronger model used the same corrections productively.
This is the central contribution. Verification effectiveness is conditional on model capability. Wrapping a weak model in a checker does not rescue it. That finding could not have come from a single-model experiment, which is why the design crosses two models against the checker.
The 36% figure is what
analysis/analyse_all.pycomputes from the committed run logs (18 of 50 baseline runs succeeded). It is reproducible from this repository.
Standard deviation of step counts, baseline versus checker:
| Task | SD base | SD + checker | Change | Mann-Whitney U p |
|---|---|---|---|---|
| WS0 | 1.26 | 0.53 | -58% | 0.008 (significant) |
| NS0 | 6.51 | 2.37 | -64% | 0.064 (not significant) |
| AC4 | 1.53 | 0.58 | -62% | 0.383 (n = 3) |
| AC0 | 0.00 | 0.67 | baseline already deterministic | 0.0002 |
The direction is consistent on the tasks with real variance to reduce. WS0 is the only comparison with enough successful runs on both sides to reach significance. NS0 and AC4 point the same way but are underpowered. On gpt-4o-mini the effect reversed, with step-count SD rising instead of falling (AC0 +453%, NS0 +94%).
The agent proposes an action. The adapter runs it against the target. Before the agent is allowed to build on a claim, the checker confirms that claim against live target state. A failed check returns a correction prompt instead of accepting the claim.
flowchart TD
A["PentestGPT agent<br/>(gpt-4.1 or gpt-4o-mini)"] -->|"proposed action + claim"| B["Adapter<br/>runner/adapter.py"]
B -->|"run command"| T["AutoPenBench target<br/>(Docker, in-vitro task)"]
T -->|"observation"| B
B -->|"verify claim"| C["Consistency Checker<br/>FastAPI, 127.0.0.1:8100"]
C -.->|"/verify/port : TCP connect"| T
C -.->|"/verify/ssh : real SSH login"| T
C -->|"/verify/cmd : output sanity"| B
C -->|"pass, continue"| B
C -->|"fail, correction prompt, retry up to 2"| A
B -->|"log every step"| L[("Per-run JSONL logs<br/>data/")]
The checker is a standalone FastAPI service. It shares no memory or state with the agent. It only reports what the real target shows. Three endpoints do the work:
POST /verify/portopens a TCP connection to confirm a service is actually listening at the claimed host and port. This catches hallucinated services and wrong target addresses.POST /verify/sshperforms a real SSH login with paramiko to confirm claimed credentials work. This catches invented usernames and passwords.POST /verify/cmdinspects command output for timeouts, empty responses, and expected content. This catches unresponsive shells and fabricated command results.
Because the checker is decoupled from the agent, it ports to other agent frameworks without modification.
A 2x2 factorial design crosses two models against two checker conditions, run over five AutoPenBench in-vitro tasks, with ten repeats each. That is 200 runs total, plus a 10-run feasibility pilot on gpt-4.1.
| Factor | Levels |
|---|---|
| Model | gpt-4o-mini, gpt-4.1 |
| Checker | off (baseline), on |
| Tasks | AC0, AC2, AC4, NS0, WS0 (AutoPenBench in-vitro) |
| Repeats | 10 per cell |
| Temperature | 0.2 |
| Step limit | 30 |
| Agent | PentestGPT, pinned at commit 6e84be8 |
Each run started from a clean target with no leftover state. Statistics use Mann-Whitney U (two-tailed) rather than a t-test, because ten runs per cell is too few to assume normality and step counts are discrete integers. Cohen's d accompanies every comparison as an effect-size measure.
agents/pentestgpt/ PentestGPT under test, pinned at commit 6e84be8 (see COMMIT.txt)
runner/
adapter.py Bridge between the model and the AutoPenBench driver; calls the checker
config.yaml Model, temperature, step limit, and the five task definitions
run_baseline.sh Batch driver for 50 baseline runs (5 tasks x 10 repeats)
prompts/ Pentest system-prompt template
checker/
app.py FastAPI consistency checker (/verify/port, /verify/ssh, /verify/cmd, /health)
rules.yaml Proof rules for step verification
benchmarks/
autopenbench/ Download helper and target hashes (no VM images committed)
hostmaps/ Host-to-IP maps
data/
gpt41_baseline/ gpt-4.1, checker off
gpt41_checker/ gpt-4.1, checker on
intervention_v2/ gpt-4o-mini, checker on (final, after the false-correction fix below)
intervention/ gpt-4o-mini, checker on (superseded first pass, kept for the record)
pilot_gpt41_baseline/ 10-run feasibility pilot
summary/ Summary CSV headers
analysis/
analyse_all.py Computes success rates, step and token stats, Mann-Whitney U, Cohen's d
full_results.txt Committed output of the analysis
figs/ Figure output directory
Note on runners. runner/adapter.py is the working entry point that produced the data, driven per task or in batch by runner/run_baseline.sh. runner/run_experiment.py and the make baseline / make checker targets are scaffolding placeholders, not the path used for the experiments.
Note on data. The gpt-4o-mini baseline logs sit under data/raw/, which is excluded by .gitignore for size. The other three conditions are committed in full. Re-running analysis/analyse_all.py against a fresh clone will need those baseline logs regenerated.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python analysis/analyse_all.py # prints the tables in analysis/full_results.txtAutoPenBench and its Docker targets are installed separately. With a target up:
# Terminal 1: start the checker
uvicorn checker.app:app --host 127.0.0.1 --port 8100
# Terminal 2: run one task (add --checker to enable verification)
cd <autopenbench-dir>
python3 -m adapter --category web_security --vm 0 --run-id 1 --checkerBatch all 50 baseline runs with bash runner/run_baseline.sh.
During early testing the checker carried a bug. When the agent scanned a subnet such as 192.168.0.0/16, the checker took the network address 192.168.0.0, treated it as a host, and tried to connect to port 22. That address is not a live host, so the check always failed. The checker kept telling the agent its scan had failed when the scan was fine. Success on that task collapsed from around 80% to 0%.
The fix was one line. The lesson is larger. A false correction actively derails the agent, worse than staying silent. Any verification layer has to tell "the agent is confused" apart from "my check is confused." The intervention and intervention_v2 data folders hold the runs from before and after this fix.
- Ten runs per task per condition. Rare successes such as AC4 at 3 of 10 produce wide confidence intervals.
- Five tasks from a single benchmark. Cryptography and multi-host scenarios were out of scope.
- No Bonferroni correction was applied. The WS0 result survives a corrected 0.01 threshold, but that correction was not formally run.
- Temperature was 0.2 rather than 0, so some variance is sampling noise by design.
CHECKER_MAX_RETRIESis defined but was not enforced as a hard per-action cap.
- The first formal measurement of run-to-run variance in an LLM penetration-testing agent, across 200 repeated runs on a standardised benchmark.
- A lightweight consistency checker that verifies agent claims against live target state, decoupled from the agent and portable to other frameworks.
- Evidence that external verification helps or hurts depending on the capability of the underlying model, established through a two-model design.
Imran Hafeez MSc Cyber Security, Edinburgh Napier University, School of Computing Supervisor: Sana Ullah Jan Dissertation submitted April 2026. Viva passed May 2026.
@mastersthesis{hafeez2026verification,
author = {Hafeez, Imran},
title = {How Does Independent Verification Affect the Reliability of LLM-Based Penetration Testing Agents},
school = {Edinburgh Napier University},
year = {2026}
}Agent under test: PentestGPT, pinned at commit 6e84be8.
Benchmark: AutoPenBench in-vitro tasks.
Apache-2.0. See LICENSE.