11# ModSecurity Pytest Testing Framework
22
3- Python/pytest front end for ModSecurity v2's test data . There are two independent suites:
3+ Python/pytest front end for ModSecurity v2's tests . There are two independent suites:
44
55- ** Unit tests** (` test_operators/ ` , ` test_transformations/ ` ): exercise individual ` @operator ` s and
6- ` t:transformation ` s directly through the ` msc_test ` C binary. No Apache required.
6+ ` t:transformation ` s directly through the ` msc_test ` C binary. No Apache required. These files are
7+ plain, hand-maintained pytest parametrize tables - add or edit cases directly.
78- ** Regression tests** (` test_regression/ ` ): exercise full ` SecRule ` /config behavior against a real
8- Apache + ` mod_security2.so ` , driven by ` conftest.py ` 's ` apache_server ` /` modsec_test ` fixtures.
9+ Apache + ` mod_security2.so ` , driven by ` conftest.py ` 's ` apache_server ` /` modsec_test ` fixtures. This
10+ suite is still generated from Perl ` .t ` data (` tests/regression/*/*.t ` ) rather than hand-ported -
11+ see "How the regression test data flows" below for why and how.
912
10- Both suites are generated from the same Perl ` .t ` data files this project has always used
11- (` tests/op/*.t ` , ` tests/tfn/*.t ` , ` tests/regression/*/*.t ` ) rather than being hand-ported line by
12- line - see "How the test data flows" below for why and how.
13+ Unit tests were originally migrated the same way (` tests/op/*.t ` /` tests/tfn/*.t ` fed through a Perl
14+ dumper into generated ` .py ` files), but that data is flat literal strings with no ongoing need for
15+ a Perl round-trip, so the ` .t ` files, the dumper, and the converter script were retired once the
16+ ` .py ` files existed - the generated files are now the source of truth and are edited directly.
1317
1418## Prerequisites
1519
@@ -28,8 +32,8 @@ line - see "How the test data flows" below for why and how.
2832 cd tests
2933 pip install -r requirements.txt
3034 ```
31- 3 . Regression tests also need Perl with ` LWP::UserAgent ` ( ` libwww-perl ` on Debian/Ubuntu) - not at
32- test-run time, but to regenerate fixtures after editing a ` .t ` file (see below) .
35+ 3 . Regenerating regression fixtures after editing a ` regression/*/*.t ` file also needs Perl with
36+ ` LWP::UserAgent ` ( ` libwww-perl ` on Debian/Ubuntu) - not needed just to run the tests .
3337
3438## Directory structure
3539
@@ -40,13 +44,13 @@ tests/
4044├── modsec_test.py # LogMatcher/ResponseMatcher/ModSecurityTestCase/UnitTestRunner
4145├── regression_fixtures.py # Loads tests/regression/fixtures/*.json into Python objects
4246│
43- ├── op/*.t, tfn/*.t # Source of truth for unit tests (Perl data, unchanged format)
44- ├── dump_unit_fixtures.pl # Evals an op/tfn .t file, emits JSON (param/input/output base64)
45- ├── convert_perl_tests.py # convert_perl_tests.py --unit-only regenerates the .py files below
46- ├── test_operators/ # One file per op/*.t (mostly generated; test_beginswith.py is
47- │ └── ... # hand-written and excluded from regeneration - see below)
48- ├── test_transformations/ # One file per tfn/*.t (same deal; test_base64decode.py is hand-written)
47+ ├── test_operators/ # Hand-maintained pytest files, one per operator
4948│ └── ...
49+ ├── test_transformations/ # Hand-maintained pytest files, one per transformation
50+ │ └── ...
51+ ├── op/pmFromFile-01.dat # Runtime data file @pmFromFile's test reads - not test *code*,
52+ │ # kept even though the op/*.t definitions that once lived
53+ │ # alongside it are gone (see git history if you need them)
5054│
5155├── regression/*/*.t # Source of truth for regression tests (Perl data, unchanged format)
5256├── dump_regression_fixtures.pl # Evals a regression .t file the way run-regression-tests.pl does
@@ -84,37 +88,28 @@ pytest -n auto
8488Markers: ` unit ` , ` regression ` , ` apache ` (needs Apache), ` slow ` . ` pytest -m unit ` / ` pytest -m regression `
8589work the same as passing the directory.
8690
87- ## How the test data flows
88-
89- Both suites are generated from Perl, not hand-ported, because the ` .t ` files use real Perl syntax
90- (` qr// ` regexes with flags, ` HTTP::Request->new(...) ` , ` $ENV{...} ` interpolation, ` \xHH ` string
91- escapes, occasional ` conf => sub {...} ` coderefs) that a text/regex-based Python parser cannot
92- reliably reproduce - the first attempt at this (` convert_perl_tests.py ` 's original Perl-parsing
93- path, still present for reference) silently mis-escaped binary data and dropped every regression
94- assertion. Instead, a small Perl script lets Perl itself evaluate the ` .t ` file (the same
95- ` @C = (...) ` trick ` run-unit-tests.pl ` /` run-regression-tests.pl ` use) and serializes the result to
96- JSON, which Python then consumes as data - no Perl semantics need to be reimplemented in Python.
97-
98- ** Unit tests** (` dump_unit_fixtures.pl ` + ` convert_perl_tests.py --unit-only ` ): base64-decodes
99- ` param ` /` input ` /` output ` into real Python ` bytes ` and emits one ` test_<name>.py ` per ` .t ` file with
100- a ` repr() ` -generated parametrize table - readable, diffable, and correct for arbitrary/invalid byte
101- sequences. Re-run after editing an ` op/*.t ` or ` tfn/*.t ` file:
102-
103- ``` bash
104- cd tests && python3 convert_perl_tests.py --unit-only
105- ```
106-
107- ` test_operators/test_beginswith.py ` and ` test_transformations/test_base64decode.py ` have hand-added
108- edge-case coverage beyond their mechanical parametrize table and are skipped by ` --unit-only `
109- (there's a ` hand_written ` set in ` convert_perl_tests.py ` if you need to add another) - update those
110- by hand instead of regenerating them.
111-
112- ** Regression tests** (` dump_regression_fixtures.pl ` ): sets up the same ` %ENV ` as
113- ` run-regression-tests.pl ` , evals the ` .t ` file, and serializes ` qr// ` → ` {pattern, flags} ` ,
114- ` HTTP::Request ` → ` {method, uri, headers, content} ` , ` conf => sub {...} ` coderefs (executed and
115- captured), etc. ` test_regression/test_fixtures.py ` then discovers every ` tests/regression/fixtures/*/*.json `
116- file and parametrizes one pytest case per entry - there is no per-` .t ` -file Python code to keep in
117- sync. Re-run after editing a ` regression/*/*.t ` file:
91+ ## Adding a unit test
92+
93+ ` test_operators/ ` /` test_transformations/ ` are plain pytest files - add a new parametrize tuple (or
94+ a whole new ` test_<name>.py ` , following an existing file's pattern) directly. ` param ` /` input_data ` /
95+ ` expected_output ` accept either a ` str ` (for real Unicode text) or a ` bytes ` literal (for exact
96+ byte sequences, including invalid UTF-8 or embedded NULs - see ` test_transformations/test_base64decode.py `
97+ for an example of both). ` unit_test.unit_runner.run_operator_test() ` /` run_transformation_test() `
98+ drive the ` msc_test ` binary directly; see ` modsec_test.py ` 's ` UnitTestRunner ` for the exact contract
99+ (` msc_test.c ` 's own ` -t op|tfn -n <name> -p <param> -r <expected_ret> ` , stdin = input).
100+
101+ ## How the regression test data flows
102+
103+ ` tests/regression/*/*.t ` files use real Perl syntax (` qr// ` regexes with flags,
104+ ` HTTP::Request->new(...) ` , ` $ENV{...} ` interpolation, ` \xHH ` string escapes, occasional
105+ ` conf => sub {...} ` coderefs) that a text/regex-based Python parser cannot reliably reproduce.
106+ Instead, ` dump_regression_fixtures.pl ` sets up the same ` %ENV ` as ` run-regression-tests.pl ` , lets
107+ Perl itself evaluate the ` .t ` file (the same ` @C = (...) ` trick ` run-regression-tests.pl ` uses), and
108+ serializes the result to JSON: ` qr// ` → ` {pattern, flags} ` , ` HTTP::Request ` → `{method, uri, headers,
109+ content}` , ` conf => sub {...}` coderefs (executed and captured), etc. - no Perl semantics need to be
110+ reimplemented in Python. ` test_regression/test_fixtures.py ` then discovers every
111+ ` tests/regression/fixtures/*/*.json ` file and parametrizes one pytest case per entry - there is no
112+ per-` .t ` -file Python code to keep in sync. Re-run after editing a ` regression/*/*.t ` file:
118113
119114``` bash
120115tests/regenerate_regression_fixtures.sh
0 commit comments