Merge pull request #60 from stacknil/stacknil/loglens-reviewer-config-contracts

Lock sample config and reviewer evidence contracts

Author: stacknil

README.md

@@ -11,7 +11,7 @@ It parses `auth.log` / `secure`-style syslog input and `journalctl --output=shor
 
 LogLens is an MVP / early release. The repository is stable enough for public review, local experimentation, and extension, but the parser and detection coverage are intentionally narrow.
 
-Reviewing the project quickly? Start with [`docs/reviewer-path.md`](./docs/reviewer-path.md) and [`docs/reviewer-brief.md`](./docs/reviewer-brief.md).
+Reviewing the project quickly? Start with [`docs/reviewer-path.md`](./docs/reviewer-path.md) and [`docs/reviewer-brief.md`](./docs/reviewer-brief.md). For detection reasoning, read the forensic-style [`Linux auth brute-force case study`](./docs/case-study-linux-auth-bruteforce.md) and the [`rule catalog`](./docs/rule-catalog.md).
 
 ## Why This Project Exists
 
@@ -90,7 +90,7 @@ Common unsupported-pattern buckets include `sshd_connection_closed_preauth`,
 `pam_faillock_account_locked`, and `pam_unix_session_closed`. These buckets keep
 non-finding evidence reviewable without counting it as detector evidence.
 
-For the parser behavior contract, supported modes, and fixture map, see [`docs/parser-contract.md`](./docs/parser-contract.md). For the deliberately noisy parser-coverage sample, see [`docs/parser-coverage-notes.md`](./docs/parser-coverage-notes.md).
+For rule-by-rule semantics and signal boundaries, see [`docs/rule-catalog.md`](./docs/rule-catalog.md). For a forensic-style evidence walkthrough, see [`docs/case-study-linux-auth-bruteforce.md`](./docs/case-study-linux-auth-bruteforce.md). For the parser behavior contract, supported modes, and fixture map, see [`docs/parser-contract.md`](./docs/parser-contract.md). For the deliberately noisy parser-coverage sample, see [`docs/parser-coverage-notes.md`](./docs/parser-coverage-notes.md).
 
 LogLens does not currently detect:
 
@@ -108,6 +108,12 @@ cmake --build build
 ctest --test-dir build --output-on-failure
 ```
 
+For Visual Studio or other multi-config generators, pass the built configuration to CTest:
+
+```bash
+ctest --test-dir build -C Debug --output-on-failure
+```
+
 For fresh-machine setup and repeatable local presets, see [`docs/dev-setup.md`](./docs/dev-setup.md).
 
 ## Run
@@ -213,6 +219,8 @@ The config file schema is intentionally small and strict:
 
 This mapping lets LogLens normalize parsed events into detection signals before applying brute-force or multi-user rules. By default, `pam_auth_failure` is treated as lower-confidence attempt evidence and does not count as a terminal authentication failure unless the config explicitly upgrades it. The `ssh_failed_keyboard_interactive` and `ssh_max_auth_tries` mapping keys are optional in older configs and default to terminal failure evidence.
 
+The checked-in [`assets/sample_config.json`](./assets/sample_config.json) is tested as a runnable default-equivalent config fixture. If default detector thresholds or signal mappings change, update that file and the related tests together.
+
 Timestamp handling is now explicit:
 
 - `--mode syslog`, `--mode syslog-legacy`, or `input_mode: syslog_legacy` requires `--year` or `timestamp.assume_year`

docs/dev-setup.md

@@ -44,6 +44,9 @@ cmake --build build
 ctest --test-dir build --output-on-failure
 ```
 
+For Visual Studio or other multi-config generators, run the test step with the
+built configuration, for example `ctest --test-dir build -C Debug --output-on-failure`.
+
 ## Windows Notes
 
 - Run from a Developer PowerShell for Visual Studio 2022, an x64 Native Tools prompt, or another shell where the MSVC toolchain is already available.
@@ -63,5 +66,5 @@ sudo apt install cmake g++ make
 ## Expected Local Outputs
 
 - Build directories under `build/dev-debug` or `build/ci-release`
-- Test runs for `parser`, `detector`, and `cli`
+- Test runs for `parser`, `detector`, `report`, `cli`, and `report_contracts`
 - `compile_commands.json` in the debug build directory when the selected generator supports it

docs/reviewer-brief.md

@@ -12,6 +12,7 @@ Linux auth logs are noisy, format-sensitive, and easy to parse incorrectly. Revi
 
 - Reproducible command: `./build/loglens --mode syslog --year 2026 ./assets/sample_auth.log ./out`
 - Deterministic outputs: `report.md`, `report.json`, optional `findings.csv`, optional `warnings.csv`, and parser coverage telemetry.
+- Detection reasoning: [`docs/rule-catalog.md`](./rule-catalog.md) documents rule inputs and boundaries; [`docs/case-study-linux-auth-bruteforce.md`](./case-study-linux-auth-bruteforce.md) traces a sanitized evidence set from raw lines to findings and warnings.
 - Tests / CI: CTest coverage plus GitHub Actions CI on Ubuntu and Windows; CodeQL is required on protected main.
 - Release evidence: changelog, release process docs, versioned release notes, and GitHub release artifacts.
 - Non-goals: live collection, SIEM replacement, cross-host correlation, exploitation, credential attack automation, or incident verdicts.

docs/reviewer-path.md

@@ -2,6 +2,17 @@
 
 This path is for reviewers who want to understand LogLens quickly without reading the whole repository first.
 
+## First choose the review question
+
+| Review question | Start here | Good stopping point |
+| --- | --- | --- |
+| What is LogLens? | [`README.md`](../README.md) and [`docs/reviewer-brief.md`](./reviewer-brief.md) | Can state scope, supported inputs, outputs, and non-goals |
+| What log formats are supported? | [`docs/parser-contract.md`](./parser-contract.md) | Can name `syslog_legacy` and `journalctl_short_full` behavior |
+| What artifacts does it produce? | [`docs/report-artifacts.md`](./report-artifacts.md) and report-contract fixtures | Can inspect Markdown, JSON, and optional CSV outputs |
+| How do rules use evidence? | [`docs/rule-catalog.md`](./rule-catalog.md) | Can explain grouping keys, windows, thresholds, and unsupported-evidence boundaries |
+| Can the parser behavior be trusted? | Parser contract, fixture matrix, and parser coverage fields | Can see known, unknown, and malformed line handling |
+| How should a finding be interpreted? | [`docs/case-study-linux-auth-bruteforce.md`](./case-study-linux-auth-bruteforce.md) | Can trace raw evidence to normalized events, findings, warnings, and non-goals |
+
 ## 30-second orientation
 
 Read:
@@ -30,6 +41,16 @@ Inspect:
 - [`tests/fixtures/report_contracts/syslog_legacy/report.json`](../tests/fixtures/report_contracts/syslog_legacy/report.json)
 - [`docs/report-artifacts.md`](./report-artifacts.md)
 - [`docs/parser-contract.md`](./parser-contract.md)
+- [`docs/rule-catalog.md`](./rule-catalog.md)
+- [`docs/case-study-linux-auth-bruteforce.md`](./case-study-linux-auth-bruteforce.md)
+
+Look for the evidence route:
+
+- raw log line
+- normalized event
+- signal mapping boundary
+- rule grouping, window, and threshold
+- report finding or parser warning
 
 Look for parser coverage fields:
 
@@ -41,7 +62,7 @@ Look for parser coverage fields:
 - `parse_success_rate`
 - `top_unknown_patterns`
 
-Good stopping point: the reviewer can explain what LogLens parses, what it reports, and how unsupported lines remain visible.
+Good stopping point: the reviewer can explain what LogLens parses, how rules count supported evidence, what the reports contain, and how unsupported lines remain visible without becoming findings.
 
 ## 15-minute local check
 
@@ -54,6 +75,9 @@ ctest --test-dir build --output-on-failure
 ./build/loglens --mode syslog --year 2026 ./assets/sample_auth.log ./out
 ```
 
+For Visual Studio or other multi-config generators, use
+`ctest --test-dir build -C Debug --output-on-failure` for the test step.
+
 Then inspect:
 
 - `out/report.md`

docs/rule-catalog.md

@@ -24,6 +24,7 @@ Metadata equivalent:
 - Rule names are stable report values.
 - Windows and thresholds are configurable through `config.json`.
 - Default values below match the built-in detector configuration.
+- The checked-in `assets/sample_config.json` is a tested default-equivalent fixture.
 
 ## Brute Force
 

tests/test_cli.cpp

@@ -225,6 +225,12 @@ int main(int argc, char* argv[]) {
         + " " + quote_argument(config_run_out))
                                                 .c_str());
     expect(config_run_exit == 0, "expected sample config run to succeed");
+    expect_report_core_fields(
+        read_file(config_run_out / "report.md"),
+        read_file(config_run_out / "report.json"),
+        "syslog_legacy",
+        true,
+        false);
 
     const auto journalctl_out = output_dir / "journalctl_cli";
     std::filesystem::create_directories(journalctl_out);

tests/test_detector.cpp

@@ -10,6 +10,7 @@
 #include <stdexcept>
 #include <string>
 #include <string_view>
+#include <vector>
 
 namespace {
 
@@ -49,6 +50,31 @@ std::vector<loglens::Event> parse_events(loglens::ParserConfig config, std::stri
     return parser.parse_stream(input).events;
 }
 
+std::filesystem::path repo_root() {
+    const std::filesystem::path source_path{__FILE__};
+    std::vector<std::filesystem::path> candidates;
+
+    if (source_path.is_absolute()) {
+        candidates.push_back(source_path);
+    } else {
+        const auto cwd = std::filesystem::current_path();
+        candidates.push_back(cwd / source_path);
+        candidates.push_back(cwd.parent_path() / source_path);
+    }
+
+    for (const auto& candidate : candidates) {
+        if (std::filesystem::exists(candidate)) {
+            return candidate.parent_path().parent_path();
+        }
+    }
+
+    throw std::runtime_error("unable to resolve repository root from test source path");
+}
+
+std::filesystem::path asset_path(std::string_view filename) {
+    return repo_root() / "assets" / std::string(filename);
+}
+
 loglens::ParserConfig make_syslog_config() {
     return loglens::ParserConfig{
         loglens::InputMode::SyslogLegacy,
@@ -121,6 +147,54 @@ std::vector<loglens::Event> build_sudo_burst_preservation_events() {
         "Mar 10 08:24:15 example-host sudo:    alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/vi /etc/ssh/sshd_config\n");
 }
 
+void expect_same_rule_threshold(const loglens::RuleThreshold& actual,
+                                const loglens::RuleThreshold& expected,
+                                const std::string& rule_name) {
+    expect(actual.threshold == expected.threshold, "expected " + rule_name + " threshold to match default");
+    expect(actual.window == expected.window, "expected " + rule_name + " window to match default");
+}
+
+void expect_same_auth_signal_behavior(const loglens::AuthSignalBehavior& actual,
+                                      const loglens::AuthSignalBehavior& expected,
+                                      const std::string& signal_name) {
+    expect(actual.counts_as_attempt_evidence == expected.counts_as_attempt_evidence,
+           "expected " + signal_name + " attempt-evidence mapping to match default");
+    expect(actual.counts_as_terminal_auth_failure == expected.counts_as_terminal_auth_failure,
+           "expected " + signal_name + " terminal-failure mapping to match default");
+}
+
+void expect_same_detector_config(const loglens::DetectorConfig& actual,
+                                 const loglens::DetectorConfig& expected) {
+    expect_same_rule_threshold(actual.brute_force, expected.brute_force, "brute_force");
+    expect_same_rule_threshold(actual.multi_user_probing, expected.multi_user_probing, "multi_user_probing");
+    expect_same_rule_threshold(actual.sudo_burst, expected.sudo_burst, "sudo_burst");
+
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.ssh_failed_password,
+        expected.auth_signal_mappings.ssh_failed_password,
+        "ssh_failed_password");
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.ssh_invalid_user,
+        expected.auth_signal_mappings.ssh_invalid_user,
+        "ssh_invalid_user");
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.ssh_failed_publickey,
+        expected.auth_signal_mappings.ssh_failed_publickey,
+        "ssh_failed_publickey");
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.ssh_failed_keyboard_interactive,
+        expected.auth_signal_mappings.ssh_failed_keyboard_interactive,
+        "ssh_failed_keyboard_interactive");
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.ssh_max_auth_tries,
+        expected.auth_signal_mappings.ssh_max_auth_tries,
+        "ssh_max_auth_tries");
+    expect_same_auth_signal_behavior(
+        actual.auth_signal_mappings.pam_auth_failure,
+        expected.auth_signal_mappings.pam_auth_failure,
+        "pam_auth_failure");
+}
+
 void test_default_thresholds() {
     const auto events = build_events();
     const loglens::Detector detector;
@@ -341,6 +415,30 @@ void test_load_valid_config() {
     expect(findings.size() == 3, "expected loaded config to preserve default findings");
 }
 
+void test_sample_config_matches_default_detector_contract() {
+    const auto config = loglens::load_app_config(asset_path("sample_config.json"));
+    expect(config.input_mode == loglens::InputMode::SyslogLegacy,
+           "expected sample config to use syslog legacy input");
+    expect(config.timestamp.assume_year == 2026,
+           "expected sample config to provide the sample syslog year");
+    expect_same_detector_config(config.detector, loglens::DetectorConfig{});
+
+    const auto events = build_events();
+    const loglens::Detector default_detector;
+    const loglens::Detector sample_config_detector(config.detector);
+    const auto default_findings = default_detector.analyze(events);
+    const auto sample_config_findings = sample_config_detector.analyze(events);
+
+    expect(sample_config_findings.size() == default_findings.size(),
+           "expected sample config to preserve default finding count");
+    expect(find_finding(sample_config_findings, loglens::FindingType::BruteForce, "203.0.113.10") != nullptr,
+           "expected sample config to preserve brute-force finding");
+    expect(find_finding(sample_config_findings, loglens::FindingType::MultiUserProbing, "203.0.113.10") != nullptr,
+           "expected sample config to preserve multi-user finding");
+    expect(find_finding(sample_config_findings, loglens::FindingType::SudoBurst, "alice") != nullptr,
+           "expected sample config to preserve sudo-burst finding");
+}
+
 void test_reject_invalid_config() {
     const auto temp_path = std::filesystem::current_path() / "invalid_config_test.json";
     {
@@ -389,6 +487,7 @@ int main() {
     test_pam_auth_failure_does_not_trigger_bruteforce_by_default();
     test_equivalent_attack_scenario_yields_same_finding_count_across_modes();
     test_load_valid_config();
+    test_sample_config_matches_default_detector_contract();
     test_reject_invalid_config();
     return 0;
 }