Skip to content

tests: add snapshot tests for feature extraction#3069

Open
williballenthin wants to merge 1 commit into
masterfrom
wb/snapshot-testing-features
Open

tests: add snapshot tests for feature extraction#3069
williballenthin wants to merge 1 commit into
masterfrom
wb/snapshot-testing-features

Conversation

@williballenthin
Copy link
Copy Markdown
Collaborator

@williballenthin williballenthin commented May 11, 2026

In order to catch unexpected changes to our feature extractors, adds snapshot testing using freeze files.

The idea is not to add lots more overhead, but simply to notify the author when capa's behavior changes. If you're expecting changes to capa's feature extractors, then you can probably just update the fixtures. But if you're not expecting changes, then double check why things have changed and make an assessment.

Introduces data-driven snapshot tests that regenerate capa freeze files for a curated set of samples in the tests/data submodule and compare the bytes against committed fixtures under tests/fixtures/freezes/. Any change that perturbs feature extraction surfaces as a test failure with a feature- count delta and a truncated unified diff.

closes #3054

related: mandiant/capa-testfiles#306

Checklist

  • This submission includes AI-generated code: R&D with Claude Code

@google-cla

This comment was marked as outdated.

gemini-code-assist[bot]

This comment was marked as low quality.

@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch from df27110 to a0a9ef4 Compare May 11, 2026 08:06
@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch 4 times, most recently from b634e12 to a73ceef Compare May 11, 2026 09:15
@williballenthin williballenthin marked this pull request as ready for review May 11, 2026 09:17
@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch 5 times, most recently from 469dc03 to ffa2565 Compare May 11, 2026 09:22
@williballenthin
Copy link
Copy Markdown
Collaborator Author

williballenthin commented May 11, 2026

tests work for me locally, but seem to fail in CI due the the testfiles PR (mandiant/capa-testfiles#306) not being merged yet (we shouldn't merge it until we're happy with this PR, they depend on each other).

image

so, please review the design and code. ignore CI failures for the time being.

Copy link
Copy Markdown
Collaborator

@mr-tz mr-tz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work, some comments inline

do we want to run this as part of CI?

Comment thread capa/features/freeze/__init__.py Outdated
Comment thread capa/features/freeze/__init__.py
Comment thread CHANGELOG.md Outdated
Comment thread CHANGELOG.md Outdated
@williballenthin
Copy link
Copy Markdown
Collaborator Author

i'll address those comments, doesn't seem like that should take much.

i would suggest running during CI. takes less than a minute on my computer i think.

@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch 2 times, most recently from ffa2565 to 67b26f3 Compare May 12, 2026 07:47
@williballenthin
Copy link
Copy Markdown
Collaborator Author

i believe the code for this is ready to go. once approved, i'll update the submodule and merge.

@mr-tz
Copy link
Copy Markdown
Collaborator

mr-tz commented May 18, 2026

the CI integration seems to be missing

@williballenthin
Copy link
Copy Markdown
Collaborator Author

the new file "test_feature_snapshots.py" should register tests that get run in CI. at least, that's what i intend. would you disagree or have another idea?

@mr-tz
Copy link
Copy Markdown
Collaborator

mr-tz commented May 18, 2026

ah gotcha, I thought we'd add a CI GH Action step but this works

@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch 3 times, most recently from 18fa543 to 90ae4d3 Compare May 20, 2026 08:10
@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch from 90ae4d3 to 42d9745 Compare May 20, 2026 08:11
Introduces data-driven snapshot tests that regenerate capa freeze files
for a curated set of samples in the tests/data submodule and compare the
bytes against committed fixtures under tests/fixtures/freezes/. Any
change that perturbs feature extraction surfaces as a test failure with
a feature-count delta and a truncated unified diff.
@williballenthin williballenthin force-pushed the wb/snapshot-testing-features branch 2 times, most recently from 42d9745 to 379d768 Compare May 20, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feature snapshot tests

2 participants