tests: add snapshot tests for feature extraction#3069
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
df27110 to
a0a9ef4
Compare
b634e12 to
a73ceef
Compare
469dc03 to
ffa2565
Compare
|
tests work for me locally, but seem to fail in CI due the the testfiles PR (mandiant/capa-testfiles#306) not being merged yet (we shouldn't merge it until we're happy with this PR, they depend on each other).
so, please review the design and code. ignore CI failures for the time being. |
mr-tz
left a comment
There was a problem hiding this comment.
nice work, some comments inline
do we want to run this as part of CI?
|
i'll address those comments, doesn't seem like that should take much. i would suggest running during CI. takes less than a minute on my computer i think. |
ffa2565 to
67b26f3
Compare
|
i believe the code for this is ready to go. once approved, i'll update the submodule and merge. |
|
the CI integration seems to be missing |
|
the new file "test_feature_snapshots.py" should register tests that get run in CI. at least, that's what i intend. would you disagree or have another idea? |
|
ah gotcha, I thought we'd add a CI GH Action step but this works |
18fa543 to
90ae4d3
Compare
90ae4d3 to
42d9745
Compare
Introduces data-driven snapshot tests that regenerate capa freeze files for a curated set of samples in the tests/data submodule and compare the bytes against committed fixtures under tests/fixtures/freezes/. Any change that perturbs feature extraction surfaces as a test failure with a feature-count delta and a truncated unified diff.
42d9745 to
379d768
Compare

In order to catch unexpected changes to our feature extractors, adds snapshot testing using freeze files.
The idea is not to add lots more overhead, but simply to notify the author when capa's behavior changes. If you're expecting changes to capa's feature extractors, then you can probably just update the fixtures. But if you're not expecting changes, then double check why things have changed and make an assessment.
Introduces data-driven snapshot tests that regenerate capa freeze files for a curated set of samples in the tests/data submodule and compare the bytes against committed fixtures under tests/fixtures/freezes/. Any change that perturbs feature extraction surfaces as a test failure with a feature- count delta and a truncated unified diff.
closes #3054
related: mandiant/capa-testfiles#306
Checklist