Skip to content

Add E2E tests, bump cloudcheck to 11.x#3210

Merged
liquidsec merged 6 commits into
devfrom
e2e-tests
Jun 25, 2026
Merged

Add E2E tests, bump cloudcheck to 11.x#3210
liquidsec merged 6 commits into
devfrom
e2e-tests

Conversation

@liquidsec

Copy link
Copy Markdown
Collaborator

Summary

  • Adds end-to-end tests that install bbot into a fresh temp venv and run real CLI scans
  • Bumps cloudcheck dep to >=11.0.0,<12 (fixes undeclared httpx runtime dep in cloudcheck 10.x)

E2E Tests

  • test_install_and_help - installs bbot from local source, verifies bbot -h works
  • test_scan_dns - DNS-only scan against one.one.one.one, asserts DNS_NAME events with resolved IPs and output files
  • test_scan_web - spins up a local HTTP server, runs a spider scan, asserts excavate discovers links across pages
  • test_clean_shutdown - verifies scan reaches FINISHED status, no [CRIT] errors, no orphaned bbot processes

E2E tests install bbot into a temp venv and run real CLI scans:
- install + help
- DNS scan against one.one.one.one
- web scan with spider preset against local HTTP server
- clean shutdown (no orphaned processes)

Bump cloudcheck >=11,<12 (10.x had undeclared httpx runtime dep).
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs e2e-tests (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 4.26ms 4.27ms +0.3%
Bloom Filter Large Scale Dns Brute Force 17.95ms 17.67ms -1.5%
Large Closest Match Lookup 366.56ms 357.48ms -2.5%
Realistic Closest Match Workload 196.75ms 189.92ms -3.5%
Event Memory Medium Scan 1402 B/event 1402 B/event +0.0%
Event Memory Large Scan 1527 B/event 1527 B/event +0.0%
Event Validation Full Scan Startup Small Batch 403.65ms 404.47ms +0.2%
Event Validation Full Scan Startup Large Batch 538.41ms 538.18ms -0.0%
Make Event Autodetection Small 24.94ms 25.22ms +1.1%
Make Event Autodetection Large 255.03ms 256.73ms +0.7%
Make Event Explicit Types 11.12ms 11.14ms +0.1%
Excavate Single Thread Small 4.097s 3.996s -2.4%
Excavate Single Thread Large 9.895s 9.852s -0.4%
Excavate Parallel Tasks Small 4.310s 4.168s -3.3%
Excavate Parallel Tasks Large 6.434s 6.300s -2.1%
Intercept Throughput Small 875.37ms 875.27ms -0.0%
Intercept Throughput Medium 933.08ms 862.11ms -7.6%
Dns Throughput Quiet 2.743s 2.712s -1.1%
Dns Throughput Loaded 1.921s 1.809s -5.8%
Dns Throughput Inherited 2.278s 2.508s +10.1% 🟡🟡 ⚠️
Is Ip Performance 2.26ms 2.24ms -0.8%
Make Ip Type Performance 237.04µs 232.98µs -1.7%
Mixed Ip Operations 2.35ms 2.33ms -0.9%
Memory Use Web Crawl 1.0 MB 1.0 MB +0.0%
Memory Use Subdomain Enum 28.3 MB 28.3 MB +0.0%
Memory Use Deep Chain 5.0 MB 5.0 MB +0.0%
Memory Use Parallel Chains 11.0 MB 10.4 MB -5.1%
Scan Throughput 100 2.802s 2.901s +3.5%
Scan Throughput 1000 20.646s 20.624s -0.1%
Typical Queue Shuffle 5.38µs 5.37µs -0.2%
Priority Queue Shuffle 25.95µs 25.82µs -0.5%

🎯 Performance Summary

! 1 regression ⚠️
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Dns Throughput Inherited: 10.1% 🐌 slower

🐍 Python Version 3.11.15

@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.66055% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 90%. Comparing base (2802c97) to head (cf92ce9).

Files with missing lines Patch % Lines
bbot/test/test_step_1/test_e2e.py 93% 8 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff           @@
##             dev   #3210    +/-   ##
======================================
- Coverage     90%     90%    -0%     
======================================
  Files        451     452     +1     
  Lines      46463   46572   +109     
======================================
+ Hits       41589   41677    +88     
- Misses      4874    4895    +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@liquidsec liquidsec requested a review from ausmaster June 18, 2026 15:26
@liquidsec liquidsec mentioned this pull request Jun 18, 2026
28 tasks
@liquidsec liquidsec added this to the BBOT 3.0 - blazed_elijah milestone Jun 19, 2026
ausmaster
ausmaster previously approved these changes Jun 25, 2026
@liquidsec liquidsec merged commit edf7c7d into dev Jun 25, 2026
17 checks passed
@liquidsec liquidsec deleted the e2e-tests branch June 25, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants