BlastDNS

[TheTechromancer]

Summary

Replace dnspython + the EngineClient/EngineServer subprocess architecture with blastdns, a Rust-backed async DNS client. DNS resolution now runs in Rust threads directly in the main process — no more subprocess IPC.

What changed

• dns.py: DNSHelper no longer inherits from EngineClient. Wraps a blastdns.Client directly. Wildcard detection, dns_omit_queries, error tracking, and connectivity checks ported from engine.py.
• helpers.py: extract_targets() and record_to_text() now delegate to Rust (record.extract_targets(), record.to_text()) instead of manually parsing rdata dicts in Python. TXT hostname extraction (SPF/DKIM) stays in Python.
• engine.py, mock.py: Deleted. blastdns handles caching, retries, and mocking natively.
• dnsresolve.py: resolve_raw_batch() replaced with resolve_multi_full() (one host, many rdtypes — resolved concurrently in Rust).
• dnsbimi.py, dnstlsrpt.py, dnscaa.py: resolve_raw() replaced with resolve_full(). CAA module reads structured rdata directly instead of regex-parsing zone-format text.
• Tests: Rewritten for the new API. Mock data uses proper zone-file format. custom_lookup_fn replaced with blastdns regex: mock patterns.
• pyproject.toml: Added blastdns dependency.

Deleted

• bbot/core/helpers/dns/engine.py (~660 lines)
• bbot/core/helpers/dns/mock.py (~75 lines)

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs blastdns (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.26ms	4.20ms	-1.3% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.39ms	17.43ms	+0.2% ⚪	✅
Large Closest Match Lookup	347.75ms	348.68ms	+0.3% ⚪	✅
Realistic Closest Match Workload	187.14ms	188.79ms	+0.9% ⚪	✅
Event Memory Medium Scan	1784 B/event	1784 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1768 B/event	1768 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	401.76ms	410.86ms	+2.3% ⚪	✅
Event Validation Full Scan Startup Large Batch	579.06ms	587.14ms	+1.4% ⚪	✅
Make Event Autodetection Small	30.71ms	31.10ms	+1.2% ⚪	✅
Make Event Autodetection Large	312.16ms	318.13ms	+1.9% ⚪	✅
Make Event Explicit Types	13.75ms	13.95ms	+1.4% ⚪	✅
Excavate Single Thread Small	3.935s	3.915s	-0.5% ⚪	✅
Excavate Single Thread Large	9.457s	9.658s	+2.1% ⚪	✅
Excavate Parallel Tasks Small	4.099s	4.062s	-0.9% ⚪	✅
Excavate Parallel Tasks Large	7.132s	7.148s	+0.2% ⚪	✅
Is Ip Performance	3.25ms	3.26ms	+0.2% ⚪	✅
Make Ip Type Performance	11.64ms	11.64ms	-0.0% ⚪	✅
Mixed Ip Operations	4.58ms	4.61ms	+0.5% ⚪	✅
Memory Use Web Crawl	41.3 MB	45.6 MB	+10.4% 🟡🟡	⚠️
Memory Use Subdomain Enum	19.4 MB	19.3 MB	-0.1% ⚪	✅
Scan Throughput 100	8.066s	7.191s	-10.8% 🟢🟢	🚀
Scan Throughput 1000	39.748s	37.162s	-6.5% ⚪	✅
Typical Queue Shuffle	63.86µs	62.03µs	-2.9% ⚪	✅
Priority Queue Shuffle	721.40µs	703.10µs	-2.5% ⚪	✅

🎯 Performance Summary

+ 1 improvement 🚀
! 1 regression ⚠️
  22 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Web Crawl: 10.4% 🐌 more memory
• Scan Throughput 100: 10.8% 🚀 faster

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 88.70056% with 60 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (a20ff02) to head (8600595).
⚠️ Report is 20 commits behind head on dev.

Files with missing lines	Patch %	Lines
bbot/core/helpers/dns/dns.py	80%	45 Missing ⚠️
bbot/scanner/target.py	45%	5 Missing ⚠️
bbot/modules/internal/dnsresolve.py	74%	4 Missing ⚠️
bbot/modules/dnscaa.py	89%	3 Missing ⚠️
bbot/modules/baddns.py	67%	2 Missing ⚠️
bbot/__init__.py	67%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3042    +/-   ##
======================================
- Coverage     91%     91%    -0%     
======================================
  Files        440     437     -3     
  Lines      37685   37431   -254     
======================================
- Hits       34104   33874   -230     
+ Misses      3581    3557    -24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheTechromancer]

This new implementation is fast, but one annoying bug is that due to the retry mechanism, black-holed / misconfigured DNS zones clog up the pipes and cause DNS resolution to slow to a crawl:

[DBUG]         - dnsresolve.handle_event(DNS_NAME("cbt-login-sg.wolf.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 23.8s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("dev-test-oss.wolf.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 20.5s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("label-studios.ugc.dev.gcp.hzn3.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 20.4s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("designs-22.preview.gcp.ugc-cs.captech.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 18.3s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("otel.dev.gcp.ugc-cs.captech.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 16.9s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("cbt-chat-sg.wolf.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 14.9s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("keycloak.luhernandez.gcp.msi.captech.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 14.7s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("console-storage-admin.msi.dev.aws.hzn3.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 13.7s:
[DBUG]         - dnsresolve.handle_event(DNS_NAME("content-reporting-api.int.gcp.ugc-cs.captech.ea.com", module=crt_db, tags={'subdomain', 'in-scope'})) running for 13.2s:

The obvious solution is to add more resolvers to your /etc/resolv.conf. But is there anything we can do to make this less bad?