Perf: eliminate hot-loop overhead in seed stretching, base58 validation, and XOR operations#685
Conversation
…hing, base58 validation, XOR operations, and block parsing Key changes: - btcrseed.py: Move type checks out of 100k-iteration Electrum1 seed stretching loop - btcrseed.py: Cache self._ attributes as locals in address derivation inner loop - btcrseed.py: Cache hmac key/type checks outside BIP39 CPU and OpenCL loops - btcrseed.py: Use list comprehension in words_to_bytes/bytes_to_words - btcrseed.py: Use context managers for file I/O in load_pathlist/load_passphraselist - btcrpass.py: Replace 6 byte-by-byte base58 validation loops with frozenset lookup - btcrpass.py: Optimize PKCS7 padding check to avoid chr().encode() in loop - aezeed.py: Replace byte-by-byte XOR loops with int.from_bytes/int.to_bytes - aezeed.py: Optimize mismatch checks using direct bytes comparison - addressset.py: Cache script pattern byte-literals as module constants Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…ary slicing in XOR, use clearer slice syntax in _aez_decrypt Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
* Fix OpenCL kernel 64-bit rotate/bitselect bugs for Apple Silicon GPUs Apple Silicon GPUs use a Metal-based OpenCL translation layer that has known bugs with the rotate() and bitselect() built-in functions for 64-bit (unsigned long) types. This causes SHA-512 and other 64-bit hash computations to produce incorrect results. Changes: - buffer_structs_template.cl: Use shift-based rotate and byte-swap for 64-bit when APPLE_GPU is defined - pbkdf2_sha1_32.cl: Same fix for its copy of the 64-bit definitions - sha512.cl: Use portable choose/majority for 64-bit on Apple Silicon - sha512-bc-kernel.cl: Disable USE_BITSELECT and use shift-based ror on Apple Silicon - opencl.py: Detect Apple vendor and pass -DAPPLE_GPU build option; add error handling around kernel compilation - btcrpass.py: Detect Apple vendor and pass -DAPPLE_GPU when building the Bitcoin Core SHA-512 kernel - opencl_helpers.py: Score Apple GPUs in auto platform selection - multibit_md5.cl: Make cl_khr_byte_addressable_store pragma conditional to avoid warnings on Apple Silicon Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/3a816aa8-2be8-4375-93f6-d721b74ace6a Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Termux CI: replace obsolete binutils-is-llvm with llvm lld Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/9ef80c42-784d-4a7f-9c95-f298913eeb09 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Update Termux install docs: replace obsolete binutils-is-llvm with llvm lld Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/8c3013f6-8f65-4cbe-854a-9161b29ca66a Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Termux Full CI: build maturin from source for maturin-based packages The pre-built maturin wheel has the wrong host triple for Termux, causing py-sr25519-bindings to fail with a cross-compilation error. Fix by building maturin from source and pre-installing py-sr25519-bindings with --no-build-isolation. Set ANDROID_API_LEVEL=24 and increase timeout to 30 minutes. Also update install docs with full requirements instructions. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/86e8deeb-7ae0-43f4-aa43-57153b529479 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Termux Full CI: export ANDROID_API_LEVEL inside entrypoint.sh bash session The /entrypoint.sh in termux-docker switches user context via su, which strips environment variables set at the job level. Fix by explicitly exporting ANDROID_API_LEVEL=24 inside the bash -c command that runs all pip installs for maturin-based packages. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/6fb564fe-473a-49b5-a340-584ad7a44eb6 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…kages only (#693) * Remove hashes and trim requirements-full.txt to only directly-imported packages Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/fb79e1a2-ab86-4533-9f0c-f537b97e57a8 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Remove --require-hashes from CI workflows to match hashless requirements-full.txt Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/f09af411-37ab-4378-800c-61e6b43f4b6e Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…ng CI jobs (#696) * Remove Python 3.9 from CI, update 3.14 to official release, fix docs build and Termux CI failures, update documentation Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/e474d63d-a297-413b-9caa-a0aa15073ac6 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Remove continue-on-error for Python 3.14 from all CI workflows Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/e71450e0-54d9-40c9-8a4d-42dc901e2c60 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix AddressSet close order for Python 3.14 Windows compatibility Close mmap before file in AddressSet.close() to avoid OSError on Python 3.14 + Windows where closing a tempfile while an mmap is still active raises 'OSError: [Errno 22] Invalid argument'. Also defer file closing from fromfile() to close() so both mmap and file resources are released in the correct order. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/23cde30b-2da5-41ef-b9cb-6afddd4cd4bb Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Termux Full CI: unset PYO3_USE_ABI3_FORWARD_COMPATIBILITY before installing requirements-full.txt The PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 env var forces PyO3 into abi3 stable ABI mode, which is needed for py-sr25519-bindings but breaks pydantic-core (transitive dep of stellar_sdk) which uses full CPython struct internals (tp_base, tp_new, PyDateTime methods, etc.). By unsetting it after py-sr25519-bindings is installed, pydantic-core can build with full CPython API access. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/59451f88-b7de-4ccd-b469-0cd20751cab8 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Termux Full CI: bump py-sr25519-bindings to 0.2.3, remove PYO3_USE_ABI3_FORWARD_COMPATIBILITY py-sr25519-bindings 0.2.3 uses PyO3 0.27.1 (up from 0.20.3 in 0.2.2) which no longer needs the PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 workaround on Termux. Removing this env var fixes the pydantic-core build failure (transitive dep of stellar_sdk) which requires full CPython API access incompatible with abi3 mode. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/59451f88-b7de-4ccd-b469-0cd20751cab8 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Windows Python 3.14 AddressSet close and update Termux install docs On Windows + Python 3.14, mmap.close() can invalidate the underlying file handle, causing self._dbfile.close() to fail with OSError. Fix by: 1. Flushing file buffer before mmap close (data integrity for WRITE mode) 2. Catching OSError from file close after mmap close Also update docs/INSTALL.md: - Bump py-sr25519-bindings from 0.2.2 to 0.2.3 in Termux instructions - Remove outdated PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 guidance Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/afa5aa77-4719-48a1-a5be-7ab2bf678a6d Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix Windows Python 3.14 AddressSet close and Hedera false-positive verification AddressSet: Reorder close() to close the mmap before writing the header. On Windows + Python 3.14, file writes/flushes fail with OSError while an mmap is active on the same file handle. Hedera: Fix _hedera_require_key_bound_match inversion. When key-bound targets (raw hex addresses/keys) are provided, weak deterministic matches alone should NOT verify a seed. The inverted flag caused random false positives in test_hedera_ed25519_account_id. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/2e9b390a-ac8a-4fee-953d-73405e1cf46c Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> * Fix AddressSet close: reopen file by name when handle invalidated on Windows+Py3.14 On Windows + Python 3.14, mmap.close() invalidates the original file handle, so writing the updated header after closing the mmap fails. The fix saves the filename and header position in fromfile(), then in close() falls back to reopening the file by name when the original handle write fails. Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/cceeb164-af57-4bf8-8828-07cd899ad8a0 Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR focuses on micro-optimizations in btcrecover’s hottest paths (seed stretching/derivation loops, base58 “looks valid” checks, and script parsing), aiming to reduce Python-level overhead while keeping behavior identical.
Changes:
- Refactors several tight loops in
btcrseed.py(seed stretching, address generation inner loops, HMAC construction) to cache locals and avoid repeated conversions. - Speeds up base58-character validation in
btcrpass.pyusing a precomputed base58 byte set andall(...). - Reworks XOR and authentication checks in
aezeed.pyand caches common script-prefix constants inaddressset.pyfor faster parsing.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| btcrecover/btcrseed.py | Reduces overhead in mnemonic/seed handling and inner derivation loops; refactors byte/word conversion helpers. |
| btcrecover/btcrpass.py | Replaces base58 range checks with set-membership validation. |
| btcrecover/aezeed.py | Replaces byte-wise XOR loops with integer-based XOR; changes AEZ authentication verification logic. |
| btcrecover/addressset.py | Caches common script-prefix/suffix byte patterns used in block parsing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| word = int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False) | ||
| words.append(word) | ||
| return words | ||
| return [int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False) |
There was a problem hiding this comment.
bytes_to_words() now assumes byte_array[i:i+4] is a bytes-like object accepted by int.from_bytes(), but there are call sites that pass a plain list of ints (e.g., decode_v3() builds [b1[2], b1[3], 0, 0]). That will raise TypeError at runtime. Consider normalizing the input inside bytes_to_words() (e.g., wrapping with bytes(...)/bytearray(...) when needed) or updating call sites to pass a bytes-like object.
| return [int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False) | |
| return [int.from_bytes(bytes(byte_array[i:i+4]), byteorder='big', signed=False) |
| # Use bytes comparison instead of byte-by-byte XOR loop | ||
| if x[:tau] != ciphertext[:tau]: | ||
| return None |
There was a problem hiding this comment.
The authentication check in _aez_decrypt() was changed from a full-length XOR accumulation to x[:tau] != ciphertext[:tau]. Plain bytes comparison is not constant-time and can reintroduce a timing side-channel for tag verification. Use a constant-time comparison (e.g., hmac.compare_digest) to preserve the original security properties while still avoiding Python-level loops where possible.
| # Check if trailing tau bytes are all zero | ||
| if any(x[-tau:]): | ||
| return None |
There was a problem hiding this comment.
if any(x[-tau:]) short-circuits on the first non-zero byte, which can leak information via timing and differs from the prior constant-time style check. If this is part of the AEZ authentication/verification step, prefer a constant-time zero check (e.g., hmac.compare_digest(x[-tau:], b"\x00" * tau) or an XOR-accumulator over the slice).
origin/master(f25290f) into branch — includes 3 new master commits:💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.