Skip to content

Perf: eliminate hot-loop overhead in seed stretching, base58 validation, and XOR operations#685

Merged
3rdIteration merged 8 commits intomasterfrom
copilot/optimize-code-performance
Apr 8, 2026
Merged

Perf: eliminate hot-loop overhead in seed stretching, base58 validation, and XOR operations#685
3rdIteration merged 8 commits intomasterfrom
copilot/optimize-code-performance

Conversation

Copy link
Copy Markdown

Copilot AI commented Mar 13, 2026


💬 Send tasks to Copilot coding agent from Slack and Teams to turn conversations into code. Copilot posts an update in your thread when it's finished.

Copilot AI and others added 2 commits March 13, 2026 03:36
…hing, base58 validation, XOR operations, and block parsing

Key changes:
- btcrseed.py: Move type checks out of 100k-iteration Electrum1 seed stretching loop
- btcrseed.py: Cache self._ attributes as locals in address derivation inner loop
- btcrseed.py: Cache hmac key/type checks outside BIP39 CPU and OpenCL loops
- btcrseed.py: Use list comprehension in words_to_bytes/bytes_to_words
- btcrseed.py: Use context managers for file I/O in load_pathlist/load_passphraselist
- btcrpass.py: Replace 6 byte-by-byte base58 validation loops with frozenset lookup
- btcrpass.py: Optimize PKCS7 padding check to avoid chr().encode() in loop
- aezeed.py: Replace byte-by-byte XOR loops with int.from_bytes/int.to_bytes
- aezeed.py: Optimize mismatch checks using direct bytes comparison
- addressset.py: Cache script pattern byte-literals as module constants

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…ary slicing in XOR, use clearer slice syntax in _aez_decrypt

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
Copilot AI changed the title [WIP] Identify performance optimizations in code base Perf: eliminate hot-loop overhead in seed stretching, base58 validation, and XOR operations Mar 13, 2026
Copilot AI requested a review from 3rdIteration March 13, 2026 03:42
Copilot AI and others added 5 commits April 8, 2026 16:59
* Fix OpenCL kernel 64-bit rotate/bitselect bugs for Apple Silicon GPUs

Apple Silicon GPUs use a Metal-based OpenCL translation layer that has
known bugs with the rotate() and bitselect() built-in functions for
64-bit (unsigned long) types. This causes SHA-512 and other 64-bit
hash computations to produce incorrect results.

Changes:
- buffer_structs_template.cl: Use shift-based rotate and byte-swap
  for 64-bit when APPLE_GPU is defined
- pbkdf2_sha1_32.cl: Same fix for its copy of the 64-bit definitions
- sha512.cl: Use portable choose/majority for 64-bit on Apple Silicon
- sha512-bc-kernel.cl: Disable USE_BITSELECT and use shift-based ror
  on Apple Silicon
- opencl.py: Detect Apple vendor and pass -DAPPLE_GPU build option;
  add error handling around kernel compilation
- btcrpass.py: Detect Apple vendor and pass -DAPPLE_GPU when building
  the Bitcoin Core SHA-512 kernel
- opencl_helpers.py: Score Apple GPUs in auto platform selection
- multibit_md5.cl: Make cl_khr_byte_addressable_store pragma
  conditional to avoid warnings on Apple Silicon

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/3a816aa8-2be8-4375-93f6-d721b74ace6a

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Termux CI: replace obsolete binutils-is-llvm with llvm lld

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/9ef80c42-784d-4a7f-9c95-f298913eeb09

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Update Termux install docs: replace obsolete binutils-is-llvm with llvm lld

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/8c3013f6-8f65-4cbe-854a-9161b29ca66a

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Termux Full CI: build maturin from source for maturin-based packages

The pre-built maturin wheel has the wrong host triple for Termux,
causing py-sr25519-bindings to fail with a cross-compilation error.
Fix by building maturin from source and pre-installing py-sr25519-bindings
with --no-build-isolation. Set ANDROID_API_LEVEL=24 and increase timeout
to 30 minutes. Also update install docs with full requirements instructions.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/86e8deeb-7ae0-43f4-aa43-57153b529479

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Termux Full CI: export ANDROID_API_LEVEL inside entrypoint.sh bash session

The /entrypoint.sh in termux-docker switches user context via su, which
strips environment variables set at the job level. Fix by explicitly
exporting ANDROID_API_LEVEL=24 inside the bash -c command that runs
all pip installs for maturin-based packages.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/6fb564fe-473a-49b5-a340-584ad7a44eb6

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…kages only (#693)

* Remove hashes and trim requirements-full.txt to only directly-imported packages

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/fb79e1a2-ab86-4533-9f0c-f537b97e57a8

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Remove --require-hashes from CI workflows to match hashless requirements-full.txt

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/f09af411-37ab-4378-800c-61e6b43f4b6e

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
…ng CI jobs (#696)

* Remove Python 3.9 from CI, update 3.14 to official release, fix docs build and Termux CI failures, update documentation

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/e474d63d-a297-413b-9caa-a0aa15073ac6

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Remove continue-on-error for Python 3.14 from all CI workflows

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/e71450e0-54d9-40c9-8a4d-42dc901e2c60

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix AddressSet close order for Python 3.14 Windows compatibility

Close mmap before file in AddressSet.close() to avoid OSError on
Python 3.14 + Windows where closing a tempfile while an mmap is
still active raises 'OSError: [Errno 22] Invalid argument'.

Also defer file closing from fromfile() to close() so both mmap
and file resources are released in the correct order.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/23cde30b-2da5-41ef-b9cb-6afddd4cd4bb

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Termux Full CI: unset PYO3_USE_ABI3_FORWARD_COMPATIBILITY before installing requirements-full.txt

The PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 env var forces PyO3 into
abi3 stable ABI mode, which is needed for py-sr25519-bindings but
breaks pydantic-core (transitive dep of stellar_sdk) which uses full
CPython struct internals (tp_base, tp_new, PyDateTime methods, etc.).
By unsetting it after py-sr25519-bindings is installed, pydantic-core
can build with full CPython API access.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/59451f88-b7de-4ccd-b469-0cd20751cab8

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Termux Full CI: bump py-sr25519-bindings to 0.2.3, remove PYO3_USE_ABI3_FORWARD_COMPATIBILITY

py-sr25519-bindings 0.2.3 uses PyO3 0.27.1 (up from 0.20.3 in 0.2.2)
which no longer needs the PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
workaround on Termux. Removing this env var fixes the pydantic-core
build failure (transitive dep of stellar_sdk) which requires full
CPython API access incompatible with abi3 mode.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/59451f88-b7de-4ccd-b469-0cd20751cab8

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Windows Python 3.14 AddressSet close and update Termux install docs

On Windows + Python 3.14, mmap.close() can invalidate the underlying
file handle, causing self._dbfile.close() to fail with OSError. Fix by:
1. Flushing file buffer before mmap close (data integrity for WRITE mode)
2. Catching OSError from file close after mmap close

Also update docs/INSTALL.md:
- Bump py-sr25519-bindings from 0.2.2 to 0.2.3 in Termux instructions
- Remove outdated PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 guidance

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/afa5aa77-4719-48a1-a5be-7ab2bf678a6d

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix Windows Python 3.14 AddressSet close and Hedera false-positive verification

AddressSet: Reorder close() to close the mmap before writing the header.
On Windows + Python 3.14, file writes/flushes fail with OSError while
an mmap is active on the same file handle.

Hedera: Fix _hedera_require_key_bound_match inversion. When key-bound
targets (raw hex addresses/keys) are provided, weak deterministic
matches alone should NOT verify a seed. The inverted flag caused
random false positives in test_hedera_ed25519_account_id.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/2e9b390a-ac8a-4fee-953d-73405e1cf46c

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

* Fix AddressSet close: reopen file by name when handle invalidated on Windows+Py3.14

On Windows + Python 3.14, mmap.close() invalidates the original file
handle, so writing the updated header after closing the mmap fails.
The fix saves the filename and header position in fromfile(), then in
close() falls back to reopening the file by name when the original
handle write fails.

Agent-Logs-Url: https://github.com/3rdIteration/btcrecover/sessions/cceeb164-af57-4bf8-8828-07cd899ad8a0

Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
Co-authored-by: 3rdIteration <2230318+3rdIteration@users.noreply.github.com>
@3rdIteration 3rdIteration marked this pull request as ready for review April 8, 2026 17:38
Copilot AI review requested due to automatic review settings April 8, 2026 17:38
@3rdIteration 3rdIteration merged commit f5e5a86 into master Apr 8, 2026
34 checks passed
@3rdIteration 3rdIteration deleted the copilot/optimize-code-performance branch April 8, 2026 17:38
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR focuses on micro-optimizations in btcrecover’s hottest paths (seed stretching/derivation loops, base58 “looks valid” checks, and script parsing), aiming to reduce Python-level overhead while keeping behavior identical.

Changes:

  • Refactors several tight loops in btcrseed.py (seed stretching, address generation inner loops, HMAC construction) to cache locals and avoid repeated conversions.
  • Speeds up base58-character validation in btcrpass.py using a precomputed base58 byte set and all(...).
  • Reworks XOR and authentication checks in aezeed.py and caches common script-prefix constants in addressset.py for faster parsing.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
btcrecover/btcrseed.py Reduces overhead in mnemonic/seed handling and inner derivation loops; refactors byte/word conversion helpers.
btcrecover/btcrpass.py Replaces base58 range checks with set-membership validation.
btcrecover/aezeed.py Replaces byte-wise XOR loops with integer-based XOR; changes AEZ authentication verification logic.
btcrecover/addressset.py Caches common script-prefix/suffix byte patterns used in block parsing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread btcrecover/btcrseed.py
word = int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False)
words.append(word)
return words
return [int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False)
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes_to_words() now assumes byte_array[i:i+4] is a bytes-like object accepted by int.from_bytes(), but there are call sites that pass a plain list of ints (e.g., decode_v3() builds [b1[2], b1[3], 0, 0]). That will raise TypeError at runtime. Consider normalizing the input inside bytes_to_words() (e.g., wrapping with bytes(...)/bytearray(...) when needed) or updating call sites to pass a bytes-like object.

Suggested change
return [int.from_bytes(byte_array[i:i+4], byteorder='big', signed=False)
return [int.from_bytes(bytes(byte_array[i:i+4]), byteorder='big', signed=False)

Copilot uses AI. Check for mistakes.
Comment thread btcrecover/aezeed.py
Comment on lines +775 to 777
# Use bytes comparison instead of byte-by-byte XOR loop
if x[:tau] != ciphertext[:tau]:
return None
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The authentication check in _aez_decrypt() was changed from a full-length XOR accumulation to x[:tau] != ciphertext[:tau]. Plain bytes comparison is not constant-time and can reintroduce a timing side-channel for tag verification. Use a constant-time comparison (e.g., hmac.compare_digest) to preserve the original security properties while still avoiding Python-level loops where possible.

Copilot uses AI. Check for mistakes.
Comment thread btcrecover/aezeed.py
Comment on lines +780 to 782
# Check if trailing tau bytes are all zero
if any(x[-tau:]):
return None
Copy link

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if any(x[-tau:]) short-circuits on the first non-zero byte, which can leak information via timing and differs from the prior constant-time style check. If this is part of the AEZ authentication/verification step, prefer a constant-time zero check (e.g., hmac.compare_digest(x[-tau:], b"\x00" * tau) or an XOR-accumulator over the slice).

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants