TSAN fixes: Refactor mutex handling

[eddyashton]

This is largely written by the robot, so I'll let the robot describe the changes:

This pull request introduces several improvements to node startup, certificate handling, and lock management in the consensus and node state code. The most significant changes include the addition of a structured startup state for consensus initialization, improved lock usage to prevent deadlocks, and asynchronous frontend opening to avoid lock contention. These changes enhance the robustness and clarity of node initialization and certificate management.

Consensus Startup Improvements:

• Added StartupRole and StartupState structs to aft::Raft for explicit initialization as primary or backup, including state info such as index, term, and view history. The consensus constructor now uses this to set up initial state and role, supporting more flexible and safer node startup scenarios.
• Modified setup_consensus in NodeState to accept and propagate StartupState, enabling correct initialization for both primary and backup roles, and ensuring backup initialization runs before other threads access consensus.

Certificate Locking and Management:

• Introduced a dedicated endorsed_cert_lock for certificate fields in NodeState, replacing the main lock in certificate-related operations to avoid lock-order-inversion and potential deadlocks. Certificate accesses and updates now use this lock, and new helper methods for safe/unsafe certificate access were added.

Frontend Initialization and Lock Avoidance:

• Replaced synchronous frontend opening with open_frontend_async, scheduling frontend opening tasks to avoid holding KV or consensus locks during potentially blocking operations. This change applies to node, member, and user frontends.

Ledger Secrets Locking Consistency:

• Moved mutex locking in ledger_secrets.h functions to after dependency checks, ensuring consistent lock ordering and reducing risk of deadlocks.

Cleanup and Deadlock Suppression:

• Removed deadlock and race suppressions for files no longer requiring them in tsan_env_suppressions, reflecting improvements in lock handling and concurrency.

[Copilot]

Pull request overview

Refactors node startup and consensus initialisation to reduce TSAN-reported deadlocks/races by making startup state explicit, adjusting lock usage around certificate/ledger-secret access, and deferring some potentially blocking work to async tasks.

Changes:

• Introduces aft::Aft::StartupState/StartupRole to initialise consensus as Primary/Backup (and optionally seed term/index/view history) inside the consensus constructor.
• Splits certificate locking in NodeState via a dedicated endorsed_cert_lock, and switches several frontend openings to open_frontend_async to avoid lock-order inversions.
• Moves LedgerSecrets mutex acquisition after KV dependency checks, and removes some TSAN suppressions.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tsan_env_suppressions	Removes deadlock/race suppressions which are presumed obsolete after lock refactors.
src/node/node_state.h	Adds dedicated cert lock, refactors consensus setup to accept startup state, and introduces async frontend opening to reduce lock contention.
src/node/ledger_secrets.h	Adjusts lock ordering by taking KV dependency before acquiring the ledger-secrets mutex.
src/consensus/aft/raft.h	Adds consensus constructor startup initialisation logic for primary/backup roles.

[eddyashton]

@copilot Review again.

[Copilot]

Pull request overview

Refactors several startup and concurrency paths to reduce TSAN reports by making consensus initialisation more explicit, reducing lock-order inversions in NodeState, and decoupling frontend opening from KV/consensus hook execution.

Changes:

• Add an explicit consensus startup description (StartupRole/StartupState) and plumb it through NodeState::setup_consensus() into AFT construction.
• Rework frontend consensus/history wiring (RpcHandler::set_consensus_and_history) and update tests accordingly.
• Adjust certificate and ledger secret locking patterns and remove TSAN suppressions that should no longer be necessary.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tsan_env_suppressions	Removes deadlock/race suppressions tied to prior locking issues.
src/node/rpc/test/frontend_test.cpp	Updates forwarding tests to explicitly initialise frontend consensus/history.
src/node/rpc/frontend.h	Stores consensus/history pointers atomically and removes per-request “update” logic in favor of explicit initialisation.
src/node/node_state.h	Adds endorsed_cert_lock, refactors startup ordering, introduces async frontend open helpers, and changes consensus setup to accept startup state.
src/node/node_client.h	Changes NodeClient to store certificate values rather than references.
src/node/ledger_secrets.h	Moves lock acquisition after dependency tracking to improve lock ordering.
src/enclave/rpc_handler.h	Extends RpcHandler interface to allow setting consensus/history pointers.
src/consensus/aft/raft.h	Adds startup-state-driven initialisation in the AFT constructor.

Comments suppressed due to low confidence (1)

src/node/rpc/frontend.h:246

• resolve_redirect_location() unconditionally dereferences consensus.load() (consensus.load()->primary()), but callers can reach this when consensus is still null (eg check_redirect() treats consensus == nullptr as !is_primary and then calls into this resolver). This can crash early during startup or in tests/embedders that haven't called set_consensus_and_history() yet. Load the pointer once into a local, return std::nullopt (or otherwise handle redirect as impossible) when it is null, and avoid any dereference unless non-null.

          std::vector<std::map<NodeId, NodeInfo>::const_iterator>
            target_node_its;
          const auto nodes = InternalTablesAccess::get_trusted_nodes(tx);
          {
            const auto primary_id = consensus.load()->primary();
            if (seeking_primary && primary_id.has_value())
            {
              target_node_its.push_back(nodes.find(primary_id.value()));
            }
            else if (seeking_backup)
            {
              for (auto it = nodes.begin(); it != nodes.end(); ++it)
              {
                if (it->first != primary_id)
                {

[Copilot]

get_self_signed_certificate_unsafe() is used here while self_signed_node_cert can be mutated concurrently under endorsed_cert_lock (eg when refreshing the self-signed cert in the node_endorsed_certificates global hook). This introduces a real data race on self_signed_node_cert. Use the locked accessor (get_self_signed_certificate()) or explicitly take endorsed_cert_lock around this read instead of the unsafe getter.

Suggested change

	get_self_signed_certificate_unsafe(),
	get_self_signed_certificate(),

[Copilot]

This SnapshotEvidence global commit hook runs during Store::compact() while Store::maps_lock is held (see src/kv/store.h), so taking NodeState::lock here risks reintroducing lock-order inversions/deadlocks with any path that takes NodeState::lock and later needs maps_lock. Consider using a dedicated mutex for backup_snapshot_fetch_task state (or making it atomic), and defer any locking to the scheduled task rather than inside the hook itself.