Skip to content

Fix permanent VDI activation lock leak after host crash#804

Open
olivierlambert wants to merge 1 commit into
xapi-project:masterfrom
olivierlambert:fix-stale-activating-lock
Open

Fix permanent VDI activation lock leak after host crash#804
olivierlambert wants to merge 1 commit into
xapi-project:masterfrom
olivierlambert:fix-stale-activating-lock

Conversation

@olivierlambert

@olivierlambert olivierlambert commented Jul 14, 2026

Copy link
Copy Markdown

Fixes #803.

The activating VDI sm_config key used by blktap2 as a pool-wide activation lock is released in a finally clause, which protects against exceptions but not against the host dying mid-activation (HA fence, power loss, watchdog). The leaked key then fails every subsequent activation of that VDI pool-wide with MAP_DUPLICATE_KEY and permanently stalls GC relinking, and no cleanup path ever removes it — unlike the sibling host_<ref> attach keys, which get crash-recovery treatment both in resetvdis.reset_sr() and in xapi's refresh_local_vdi_activations. xe cannot clear it either (VDI sm-config is a read-only map in the CLI); only raw VDI.remove_from_sm_config calls could recover. See the linked issue for the full analysis and reproduction.

This PR gives activating the same owner encoding and crash-recovery treatment, without renaming the key (the GC's DB_VDI_ACTIVATING check and the snapshot sm_config filter in LVHDSR.py depend on the literal name):

  • _add_tag() stores the owning host ref as the key's value instead of 'True', so staleness can be attributed;
  • resetvdis.reset_sr(), already run on every sr_attach, now removes an activating entry owned by the re-attaching host — a crashed host cleans up after itself as soon as it replugs its PBDs, before anything can attempt an activation;
  • resetvdis.reset_vdi() now removes the entry when --force is given or when the recorded owner is no longer part of the pool, making resetvdis single <uuid> a supported remedy for stale locks left by hosts that never come back.

Legacy bare 'True' values written by older code cannot be attributed to an owner and are deliberately only cleared by --force, so mixed pools during rolling upgrade behave exactly as before.

Acquisition-time arbitration (stealing the key on MAP_DUPLICATE_KEY when no host_* key is present) was considered and rejected: a concurrent starter can observe the window between the two adjacent add_to_sm_config calls of a healthy activation and steal a live lock, allowing two RW tapdisks on shared storage.

Testing: new unit tests in tests/test_resetvdis.py cover the reset_sr owner-match/mismatch/legacy cases and the reset_vdi force/invalid-owner/legacy cases; tests/test_blktap2.py expectations updated for the new value. Full suite passes with no new failures (the 5 failures in test_HBASR/test_cleanup are pre-existing on master in my environment).

The 'activating' VDI sm_config key used by blktap2 as a pool-wide
activation lock is released in a finally clause, which protects
against exceptions but not against the host dying mid-activation
(fence, power loss, watchdog). The leaked key then fails every
subsequent activation of that VDI pool-wide with MAP_DUPLICATE_KEY,
and blocks GC relinking. Unlike the sibling host_<ref> attach keys,
it carried no owner and no cleanup path ever removed it: xe cannot
touch it (sm-config is read-only in the CLI) and only raw
VDI.remove_from_sm_config calls could recover.

Encode the owning host ref as the key's value (instead of 'True') so
staleness can be arbitrated, and clean the key up in the two places
that already give host_<ref> keys their crash-recovery treatment:

- resetvdis.reset_sr, run on every sr_attach, now removes an
  'activating' entry owned by the re-attaching host, so a crashed
  host cleans up after itself as soon as it replugs its PBDs;
- resetvdis.reset_vdi now removes the entry when --force is given or
  when the recorded owner is no longer part of the pool, making
  'resetvdis single <uuid>' a supported remedy for stale locks left
  by hosts that never come back.

Legacy bare 'True' values written by older code cannot be attributed
to an owner and are deliberately only cleared by --force.
@olivierlambert
olivierlambert force-pushed the fix-stale-activating-lock branch from 2865ff0 to a70508a Compare July 14, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Host crash mid-activation permanently leaks the 'activating' VDI sm_config lock, blocking activations pool-wide (MAP_DUPLICATE_KEY)

1 participant