fix(attest): increase attestation heap from 512K to 2M#877
Merged
Conversation
The C verifier (servtd_attest) allocates from a private dlmalloc/sbrk
arena whose high-water mark never shrinks. Two effects compound:
1. dlmalloc's auto-trim is gated on the top free chunk exceeding
DEFAULT_TRIM_THRESHOLD (2 MiB). With a sub-2 MiB heap that gate is
never satisfied, so sys_trim never calls sbrk(-n) and sbrk's
high-water mark is monotonically non-decreasing — even after every
chunk is freed. HAVE_MMAP=0, so there is no second release path.
free() does coalesce, so freed space is reusable inside the arena,
but it is never returned to sbrk.
2. OpenSSL, sgxssl, and libstdc++ inside the verifier lazily allocate
per-process state on first use (error strings, OBJ_NAME tables, EC
precompute, C++ exception storage, locale facets, ...). Those
chunks stay live for the lifetime of the MigTD process and the set
grows as new quote / cert / policy code paths are exercised.
With LOCAL_TCB_INFO caching removed in 673fe2c, each migration now runs
verify_quote_integrity_ex() up to 4 times in the same MigTD process
(loopback). With the old 512 KiB arena, the cumulative live set plus
the per-call peak working set exhausted the budget by call ~3: dlmalloc
returned NULL, an uncaught std::bad_alloc reached std::terminate ->
abort(), and servtd_attest's abort() is literally `__asm__("ud2")`, so
the failure surfaced as a #UD with no error code propagated to the
Rust caller.
Bump ATTEST_HEAP_SIZE from 0x80000 to 0x200000. This raises the ceiling
so the cumulative footprint fits with comfortable headroom; it does not
enable trimming (that would require lowering trim_threshold via
mallopt() or providing a heap-reset entry point — neither is wired up).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Haitao Huang <haitaohuang@microsoft.com>
haitaohuang
force-pushed
the
upstream/pr2a-heap-increase
branch
from
813482a to
75bf5a8
Compare
Contributor
|
Changes look good |
sgrams
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The C verifier (servtd_attest) allocates from a private dlmalloc/sbrk arena whose high-water mark never shrinks. Two effects compound:
dlmalloc's auto-trim is gated on the top free chunk exceeding DEFAULT_TRIM_THRESHOLD (2 MiB). With a sub-2 MiB heap that gate is never satisfied, so sys_trim never calls sbrk(-n) and sbrk's high-water mark is monotonically non-decreasing — even after every chunk is freed. HAVE_MMAP=0, so there is no second release path. free() does coalesce, so freed space is reusable inside the arena, but it is never returned to sbrk.
OpenSSL, sgxssl, and libstdc++ inside the verifier lazily allocate per-process state on first use (error strings, OBJ_NAME tables, EC precompute, C++ exception storage, locale facets, ...). Those chunks stay live for the lifetime of the MigTD process and the set grows as new quote / cert / policy code paths are exercised.
With LOCAL_TCB_INFO caching removed in 673fe2c, each migration now runs verify_quote_integrity_ex() up to 4 times in the same MigTD process (loopback). With the old 512 KiB arena, the cumulative live set plus the per-call peak working set exhausted the budget by call ~3: dlmalloc returned NULL, an uncaught std::bad_alloc reached std::terminate -> abort(), and servtd_attest's abort() is literally
__asm__("ud2"), so the failure surfaced as a #UD with no error code propagated to the Rust caller.Bump ATTEST_HEAP_SIZE from 0x80000 to 0x200000. This raises the ceiling so the cumulative footprint fits with comfortable headroom; it does not enable trimming (that would require lowering trim_threshold via mallopt() or providing a heap-reset entry point — neither is wired up).