Skip to content
This repository was archived by the owner on Sep 8, 2025. It is now read-only.

consolidate unsafe code in concurrent.rs and friends#159

Merged
dicej merged 8 commits into
mainfrom
dicej/async-consolidation
May 16, 2025
Merged

consolidate unsafe code in concurrent.rs and friends#159
dicej merged 8 commits into
mainfrom
dicej/async-consolidation

Conversation

@dicej
Copy link
Copy Markdown
Collaborator

@dicej dicej commented May 12, 2025

This adds a few abstractions to reduce the amount of unsafe code and make what remains easier to reason about locally (i.e. without relying as much on functions only being called in certain contexts, or on non-local safe code upholding invariants).

This introduces a StoreToken abstraction for witnessing the data type parameter T for a StoreContextMut<T> and later safely casting from a &mut dyn VMStore to a &mut StoreInner<T> within a closure by asserting that the StoreIds match.

This also eliminates various transmutes and other unsafe shenanigans to avoid a T: 'static bound since we've finally decided avoiding such a bound is impractical.

This adds a few abstractions to reduce the amount of unsafe code and make what
remains easier to reason about locally (i.e. without relying as much on
functions only being called in certain contexts, or on non-local safe code
upholding invariants).

This introduces a `StoreToken` abstraction for witnessing the data type
parameter `T` for a `StoreContextMut<T>` and later safely casting from a `&mut
dyn VMStore` to a `&mut StoreInner<T>` within a closure by asserting that the
`StoreId`s match.

This also eliminates various transmutes and other unsafe shenanigans to avoid a
`T: 'static` bound since we've finally decided avoiding such a bound is
impractical.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej marked this pull request as draft May 12, 2025 16:40
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 12, 2025

There are a bunch of outdated comments that need updating to match the new code; will work on that next.

dicej added 2 commits May 12, 2025 11:18
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej marked this pull request as ready for review May 12, 2025 20:10
@dicej dicej added this pull request to the merge queue May 12, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 12, 2025
1.85.0 adds async closure support and should be two versions older than the
current stable release once this is merged, per Wasmtime's current policy.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej enabled auto-merge May 12, 2025 20:47
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 12, 2025

Looks like it's going to take some work to convince CI to let me bump the MSRV to 1.85.0 given that it's newer than $CURRENT_STABLE minus 2; I'll probably just wait until 1.87.0 is released in a few days, which should satisfy it.

This consolidates some unsafe code to avoid the need for non-local reasoning
about invariants.  Specifically, we can now efficiently transfer ownership of
buffered items for host-based streams and futures without requiring unsafe
blocks.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@alexcrichton
Copy link
Copy Markdown
Member

If you want to test a bit sooner you can try decrementing this to 1 to increase the msrv to 1.85

@dicej dicej added this pull request to the merge queue May 15, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 15, 2025
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

@alexcrichton Does this failure mean anything to you? https://github.com/bytecodealliance/wasip3-prototyping/actions/runs/15052610459/job/42310963180

I can repro locally both on this branch and on main, so I believe it's unrelated to any of my changes, but I'm not sure why it's popping up now.

I thought maybe it could be related to me bumping the MSRV, but I can repro on Rust 1.84, too. EDIT: nevermind, I hit a different error on 1.84 and 1.86, which might be a qemu thing. So it does appear to be related to 1.87. Specifically, I get one test failure on 1.84 and 1.86, but two test failures on 1.87.

@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

I think I'll just ignore the test on riscv for now with a TODO comment.

@dicej dicej enabled auto-merge May 15, 2025 19:32
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

Oof, it's broken on s390x too, I guess.

@dicej dicej force-pushed the dicej/async-consolidation branch from a199faa to 53193a3 Compare May 15, 2025 19:47
@dicej dicej added this pull request to the merge queue May 15, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 15, 2025
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

CI and I are just not getting along today. Now the AddressSanitizer tests are failing with a bunch of leaks reported. Could be a real regression or could be another MSRV thing. I can't seem to use AddressSanitizer on my Linux/ARM64 machine, whether natively or via qemu, so I haven't been able to repro and bisect yet. Might need to borrow a Linux/x86_64 machine and try there.

dicej added a commit that referenced this pull request May 15, 2025
I'm splitting this out of #159 to see if the CI failures there are due to this
change or the other ones.

This started failing when I bumped the MSRV.  Presumably this will be addressed
upstream, but for now I'm disabling the test so it doesn't block other work.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej mentioned this pull request May 15, 2025
@alexcrichton
Copy link
Copy Markdown
Member

re riscv64/s390x, yeah that definitely looks like the msrv update, and is something we'll have to fix in upstream wasmtime

re asan, that may also be an msrv update with a newer llvm and a new ASAN build detecting something. Maybe threads related? Unsure, would need to try reproducing locally

(if possible I'd recommend decoupling the update to 1.87 and just update the msrv window to 1 to make the msrv here 1.85 while still using 1.86 as before today)

This will hopefully sidestep the various 1.87 upgrade headaches that will be
addressed upstream.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej force-pushed the dicej/async-consolidation branch from 53193a3 to dfcef3d Compare May 15, 2025 23:02
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

Okay; I just pushed an update to reduce msrv_range 🤞

@dicej dicej enabled auto-merge May 15, 2025 23:09
@dicej dicej added this pull request to the merge queue May 15, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 15, 2025
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 15, 2025

Now the MinGW build is broken, both here and in #166. I'm going to just step slowly away from the computer for now.

Some of the GitHub images have a broken MinGW, so we'll skip this for now.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej enabled auto-merge May 16, 2025 14:31
@dicej dicej added this pull request to the merge queue May 16, 2025
auto-merge was automatically disabled May 16, 2025 14:36

Pull Request is not mergeable

@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 16, 2025
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 16, 2025

Looks like the address sanitizer leak reports are a real regression and not related to the MSRV bump; they passed in #166 but failed here. I'll borrow a Linux/x86_64 device and see if I can repro and debug locally.

@dicej dicej enabled auto-merge May 16, 2025 19:12
@dicej dicej force-pushed the dicej/async-consolidation branch from 1ed8892 to 0f958c9 Compare May 16, 2025 19:21
@dicej dicej added this pull request to the merge queue May 16, 2025
@dicej
Copy link
Copy Markdown
Collaborator Author

dicej commented May 16, 2025

I've fixed some of the leaks flagged by AddressSanitizer; working on the others now.

@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 16, 2025
@dicej dicej force-pushed the dicej/async-consolidation branch from 0f958c9 to 115d0ab Compare May 16, 2025 21:35
In `ComponentInstance::from_vmctx` and
`StoreContextMut::with_detached_instance[_async]` we were leaking memory due to
having taken `InstanceData` out of the `Store` and making it unreachable except
via a raw pointer, meaning there was nothing responsible for dropping it on
panic.  This commit stashes the data elsewhere in the store to make sure it gets
dropped.

Signed-off-by: Joel Dice <joel.dice@fermyon.com>
@dicej dicej force-pushed the dicej/async-consolidation branch from 115d0ab to 49f24c1 Compare May 16, 2025 21:43
@dicej dicej enabled auto-merge May 16, 2025 21:44
@dicej dicej added this pull request to the merge queue May 16, 2025
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 16, 2025
@dicej dicej added this pull request to the merge queue May 16, 2025
Merged via the queue into main with commit a176e9e May 16, 2025
44 checks passed
@alexcrichton alexcrichton mentioned this pull request May 19, 2025
@dicej dicej deleted the dicej/async-consolidation branch June 10, 2025 13:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants