interpret: properly check for inhabitedness of nested references by RalfJung · Pull Request #156977 · rust-lang/rust

RalfJung · 2026-05-26T15:41:06Z

This implements the opsem from the ongoing FCP in rust-lang/unsafe-code-guidelines#413. The bit we were previously missing is that transmuting a &&! into existence was not caught as being immediate UB -- only the &! case behaved as expected.

I did not adjust the layout computation because when we compute the layout of &T, we cannot know the layout of T (as that might be recursive).

r? @oli-obk

rustbot · 2026-05-26T15:41:10Z

The Miri subtree was changed

cc @rust-lang/miri

Some changes occurred to the CTFE / Miri interpreter

cc @rust-lang/miri

Some changes occurred to the CTFE machinery

cc @oli-obk, @lcnr

RalfJung · 2026-05-26T15:42:23Z

+        ty::Coroutine(..) => {
+            true // FIXME should these really be trivially inhabited?
+        }
+        ty::CoroutineClosure(..) => {
+            true // FIXME should these really be trivially inhabited?
+        }


I wasn't sure how to recurse into these. Are coroutines always inhabited via a trivial start state, or can they be uninhabited due to capturing ! as an "upvar"? How do coroutine closures work?

View changes since the review

Looks like yes they can be uninhabited

Coroutine(DefId(0:13 ~ diverges[9ce3]::async_let::{closure#0}), [(), std::future::ResumeTy, (), !, (Void,)]) is ABI-uninhabited but not opsem-uninhabited?

I now made them all check the upvar_tys, but I am not entirely sure if that is enough.

For coroutines, checking upvars is enough. The coroutine state is pretty much struct { upvars, enum { Unresumed, Returned, Panicked, Suspend0 { .. }, Suspent1 { .. }, .. } }

Note that #135527 proposes to change this to enum { Unresumed { upvars }, Returned, Panicked, Suspend0 { .. }, Suspent1 { .. }, .. }. Starting from that PR, the enum state will be inhabited in all cases.

Coroutine closures work exactly like closures.

Thanks for confirming!

RalfJung · 2026-05-26T15:43:37Z

+            len.try_to_target_usize(tcx).unwrap() == 0
+                || is_opsem_inhabited_recursor(elem, tcx, root, adt_handler)
+        }
+        ty::Pat(inner, _pat) => is_opsem_inhabited_recursor(inner, tcx, root, adt_handler),


I guess in theory the pattern could make a type uninhabited... so technically if we ever want to use that for the opsem we have to add it has a check here before pattern types get stabilized.

View changes since the review

none of the current patterns can, but yes, that may be possible in the future

RalfJung · 2026-05-26T15:44:57Z

+///
+/// When we git an ADT, we call `adt_handler`, giving it as its last argument a closure that it
+/// can invoke to continue the recursion.
+fn is_opsem_inhabited_recursor<'tcx>(


Do we need to do something to bound this recursion? We already stop when encountering the same ADT again, so recursion is bounded by the depth of ADT field types until it comes back to the original type.

Do we need to do some stack growing magic?

View changes since the review

it's possible this can't be hit due to other stack growth protections, but theoretically you can create a very nested tuple, or just &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&T with a lot of repetitions of the reference. I'd honestly just wait for an issue

oli-obk · 2026-05-27T10:30:57Z

+///
+/// When we git an ADT, we call `adt_handler`, giving it as its last argument a closure that it
+/// can invoke to continue the recursion.
+fn is_opsem_inhabited_recursor<'tcx>(


it's possible this can't be hit due to other stack growth protections, but theoretically you can create a very nested tuple, or just &&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&T with a lot of repetitions of the reference. I'd honestly just wait for an issue

oli-obk · 2026-05-27T10:37:38Z

+            len.try_to_target_usize(tcx).unwrap() == 0
+                || is_opsem_inhabited_recursor(elem, tcx, root, adt_handler)
+        }
+        ty::Pat(inner, _pat) => is_opsem_inhabited_recursor(inner, tcx, root, adt_handler),


none of the current patterns can, but yes, that may be possible in the future

RalfJung · 2026-05-27T14:30:49Z

+            // Bailing out here is unfortuante as it means that the recursion limit affects the
+            // operational semantics... but what else could we do?
+            return true;
+        }


I pushed a proper recursion check including a recursion limit check.

But... I am not sure it's sound. This means that depending on the recursion limit, different crates might consider the same type inhabited or not.

OTOH if we just hard-code e.g. 256 here, then with a sufficiently high query recursion limit one can write a 257-level-nested type (Wrap<Wrap<...<Wrap<!>>...>>) that layout will still consider uninhabited but this check will give up.

View changes since the review

Why exactly is this check soundness-critical? I don't quite follow...

As far as I can tell, it seems like this is just a courtesy best-effort check for UB in consteval, and a sanity check after computing layout?

This is also used by Miri so it is supposed to define what is and isn't UB.

Also our query system relies on cross-crate-consistent results, doesn't it?

@camsteffen proposed to do something more like inhabited_predicate_adt. I must admit that I do not understand how that query works -- I can see that it produces a predicate and ships it to the trait solver, but how is that helpful for recursive types?

Sorry, that was kind of a red herring.

More to the point is check_representability. That query uses query cycle detection to detect an infinite sized type and abort compilation.

So I think we could have a check_opsem_inhabited query which also has query cycle detection, but instead of aborting, return OpsemInhabited::False or something like that for the query result.

check_representability uses params_in_repr, so check_opsem_inhabited could use something similar like params_in_repr_or_ref to include references.

Another idea, that lcnr proposed, is that we can error on overflow. I.e. when computing inhabitedness of a type, if we reach the recursion limit, emit a hard error.

Since an error prevents the code from being compiled, there is no problem with linking two different crates with different recursion limits -- they either both get the same result, or one of them doesn't compile.

We can still support (assume inhabited) types which refer to themselves directly like

struct A(&'static A);

And error only if the same type is mentioned with different generic parameters and overflows:

struct B<T: 'static>(&'static B<(T, T)>);

This is technically a breaking change, but it seems fine not to support types which infinitely grow like this (or am I missing something?).

But this seems fine, considering how much of an edge case this is?

Given,

struct Wrap<T>(T);

I believe that the algorithm you proposed would say that &Wrap<Wrap<!>> is inhabited, which doesn't seem like a particularly strange edge case. @WaffleLapkin

This is technically a breaking change, but it seems fine not to support types which infinitely grow like this (or am I missing something?).

Given that #138599 fixed multiple issues, it seems that there exist some use cases where people want such strange recursive types. We can't know how much that is before we run crater though.

I believe that the algorithm you proposed would say that &Wrap<Wrap<!>> is inhabited, which doesn't seem like a particularly strange edge case. @WaffleLapkin

I personally think that it's "fine", especially given that this improves over the status quo. But I suppose we could also allow recursing through the same definition a set number of times (i.e. setting a recursion limit higher than 1)...

@WaffleLapkin I like that first version, and I think I can combine it with an idea I had to avoid the problem @theemathas mentioned. This avoids any dependency on the recursion limit while also still satisfying the desired implication "layout inhabited => opsem inhabited". I pushed that now.

craterbot · 2026-06-03T17:36:06Z

👌 Experiment pr-156977 created and queued.
🤖 Automatically detected try build 05c0158
⚠️ Try build based on commit 5f63b50, but latest commit is d1a8249. Did you forget to make a new try build?
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

RalfJung · 2026-06-03T17:36:12Z

@rustbot ready

theemathas · 2026-06-08T03:49:41Z

+        | ty::Param(..)
+        | ty::Alias(..)
+        | ty::CoroutineWitness(..) => {
+            bug!("non-normalized type in `is_opsem_uninhabited_raw::rec`: `{ty}`")


Suggested change

bug!("non-normalized type in `is_opsem_uninhabited_raw::rec`: `{ty}`")

bug!("non-normalized type in `is_opsem_uninhabited_recursor`: `{ty}`")

View changes since the review

theemathas · 2026-06-08T10:08:26Z

+            // ADTs need a special handler to avoid infinite recursion. That handler is meant to
+            // call back into the recursor. Ideally it'd just call `is_opsem_inhabited_recursor` but
+            // then it would have to pass itself as the adt_handler argument which is not possible
+            // in Rust... so we provide the handler with a callback that it can use to continue the
+            // recurison with the same `adt_handler`.


Maybe it would be cleaner to have a struct that manually implements a trait? I kinda hate that less than having this confusing knot of callbacks.

View changes since the review

Alternatively, we could have a function that takes some arguments and returns the needed adt_handler. Each time we need an adt_handler, we call this function instead of trying (and failing) to use the closure itself.

Returning a closure is awkward and requires Box. And personally I prefer the current approach over a new trait. But I can add a trait if that becomes a blocking concern.

theemathas · 2026-06-12T16:29:43Z

@craterbot cancel

See #157814

craterbot · 2026-06-12T16:30:00Z

🗑️ Experiment pr-156977 deleted!

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

theemathas · 2026-06-16T09:22:56Z

@craterbot check p=1 crates=https://crater-reports.s3.amazonaws.com/pr-157814-crater-rollup/retry-regressed-list.txt

craterbot · 2026-06-16T09:23:01Z

👌 Experiment pr-156977 created and queued.
🤖 Automatically detected try build 05c0158
⚠️ Try build based on commit 5f63b50, but latest commit is 09db86e. Did you forget to make a new try build?
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-06-19T02:44:08Z

🚧 Experiment pr-156977 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

craterbot · 2026-06-19T05:24:40Z

🎉 Experiment pr-156977 is completed!
📊 0 regressed and 0 fixed (14695 total)
📊 610 spurious results on the retry-regressed-list.txt, consider a retry¹ if this is a significant amount.
📰 Open the summary report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

re-run the experiment with crates=https://crater-reports.s3.amazonaws.com/pr-156977/retry-regressed-list.txt ↩

RalfJung · 2026-06-20T00:56:28Z

Crater looks good, so @WaffleLapkin this is ready for review. :)

WaffleLapkin · 2026-06-22T11:09:48Z

+        ty::Pat(inner, _pat) => {
+            is_opsem_inhabited_recursor(inner, tcx, seen, stop_at_ref, adt_handler)
+        }


When pattern types start supporting enums, we'll need to decide if type X here is inhabited or not:

#![feature(pattern_types)] #![feature(pattern_type_macro)] #![feature(never_type)] enum E { A, B(!), } type X = pattern_type!(E is E::B(_));

View changes since the review

WaffleLapkin · 2026-06-22T11:19:56Z

+    }
+}
+
+fn is_opsem_inhabited_raw<'tcx>(


I feel like the code is fairly confusing with the closures & mutual recursion.

This query is only called for ADTs. I think if you change it to be is_adt_opsem_inhabited(tcx, adt_def, adt_args, seen) the code will become a lot clearer.

Then you can have

is_opsem_inhabited calls is_opsem_inhabited_recursor

is_opsem_inhabited_recursor calls is_adt_opsem_inhabited for adts

is_adt_opsem_inhabited calls is_opsem_inhabited_recursor directly

I think this should work & make the code a lot easier to follow.

View changes since the review

WaffleLapkin · 2026-06-22T11:30:15Z

+            // If we have seen this ADT before, stop at the next reference to avoid infinite
+            // recursion. We can't stop here since we have to ensure that "layout inhabited"
+            // implies "opsem inhabited".
+            let stop_at_ref = !new_adt;


Q: wasn't the thing that we want "layout uninhabited" => "opsem uninhabited"?

View changes since the review

WaffleLapkin · 2026-06-22T11:32:40Z

+
+/// Recurse over a type to determine whether it is inhabited on the opsem level.
+/// Key constraints are:
+/// - if a type's validity invariant is satisfiable, it must be opsem-inhabited.
+/// - if a type's layout is marked uninhabited, it must be opsem-uninhabited.
+///
+/// Beyond that, the value returned by this function is not a stable guarantee.


I feel like this should be documented on the public function.

View changes since the review

WaffleLapkin · 2026-06-22T11:34:19Z

@@ -3,7 +3,7 @@ use std::mem::{forget, transmute};

 fn main() {
    unsafe {
-        let x: Box<!> = transmute(&mut 42); //~ERROR: encountered a box pointing to uninhabited type !
+        let x: Box<!> = transmute(&mut 42); //~ERROR: encountered a box pointing to uninhabited type `!`


Q: Do we need to special case Box in is_opsem_inhabited?

View changes since the review

WaffleLapkin · 2026-06-22T11:34:50Z

@@ -4,6 +4,6 @@ enum Void {}

 fn main() {
    unsafe {
-        let _x: &(i32, Void) = transmute(&42); //~ERROR: encountered a reference pointing to uninhabited type (i32, Void)
+        let _x: &&(i32, Void) = transmute(&&42); //~ERROR: encountered a reference pointing to uninhabited type `&(i32, Void)`


Q: why change this?

View changes since the review

WaffleLapkin · 2026-06-22T11:38:14Z

+// If we just unfold this type going down the first variant of every enum, we'll never stop; we'll
+// never even encounter the same type a second time.
+struct S<T: 'static>(&'static S<(T, T)>, PhantomData<T>);
+const C: &Result<S<()>, ()> = &Err(());


"every enum"?

View changes since the review

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 26, 2026

rustbot assigned oli-obk May 26, 2026

RalfJung commented May 26, 2026

View reviewed changes

Comment thread compiler/rustc_middle/src/ty/inhabitedness/mod.rs

RalfJung commented May 26, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from 4e90bd5 to e6d1440 Compare May 26, 2026 15:51

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from e6d1440 to 33dc892 Compare May 26, 2026 17:12

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from 33dc892 to 846fa4f Compare May 26, 2026 18:07

RalfJung mentioned this pull request May 26, 2026

Can references to uninhabited types ever be valid? rust-lang/unsafe-code-guidelines#413

Open

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from 846fa4f to 8d272fd Compare May 26, 2026 20:45

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from 8d272fd to 06e6db9 Compare May 26, 2026 21:42

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from 06e6db9 to f328265 Compare May 27, 2026 06:27

oli-obk requested changes May 27, 2026

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 27, 2026

theemathas reviewed May 27, 2026

View reviewed changes

Comment thread compiler/rustc_middle/src/ty/inhabitedness/mod.rs Outdated

RalfJung force-pushed the interpret-opsem-inhabited branch from f328265 to bffa3cd Compare May 27, 2026 14:28

RalfJung commented May 27, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

RalfJung force-pushed the interpret-opsem-inhabited branch from bffa3cd to 2552ff6 Compare May 27, 2026 14:56

This comment has been minimized.

Sign in to view

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 3, 2026

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 3, 2026

theemathas reviewed Jun 8, 2026

View reviewed changes

RalfJung force-pushed the interpret-opsem-inhabited branch 2 times, most recently from 84e3817 to 19c1da7 Compare June 8, 2026 10:04

theemathas reviewed Jun 8, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

interpret: properly check for inhabitedness of nested references

09db86e

RalfJung force-pushed the interpret-opsem-inhabited branch from 19c1da7 to 09db86e Compare June 8, 2026 10:22

WaffleLapkin self-assigned this Jun 9, 2026

oli-obk removed their assignment Jun 9, 2026

theemathas mentioned this pull request Jun 12, 2026

Crater rollup #157814

Closed

craterbot removed the S-waiting-on-crater Status: Waiting on a crater run to be completed. label Jun 12, 2026

craterbot added S-waiting-on-crater Status: Waiting on a crater run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 16, 2026

craterbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-crater Status: Waiting on a crater run to be completed. labels Jun 19, 2026

WaffleLapkin requested changes Jun 22, 2026

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 22, 2026

	bug!("non-normalized type in `is_opsem_uninhabited_raw::rec`: `{ty}`")
	bug!("non-normalized type in `is_opsem_uninhabited_recursor`: `{ty}`")

Uh oh!

Conversation

RalfJung commented May 26, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented May 26, 2026

Uh oh!

RalfJung May 26, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RalfJung May 26, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung May 26, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RalfJung May 27, 2026 • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theemathas May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RalfJung May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theemathas Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theemathas Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

craterbot commented Jun 3, 2026

RalfJung commented May 26, 2026 •

edited by rustbot

Loading

RalfJung May 26, 2026 •

edited by rustbot

Loading

RalfJung May 26, 2026 •

edited by rustbot

Loading

RalfJung May 26, 2026 •

edited by rustbot

Loading

RalfJung May 27, 2026 •

edited by rustbot

Loading

theemathas May 27, 2026 •

edited

Loading

RalfJung May 27, 2026 •

edited

Loading

theemathas Jun 1, 2026 •

edited

Loading

theemathas Jun 1, 2026 •

edited

Loading

theemathas Jun 8, 2026 •

edited by rustbot

Loading

theemathas Jun 8, 2026 •

edited by rustbot

Loading

WaffleLapkin Jun 22, 2026 •

edited by rustbot

Loading

WaffleLapkin Jun 22, 2026 •

edited by rustbot

Loading

WaffleLapkin Jun 22, 2026 •

edited by rustbot

Loading

WaffleLapkin Jun 22, 2026 •

edited by rustbot

Loading

WaffleLapkin Jun 22, 2026 •

edited by rustbot

Loading