Draft: Support deterministic body IDs when restoring state

[JonECG]

As a continuation from #1930
I ended up disappearing for GDC and never updated. But following that discussion I have been successful synchronizing ids and context for shape/material out of band, while sending a full staterecorder over the wire only once at the beginning of each client connection. In addition, storing one for each tick within history for reprediction. I quickly noticed that id generation was not deterministic after restoring, which inspired these changes:

I added a EStateRecorderState::IdSequences to save/restore body id sequence numbers and the free list order, and then an option to PhysicsSystem::RestoreState to destroy bodies not present in the state.

Together these make body ID allocation after a restore identical to the original run, which I needed for rollback/reprediction. Also adds some state recorder tests to verify.

Currently a draft because I'm also looking for a bit guidance on expected guarantees for serialization format of the state recorder. I put all of this in the new IdSequences so theoretically it could be backwards compatible if folks were saving previous recordings to disk. Also, maybe it makes sense to not care about deterministic body id on replay. It's not as important if you're just using it for like visual debugging or forking. So I wanted to keep it as a separate option. You can also tell I hesitated to add it to All and that it awkwardly tags onto the body logic. Happy to clean this up or drop it altogether if you prefer another approach. I'm sure it also needs extra warning/handling for filters or other partial state saves. Or the free list restore not working if you don't have the entity destruction enabled.

I've been using these changes successful for a while in my prototype for the past few weeks, but it's not incredibly complex at the moment.

Ultimately I needed 2 things:

• When rewinding the simulation (for network reprediction) body id generation is replayed deterministically for all observers once all inputs converge
• An easy way to remove bodies that were not present in a state recording, ideally without rewriting or rereading the snapshot

[CLAassistant]

All committers have signed the CLA.

[jrouwe]

Hello,

I've looked at the changes and I appreciate the work you've put in this, but to be honest, I don't want to add this kind of complexity to the core Jolt library.

As far as I can see the change does:

• Synchronization of the ID freelist: I think the BodyIDs should be managed externally for your scenario. This is why e.g. the CreateBodyWithID function exist.
• Deletes bodies that are not in the snapshot: Again, I think this should be managed externally. A previous state could also have a body that doesn't exist in the current state. Since we don't store the BodyCreationSettings, this is not solvable by the RestoreState function and you'll be missing a body.
• Updates IDs of bodies that have changed sequence number: Sequence numbers only change if you destroy and recreate a body with the same ID. I'd say that when this happens, you're dealing with a new body with new settings, so it feels like you're going to restore state on the wrong body.

I think you can implement the same behavior in your own application:

• Manage your own BodyIDs and exclusively use BodyInterface::CreateBodyWithID to assign your BodyIDs, bypassing the freelist completely.
• Unless you're working with stale BodyIDs (e.g. when storing a BodyID from a collision query and use it after the body may have been deleted and a new body with the same BodyID created), there's no reason to have sequence numbers. Just use 0 for all your BodyIDs.
• You can use PhysicsSystem::GetBodies to get all the BodyIDs that are currently in use and implement the logic you have created in this PR. A cheaper solution would be to remember the BodyIDs + BodyCreationSettings that were added/removed for every frame together with the snapshot. Store the BodyCreationSettings that was used to create a body and use Body::GetBodyCreationSettings just before removing a body so you can recreate it when rewinding. Note that this is assuming that whatever you're doing to the bodies from your application (e.g. apply forces) is replayed through other means.

[JonECG]

Appreciate your time to take a look and the comments.

Funnily enough, the start of the experiment was doing just that, having a monotonically increasing body id that was synchronized/reset, but with sequence id of 0, meaning it wouldn't reuse a slot. I was initially worried (and maybe a bit lazy 😉 ) about performance penalty for how sparse that would ultimately become. But I can probably move some version of the freelist into my game layer as you mentioned. Or at the very least lease slots past destruction so I know they can't be used until they're VERY stale and out of history before being reused.

Again, Jorrit, thanks so much for everything you do.

[jrouwe]

Having a monitonically increasing body ID is probably not a good idea if it means there will be lots of sparsity. If you create a body with a custom ID then it currently does an O(N) loop through the freelist to remove that ID from the freelist. So it's best to keep the freelist small. Ideally you track the body IDs you free and reuse the last freed body ID first.