Discussion: Multithreading in ECS

[tillpp]

Let us discuss how we want to handle multithreading in the entityComponentSystem
My idea is to Let each Component and the EntityManager have Its own mutex, that has to be locked when interacting with it.

The question is, should component Pointer be given out?

Idea 1: The simple approach

Component pointers are given out only if the Mutex is locked. ( get(id) asserts mutex for entityManager is locked )
benefits

• doesn't confuse your IDE

downsides

• someone might accidently use entityPointers after they have unlocked the entityManager (not fool proof)

example:

healthComponent.mutex.lock();
const component:*HealthComponent = healthComponent.get(id); //assert lock
component.health = 15;
healthComponent.mutex.unlock();
//Illegal / not fool proof when:
component.health=17;

idea 2: The more complex approach

have get/set functions

component.set(id,.attribute,newvalue),
component.get(id,.attribute),

example:

healthComponent.mutex.lock();
healthComponent.set(id,.health,15); //assert lock
healthComponent.mutex.unlock();
healthComponent.set(id,.health,17); //assert lock, fails successfully! more fool proof.

which internally asserts the locks for the component, and set/gets an attribute for the component
At no point is an component pointer even given out.

benefit

• more fool proof. (100% fool proof is not reasonably possible, like for attributes of components that are pointers)
• We could integrate it with the network in the future: when component.set(..) then component gets retransmitted to all users automatically, without having to explicity write it every time.

downside:

• confuses Intellisense

Idea 3:

insert your idea here

[Argmaster]

I think ECS libs tend to treat components as values, thus require explicit get and set ops (like idea 2)
However they also differentiate between flag components and those which actually store data, for performance reasons.
They also do caching of those edit operations and they flush them only now and then to avoid moving entities between archetypes too often.
If you write the ops into the cache you can do this without mutex, just with atomic stack and lock for longer only on flush.
Look at flecks

[Argmaster]

After reviewing this part of documentation of flecs https://www.flecs.dev/flecs/md_docs_2EntitiesComponents.html
I found that it technically also allows for obtaining mutable references, further research is needed, since I clearly have no idea what I am talking about 😅

[IntegratedQuantum]

My goal is to have a lock-free system, see #2420

For entities this would mean: All expensive (O(n²)) entity updates (except from component addition/removal) are done in parallel based on their position, everything else (cheap O(n) stuff, like checking if your health is negative, ticking down effect timers) and death/component removal is done sequentially on the server thread.

To make this safe, you can use sync.threadContext.assertCorrectContext (which is not fully fledged out yet).

With this we also won't even run into the problem you described above, since the thread will be in the correct context for as long as the function runs.

[tillpp]

With this we also won't even run into the problem you described above,

Not really, problem 1 also includes:

healthComponent.mutex.lock();
const component:*HealthComponent = healthComponent.get(id); //assert lock
component.health = 15;
healthComponent.mutex.unlock();
//Illegal / not fool proof when:
somevariablesomewhereelse = component;
...
use in some other function of somevariablesomewhereelse

My goal is to have a lock-free system, see #2420

For entities this would mean: All expensive (O(n²)) entity updates (except from component addition/removal) are done in parallel based on their position, everything else (cheap O(n) stuff, like checking if your health is negative, ticking down effect timers) and death/component removal is done sequentially on the server thread.

Can you expand on that?

[IntegratedQuantum]

Can you expand on that?

Which part is unclear?

[Argmaster]

Oh yeah let's pick up that issue for no reason!

healthComponent.mutex.lock();
const component:*HealthComponent = healthComponent.get(id); //assert lock
component.health = 15;
healthComponent.mutex.unlock();

In general I don't think that is how API should work;
IMO only valid way to update a component would be to explicitly set it via set(newComponentValue).
I would assume that most of the components are small, ie. They fit in a single cache line. This means that cost of changing it is capped by memory speed. That would also quickly answer many questions of how many attributes should a component have - not many, four u64 to be exact. Enough for Vec3i, not enough for every weird attribute we might want to place inside.
Components shouldn't contain hash maps, nor lists. Most they can have is a pointer into data structure held by a System.
There is also a (arguably) good technical reason to do so; it allows ECS a fine grained control of what and when to cache and synchronize.
Most likely changes to the entity state should be cached and applied as a batch, eg. After system has done full iteration over all entities. This would guarantee that order of the iteration doesn't change the outcome.
I think it would reduce the risk of client desync and possibly number of packets send by networking layer, even though it has its own buffers (?).
That result can be achieved via left-right data structure, where we get two exact copies of the same thing, but we read left copy and write to right copy. Then we sync (copy contents of right to left) and swap. This means unobstructed reading and no cache invalidation (in ideal case).
In most cases systems will be checking flags or current values, not necessarily editing immediately. It should be much faster than sharing original data structure between threads.