Scaling / IMC documentation

[ansien]

Hi,

I'm looking for any available documentation on scaling / IMC.

I know BitCraft uses IMC itself, but I can't find any documentation on it. Looking at the public code only gives a limited view on how it works. E.g. how does the relay work between different databases?

My goal: Assigning "room" tables to different databases (or regions) and having communication between them.

Thanks!

[bfops]

Hey @ansien, we are working on adding inter-database communication into SpacetimeDB as a proper feature, but it's still in the design phase.

In the meantime, the basic pattern of BitCraft's inter-database communication is:

• every database has an inter_database_message table which acts as an outbox
• reducers insert typed messages into that table
• an external relay service subscribes to that table on all databases
• when the relay sees a row, it calls process_inter_database_message(sender, message) on the destination database
• the destination reducer applies the message
• the relay observes the reducer result and calls on_inter_database_message_processed(id, error) back on the sender
• the sender deletes the outbox row once it has been acknowledged

The relay is intentionally simple: it maintains connections to the global database and all region database, subscribes to their outbox tables, sorts newly inserted messages by id, and forwards them by reducer call. If a destination
is disconnected, messages stay queued. If the relay reconnects and sends a message twice, the destination database ignores already-processed messages using a per-sender last_processed_message_id table.

This gives at-least-once delivery with application-level idempotency, not exactly-once delivery. Cross-database operations are not one atomic transaction across databases; each destination reducer is atomic only within its own
database.

[ansien]

@bfops Thank you, that clears up a lot of it! Would be great if this became a "native" functionality in Spacetime itself.

[pinkertond]

I would also really love to see the IMC feature natively built in.

[bfops]

We are working on it! There will absolutely be a big announcement when we release it!