Add NodeJS bindings to `synapse_auto_compressor` by jaywink · Pull Request #167 · matrix-org/rust-synapse-compress-state

jaywink · 2026-04-20T08:29:53Z

Similarish to the Python bindings, we'll expose Node bindings to run synapse_auto_compressor tool. The function naming and parameters hav been kept the same as the Python equivelent. Currently the Node bindings don't set up logging. Also due to the Python bindings sitting at the synapse_auto_compressor root, the node bindings are in a submodule node.

Also adds some structured result information for the compress_chunks_of_database, which is useful for more programmatic analysis of what was done. This has also been reflected in the Python bindings.

TODO: needs NPM publish steps

Similarish to the Python bindings, we'll expose Node bindings to run `synapse_auto_compressor` tool. The function naming and parameters hav been kept the same as the Python equivelent. Currently the Node bindings don't set up logging. Also due to the Python bindings sitting at the `synapse_auto_compressor` root, the node bindings are in a submodule `node`. Also adds some structured result information for the `compress_chunks_of_database`, which is useful for more programmatic analysis of what was done. This has also been reflected in the Python bindings. TODO: needs NPM publish steps

jaywink · 2026-04-21T10:59:24Z

clippy seems happy locally:

$ cargo clippy -- -D warnings
    Checking futures-util v0.3.31
    Checking openssl v0.10.72
    Checking tokio-postgres v0.7.13
    Checking tokio-openssl v0.6.5
    Checking postgres-openssl v0.5.1
    Checking postgres v0.19.10
    Checking synapse_compress_state v0.1.4 (/home/jaywink/workspace/rust-synapse-compress-state)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2.18s

reivilibre

Generally happy with this, what do you think?

reivilibre · 2026-04-21T11:42:36Z

+
+`$ cd synapse_auto_compressor`   
+
+2. Build and install the library  


Any objection to using pnpm instead?
On the team, we are more familiar with that, so if possible it's nice to keep our tools in alignment.

Have converted 👍

Hmm, trying to use the node bindings appears to be causing this now after converting to pnpm:

Error initializing: Error: Cannot find native binding. npm has a bug related to optional dependencies (npm/cli#4828). Please try npm i again after removing both package-lock.json and node_modules directory.

Unsure if this is due to the way I'm bundling the build locally to our nodejs app, but I suspect pnpm does something differently to the napi suggested yarn 🤔 Any ideas?

Looks like the yarn built node native binding is approx 3.3MB, while the one built by pnpm is ~700KB (+ a node_modules directory, which the yarn version doesn't generate).

Actually no, reverting to yarn doesn't seem to fix things, so broke something with the feature refactoring probably.

Right, after some amount of hair tearing, it seems I had dropped the node module declaration 🤦

Thanks and sorry for the trouble!

reivilibre · 2026-04-21T11:43:44Z

+napi = "3.0.0"
+napi-derive = "3.0.0"


would it be reasonable to make these optional dependencies (see pyo3 for instance)?

This way if someone only wants the Python bindings they don't have to bring along the Node ones.

I think it would be a matter of adding a feature flag for this, making these optional and 3 or 4 #[cfg] / #[cfg_attr] conditional compilation directives

Have done so, hopefully the code makes sense, had some issues getting things to compile in a way that the features can also return the result into Python and Node, basically gave each their own struct and copied things over. I've not done a lot of rust so happy to receive feedback if something doesn't make sense. Will do some testing to ensure things actually work.

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Also fix the Python bindings for the new result return.

jaywink · 2026-04-23T09:09:46Z

Right I think my local clippy is now updated to a version that fails it seems:

$ cargo clippy -- -D warnings
    Checking synapse_compress_state v0.1.4 (/home/jaywink/workspace/rust-synapse-compress-state)
error: iterating on a map's values
   --> src/database.rs:236:39
    |
236 |           let mut missing_sgs: Vec<_> = state_group_map
    |  _______________________________________^
237 | |             .iter()
238 | |             .filter_map(|(_sg, entry)| {
239 | |                 entry
240 | |                     .prev_state_group
241 | |                     .filter(|&prev_sg| !state_group_map.contains_key(&prev_sg))
242 | |             })
    | |______________^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/rust-1.95.0/index.html#iter_kv_map
    = note: `-D clippy::iter-kv-map` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::iter_kv_map)]`
help: try
    |
236 ~         let mut missing_sgs: Vec<_> = state_group_map.values().filter_map(|entry| {
237 +                 entry
238 +                     .prev_state_group
239 +                     .filter(|&prev_sg| !state_group_map.contains_key(&prev_sg))
240 +             })
    |

error: could not compile `synapse_compress_state` (lib) due to 1 previous error

Probably shouldn't be touching that in this pr tho 🤔

jaywink · 2026-04-23T11:15:04Z

Probably shouldn't be touching that in this pr tho 🤔

Have opened #169

reivilibre · 2026-04-24T16:09:47Z

+
+`$ cd synapse_auto_compressor`   
+
+2. Build and install the library  


Thanks and sorry for the trouble!

reivilibre · 2026-04-24T16:11:59Z

 }

+#[allow(dead_code)]
+pub struct CompressedChunkResult {


I'm not sure which one would be generally preferable (I'm not against duplicating this struct for each binding), but if interested you could

#[cfg_attr(feature = "pyo3", pyclass)] #[cfg_attr(feature = "node", napi(constructor))]

etc

Well, yes, that is much simpler, thanks for the suggestion!

reivilibre · 2026-04-24T16:19:48Z

+
+## Publishing to NPM
+
+TODO TBD


Is this something you intend to PR after?

I was thinking in a follow-up pr, yes, if that is ok.

jaywink added 6 commits April 20, 2026 11:27

Fix reporting original_num_rows from compress_chunks_of_database

83e853b

Make run_compression async

7c7186d

Merge remote-tracking branch 'upstream/main' into jaywink/node-bindings

bbba5e6

Regenerate Cargo.lock

a117017

Run rustfmt

e55b0c5

jaywink marked this pull request as ready for review April 21, 2026 11:16

jaywink requested a review from a team as a code owner April 21, 2026 11:16

reivilibre requested changes Apr 21, 2026

View reviewed changes

jaywink and others added 5 commits April 21, 2026 18:33

Ok(results?) -> results

b98c741

Co-authored-by: Olivier 'reivilibre' <olivier@librepush.net>

Sneak .idea into .gitignore

9757a61

yarn -> pnpm

7d47f3d

Put the Node bindings behind the node feature

25597cf

Also fix the Python bindings for the new result return.

Rstfmt

8ee0c73

jaywink requested a review from reivilibre April 22, 2026 14:53

jaywink added 3 commits April 23, 2026 11:59

Re-add missing module declaration

5ac82e3

Install latest @napi-rs/cli

982bb4b

Fix docs/node.md

3604832

reivilibre approved these changes Apr 24, 2026

View reviewed changes

Don't duplicate CompressedChunkResult

e4af53f

jaywink mentioned this pull request Apr 28, 2026

Return result information from synapse_auto_compressor.compress_chunks_of_database #166

Closed


		`$ cd synapse_auto_compressor`

		2. Build and install the library

Conversation

jaywink commented Apr 20, 2026

Uh oh!

jaywink commented Apr 21, 2026

Uh oh!

reivilibre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaywink commented Apr 23, 2026

Uh oh!

jaywink commented Apr 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants