Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@
*.old
**.sql
*.csv
**.log
**.log
.idea/
140 changes: 140 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 33 additions & 0 deletions docs/node.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Running the compressor tools from nodejs

The automatic compressor tool is available through Node bindings.

Currently, the Node bindings wont produce logging. To see output,
look at the response array for chunk (room) specific results.

## Requirements

NodeJS 16+

## Setting things up

1. Navigate to the correct location

`$ cd synapse_auto_compressor`

2. Build and install the library
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any objection to using pnpm instead?
On the team, we are more familiar with that, so if possible it's nice to keep our tools in alignment.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have converted 👍

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, trying to use the node bindings appears to be causing this now after converting to pnpm:

Error initializing: Error: Cannot find native binding. npm has a bug related to optional dependencies (npm/cli#4828). Please try npm i again after removing both package-lock.json and node_modules directory.

Unsure if this is due to the way I'm bundling the build locally to our nodejs app, but I suspect pnpm does something differently to the napi suggested yarn 🤔 Any ideas?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the yarn built node native binding is approx 3.3MB, while the one built by pnpm is ~700KB (+ a node_modules directory, which the yarn version doesn't generate).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually no, reverting to yarn doesn't seem to fix things, so broke something with the feature refactoring probably.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, after some amount of hair tearing, it seems I had dropped the node module declaration 🤦

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks and sorry for the trouble!

`$ pnpm i && pnpm build`

## Automatic tool example

```node
const synapseAutoCompressor = require('synapse_auto_compressor');

synapseAutoCompressor.runCompression(
"postgresql://localhost/synapse",
500,
100,
).then(results => {
console.log(JSON.stringify(results));
});
```
3 changes: 2 additions & 1 deletion docs/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ Both the automatic and manual tools use PyO3 to allow the compressor
to be run from Python.

To see any output from the tools, logging must be setup in Python before
the compressor is run.
the compressor is run. You can also inspect the returned list of chunks (rooms)
for information taken on them.

## Setting things up

Expand Down
6 changes: 6 additions & 0 deletions synapse_auto_compressor/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.pnp.*
.yarn/
synapse_auto_compressor.*
index.d.ts
index.js
node_modules/
6 changes: 6 additions & 0 deletions synapse_auto_compressor/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ clap = { workspace = true, optional = true }
env_logger.workspace = true
log-panics.workspace = true
log.workspace = true
napi = { version = "3.0.0", optional = true }
napi-derive = { version = "3.0.0", optional = true }
openssl.workspace = true
postgres-openssl.workspace = true
postgres.workspace = true
Expand All @@ -28,11 +30,15 @@ path = "../"
default-features = false
features = ["no-progress-bars"]

[build-dependencies]
napi-build = "2"

# Needed for pyo3 support
[lib]
crate-type = ["cdylib", "rlib"]

[features]
default = ["clap", "jemalloc"]
jemalloc = ["tikv-jemallocator", "synapse_compress_state/jemalloc"]
node = ["dep:napi", "dep:napi-derive"]
pyo3 = ["dep:pyo3", "dep:pyo3-log", "synapse_compress_state/pyo3"]
4 changes: 4 additions & 0 deletions synapse_auto_compressor/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,7 @@ Bump the version number and run from the root directory of the repo:
```
docker run -it --rm -v $(pwd):/io -e OPENSSL_STATIC=1 konstin2/maturin publish -m synapse_auto_compressor/Cargo.toml --cargo-extra-args "\--features='openssl/vendored'"
```

## Publishing to NPM

TODO TBD
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something you intend to PR after?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking in a follow-up pr, yes, if that is ok.

4 changes: 4 additions & 0 deletions synapse_auto_compressor/build.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
// Build NodeJS bindings
fn main() {
napi_build::setup();
}
33 changes: 33 additions & 0 deletions synapse_auto_compressor/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"name": "synapse_auto_compressor",
"version": "0.1.4",
"description": "A tool to compress some state in a Synapse instance's database",
"main": "index.js",
"repository": {
"type": "git",
"url": "git+ssh://git@github.com/matrix-org/rust-synapse-compress-state.git"
},
"license": "Apache-2.0",
"files": [
"index.d.ts",
"index.js"
],
"napi": {
"binaryName": "synapse_auto_compressor",
"targets": [
"x86_64-unknown-linux-gnu"
]
},
"engines": {
"node": ">= 12.22.0 < 13 || >= 14.17.0 < 15 || >= 15.12.0 < 16 || >= 16.0.0",
"pnpm": ">=8"
},
"scripts": {
"build": "napi build --features node --platform --release",
"build:debug": "napi build --features node --platform"
},
"devDependencies": {
"@napi-rs/cli": "^3.6.2",
"typescript": "^6.0.0"
}
}
Loading
Loading