feat: Invite over internet

[RangerMauve]

Closes #1207

[awana-lockfile-bot]

package-lock.json changes

Summary

Status	Count
	30
	8
	47

Click to toggle table visibility

Name	Status	Previous	Current
@bufbuild/buf-darwin-arm64		1.26.1	-
@bufbuild/buf-darwin-x64		1.26.1	-
@bufbuild/buf-linux-aarch64		1.26.1	-
@bufbuild/buf-win32-arm64		1.26.1	-
@bufbuild/buf-win32-x64		1.26.1	-
@emnapi/core		1.3.1	-
@emnapi/runtime		1.3.1	-
@emnapi/wasi-threads		1.0.1	-
@esbuild/aix-ppc64		0.25.10	-
@esbuild/android-arm		0.25.10	-
@esbuild/android-arm64		0.25.10	-
@esbuild/android-x64		0.25.10	-
@esbuild/darwin-arm64		0.25.10	-
@esbuild/darwin-x64		0.25.10	-
@esbuild/freebsd-arm64		0.25.10	-
@esbuild/freebsd-x64		0.25.10	-
@esbuild/linux-arm		0.25.10	-
@esbuild/linux-arm64		0.25.10	-
@esbuild/linux-ia32		0.25.10	-
@esbuild/linux-loong64		0.25.10	-
@esbuild/linux-mips64el		0.25.10	-
@esbuild/linux-ppc64		0.25.10	-
@esbuild/linux-riscv64		0.25.10	-
@esbuild/linux-s390x		0.25.10	-
@esbuild/netbsd-arm64		0.25.10	-
@esbuild/netbsd-x64		0.25.10	-
@esbuild/openbsd-arm64		0.25.10	-
@esbuild/openbsd-x64		0.25.10	-
@esbuild/openharmony-arm64		0.25.10	-
@esbuild/sunos-x64		0.25.10	-
@esbuild/win32-arm64		0.25.10	-
@esbuild/win32-ia32		0.25.10	-
@esbuild/win32-x64		0.25.10	-
@mapeo/crypto		1.0.0-alpha.10	1.1.0
@napi-rs/wasm-runtime		0.2.6	-
@node-rs/crc32-android-arm-eabi		1.10.6	-
@node-rs/crc32-android-arm64		1.10.6	-
@node-rs/crc32-darwin-arm64		1.10.6	-
@node-rs/crc32-darwin-x64		1.10.6	-
@node-rs/crc32-freebsd-x64		1.10.6	-
@node-rs/crc32-linux-arm-gnueabihf		1.10.6	-
@node-rs/crc32-linux-arm64-gnu		1.10.6	-
@node-rs/crc32-linux-arm64-musl		1.10.6	-
@node-rs/crc32-wasm32-wasi		1.10.6	-
@node-rs/crc32-win32-arm64-msvc		1.10.6	-
@node-rs/crc32-win32-ia32-msvc		1.10.6	-
@node-rs/crc32-win32-x64-msvc		1.10.6	-
@tybys/wasm-util		0.9.0	-
adaptive-timeout		-	1.0.1
bare-addon-resolve		-	1.10.0
bare-ansi-escapes		-	2.2.3
bare-assert		-	1.2.0
bare-events		2.4.2	2.8.2
bare-inspect		-	3.1.4
bare-module-resolve		-	1.12.1
bare-semver		-	1.0.2
bare-stream		2.1.3	2.10.0
bare-type		-	1.1.0
bits-to-bytes		-	1.3.0
blind-relay		-	1.4.0
bogon		1.1.0	1.2.0
compact-encoding-bitfield		-	1.0.0
compact-encoding		2.15.0	2.19.2
crockford-base32		-	2.1.0
devlop		-	1.1.0
dht-rpc		-	6.26.3
events-universal		-	1.0.1
hypercore-id-encoding		-	1.3.0
hyperdht		-	6.29.4
hyperswarm		-	4.17.0
kademlia-routing-table		-	1.0.6
nat-sampler		-	1.0.1
record-cache		-	1.2.0
require-addon		-	1.2.0
shiki		-	1.17.7
shuffled-priority-queue		-	2.1.0
signal-promise		-	1.0.3
streamx		2.19.0	2.25.0
teex		-	1.0.1
time-ordered-set		-	2.0.1
uc.micro		-	2.1.0
udx-native		-	1.19.2
unordered-set		-	2.0.1
xache		1.1.0	1.2.1
z32		1.0.1	1.1.0

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​mapeo/​crypto@​1.1.0
	bogon@​1.1.0 ⏵ 1.2.0	+1		+1	+3
	crockford-base32@​2.1.0
	hyperswarm@​4.17.0
	streamx@​2.19.0 ⏵ 2.25.0	+1			-2
	compact-encoding@​2.15.0 ⏵ 2.19.2	+1

View full report

[gmaclennan]

I left some comments on the approach. I'm not sure the division of responsiblities between the member-api and the remote-discovery class is quite right, but it depends on how you resolve the validation / authentication of incoming connections, which is the most important challenge to address.

[RangerMauve]

I've got the connection and invite flow going. We want to make it harder to track specific peers via the DHT, therefore we want to use a fresh DHT keypair each time. The difficult part is that the keypair is used for the deviceId during the RPC API and that's tied with the member record. We'd need to either tie all members with an additional keypair, or somehow override the deviceID at the RPC layer, or something along those lines.

Gregor proposed we have something like this to send the real identity as the first packet down the noise stream after handshaking.

// Stable identity — persisted across sessions
const stableKeyPair = DHT.keyPair(someSeed)

// Ephemeral DHT presence — rotated each session
const swarm = new Hyperswarm({ keyPair: DHT.keyPair() })

swarm.join(topic, { server: true, client: true })

swarm.on('connection', async (conn, peerInfo) => {
  try {
    const remote = await handshakeIdentity(conn, stableKeyPair)
    console.log('Authenticated:', b4a.toString(remote.publicKey, 'hex'))
  } catch (err) {
    conn.destroy()
  }
})

function handshakeIdentity (conn, keyPair) {
  return new Promise((resolve, reject) => {
    // Sign the Noise handshake hash with our stable key
    const sig = b4a.allocUnsafe(64)
    sodium.crypto_sign_detached(sig, conn.handshakeHash, keyPair.secretKey)

    // Send stable public key + proof in a single message
    conn.write(encodeHandshake({
      publicKey: keyPair.publicKey,
      signature: sig
    }))

    conn.once('data', (data) => {
      const msg = decodeHandshake(data)

      const valid = sodium.crypto_sign_verify_detached(
        msg.signature,
        conn.handshakeHash,  // same hash on both sides
        msg.publicKey
      )

      if (!valid) return reject(new Error('Invalid identity proof'))
      resolve({ publicKey: msg.publicKey })
    })

    setTimeout(() => reject(new Error('Auth timeout')), 10000)
  })
}

This could leave the option to join by remote public key instead of a fresh topic.

We will also need some sort of access control step to have the invite ID before we expose the protomux channel for RPC.

[RangerMauve]

Been wrestling with some sort of race condition for several hours now. I think I traced it down to the protomux channel recieving data before it's fully opened.

[gmaclennan]

I still think we need an authentication step before connecting to the RPC channel:

Validating the inviteID and allowing the user (invitor) to confirm before connecting, which means queueing the connection somewhere until the user confirms

[RangerMauve]

Been blocked getting the tests to work with the handshake logic. Seems the connection is breaking when we try to write the public key down the wire and isn't draining.

[RangerMauve]

Might do a flag in local-peers to ignore incoming RPC messages until we get authenticated? I think this extra handshake step is making everything more fragile.

[RangerMauve]

Been trying a bunch of ways to rework this and I'm getting a bit stuck. Gonna write out some thoughts on approaches here.

Initial approach:

• Invitor creates invite id
• Invitor starts swarm to listen on connections with same identity keypair (with deviceId ad publicKey) as local connections
• Invitor sends URL with own deviceId and the inviteId to invitee out of band
• Invitee takes deviceID and inviteId from URL
• Invitee starts swarm with own identity keypair
• Invitee calls "joinPeer" with invitor deviceId and waits for a connection to be established to them
• Invitee calls RPC API to send inviteID to the invitor deviceID to redeem the invite
• Meanwhile, the invitor was replicating all incoming swarm connections to localPeers and was waiting for an invite redeem call
• Invitor runs regular invite flow to the deviceID if the inviteID is valid
• Invitee auto-accepts invite if it is from the invitor deviceID
• Invitee becomes part of the project

This worked fine

Ephemeral keys approach

• Invitor initializes a fresh ephemeral keypair for the swarm
• Invitor creates invite id
• Invitor starts swarm to listen on connections with swarm keypair
• Invitor sends URL with swarm public key and the inviteId to invitee out of band
• Invitee takes deviceID and inviteId from URL
• Invitee starts swarm with own ephemeral swarm keypair
• Invitee calls "joinPeer" with invitor swarm id and waits for a connection to be established to the ephemeral keypair
• On both sides we wait for the first "network packet" and fetch a proof for them owning a deviceId keypair (this is their identity key). This Id is used to identify them in local-peers instead of the remotePublic key of their ephemeral swarm keypair in the connection
• While waiting for this proof, we generate our own proof and send it down the wire
• Invitee calls RPC API to send inviteID to the invitor deviceID to redeem the invite
• Invitor runs regular invite flow
• Invitee auto-accepts invite if it is from the invitor deviceID
• Invitee becomes part of the project

This didn't work because the connection was "breaking" after sending the first packet. I think it's got something to do with how we're reading a packet of the stream before sending it off elsewhere, but it's been really difficult to debug.

Edit: I got this working! I needed to pause the stream after getting the first packet, else we'd drop an event.

Queue up connections before RPC

In order to guard the RPC methods from spam, we should prompt the user to verify them before replicating anything. If we queue up the connections (at the manager level) we need a new place to track them that isn't local peers and would need to somehow tie them to the project they're trying to join via the member API. This extra book keeping and connection management is IMO not ideal. We'd likely need to add the invite ID into the handshake after establishing hyperswarm, which would also mean getting the remote discovery module to ask each project's member-api if this is a valid invite or not. Kinda leaky IMO

Alternate: Disable RPC until connection is verified

Instead we could mark peers inside localPeers as being untrusted which limits which types of RPC events they are allowed to send. We could keep the invite redeem event open and give member-api a new trustConnection(peerId) method which would unlock the RPC channels. On the invitee side we can mark the connection as trusted right off the bat since we validate the remote ID. I think I'll go with this approach first since it's got the least amount of code changes and back-and-forths.

[RangerMauve]

TODO: Test all the edge cases

[RangerMauve]

TODO: Convert all new Error to proper Error classes inside errors.js

[gmaclennan]

I see there are some checks that are failing that need to pass before we can merge. Also, some API feedback and some comments that were not resolved from the last review pass, and some new comments.

[gmaclennan]

still unresolved I think?

[gmaclennan]

should we test this maxPeers? What happens if it's exceeded? Would be rare, e.g. workshop-based setup where invites are sent out to multiple invitees at the same time. I think it's fine to restrict this, just want to understand what the consequence is if it is exceeded, particularly for the "spamming" case (someone on the swarm, attempting multiple connections opportunistically - we block them doing anything, but they would be connected)

[RangerMauve]

Actually, it seems to limit outgoing connections, not incoming.

https://github.com/holepunchto/hyperswarm/blob/48d4241f69f848bb4c9dccb65b6fce5a9f40009d/index.js#L300

[RangerMauve]

what I mean is that if we get spammed with connections, once we reach 16 we won't be able to make new outgoing connections to join a link. Pruning the untrusted connections (and maybe banning them?) should reduce the risks further

[gmaclennan]

We should use our new type RemoteAuthedNoiseStream - right now this is fragile for future maintenance - isTrusted defaults to true, so there is implied knowledge that you must only call #replicate with a trusted stream.

Suggestion: define the shared type in the noise strem helpers:

/** @typedef {OpenedNoiseStream & { authenticatedPublicKey: Buffer, isTrusted: boolean }} AuthedNoiseStream */

Then in LocalDiscovery, set socket.authenticatedPublicKey = socket.remotePublicKey, and isTrusted=true.

then you can simplify the checks in local-peers.js:1248-1257, mapeo-manager.js:408 and remote-discover.js:140-144. It avoids a future maintainer accidentially passing a stream through connect() and defaulting to trusted.

[gmaclennan]

A few edge-cases and remaining security issues to clean up

[gmaclennan]

I don't think you attach an error listener to the remoteDiscovery instance, so this will throw an unhandled 'error' and crash node.

[gmaclennan]

Have you maybe not pushed this change? The code I see is still unbounded length.

[gmaclennan]

Sorry to come late with some additional architecture comments, but I've been thinking through more details about privacy and security related to connections over the internet, and how we manage future internet sync. I think it merits some changes in this PR before we roll it out because it reduces changes that are needed in the future.

Block project replication based on trust
Right now, when an invitee connects, it shares the discovery keys of all the projects it is a member of with the invitor, and vice-versa. This unnecessarily leaks identifiable information - it leaks to anyone who connects, before even validating their inviteId (the project creator core replicates into the stream on peer-add).

Fix: localPeers.on('peer-add', onPeerAdd) shouldn't replicate to non-trusted (e.g. non-local) peers, and the discovery-key event should be ignored. An invitee does need, however, to replicate the project they are invited to.

Test: during invite, with either/both invitee and invitor with multiple projects, no hypercore/alpha channel should open before invite accept, and after accept it should only be the joined project.

New protomux auth channel
Currently the auth is a hand-rolled readHandshakeBuffer message that runs before Protomux attaches, with no version negotiation. For future internet sync, and more private local sync, we need a channel to negotiate membership of every project to avoid revealing stable identifiers (e.g. project discovery key) to anyone who connects. If we move the identify proofs from this branch into a new protomux channel, e.g. named comapeo/auth, then we can roll out future changes without breaking changes to this invite-over-the-internet protocol, or without maintaining two separate protocols.

Fix: Create a protomux in remote-discovery.js and store in noiseStream.userData like we already do for local, and open comapeo/auth. The first message should be a hello with { protocolVersion, features } - protocolVersion should never change, but we should treat anything other than 1 as "disconnect - you need to update to connect to this peer". features should list the supported message types. An IdentityProof { publicKey, signature } message can be identical to the current waterfall message that we have. We can remove the manual implementation for reading length-prefixed messages since protomux covers that for us.

Defer invitor's IndentityProof until acceptance
The invitor eagerly sends its identity proof on connection, without user intervention. This means that anyone that obtains a leaked invite URL can establish the identity of the invitor. It's a small risk, but significant in some scenarios.

Fix: only send the IdentityProof after the invitor calls acceptInviteLinkRequest. This means from the invitee side the invitor peer has no identity until after accept. I don't think this matters, but we should check the code.