fix(daemon): mitigate onion message flooding DoS attack

[m0wer]

Problem

An ongoing DoS attack targets JoinMarket makers by connecting directly to makers' onion services and flooding !orderbook requests.
Each request can trigger expensive orderbook response handling and failed reply routing, causing:

• disk growth from repeated logs
• high CPU/memory usage
• maker instability / OOM kills

What this patch does

This is a small stopgap mitigation in src/jmdaemon/onionmc.py:

1. Per-connection inbound rate limit
   • Drop a connection if it sends more than 45 messages in 15 seconds.
2. Handshake gate for JM messages
   • For non-directory peers, ignore JM messages unless the peer has completed handshake.
   • Prevents unauthenticated inbound peers from triggering expensive JM message handling.
3. Reduce noisy warning
   • Change "Failed to send privmsg because no directory peer is connected." from warning to debug.

Directory classification / spoofing notes

• A peer is treated as a directory based on local configured directory nodes, not by arbitrary handshake claims.
• Inbound peers are non-directory by default.
• dn-handshake is only accepted from peers already marked as directory.
• Sending {"directory": true} in a handshake does not grant directory privileges.

Limitations

• Attackers can reconnect after disconnect and continue probing.
• Attackers can still flood !orderbook through trusted directory relay paths; this bypasses per-connection direct-connection limiting.
• Fully addressing that requires additional controls (not in this emergency patch), especially per-nick / higher-layer rate limiting. And ideally Tor PoW defense. Implemented in https://github.com/joinmarket-ng/joinmarket-ng/
• This is an urgent mitigation, not the final comprehensive defense.

Testing

• Added test/jmdaemon/test_onionmc_dos.py with 11 tests:
  • per-connection rate limiting behavior
  • handshake-gating behavior for JM messages
• New tests pass in local daemon-focused run.

Applying patch manually

curl -sL https://github.com/m0wer/joinmarket-clientserver/commit/b391a29e5f3c28e93fc8e80bb261830adbb7ed86.patch -o /tmp/onionmc.patch && FILE=$(find / -type f -name onionmc.py 2>/dev/null | head -n 1) && DIR=$(dirname $(dirname $(dirname "$FILE"))) && echo "Patching in: $DIR" && cd "$DIR" && patch -p1 -i /tmp/onionmc.patch && echo "Success"

Don't trust, verify!

[m0wer]

@AdamISZ @kristapsk @PulpCattel

This problem is bringing down most makers at the moment.

[AdamISZ]

Interesting about the orderbook part. Note that a very small feature was added early on by @chris-belcher to limit : not allow more that one !orderbook command in a single request. It's still in the messagechannel code in the old repo.

General rate limiting certainly makes a ton of sense, I vaguely remember thinking about doing that.

I'm curious about the 'don't treat a peer as a directory unless.. x'; is it the case that directory peers have privileges in terms of what they can send (I must admit I've forgotten how the protocol works; is it because the directories are sending us peer lists or something? Isn't orderbook a command sent from the taker peer to the directory node?)

[m0wer]

Interesting about the orderbook part. Note that a very small feature was added early on by @chris-belcher to limit : not allow more that one !orderbook command in a single request. It's still in the messagechannel code in the old repo.

General rate limiting certainly makes a ton of sense, I vaguely remember thinking about doing that.

I'm curious about the 'don't treat a peer as a directory unless.. x'; is it the case that directory peers have privileges in terms of what they can send (I must admit I've forgotten how the protocol works; is it because the directories are sending us peer lists or something? Isn't orderbook a command sent from the taker peer to the directory node?)

Right, Chris's check:

joinmarket-clientserver/src/jmdaemon/message_channel.py

Lines 887 to 889 in 5252017

	#DOS vector: repeated !orderbook requests, see #298.
	if commands.count('orderbook') > 1:
	return

(which came from JoinMarket-Org/joinmarket#298) blocks multiple !orderbook commands packed into a single message. But the attack we observed was sending many separate messages, each with a single !orderbook. That check doesn't help there, since it's one !orderbook per message, one message per line, many lines per second.

In the base code

joinmarket-clientserver/src/jmdaemon/onionmc.py

Lines 172 to 180 in 5252017

	def lineReceived(self, line: bytes) -> None:
	try:
	msg = OnionCustomMessage.from_string_decode(line)
	except OnionCustomMessageDecodingError:
	log.debug("Received invalid message: {}, "
	"dropping connection.".format(line))
	self.transport.loseConnection()
	return
	self.factory.receive_message(msg, self)

passes everything straight through with no rate limit. And

joinmarket-clientserver/src/jmdaemon/onionmc.py

Lines 982 to 989 in 5252017

	# real JM message; should be: from_nick, to_nick, cmd, message
	try:
	nicks_msgs = msgval.split(COMMAND_PREFIX)
	from_nick, to_nick = nicks_msgs[:2]
	msg = COMMAND_PREFIX + COMMAND_PREFIX.join(nicks_msgs[2:])
	if to_nick == "PUBLIC":
	self.on_pubmsg(from_nick, msg)

parses the from_nick directly from the message payload without validating it against the peer's handshake nick, so a single connection could even vary the from_nick across requests (though for !orderbook specifically that doesn't change the cost much, since the maker generates the same response regardless). And that's why it's important to ban per connection instead of per nick!

Each !orderbook triggers the full on_orderbook_requested path at

joinmarket-clientserver/src/jmdaemon/daemon_protocol.py

Lines 759 to 772 in 5252017

	@maker_only
	def on_orderbook_requested(self, nick, mc=None):
	"""Dealt with by daemon, assuming offerlist is up to date
	"""
	if self.use_fidelity_bond:
	taker_nick = nick
	maker_nick = self.mcc.nick
	d = self.callRemote(JMFidelityBondProofRequest,
	takernick=taker_nick,
	makernick=maker_nick)
	self.defaultCallbacks(d)
	else:
	self.mcc.announce_orders(self.offerlist, nick, fidelity_bond_proof_msg=None,
	new_mc=mc)

which builds and sends the complete offerlist (including fidelity bond proof if enabled). And probably is the cause behind the memory "leak" that leads to OOM crashes.

---

About directory peer privileges, you're right that it relates to message relay. The directory is not sending its own !orderbook, it's forwarding taker requests to makers. So it's not as trivial to limit. The flow is:

1. Taker calls _pubmsg which sends !orderbook to all directory nodes
   

   joinmarket-clientserver/src/jmdaemon/onionmc.py

   Lines 771 to 786 in 5252017

   	def _pubmsg(self, msg:str) -> None:
   	""" Best effort broadcast of message `msg`:
   	send the message to every known directory node,
   	with the PUBLIC message type and nick.
   	"""
   	dps = self.get_directory_peers()
   	msg = OnionCustomMessage(self.get_pubmsg(msg),
   	JM_MESSAGE_TYPES["pubmsg"])
   	for dp in dps:
   	# currently a directory node can send its own
   	# pubmsgs (act as maker or taker); this will
   	# probably be removed but is useful in testing:
   	if dp == self.self_as_peer:
   	self.receive_msg(msg, "00")
   	else:
   	self._send(dp, msg)

2. Directory receives it in receive_msg, processes it, and calls forward_pubmsg_to_peers to broadcast to all connected non-directory peers
   

   joinmarket-clientserver/src/jmdaemon/onionmc.py

   Lines 988 to 991 in 5252017

   	if to_nick == "PUBLIC":
   	self.on_pubmsg(from_nick, msg)
   	if self.self_as_peer.directory:
   	self.forward_pubmsg_to_peers(msg, from_nick)

3. That forwarding function iterates over all connected makers and sends them each a copy with the original taker's nick preserved
   

   joinmarket-clientserver/src/jmdaemon/onionmc.py

   Lines 1028 to 1050 in 5252017

   	def forward_pubmsg_to_peers(self, msg: str, from_nick: str) -> None:
   	""" Used by directory nodes currently. Takes a received
   	message that was PUBLIC and broadcasts it to the non-directory
   	peers.
   	"""
   	assert self.self_as_peer.directory
   	pubmsg = self.get_pubmsg(msg, source_nick=from_nick)
   	msgtype = JM_MESSAGE_TYPES["pubmsg"]
   	# NOTE!: Specifically with forwarding/broadcasting,
   	# we introduce the danger of infinite re-broadcast,
   	# if there is more than one party forwarding.
   	# For now we are having directory nodes not talk to
   	# each other (i.e. they are *required* to only configure
   	# themselves, not other dns). But this could happen by
   	# accident.
   	encoded_msg = OnionCustomMessage(pubmsg, msgtype)
   	for peer in self.get_connected_nondirectory_peers():
   	# don't loop back to the sender:
   	if peer.nick == from_nick:
   	continue
   	log.debug("Sending {}:{} to nondir peer {}".format(
   	msgtype, pubmsg, peer.peer_location()))
   	self._send(peer, encoded_msg)

So from a maker's perspective, the directory connection legitimately carries !orderbook messages from many different takers. That's why the handshake gate this PR adds exempts directory peers. Applying a per-connection rate limit or handshake requirement to the directory connection would break normal operation during busy periods. Rate limiting directory-forwarded messages would need to be per-nick rather than per-connection, which is a bigger change (and out of scope for this stopgap). JM-NG implements this at the directory level and at the directory clients as well, to not trust directories more than necessary.

The distinction in this PR is: for direct (non-directory) inbound connections, there's no reason to accept JM messages before the handshake completes, and there's no reason a single direct peer should send 45+ messages in 15 seconds. Those are the two checks this patch adds.

[AdamISZ]

Yes. I believe I understand all the logic, and it makes sense.

The historical progression is something like: per-message rate limits were enforced by the IRC servers so we were only really worried (if at all) about intra-message edge cases like the !orderbook > 1 mentioned above (and only then because of its magnifying effect; individual line lengths were also of course limited in the IRC version). Then when the onion message channel stuff was added by me, my attitude was 'let's see if this even works' at first; it did for a while, then got hammered by the non-JM specific onion service DOS of the 2021-2023 period (I don't remember the exact time frame) and onionmc wasn't working too well. I should have put some rate limiting in right at the start; the system is clearly much too weak without it.

And then we get to your further developed point: intrinsically directory nodes are different and can't be subject to general/normal rate limiting rules (should they not still have some rate limiting rules, though, even if different?). Which means we don't just trust anyone to be a d-node; this does leave open the question of (this was part of the original high level thinking): can directory nodes gossip other directory nodes? Think of the analogy with seeding peers in bitcoin IBD bootstrap. (edit: for anyone reading who doesn't see why that was part of the original thinking, consider the censorship resistance angle; one thing we're really trying hard to avoid here is giving directory nodes the power to censor makers). It's fine if not, just a question.

[m0wer]

I think we talked about the censorship resistance/gossiping somewhere but I don't remember. Open to ideas!

Short version of how joinmarket-ng does it:

• Directory discovery/gossip: none today.
• Directory addresses are static config/defaults, not learned from peers. 1
• Directory servers also refuse directory-as-client handshakes and do not include directories in peerlists. 3
• So right now, directory nodes cannot gossip other directory nodes.

Rate limiting in jm-ng is layered, with different scopes:

• Layer 1 (directory server ingress, per connection): default 500 msg/s sustained, 1000 burst, disconnect threshold 0 (so normally drop/slow instead of disconnect). 5
• Layer 2 (maker generic, per nick): default 10 msg/s, 100 burst. 7
• Layer 3 (maker !orderbook, per nick): default 1 request per 10s; escalation is 10 violations -> 60s interval, 50 -> 300s, 100 -> 1h ban. 7
• Layer 4 (direct maker onion connections, per connection): stricter defaults: 5 msg/s, 20 burst, plus !orderbook 1 per 30s, ban after 10 violations (1h ban duration). 10

Conceptually:

• Connection-keyed limits protect against nick spoofing/rotation at transport ingress.
• Nick-keyed + command-specific limits protect expensive maker paths (!orderbook) behind directories.

And of course all of this needs something to make new connections expensive, otherwise rate limiting doesn't really work! And that is Tor's PoW defense: joinmarket-ng/joinmarket-ng#108 Which is very cool because it only kicks in when needed, and is transparent to the user and jm-ng itself.

[AdamISZ]

Right. Indeed the Tor pow is a big part.

It's just that, somewhere in the stack, the user has to feel at least somewhat comfortable that a random maker out there can communicate their offer to them. The more limited/controlled the message layer is, the less they can be sure of that.

[m0wer]

On that topic, I would like to explore what can be done with Nostr relays. Since there are there anyway and there's lots of them.

[takinbrrrr]

Thought I should add my 2 sats worth on this attack. Rate limiting is a part of the solution but certain behaviours suggest non-standard usage. In my patch, my approach is simple: disconnect any non-directory peers that send a pubmsg. Normal taker behaviour does not involve sending a pubmsg directly to makers. Pubmsgs are meant to be public and should only be sent in the public IRC channel or through a directory. As @AdamISZ mentioned, the IRC servers already rate-limit messages from clients so implementing the same on directories would be effective.

Following this logic, accepting pubmsgs only from directories mitigates this attack, since it prevents attackers from sending these messages directly to peers; privmsgs sent directly are still allowed. I've observed this attack uses either different takers or the same taker with multiple connections, making it easy to bypass per-connection rate limits.

On the privilege concern: directories being the only peers allowed to forward pubmsgs is acceptable because makers connect to directories, not vice versa. This means attackers cannot exploit this channel, as it would require makers to connect to them. There is a potential risk of directory censorship, but makers mitigate this by connecting to multiple directories.

[m0wer]

Thought I should add my 2 sats worth on this attack. Rate limiting is a part of the solution but certain behaviours suggest non-standard usage. In my patch, my approach is simple: disconnect any non-directory peers that send a pubmsg. Normal taker behaviour does not involve sending a pubmsg directly to makers. Pubmsgs are meant to be public and should only be sent in the public IRC channel or through a directory. As @AdamISZ mentioned, the IRC servers already rate-limit messages from clients so implementing the same on directories would be effective.

Following this logic, accepting pubmsgs only from directories mitigates this attack, since it prevents attackers from sending these messages directly to peers; privmsgs sent directly are still allowed. I've observed this attack uses either different takers or the same taker with multiple connections, making it easy to bypass per-connection rate limits.

On the privilege concern: directories being the only peers allowed to forward pubmsgs is acceptable because makers connect to directories, not vice versa. This means attackers cannot exploit this channel, as it would require makers to connect to them. There is a potential risk of directory censorship, but makers mitigate this by connecting to multiple directories.

Sooner or later, makers will be able to receive a command directly. If not !orderbook then !fill, which is unavoidable. Or do the attack through a directory.

Opening Tor connections takes a bit of time and resources, and that seems enough to slow down the current attack to manageable for most hardware.

Would be nice to agree on the protocol specification about direct pubmsg though. For compatibility.