multiline: Specify behavior with respect to formatting.

[progval]

This is the only way to format the fallback for multiline-unaware clients the same way as for multiline-aware clients

[slingamn]

lgtm

[jwheare]

I'm not sure. This puts a significant burden on clients when interpreting user input and composing a multiline message.

[progval]

So currently, if someone sends this:

BATCH +123 draft/multiline #channel
@batch=123 PRIVMSG #channel :not-bold <0x02>bold
@batch=123 PRIVMSG #channel :unclear
BATCH -123

then irccloud and the current weechat master will display it like this, because they don't implement this PR:

not-bold bold
unclear

and assuming servers don't do any reformatting, then every client not supporting multiline will get this:

PRIVMSG #channel :not-bold <0x02>bold
PRIVMSG #channel :unclear

so they will display it like this:

not-bold bold
unclear

I understand this can be annoying to implement, but we need to either:

1. accept this solution
2. propose a different solution
3. decide it's not a problem

Could we resolve this before weechat ships version 4.0 with draft/multiline support?

[progval]

Update: weechat released v4.0 without changing its behavior with respect to formatting.

[progval]

@flashcode As implementer of the multiline spec, could you weigh in on this question?

[JustAnotherArchivist]

This has been stalled for some time.

I agree that this is the only sensible way of doing it. Any other approach has no sane fallback mechanism.

There is however one complication that hasn't been addressed: the current draft says says this about fallback clients:

When delivering multiline batches to clients that have not negotiated the multiline capability, [...]

Servers SHOULD maintain the line composition sent by the client instead of combining to a normalised form before re-splitting. This ensures that steps taken to split long lines appropriately are preserved.

If the servers don't maintain the lines as sent, this would again lead to different formatting between multiline and non-multiline clients. The only sensible approach to me here seems to be to turn that recommendation into a requirement, i.e. change the 'SHOULD' into a 'MUST'.

[JustAnotherArchivist]

Actually, the above would apply not only to fallback clients but also to multiline-capable ones; their representation would also change if the server didn't maintain what the author client sent.

[progval]

And that's fine. Servers have always been allowed to tamper with the text and color codes if they want to.

[JustAnotherArchivist]

I suppose so. Would a recommendation to insert formatting reset codes during such rearrangements (if needed) be reasonable?

[jwheare]

Have any client developers made an attempt at implementing this suggestion and have any thoughts?

[skizzerz]

Alternative server-side implementation (not fully fleshed-out wording wise, just an idea at the moment). I'm interested to hear what ircd devs think of this feasibility-wise, etc.

For each line in a multiline batch, keep track if there's an unterminated format code. When the server processes a new line in the batch, it will synthesize and inject a multline-concat line to clients supporting multiline which terminates those format codes or CTCP. Clients that do not support multiline do not receive any additional fallback behavior, as they are already processing each line independently. (side-note: instead of closing CTCP, it may be better to just reject unclosed CTCP entirely, I'm not sure client-side if the \x01 at the end will cause some clients to process it differently based on whether they support multiline and thus get the extra synthesized line or not).

Example:

Client to server:
BATCH +123 draft/multiline #channel
@batch=123 PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
@batch=123;draft/multiline-concat PRIVMSG #channel :<0x02>this is still bold,<SPACE>
@batch=123;draft/multiline-concat PRIVMSG #channel :but this is not
BATCH -123

Server to client with draft/multiline CAP enabled:
:client!user@host BATCH +123 draft/multiline #channel
@batch=123 :client!user@host PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x0F>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x02>this is still bold,<SPACE>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x0F>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :but this is not
BATCH -123

Server to client without draft/multiline CAP:
:client!user@host PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
:client!user@host PRIVMSG #channel :<0x02>this is still bold,<SPACE>
:client!user@host PRIVMSG #channel :but this is not

Repeat example but replace <0x02> with <0x01>ACTION and <0x0F> with <0x01> for CTCP. As a note, these injected lines could cause the batch being sent to clients to exceed the stated max-lines or max-bytes limits.

[progval]

What makes it easier for servers to insert <0x0F> than for receiving clients who implement draft/multiline?

[skizzerz]

What makes it easier for servers to insert <0x0F> than for receiving clients who implement draft/multiline?

imo it's not easier to do this server-side, doing it client-side would be easier because there'd be no concerns about impacting send queue and the like (plus a lot of extra overhead on the wire to add one character). However if this is stalled because clients don't want to go through the effort or we're afraid of client fragmentation with some following spec and others not doing the format reset, then there are a lot fewer servers in active development than clients, so having consistency would involve fewer people to do said implementation.

[progval]

However if this is stalled because clients don't want to go through the effort or we're afraid of client fragmentation with some following spec and others not doing the format reset

Inserting <0x0F> does not fix this issue. If IRCCloud and Weechat don't change their current behavior, then it's still broken, because when they send messages like this:

BATCH +123 draft/multiline #channel
@batch=123 PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
@batch=123;draft/multiline-concat PRIVMSG #channel :<0x02>this is not in bold,<SPACE>
@batch=123;draft/multiline-concat PRIVMSG #channel :and this also is not
BATCH -123

and the server inserts <0x0F> then it would make the message into this:

:client!user@host BATCH +123 draft/multiline #channel
@batch=123 :client!user@host PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x0F>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x02>this is not in bold,<SPACE>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :<0x0F>
@batch=123;draft/multiline-concat :client!user@host PRIVMSG #channel :and this also is not
BATCH -123

to clients who support batch/multiline, which is wrong because they would bold the second line

and this:

:client!user@host PRIVMSG #channel :this is not bold, <0x02>but this is,<SPACE>
:client!user@host PRIVMSG #channel :<0x02>this is still bold,<SPACE>
:client!user@host PRIVMSG #channel :but this is not

to clients who don't support batch/multiline, which is also wrong for the same reason.

[Steering7253]

There are literally and objectively only 3 options here: either the server adds formatting codes to subsequent lines for fallback clients, or the non-fallback clients treat formatting as ending at the end of each line, or the behavior is different for fallback and non-fallback clients.

If the desire is to have the server handle making sure the formatting is applied to subsequent lines for fallback clients - which I will note is not the one and only objection noted in this thread - then the server will have to wrap lines which become too long as a result of that formatting, which means multiline-concat should have just been clients opting into longer lines in the first place, and then servers could have wrapped lines being sent to fallback clients. (As has been pointed out before, of course, yet multiline has consensus...)

[Celelibi]

Here's a hot take: we need less server-side message tampering, not more.
It hasn't been mentioned yet in this PR, but any server-side tampering wouldn't play well with E2EE in the future if we want such feature. This formatting issue can easily be dealt with in the client. The client can also, at the same time, make sure the message still meets the max-bytes and max-lines limits after adding the additional formatting codes. Doing it on the server would mean rejecting multiline messages that the sender would see as perfectly valid just because the added formatting bytes makes it go over the limit.

[andymandias]

Implementing these suggestions in Halloy brings up a complication. For spoiler text (and potentially monospaced text, depending on how the client handles it) two spans directly next to each other are not equivalent to one span covering the same text. The former will reveal the spoiler for each span individually (on user interaction), while the latter will reveal the entire text at once. This means we can't just shove a reset between each received message if we want to preserve message intent, instead it's necessary to check the formatting at the end of the previous message and strip that from the beginning of the current message (if present).

[slingamn]

As a server vendor, I'm very skeptical of any proposal that requires server modification of messages here: avoiding server modification was one of the core design goals of multiline in the first place.

[progval]

@andymandias What you are saying is consistent the spirit of this PR. Can you suggest a better wording?

[andymandias]

I agree it is consistent with the spirit of the PR. I don't believe it to be necessarily prohibitive either, just an increase in complexity I felt was worth noting since it wasn't apparent to me on first read (and based on some of the earlier comments I suspect I wasn't the only one). A note along the lines of "Clients will be expected to strip the formatting that carries over from the previous message in the batch in order to reproduce the original formatting spans (e.g. when creating a spoiler text span from multiple messages)." perhaps?

[skizzerz]

I agree it is consistent with the spirit of the PR. I don't believe it to be necessarily prohibitive either, just an increase in complexity I felt was worth noting since it wasn't apparent to me on first read (and based on some of the earlier comments I suspect I wasn't the only one). A note along the lines of "Clients will be expected to strip the formatting that carries over from the previous message in the batch in order to reproduce the original formatting spans (e.g. when creating a spoiler text span from multiple messages)." perhaps?

The goal is so that the text your client sends renders the same regardless of whether the receiving side supports multiline or not. In the fallback case, each line of the multiline batch (including multiline-concat lines) is rendered as its own independent line, and clients historically do not merge formatting between independent lines; it's reset for each line. The suggested wording in this PR serves to emulate that behavior for receiving clients which do support multiline.

As a sending client, you should therefore be repeating any formatting that needs to cross protocol line boundaries in each relevant line. As a receiving client, you are free to merge those sections together, and if you have suggested wording for that then feel free to propose it.

The option that doesn't work is "do nothing and interpret format codes across protocol line boundaries" because that makes it impossible to specify a rendering that works for all receiving clients.