NIP-67: EOSE Completeness Hint

[mattn]

NIP-67: EOSE Completeness Hint

Problem

EOSE in NIP-01 marks the boundary between stored and real-time events for a subscription. It does not tell the client whether every stored event matching the filter has been delivered, or whether the relay stopped partway through because of an internal limit.

In practice, relays enforce a per-subscription cap (commonly 300–1000 events) that is independent of the client's limit. Clients today cannot observe this cap directly, so they fall back to a heuristic: compare the number of events received against the limit they requested, and paginate with until if the counts appear to match.

This heuristic is both unreliable and wasteful.

Concrete failure cases

1. Silent data loss when the relay's cap is below the client's limit.

A client asks for the last 500 notes. The relay caps responses at 300 and returns 300 events. Because 300 < 500, the client concludes the result is complete and stops. The 201st through Nth oldest matching notes — which the relay does hold — are never retrieved. The client has no way to know.

2. Mandatory extra round trip when the cap is ≤ the total number of matching events.

A relay caps at 300 events. Any subscription that exhausts the cap requires at least one redundant REQ:

REQ  limit=300          → 300 events + EOSE
REQ  until=<oldest>     → 0 events + EOSE   (just to confirm completion)

If the filter happens to match exactly 300 events in total, the second REQ is pure waste — for the client (latency, bandwidth) and for the relay (another full filter scan to return zero events). Every subscription against such a relay pays this cost.

3. Boundary tie ambiguity.

Multiple events can share the oldest created_at in a batch. Paginating with until = oldest_created_at risks either missing events with that timestamp or double-fetching them, depending on the relay's comparison semantics.

Change

This PR adds NIP-67, which extends EOSE with an optional third element:

["EOSE", <subscription_id>, "finish"]   // all matching stored events have been sent
["EOSE", <subscription_id>]              // no claim of completeness (legacy-compatible)

Only the positive "done" signal is specified. A relay that advertises NIP-67 support but omits the hint is telling the client there is more; a relay that does not advertise support falls through to the existing legacy heuristic. One optional string — a single flag — is enough to eliminate both the silent-truncation bug and the guaranteed-wasted round trip.

The third element is optional. Legacy relays omit it, legacy clients ignore trailing array elements, so the change is backward compatible in both directions.

Why this matters

• Correctness: Today a client using a reasonable default limit against a cap-enforcing relay can silently lose data. This is not a client bug — the protocol provides no way to detect the truncation.
• Efficiency at scale: The "always one extra REQ to confirm" pattern is paid by every client on every exhausted subscription. Across the network this is a significant amount of avoidable query load on relays, which are the constrained resource.
• Fixes a real gap in NIP-01: limit is a client-side hint, not a completeness contract. EOSE signals a phase boundary, not a completeness contract either. Nothing in the core protocol currently lets a relay communicate "I have more for you" or "that's everything." This small addition closes that gap.
• Minimal surface: One optional string on one existing message. No new verbs, no new event kinds. Opt-in via NIP-11 supported_nips.

Non-goals

• This NIP does not change the meaning of EOSE as the stored/real-time boundary.
• It does not define a cursor or pagination token. until-based pagination remains the mechanism; this NIP only tells the client whether pagination is worth attempting.
• It does not require relays to change their internal caps, only to report whether a cap was hit.

---

Note: I used AI to help translate this pull request into English, but I have reviewed the content myself.

[fiatjaf]

Can this be used for when there is content behind AUTH, for example?

It would come in very handy for #2156.

We could just add another hint that says "auth" or something like that. Do you think this makes sense?

[Semisol]

Can this be used for when there is content behind AUTH, for example?

It would come in very handy for #2156.

We could just add another hint that says "auth" or something like that. Do you think this makes sense?

In this case, I would propose an object here instead of a field, with {"hints":[...]}. Since for AUTH/Paywall you might want to attach additional information, you could do that as additional fields in tht object.

[fiatjaf]

No, I think just a string is enough, we can leave the object for the next item, if necessary.

["EOSE", "<subid>", "auth", {...}]

[Semisol]

No, I think just a string is enough, we can leave the object for the next item, if necessary.

["EOSE", "<subid>", "auth", {...}]

Another reason is that you may have auth and finish at once, or even other combinations of flags. I guess we could use , to split them in the string, but if so, we should define it from the outset.

[fiatjaf]

Makes sense, but I don't think having auth and finish at once will ever happen. Because each different hint will require a completely different implementation I'd say we delay specifying multiple simultaneous hints for when we have other hints that may require that.

[Semisol]

Makes sense, but I don't think having auth and finish at once will ever happen. Because each different hint will require a completely different implementation I'd say we delay specifying multiple simultaneous hints for when we have other hints that may require that.

What about the simple query {"#p": [notauthed], "kinds": [4]}? You are at the end of the query (for your current privilege level) but also need AUTH to access anything.

Or even drop the kinds. A client may want to download everything mentioning someone, and you can have both auth and finish there. There is no real reason to deny the query either (read-only mode, "events tagging these people" feeds, etc.)

[mattn]

Instead of changing the third element into an array/object (which changes its type and breaks positional indexing), I'd keep it a plain string but define it as a comma-separated list of tokens from the start:

["EOSE", "<subid>", "finish"]
["EOSE", "<subid>", "auth,finish"]

Fully backward compatible, and it covers @Semisol's auth+finish case (e.g. {"#p": [<notauthed>], "kinds": [4]}). Only finish is defined for now; tokens are ,-separated with no whitespace, order-insensitive, and unknown/duplicate tokens MUST be ignored. Single hint = single token, so the common case is unchanged. Richer per-hint metadata can still go in a future 4th object element.

[fiatjaf]

Makes sense, but then I think it is better to do the array. We already have various types of things in these messages anyway. EVENT has strings and objects, OK has booleans, COUNT has objects. I specially dislike objects because developers tend to shove garbage in them, but an array would be ok here:

["EOSE", "<subid>", ["finish", "auth"]]

[mattn]

@fiatjaf just to confirm the array form you suggested:

Backward compat is fine since clients only read index 1, and nothing implements the string form yet. OK to update the spec to array-only?

Edit: went ahead and updated the spec to the array form (array-only). "finish" stays the only defined hint — the ["auth", "finish"] above was just to show the array can hold multiple hints; auth is not defined by this NIP.

[fiatjaf]

Oh, I need "auth" to be defined by this NIP!

[fiatjaf]

But let's merge this as it is and add "auth" later.

[Semisol]

Sounds good to me.

[fiatjaf]

Don't you think it makes more sense to have a "more" hint as well as the "finish"? That way clients can immediately know there is more stuff to paginate without having to query NIP-11.

[Semisol]

Don't you think it makes more sense to have a "more" hint as well as the "finish"? That way clients can immediately know there is more stuff to paginate without having to query NIP-11.

Doesn't not having "finish" imply there is more?
Even if you are at the boundary (500 events returned, 500 actually stored), every DB that I know of makes it really cheap to know if you ran out of keys to scan in your index (or you can just do limit: real_limit+1 internally).

---

Also, I think it was a bad idea to have a max-limit and default-limit listed in NIP-11.

Relays might have more events, but return less than their advertised max limit if the query exceeded a hidden limit, like total scanned index entries.

[vitorpamplona]

I am not sure this adds much. The way I got around this was to always paginate queries and re-sub a modified filter until it hits the number of records I want, or the subscription returns EOSE without any event.

This PR seems to inform if "there is more", but you still have to paginate to get them, so just paginate until nothing is left on the relay. I am not sure if the added information is doing much.

[Semisol]

This PR seems to inform if "there is more", but you still have to paginate to get them, so just paginate until nothing is left on the relay. I am not sure if the added information is doing much.

This allows you to not have to have another round trip, when your limit is higher than the number of events on the relay (which is a lot of the time for reactions, etc.).

It also opens up the door to a much more useful use case, which is to tell clients that you can do this REQ without AUTH, but you will see less results.

For example, a filter mixing non-public and public kinds, or aggregators, pay-for-read relays, etc.

[vitorpamplona]

It also opens up the door to a much more useful use case, which is to tell clients that you can do this REQ without AUTH, but you will see less results.

Isn't that already solved on a Notice? I thought I had already seen this somewhere.

[Semisol]

It also opens up the door to a much more useful use case, which is to tell clients that you can do this REQ without AUTH, but you will see less results.

Isn't that already solved on a Notice? I thought I had already seen this somewhere.

Yes, you have, from my relay! The problem with notices is that they are unstructured and are not attached to any specific subscription.

Clients can instead be told to AUTH for certain subscriptions instead, and clients can then show the AUTH request to be approved depending on the context. (for a feed / for DMs / etc.)

[mattn]

just paginate until nothing is left on the relay

That workaround is the cost this PR removes. "Paginate until an empty EOSE" forces one extra REQ returning zero events on every completed subscription, just to confirm there's nothing left — wasted latency for the client and a wasted filter scan for the relay. finish lets the relay say "that's everything" on the last useful response, so you stop there instead of doing one more query to be sure. Same data, one fewer round trip every time.

It also helps clients that don't paginate like you do. Many use the count < limit ⇒ "done" heuristic. When a relay's cap is below the client's limit (cap 300, limit 500), they get 300 and assume completeness — silently losing older events the relay holds, with no way to detect it. finish makes that truncation observable: no finish ⇒ there's more, regardless of count.

One optional string, fully backward compatible. (And per Semisol, the same mechanism generalizes to a per-subscription auth hint — which a NOTICE can't do.)

[fiatjaf]

Doesn't not having "finish" imply there is more?

No, it could be just a relay that doesn't implement this.

Also, I think it was a bad idea to have a max-limit and default-limit listed in NIP-11.

I tried to remove that once but tons of people, you included, said those were very important fields. Just ignore that. NIP-11 is mostly useless.

[fiatjaf]

When a relay's cap is below the client's limit (cap 300, limit 500), they get 300 and assume completeness

Shouldn't they?

[vitorpamplona]

"Paginate until an empty EOSE" forces one extra REQ returning zero events on every completed subscription, just to confirm there's nothing left

To be fair, the cost of that final EOSE is virtually irrelevant. Parsing and verifying the previous 300 or 500 events (~20 ms in native code) will take more time than another round trip, which happens in parallel. This will be an optional NIP, so I don't think that it will change a lot in practice. To use this PR, devs will need to code two solutions: my way and this PR's new way when available.

I don't know... this does feel like another NIP-11-like thing that is not reliable, and thus nobody uses them. It's just more bureaucracy to the spec with minimal gains.

[mattn]

@fiatjaf fair point that absence is ambiguous (indistinguishable from a relay that doesn't implement this), so I'll add more as the counterpart to finish, but as a MAY:

• finish: returned everything
• more: there are additional matching events
• neither: unknown (can't tell cheaply, or doesn't implement this)

MAY because limit+1 probing or counting past the cap isn't always cheap. A relay that can emits more; one that can't stays silent and the client paginates as today. No regression.

@vitorpamplona I don't think the round trip is negligible. That measures only the client-side parse cost and ignores the server side. That final empty REQ makes the relay run its whole pipeline: receive the WS message, parse JSON, acquire a DB connection, issue SQL, scan an empty result, build and send the EOSE. All for zero events, and it repeats for every completed subscription across every client. Small on one client, far from negligible on the relay.

On top of that, what pagination can't replicate is the auth/partial signal: a loop that stops on an empty EOSE concludes "got everything" even when auth-gated events it never saw exist. That's correctness, not latency, so it belongs on the subscription rather than a NOTICE.

[vitorpamplona]

If it is correctness, not latency, then it should not be an optional NIP.

[mattn]

It's optional for backward compatibility. Appending the third element doesn't break older clients (they keep indexing EOSE by position). That's protocol-level compatibility, not "unimportant, ignore at will."

[vitorpamplona]

Still, if it is correctness, this should be on NIP-01 or NIP-42

[mattn]

I'm not familiar with the nips conventions, but as far as I know EOSE itself started as its own NIP (NIP-15) and was merged into NIP-01 later, so starting this as its own NIP seems natural to me. That said, if putting it directly into NIP-01 is better, I'm fine with that too.

Either way, nostr has no versioning on the protocol spec like internet drafts do, so client and relay implementers have to keep watching the latest spec regardless.

[Semisol]

Doesn't not having "finish" imply there is more?

No, it could be just a relay that doesn't implement this.

Relays that do not implement this do not send an array. This could be a mandatory hint for this spec, and relays could send more without there being more (for example index scan stopped, but all other entries do not match all conditions).

[Semisol]

When a relay's cap is below the client's limit (cap 300, limit 500), they get 300 and assume completeness

Shouldn't they?

They should not. Relays may return less events than requested for many reasons

[fiatjaf]

When a relay's cap is below the client's limit (cap 300, limit 500), they get 300 and assume completeness

Shouldn't they?

They should not. Relays may return less events than requested for many reasons

Good point, actually.

I think some of my relays do this in some special cases:

• receive REQ with limit=500
• read 500 events from database
• filter out some of those in memory
• return only 450 events to client
• EOSE

[vitorpamplona]

@fiatjaf I think you should fix that and return 450 + 50 new events. If the client is asking for 500, you should try to deliver it, not just cut it short. You can even send the 450 in parallel while you gather more 50 before the EOSE.

[fiatjaf]

Yes, in the ideal world I should, but these clients are quite obnoxious aren't they?

[vitorpamplona]

Yes, in the ideal world I should, but these clients are quite obnoxious aren't they?

If you prefer to be hit by another query just to get the next 50 events... be my guest...

[mattn]

Your "minimal gains" holds only when the client always sends a limit. That's what lets count < limit decide completeness, which shrinks the hint's value to the exact boundary case. But the limit isn't even reliable: whether it's honored depends on the relay's real cap (default_limit/max_limit in NIP-11), and most clients never read NIP-11. So count < limit rests on a hidden cap the client can't see. Three cases:

1. limit: 500, relay cap C >= 500 (500 is honored)

records	hint	EOSE returned	client concludes	extra query
300	no	[EOSE, sub]	300 < 500, complete	0
300	yes	[EOSE, sub, ["finish"]]	complete, for sure	0
500	no	[EOSE, sub]	500 == 500, unknown, must probe	1 wasted
500	yes	[EOSE, sub, ["finish"]]	complete, for sure	0 (saved)
800	no	[EOSE, sub] then next 300	paginate, last page 300 < 500, done	2 (needed)
800	yes	["more"] then ["finish"]	knows there's more up front	2 (needed)

Here the hint only clearly pays off at the boundary (records == cap). But this whole table assumes C >= 500, which the client can't verify.

2. limit: 500, but the relay's real cap is 300 (the client doesn't know that)

records	hint	relay returns	client concludes	outcome
200	no	200 + [EOSE]	200 < 500, complete	right, but a blind guess
200	yes	200 + ["finish"]	complete, for sure	right
300	no	300 + [EOSE]	300 < 500, complete	right only by luck; indistinguishable from the 800 row (same count, bare EOSE)
300	yes	300 + ["finish"]	complete, for sure	right
800	no	300 + [EOSE]	300 < 500, complete	wrong: silently loses 500 events; indistinguishable from the 300 row (same count, bare EOSE)
800	yes	300 + ["more"]	there's more, paginate	right (no loss)

For completeness detection the 300-records and 800-records rows are indistinguishable without the hint: the returned events differ, but the only signal the client can use is the same in both, 300 events plus a bare EOSE, no hint either way, so it concludes "complete" in both. It's right in one and loses 500 events in the other, purely by luck of the true count. finish vs more is the only thing that separates them.

3. Without a limit (relay caps at C, also unknown to the client)

records	hint	EOSE returned	client concludes	extra query
< C	no	[EOSE, sub]	no limit and C unknown, can't tell, must probe	1 wasted
< C	yes	[EOSE, sub, ["finish"]]	complete, for sure	0 (saved)
== C	no	[EOSE, sub]	unknown, must probe	1 wasted
== C	yes	[EOSE, sub, ["finish"]]	complete, for sure	0 (saved)
> C	no	[EOSE, sub] then C more then probe	never sure until an empty page	1 wasted
> C	yes	["more"] then ... then ["finish"]	more then complete, explicit	0 (saved)

So count < limit is never actually trustworthy: with a limit it silently breaks when the cap is below the true match count (table 2), and without a limit there's no reference at all (table 3). Either way the client can't see C. The hint states complete vs more directly, removing both the silent loss and the redundant probe.

[vitorpamplona]

What if, instead of doing this is an optional nip hack, we fully develop pagination into nip01?

We have other issues that we are not addressing, like when all created_at's are all the same within the limit. There is no way to filter by anything before "created at + id". Client either gets stuck downloading the same events over and over again or skips some events to unstuck itself

That, to me, is more problematic than the gains of this discussion..

Then in that pagination solution, we can make the "finish", "auth" and other replies a first class citizen.

[mattn]

@fiatjaf Keep this as NIP-67, or fold it into NIP-01 (with auth in NIP-42)? Happy to rewrite either way.

[fiatjaf]

Let's do this for now.

I agree we should also have better pagination primitives, but that's independent from this, and can wait more.

[Semisol]

Let's do this for now.

I agree we should also have better pagination primitives, but that's independent from this, and can wait more.

For pagination primitives, there is the issue that there could be more than limit events at one instant, due to for example a batch generation job.

I think we can fix that mostly with Negentropy.