spec.bs

<pre class='metadata'>
Title: SPARQL 1.1 Incremental Protocol
Shortname: sparql-1-1-incremental-protocol
Level: none
Status: LD
Editor: Maarten Vandenbrande, Ghent University - imec http://idlab.ugent.be/, maarten.vandenbrande@ugent.be
Abstract: This document defines an extension to the SPARQL 1.1 Protocol that enables incremental delivery of SPARQL query results over Server-Sent Events. Instead of a single response, the service emits a stream of events that convey an initial query results followed by incremental updates and explicit "up-to-date" signals. This extension is intended for use by incremental view maintenance (IVM) engines and other systems that require low-latency maintenance of SPARQL query results under updates.
Markup Shorthands: markdown yes, css no
Canonical URL: _ORCHESTRATOR_CURRENT_BUILD_FULL_LINK_
Previous Version: _ORCHESTRATOR_PREVIOUS_BUILD_FULL_LINK_
Local Boilerplate: logo yes
</pre>

<pre class="anchors">
spec: SPARQL11-QUERY; urlPrefix: https://www.w3.org/TR/sparql11-query/#
    type: dfn; text: SPARQL Query
spec: SPARQL11-PROTOCOL; urlPrefix: https://www.w3.org/TR/sparql11-protocol/#
    type: dfn; text: SPARQL Protocol
    type: dfn; text: SPARQL Protocol service
spec: SPARQL11-RESULTS-JSON; urlPrefix: https://www.w3.org/TR/sparql11-results-json/#
    type: dfn; text: SPARQL Results JSON
spec: SPARQL11-RESULTS-XML; urlPrefix: https://www.w3.org/TR/sparql11-results-xml/#
    type: dfn; text: SPARQL Results XML
spec: SPARQL11-RESULTS-CSV-TSV; urlPrefix: https://www.w3.org/TR/sparql11-results-csv-tsv/#
    type: dfn; text: SPARQL Results CSV
    type: dfn; text: SPARQL Results TSV
spec: HTML; urlPrefix: https://html.spec.whatwg.org/multipage/#
    type: dfn; text: server-sent events
</pre>

Introduction {#intro}
=====================
The [=SPARQL Protocol=] defines how clients issue SPARQL queries and receive results over HTTP.
In its standard form, each query produces a single response containing the query results, serialized using formats such as [=SPARQL Results JSON=] or [=SPARQL Results XML=].

For applications that maintain long-lived queries over changing data, repeatedly reissuing queries and re-fetching full results is inefficient and adds latency.

This specification defines an extension to the [=SPARQL Protocol=] that allows a [=SPARQL Protocol service=] to deliver results as an [=Incremental Result Stream=] using [=server-sent events=].
Instead of returning a single result document, the service emits:

* an `initial` event containing the complete results at that time.
* zero or more `processing` event indicating that the service is working on new changes.
* zero or more `update` events conveying incremental additions and deletions.
* one or more `up-to-date` events indicating that, at the moment of emission, the service has processed all known changes for the query.
* zero or one `error` event indicating that an error has occurred, after which no further events will be sent.

This version of the specification normatively defines different serializations for query results over [=server-sent events=].

Terminology {#terminology}
--------------------------
We use the terms specified in the [=SPARQL Protocol=] and [=SPARQL Query=] specifications, and additionally define:
* <dfn>Incremental SPARQL Query</dfn>: A SPARQL query that is intended to be maintained incrementally over time as the underlying data changes.
* <dfn>Incremental SPARQL Service</dfn>: A SPARQL service advertising `sip:incrementalProtocol` (typed as `sip:IncrementalService`) that exposes incremental results. Consisting of additions and deletions over time updating the results of [=Incremental SPARQL Queries=].
* <dfn>Incremental Result Stream</dfn>: The [=server-sent events=] stream for a single query that carries the `initial`, `processing`, `update`, `up-to-date`, and `error` events defined in this specification.

Namespaces {#namespaces}
------------------------
The following namespace prefixes are used throughout this document.

<table class="complex">
  <thead>
    <tr>
      <th>Prefix &nbsp;&nbsp;&nbsp;&nbsp;</th>
      <th>IRI</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>`sip:`</td>
      <td>`https://www.w3.org/ns/sparql-incremental#`</td>
    </tr>
    <tr>
      <td>`sd:`</td>
      <td>`http://www.w3.org/ns/sparql-service-description#`</td>
    </tr>
    <tr>
      <td>`xsd:`</td>
      <td>`http://www.w3.org/2001/XMLSchema#`</td>
    </tr>
    <tr>
      <td>`rdfs:`</td>
      <td>`http://www.w3.org/2000/01/rdf-schema#`</td>
    </tr>
    <tr>
      <td>`rdf:`</td>
      <td>`http://www.w3.org/1999/02/22-rdf-syntax-ns#`</td>
    </tr>
    <tr>
      <td>`owl:`</td>
      <td>`http://www.w3.org/2002/07/owl#`</td>
    </tr>
  </tbody>
</table>

ISSUE: The `http://www.w3.org/ns/sparql-incremental#` namespace needs to be registered.

SPARQL Incremental Protocol {#protocol}
===========================
This protocol only changes three aspects of the [=SPARQL Protocol=]:
1. Content negotiation to select the [=server-sent events=]-based incremental result format.
2. The body of the success response format itself, which is now a [=server-sent events=] stream.
3. The service description to advertise support for this protocol extension.

Content Negotiation {#content-negotiation}
------------------------------------------
To request incremental results over [=server-sent events=], the client MUST include the `text/event-stream` media type in the `Accept` header as per the requirements for [=server-sent events=] in [[!HTML]].
By default, the the [=Incremental SPARQL Service=] MUST respond with the JSON-based serialization (see [[#events-json]]) for binding and boolean results and the RDF serialization (see [[#events-rdf]]) for graph results.
This is to have the best compatibility with the EventSource API in web browsers, where JSON is the most commonly supported format.
To support other serializations, the client MAY add the `Accept` header string to a `accept` HTTP query string parameter of the request.
In case of a URL-encoded POST requests, the `accept` parameter MUST be included in the body of the request.
This parameter is **not** part of HTTP content negotiation; it is interpreted exclusively by SPARQL 1.1 Incremental Protocol services.

Events Semantics {#events-semantics}
--------------------------------
The [=server-sent events=] stream consists of the following event types.
Implementations MUST support these event types, but MAY emit additional event types not defined in this specification.
The `id` field defined by the [=server-sent events=] specification MAY be supported by the implementation.

: <dfn noexport>initial</dfn>
::  This event conveys the initial state of the query results.
    It MUST be the first event emitted by the [=Incremental SPARQL Service=] upon successfully establishing the stream.
    The data payload contains the complete results, serialized according to the negotiated format.

: <dfn noexport>processing</dfn>
::  This event MUST be emmited when the [=Incremental SPARQL Service=] is processing updates to the underlying data.
    The payload MAY contain a timestamp representing the latest data change being processed.
    This event does not guarantee that an `update` event will follow (e.g., if the processed changes do not affect the query result).

: <dfn noexport>update</dfn>
::  This event conveys incremental changes to the query results since the last `initial` or `update` event.
    The payload describes the additions and/or deletions to the query results.
    The head with variable names and the optional link array SHOULD NOT be included in these events.
    The [=Incremental SPARQL Service=] MAY emit multiple `update` events before an `up-to-date` event to batch changes.

NOTE: When processing updates, the additions should be applied before deletions to avoid transient inconsistencies in the results.

: <dfn noexport>up-to-date</dfn>
::  This event signals that the [=Incremental SPARQL Service=] has processed all changes to the underlying data.
    It indicates that all changes known up to the included timestamp have been processed and any resulting `update` events have been emitted.
    The payload MUST include a timestamp indicating this consistency point.

: <dfn noexport>error</dfn>
::  This event indicates a terminal failure in the stream generation.
    After emitting this event, the [=Incremental SPARQL Service=] MUST close the connection.
    The payload MUST include an error code and a description.
    This event is reserved for errors occurring *after* the stream has started; initial request errors (like SPARQL syntax errors) MUST be handled via standard HTTP status codes.

Events JSON {#events-json}
---------------------

This section defines the JSON serialization for the incremental events.
The `event` field of the [=server-sent events=] message corresponds to the event type defined in [[#events-semantics]].
The `data` field contains the JSON payload.

### `initial` Event ### {#json-initial}

The `initial` event payload follows the [=SPARQL Results JSON=] format.

<div class="example">
```
event: initial
data: {
data:   "head": { "vars": [ "book", "title" ] },
data:   "results": {
data:     "bindings": [
data:       {
data:         "book": { "type": "uri", "value": "http://example.org/book/book1" },
data:         "title": { "type": "literal", "value": "SPARQL Tutorial" }
data:       }
data:     ]
data:   }
data: }
```
</div>

### `processing` Event ### {#json-processing}

The `processing` event payload is a JSON object that MAY contain a `timestamp` property (xsd:dateTime) indicating the time of the latest processed change.

<div class="example">
```
event: processing
data: { "timestamp": "2024-11-24T10:00:00Z" }
```
</div>

### `update` Event ### {#json-update}

The `update` event payload is a JSON object containing `additions` and `deletions` arrays.
Each array contains bindings formatted exactly as the `bindings` array in [=SPARQL Results JSON=].

<div class="example">
```
event: update
data: {
data:   "additions": [
data:     {
data:       "book": { "type": "uri", "value": "http://example.org/book/book2" },
data:       "title": { "type": "literal", "value": "The Semantic Web" }
data:     }
data:   ],
data:   "deletions": [
data:     {
data:       "book": { "type": "uri", "value": "http://example.org/book/book1" },
data:       "title": { "type": "literal", "value": "SPARQL Tutorial" }
data:     }
data:   ]
data: }
```
</div>

### `up-to-date` Event ### {#json-up-to-date}

The `up-to-date` event payload is a JSON object that MUST contain a `timestamp` property (xsd:dateTime) indicating the consistency point.

<div class="example">
```
event: up-to-date
data: { "timestamp": "2024-11-24T10:05:00Z" }
```
</div>

### `error` Event ### {#json-error}

The `error` event payload is a JSON object containing `status` (integer) and `statusText` (string) properties.

<div class="example">
```
event: error
data: {
data:   "status": 500,
data:   "statusText": "Internal Server Error: Query execution timeout"
data: }
```
</div>

Events XML {#events-xml}
---------------------

This section defines the XML serialization for the incremental events.
The `event` field of the [=server-sent events=] message corresponds to the event type defined in [[#events-semantics]].
The `data` field contains the XML payload.
The incremental protocol elements are defined in the namespace `http://www.w3.org/ns/sparql-incremental#`.
Standard SPARQL results elements are used within this wrapper in the `http://www.w3.org/2005/sparql-results#` namespace.

### `initial` Event ### {#xml-initial}

The `initial` event payload follows the [=SPARQL Results XML=] format.
It uses the standard SPARQL results namespace.

<div class="example">
```
event: initial
data: <?xml version="1.0"?>
data: <sparql xmlns="http://www.w3.org/2005/sparql-results#">
data:   <head>
data:     <variable name="book"/>
data:     <variable name="title"/>
data:   </head>
data:   <results>
data:     <result>
data:       <binding name="book"><uri>http://example.org/book/book1</uri></binding>
data:       <binding name="title"><literal>SPARQL Tutorial</literal></binding>
data:     </result>
data:   </results>
data: </sparql>
```
</div>

### `processing` Event ### {#xml-processing}

The `processing` event payload is an XML element `processing` in the incremental namespace that MAY contain a `timestamp` attribute (xsd:dateTime).

<div class="example">
```
event: processing
data: <processing xmlns="http://www.w3.org/ns/sparql-incremental#" timestamp="2024-11-24T10:00:00Z"/>
```
</div>

### `update` Event ### {#xml-update}

The `update` event payload is an XML element `update` in the incremental namespace containing `additions` and `deletions` child elements.
These elements contain `result` elements formatted exactly as in [=SPARQL Results XML=], using the standard SPARQL results namespace.

<div class="example">
```
event: update
data: <update xmlns="http://www.w3.org/ns/sparql-incremental#">
data:   <additions>
data:     <result xmlns="http://www.w3.org/2005/sparql-results#">
data:       <binding name="book"><uri>http://example.org/book/book2</uri></binding>
data:       <binding name="title"><literal>The Semantic Web</literal></binding>
data:     </result>
data:   </additions>
data:   <deletions>
data:     <result xmlns="http://www.w3.org/2005/sparql-results#">
data:       <binding name="book"><uri>http://example.org/book/book1</uri></binding>
data:       <binding name="title"><literal>SPARQL Tutorial</literal></binding>
data:     </result>
data:   </deletions>
data: </update>
```
</div>

### `up-to-date` Event ### {#xml-up-to-date}

The `up-to-date` event payload is an XML element `up-to-date` in the incremental namespace that MUST contain a `timestamp` attribute (xsd:dateTime).

<div class="example">
```
event: up-to-date
data: <up-to-date xmlns="http://www.w3.org/ns/sparql-incremental#" timestamp="2024-11-24T10:05:00Z"/>
```
</div>

### `error` Event ### {#xml-error}

The `error` event payload is an XML element `error` in the incremental namespace containing `status` and `statusText` attributes.

<div class="example">
```
event: error
data: <error xmlns="http://www.w3.org/ns/sparql-incremental#" status="500" statusText="Internal Server Error: Query execution timeout"/>
```
</div>


Events CSV and TSV {#events-csv-tsv}
---------------------

This section defines the CSV and TSV serializations for the incremental events.
The `event` field of the [=server-sent events=] message corresponds to the event type defined in [[#events-semantics]].
The `data` field contains the payload formatted according to [=SPARQL Results CSV=] or [=SPARQL Results TSV=].
For TSV, the values MUST be encoded as defined in [=SPARQL Results TSV=] (e.g., IRIs enclosed in `<>`).

### `initial` Event ### {#csv-tsv-initial}

The `initial` event payload follows the standard [=SPARQL Results CSV=] or [=SPARQL Results TSV=] format.

<div class="example">
```
event: initial
data: book,title
data: http://example.org/book/book1,"SPARQL Tutorial"
```
</div>

### `processing` Event ### {#csv-tsv-processing}

The `processing` event payload is a CSV/TSV document with a single column `timestamp`.

<div class="example">
```
event: processing
data: timestamp
data: 2024-11-24T10:00:00Z
```
</div>

### `update` Event ### {#csv-tsv-update}

The `update` event payload is a CSV/TSV document.
To distinguish between additions and deletions, the payload MUST include a special column named `_op` as the first column.
The values in this column MUST be `add` for additions and `del` for deletions.
The remaining columns correspond to the variables in the query results.

<div class="example">
```
event: update
data: _op,book,title
data: add,http://example.org/book/book2,"The Semantic Web"
data: del,http://example.org/book/book1,"SPARQL Tutorial"
```
</div>

NOTE: This serialization does not allow to update the head information (variable names).

### `up-to-date` Event ### {#csv-tsv-up-to-date}

The `up-to-date` event payload is a CSV/TSV document with a single column `timestamp`.

<div class="example">
```
event: up-to-date
data: timestamp
data: 2024-11-24T10:05:00Z
```
</div>

### `error` Event ### {#csv-tsv-error}

The `error` event payload is a CSV/TSV document with columns `status` and `statusText`.

<div class="example">
```
event: error
data: status,statusText
data: 500,"Internal Server Error: Query execution timeout"
```
</div>

Events RDF {#events-rdf}
---------------------

This section defines the RDF serialization for the incremental events.
This serialization is used when the client negotiates an RDF media type (e.g., `text/turtle`, `application/trig`, `application/ld+json`).
It is applicable only to `CONSTRUCT` and `DESCRIBE` queries, which return an RDF Graph.
The `event` field of the [=server-sent events=] message corresponds to the event type defined in [[#events-semantics]].
The `data` field contains the RDF payload serialized in the negotiated format.

The incremental protocol terms are defined in the namespace `http://www.w3.org/ns/sparql-incremental#` (prefix `sip`).

### `initial` Event ### {#rdf-initial}

The `initial` event payload contains the complete RDF Graph result.

<div class="example">
```
event: initial
data: @prefix : <http://example.org/> .
data: :book1 :title "SPARQL Tutorial" .
```
</div>

### `processing` Event ### {#rdf-processing}

The `processing` event payload contains an instance of [sip:Processing](#Processing).

<div class="example">
```
event: processing
data: @prefix sip: <http://www.w3.org/ns/sparql-incremental#> .
data: @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
data:
data: [] a sip:Processing ;
data:    sip:timestamp "2024-11-24T10:00:00Z"^^xsd:dateTime .
```
</div>

### `update` Event ### {#rdf-update}

The `update` event payload conveys additions and deletions.
The additions and deletions MUST be described using standard RDF Reification.
The [sip:Update](#Update) instance links to the reified statements via [sip:additions](#additions) and [sip:deletions](#deletions).

<div class="example">
```
event: update
data: @prefix sip: <http://www.w3.org/ns/sparql-incremental#> .
data: @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
data: @prefix : <http://example.org/> .
data:
data: [] a sip:Update ;
data:    sip:additions [
data:      a rdf:Statement ;
data:      rdf:subject :book2 ; rdf:predicate :title ; rdf:object "The Semantic Web"
data:    ] ;
data:    sip:deletions [
data:      a rdf:Statement ;
data:      rdf:subject :book1 ; rdf:predicate :title ; rdf:object "SPARQL Tutorial"
data:    ] .
```
</div>

NOTE: A future version of this specification will adopt the RDF 1.2 terminology of *RDF terms* and will use RDF 1.2 quoted triples for describing additions and deletions.

### `up-to-date` Event ### {#rdf-up-to-date}

The `up-to-date` event payload contains an instance of [sip:UpToDate](#UpToDate).

<div class="example">
```
event: up-to-date
data: @prefix sip: <http://www.w3.org/ns/sparql-incremental#> .
data: @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
data:
data: [] a sip:UpToDate ;
data:    sip:timestamp "2024-11-24T10:05:00Z"^^xsd:dateTime .
```
</div>

### `error` Event ### {#rdf-error}

The `error` event payload contains an instance of [sip:Error](#Error).

<div class="example">
```
event: error
data: @prefix sip: <http://www.w3.org/ns/sparql-incremental#> .
data:
data: [] a sip:Error ;
data:    sip:status 500 ;
data:    sip:statusText "Internal Server Error: Query execution timeout" .
```
</div>

Service Description {#service-description}
---------------------------
Endpoints implementing this extension SHOULD advertise their support for the SPARQL 1.1 Incremental Protocol through the SPARQL Service Description vocabulary.
Concretely, an endpoint MAY assert the feature [sip:incrementalProtocol](#incrementalProtocol) on its [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service) description, MAY type the service as a [sip:IncrementalService](#IncrementalService) (making it an [=Incremental SPARQL Service=]), and MAY provide the incremental metadata defined in [[#sip-vocabulary]].

<div class="example">
```turtle
@prefix sd:  <http://www.w3.org/ns/sparql-service-description#> .
@prefix sip: <https://www.w3.org/ns/sparql-incremental#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

[] a sd:Service, sip:IncrementalService ;
   sd:endpoint <https://example.org/sparql> ;

   # Advertise incremental protocol support
   sd:feature sip:incrementalProtocol ;

   # Additional incremental metadata
   sip:streamingEndpoint <https://example.org/sparql/stream> ;
   sip:resultFormat "text/event-stream+sparql-results+json" ;
   sip:supportsLastEventID true .
```
</div>

Vocabulary {#sip-vocabulary}
============================
The SPARQL 1.1 Incremental Protocol vocabulary is defined in the `sip:` namespace (`https://www.w3.org/ns/sparql-incremental#`). 
The following terms are used in this specification.

Classes {#sip-vocabulary-classes}
---------------------------

### sip:IncrementalService ### {#IncrementalService}
An instance of `sip:IncrementalService` indicates that the service supports the SPARQL 1.1 Incremental Protocol.

**type:** [rdfs:Class](https://www.w3.org/TR/rdf-schema/#ch_class)

### sip:Processing ### {#Processing}
The class of resources signalling that the [=Incremental SPARQL Service=] is processing updates to the underlying data.

**type:** [rdfs:Class](https://www.w3.org/TR/rdf-schema/#ch_class)

### sip:Update ### {#Update}
The class of resources that describe incremental changes to the results via additions and deletions.

**type:** [rdfs:Class](https://www.w3.org/TR/rdf-schema/#ch_class)

### sip:UpToDate ### {#UpToDate}
The class of resources indicating that the [=Incremental SPARQL Service=] has processed all updates to the underlying data.

**type:** [rdfs:Class](https://www.w3.org/TR/rdf-schema/#ch_class)

### sip:Error ### {#Error}
The class of resources conveying terminal stream errors emitted after a stream has started.

**type:** [rdfs:Class](https://www.w3.org/TR/rdf-schema/#ch_class)

Properties {#sip-vocabulary-properties}
---------------------------

### sip:incrementalProtocol ### {#incrementalProtocol}
Feature IRI advertised on a [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service) to signal support for the SPARQL 1.1 Incremental Protocol.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)<br>
**domain:** [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service)

### sip:streamingEndpoint ### {#streamingEndpoint}
Relates a [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service) to the IRI of the streaming endpoint that serves incremental results using the SPARQL 1.1 Incremental Protocol.

**type:** [owl:InverseFunctionalProperty](https://www.w3.org/TR/owl-ref/#InverseFunctionalProperty-def)<br>
**domain:** [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service)

### sip:resultFormat ### {#resultFormat}
Relates a [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service) to the serialization used for incremental results (e.g., `text/event-stream+sparql-results+json`).

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)<br>
**domain:** [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service)<br>
**range:** [http://www.w3.org/ns/formats/Format](https://www.w3.org/ns/formats/Format)

### sip:supportsLastEventID ### {#supportsLastEventID}
Relates a [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service) to a boolean value indicating whether the [=Incremental SPARQL Service=] supports the `Last-Event-ID` mechanism of Server-Sent Events for resuming streams.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)<br>
**domain:** [sd:Service](https://www.w3.org/TR/sparql11-service-description/#sd-Service)<br>
**range:** [xsd:boolean](https://www.w3.org/TR/xmlschema11-2/#boolean)

### sip:additions ### {#additions}
Relates a [sip:Update](#Update) instance to reified statements describing triples that were added to the result.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)

### sip:deletions ### {#deletions}
Relates a [sip:Update](#Update) instance to reified statements describing triples that were removed from the result.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)

### sip:timestamp ### {#timestamp}
Records the [xsd:dateTime](https://www.w3.org/TR/xmlschema11-2/#dateTime) at which the [=Incremental SPARQL Service=] is up to date or processing a specific change, used on [sip:Processing](#Processing) and [sip:UpToDate](#UpToDate).

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)

### sip:status ### {#status}
Captures the HTTP status code associated with a [sip:Error](#Error) event payload.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)

### sip:statusText ### {#statusText}
Provides a textual explanation associated with a [sip:Error](#Error) status code.

**type:** [rdf:Property](https://www.w3.org/TR/rdf-schema/#ch_property)