add Type Index Section

[ebremer]

The type index is the simplest, basic "query" mechanism.

It is a registry that advertises which types exist in storage and exactly which containers hold resources of each type. Clients use it to discover types and retrieve matching resource URIs, - all server-enforced for authorization.

Advanced queries are out of scope.

additional reference

---

Preview | Diff

[acoburn]

I have a few high-level questions about how this service is intended to work.

First, the server will need to identify which resources are indexed with which types. Does every resource get indexed? What is used as a type value? A link header (e.g., rel=type) or an RDF property (for RDF documents)? What happens if a client submits a JSON-LD document? MUST the server perform a full parse of that input (this has significant security implications). In the input document, a client decides which resources are indexed, but I do not see a mechanism by which a client can signal that certain resources are indexed and other are not. Here, the question is primarily: do clients populate the type index or do servers?

Second, where does the default field come from? Who sets that and how is it managed or changed? There is no definition/explanation for that property.

Third, the location field is unclear. Does the value of that field indicate where such resource types can be found or stored, or does that field indicate resources with that type? In other words, if I am a client and I want to add an entry to a resource of type "Foo", are the location values containers that I would POST to, or are they resources that I would append to (via PUT or PATCH)?

Fourth, the TypeIndexService points to a serviceEndpoint, and that serviceEndpoint points to a separate searchEndpoint. It seems like those are two separate features that have related, but slightly different APIs. The purpose of each is not clearly articulated.

Finally, should this feature be defined in a separate document, as is the case for the Access Request feature?

[gibsonf1]

The type index is the simplest, basic "query" mechanism.

It is a registry that advertises which types exist in storage and exactly which containers hold resources of each type. Clients use it to discover types and retrieve matching resource URIs, - all server-enforced for authorization.

Advanced queries are out of scope.

additional reference

Preview | Diff

The Solid client Type Index spec was trying to simulate type search in the only way they could as clients using containers. We should have a simple type search that has nothing to do with CRUD - that is covered elsewhere in the spec. How an implementation implements the search should not be specified. The discussion with respect to search at the last FTF meeting was to have type search rather than a more comprehensive search. The Type Index using containers as clients implementation doesn't make much sense when a server is involved.

I think we should rename this Type Search (and not reference the client based Type Index implementation terminology)

[elf-pavlik]

I think there should be some required consistency between Access Grants and type indexes. In SAI access grants also serve as type index which besides type and location also provides information about access modes and in the future can include usage policies. Currently there is no default equivalent but I see a need to incubate this future with focus on UX. As for consistency, Besides public read and indexed resources. For all other protected resources which are not public read and not publicly indexed. Is application expected to check both, access grants where it is the assignee and known type indexes? What if those two diverge? I understand that the intention is to have access grants and type indexes as two independent features but they should still reflect some common source of truth.

[gibsonf1]

It makes no sense to be putting things in containers by type - what if something has more than one type? (Most things do) etc. This is an implementation idea best left to implementations or to clients. To create containers and put things in them, that seems to already be covered by other parts of the spec.. What is not currently possible (solid spec) is search on a storage by type no matter where something is - this is only possible with a server type search. Search is the foundational feature for most apps, so having a simple type search will get a foot in the door to providing what all apps actually need.

[elf-pavlik]

@gibsonf1 I'm not sure if your reply is specific to my comment, type index seems to assume some kind of partitioning by type, which is not a new idea https://www.w3.org/TR/void/#class-property-partitions

My point was mostly abuout consistency with AuthZ. It also isn't a new idea to use access control artifacts for the target resource(s) location discovery

[gibsonf1]

@gibsonf1 I'm not sure if your reply is specific to my comment, type index seems to assume some kind of partitioning by type, which is not a new idea https://www.w3.org/TR/void/#class-property-partitions

@elf-pavlik My point here is that the "Type Index" is a specific client side implementation of an attempt to do type search - which makes no sense in the LWS server spec. We should have a Type Search spec here instead which has no specifics on how an implementation might do things, as opposed to the case in the Solid Client Type Index implementation spec.

[elf-pavlik]

I think my other comment could be releavant here. Without references to requirements, and through them to use cases, it may not be clear what this proposal intends to satisfy.

[acoburn]

I agree with @elf-pavlik. The field of search is extremely broad, and without clear reference to requirements, it will be challenging to define a proposal that satisfies the entire WG.

My understanding is that the specific requirements are based on a need for basic data discovery (distinct from service or capability discovery). In addition, this needs to be implementable by the widest range of servers in an interoperable way. Furthermore the design needs to draw on prior art so that we don't define something that no one will use.

In no way should the type index mechanism interfere with an implementation offering other query endpoints.

[gibsonf1]

I agree with @elf-pavlik. The field of search is extremely broad, and without clear reference to requirements, it will be challenging to define a proposal that satisfies the entire WG.

My understanding is that the specific requirements are based on a need for basic data discovery (distinct from service or capability discovery). In addition, this needs to be implementable by the widest range of servers in an interoperable way. Furthermore the design needs to draw on prior art so that we don't define something that no one will use.

In no way should the type index mechanism interfere with an implementation offering other query endpoints.

The agreed to requirement at the F2F was to limit search to a simple type search. This is far less precarious than trying to spec out a client implementation of a type index to try to simulate type search. And is there any in production server based "type index" anywhere? I think not.

[elf-pavlik]

Which use cases and requirements the default property intends to address?
I understand that it intends to indicate a default container for given storage. Are there any restrictions on how many default containers for given type can be indicated in a storage, I would expect just one.
When user has multiple storages, if application has access to discover indexes for given type across some of those storages, there still can be a friction with identifying default of which storage should be used. I think the design of default needs to be evaluated against use cases and requirements it intends to address.

[gibsonf1]

@pchampin I just saw your presentation at Solid World, and the whole issue you are trying to solve with dealing with files to try to get query is solved with server based /search actually https://www.youtube.com/watch?v=0Vrqn1Ie3c0. (On TwinPod, we would just use search to do all you need there) This is specifically why search is a foundational feature for any app that needs it.

[ebremer]

The "default" and "location" fields have been removed from the typeIndexService proposal. The filters for the typeIndexService are, therefore, no longer needed and have been removed. The POST operation on typeIndexService no longer made any sense and has been removed. The typeSearchService, because multiple filters can be added, can be used for finding both lws:Containers and/or lws:DataResource with pagination being handled. Other text clarifications made.

[elf-pavlik]

How does this proposal address sharing resources via Capability URLs?
Some resources will be have public read permission but they should not appear in an index readable by the public.