Skip to content

Commit c821c53

Browse files
Fix rerank usageDetails note (PR review)
The 0060 conformance note said Langfuse usageDetails is an empty map when usage is null, but the observer omits usageDetails entirely (usage=None, so the adapter adds no usage_details), which fixture 142 asserts. Correct the note to 'omitted when usage = null'.
1 parent 7ede3d3 commit c821c53

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

conformance.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -614,7 +614,7 @@ note = "Retrieval-provider embedding capability. The EmbeddingProvider protocol
614614
[proposals."0060"]
615615
status = "implemented"
616616
since = "0.16.0"
617-
note = "Retrieval-provider rerank protocol. The RerankProvider protocol (retrieval-provider §5) + the RerankResponse / ScoredDocument / RerankUsage / RerankRuntimeConfig response types (§6) + the RerankRuntimeConfig runtime config (§2) ship, alongside the CohereRerankProvider reference reranker against POST /v2/rerank (§8.4, Cohere-shape): the request body is {model, query, documents (string array), top_n from top_k}, return_documents is a silent no-op (the Cohere wire has no such field), and max_tokens_per_doc rides the extras pass-through bag. The provider parses {id, model?, results: [{index, relevance_score, document?}], meta.billed_units.{search_units, input_tokens}}, sorts results by relevance_score descending, reads the document echo only when present (never auto-filled), builds a RerankUsage record only when the provider surfaces >= 1 usage figure (else usage = null, never fabricated), and enforces the §6 invariants -- valid index into the input documents, no duplicate index, len(results) <= top_k when supplied -- raising provider_invalid_response otherwise. Pre-send validation raises provider_invalid_request on an empty query, an empty documents list, or top_k <= 0 (top_k MAY exceed len(documents)). The provider dispatches the typed RerankEvent / RerankFailedEvent (graph-engine §6) per rerank() call, mutually exclusive, alongside the §7-category exception on failure; RerankEvent.output_results is populated unconditionally on success (proposal 0089). The §7 categories reuse the shared llm-provider taxonomy. Both bundled observers render the rerank events (0060b): the OTelObserver emits an openarmature.rerank.complete span (observability §5.5.13, SpanKind.CLIENT) parented lineage-aware under the calling node, carrying the GenAI semconv subset (gen_ai.system / request.model / response.model / response.id + the conditional usage.input_tokens, gated by disable_genai_semconv) + the OA openarmature.rerank.query_length (UTF-8 bytes) / .document_count / .top_k (omitted when null) / .result_count identity attrs + the conditional .search_units + the payload-gated (disable_provider_payload) .query / .documents / .results (the results sourced from RerankEvent.output_results per proposal 0089) -- no gen_ai.operation.name (deferred, no upstream rerank coverage), not gated by disable_llm_spans; failure renders ERROR status + error.type + an exception event + openarmature.error.category. The LangfuseObserver renders a dedicated Retriever observation (asType retriever, §8.4.7) with model = response_model-or-model, usageDetails.{input, searchUnits} (empty map when usage = null), the openarmature_query_length / _document_count / _top_k / _result_count / _response_id metadata, and the payload-gated input {query, documents} + output results; failure renders an ERROR-level Retriever observation with error_category as statusMessage, error_type / error_message in metadata, and no output. Protocol fixtures 006-012 + observability fixtures 099-108 / 138 / 141 / 142 pass. The §11 rerank-metrics path (fixture 109) is proposal 0067's metrics surface (one GenAI instrument across LLM + embedding + rerank calls), tracked and deferred under 0067 (cf. 089), not part of 0060's own observability surface. Wire-format details (0090 Cohere rerank fixtures 028-031) are a separate proposal."
617+
note = "Retrieval-provider rerank protocol. The RerankProvider protocol (retrieval-provider §5) + the RerankResponse / ScoredDocument / RerankUsage / RerankRuntimeConfig response types (§6) + the RerankRuntimeConfig runtime config (§2) ship, alongside the CohereRerankProvider reference reranker against POST /v2/rerank (§8.4, Cohere-shape): the request body is {model, query, documents (string array), top_n from top_k}, return_documents is a silent no-op (the Cohere wire has no such field), and max_tokens_per_doc rides the extras pass-through bag. The provider parses {id, model?, results: [{index, relevance_score, document?}], meta.billed_units.{search_units, input_tokens}}, sorts results by relevance_score descending, reads the document echo only when present (never auto-filled), builds a RerankUsage record only when the provider surfaces >= 1 usage figure (else usage = null, never fabricated), and enforces the §6 invariants -- valid index into the input documents, no duplicate index, len(results) <= top_k when supplied -- raising provider_invalid_response otherwise. Pre-send validation raises provider_invalid_request on an empty query, an empty documents list, or top_k <= 0 (top_k MAY exceed len(documents)). The provider dispatches the typed RerankEvent / RerankFailedEvent (graph-engine §6) per rerank() call, mutually exclusive, alongside the §7-category exception on failure; RerankEvent.output_results is populated unconditionally on success (proposal 0089). The §7 categories reuse the shared llm-provider taxonomy. Both bundled observers render the rerank events (0060b): the OTelObserver emits an openarmature.rerank.complete span (observability §5.5.13, SpanKind.CLIENT) parented lineage-aware under the calling node, carrying the GenAI semconv subset (gen_ai.system / request.model / response.model / response.id + the conditional usage.input_tokens, gated by disable_genai_semconv) + the OA openarmature.rerank.query_length (UTF-8 bytes) / .document_count / .top_k (omitted when null) / .result_count identity attrs + the conditional .search_units + the payload-gated (disable_provider_payload) .query / .documents / .results (the results sourced from RerankEvent.output_results per proposal 0089) -- no gen_ai.operation.name (deferred, no upstream rerank coverage), not gated by disable_llm_spans; failure renders ERROR status + error.type + an exception event + openarmature.error.category. The LangfuseObserver renders a dedicated Retriever observation (asType retriever, §8.4.7) with model = response_model-or-model, usageDetails.{input, searchUnits} (omitted when usage = null), the openarmature_query_length / _document_count / _top_k / _result_count / _response_id metadata, and the payload-gated input {query, documents} + output results; failure renders an ERROR-level Retriever observation with error_category as statusMessage, error_type / error_message in metadata, and no output. Protocol fixtures 006-012 + observability fixtures 099-108 / 138 / 141 / 142 pass. The §11 rerank-metrics path (fixture 109) is proposal 0067's metrics surface (one GenAI instrument across LLM + embedding + rerank calls), tracked and deferred under 0067 (cf. 089), not part of 0060's own observability surface. Wire-format details (0090 Cohere rerank fixtures 028-031) are a separate proposal."
618618

619619
# Spec v0.55.0 (proposal 0065; repo pins v0.55.1). Failure-isolation
620620
# cause fidelity at non-node placements (pipeline-utilities §6.3 /

0 commit comments

Comments
 (0)