feat: connector capability markers (location shape/identity/record-version)#752
Open
tabossert wants to merge 11 commits into
Open
feat: connector capability markers (location shape/identity/record-version)#752tabossert wants to merge 11 commits into
tabossert wants to merge 11 commits into
Conversation
Introduce LocationShape and connector-level capability markers (location_shape, location_identity, emits_record_version, supports_recursion) on the registry entry dataclasses, with defaults that preserve today's fsspec blob behavior for unannotated connectors. Mark the fsspec location leaves (remote_url, recursive) x-runtime-eligible in the shared base config so every fsspec connector's JSON schema carries it, and set entry-level markers on the s3/azure/gcs/box/dropbox/sftp source and destination entries. Sources emit a record version except sftp. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
location_shape None means opted-out, so consumers fall back rather than mistake an unannotated connector for an explicit fsspec declaration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tions Annotate the sql-table cohort's registry entries (the entry platform-api picks: source-preferred) with location_shape=SQL_TABLE, the equality-only location_identity, supports_recursion=False, and x-runtime-eligible on the non-credential location fields so platform-api's derivation reproduces the hand-seeded capability table. Clean matches: postgres, snowflake, teradata, couchbase, vastdb, motherduck, kdbai, ibm_watsonx_s3, databricks_volume_delta_tables. Table-name markers live on the connector-specific IndexerConfig subclass (not the shared SQL base) so vastdb, whose identity is bucket/schema, is not over-marked. mongodb and singlestore carry shape/identity/emits/recursion but their hand-table runtime path references a section the picked source entry cannot derive (indexer-hosted database/collection; dest-only table_name); flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…initions Annotate the search-index cohort with location_shape=SEARCH_INDEX, the index/collection identity, supports_recursion=False, and x-runtime-eligible on the mutable index/collection leaf (never the host/url credentials). Clean matches: pinecone, weaviate-cloud, qdrant-cloud, chroma, milvus, vectara. elasticsearch emits a record version. Collection leaves are marked on the shared weaviate/qdrant uploader base configs (only the cloud variants are hand-seeded). elasticsearch, opensearch, and astradb keep index_name/keyspace/collection_name on the source indexer config, so their derived runtime path is indexer_config.* while the hand table labels them connector_config.*/uploader_config.*; flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…itions Annotate the api-folder cohort with location_shape=API_FOLDER, the folder-path identity, and x-runtime-eligible on the path/scope leaves. onedrive, sharepoint, and google_drive have real folder trees (supports_recursion=True) and emit a record version; confluence/jira/salesforce/outlook/zendesk/gitlab are flat (supports_recursion=False). sharepoint's recursive is inherited from the onedrive indexer config; its overridden path leaf is marked directly. slack keeps its channels list on the source indexer config, so the derived runtime path is indexer_config.channels while the hand table labels it connector_config.channels; flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…initions Annotate the remaining cohort: local/redis/neo4j (location_shape=OTHER, equality identity, no recursion) and the destination-only delta_table (fsspec-url; table_uri is an s3 URL, so recursion stays on). x-runtime-eligible marks the non-credential location leaves. airtable, kafka-cloud, and databricks_volumes carry shape/identity/emits/ recursion, but their hand-table runtime path cannot be reproduced: airtable's base_id/table_id are parsed out of indexer list_of_paths (no such fields), kafka's field is bootstrap_server (singular) with the topic on the indexer, and databricks_volumes keeps catalog/schema/volume on the indexer config. Flagged for platform-api-side reconciliation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…le markers Several registry entries pointed location_identity at a census-guessed settings section while their x-runtime-eligible markers lived on the real field. Make each entry's identity reference the same fields/sections the markers actually annotate: indexer_config.* for source location leaves, uploader_config.* for destination leaves, connector_config.* only for fixed connection-level identity fields. Also splits the opensearch and singlestore dual-role connectors so the source identity resolves onto the indexer config and the destination identity onto the uploader config, marking the destination location leaves runtime-eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… common write targets Twelve commonly used write-target connectors had an unannotated destination registry entry (location_shape=None). Annotate each with the shape of its cohort and a location_identity that names its real write location, marking the write-location leaves x-runtime-eligible: - sql-table: postgres/snowflake/teradata/vastdb -> uploader_config.table_name (+ connection database/schema/bucket fixed identity); mongodb -> uploader_config.database+collection; couchbase writes to its connection-level bucket/scope/collection (no uploader location field). - search-index: elasticsearch -> connector_config.hosts + uploader_config.index_name; astradb -> uploader_config.keyspace+collection_name. - other: kafka-cloud -> connector_config.bootstrap_server + uploader_config.topic; databricks_volumes -> uploader_config.catalog/schema/volume; local -> uploader_config.output_dir. - fsspec-url: onedrive -> uploader_config.remote_url. table_name eligibility now lives on the shared SQLUploaderConfig, so the singlestore uploader redeclaration added earlier is dropped. Preserves the PART A invariant: every identity leaf is a real field on its section, eligible unless it is a fixed connection-level identity field. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…markers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 53 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
Address valid cubic findings where a marker contradicted the connector's actual runtime behavior. Only unambiguous, test-green corrections are applied; identity leaves stay real fields on their named section. Record-version emission (metadata.version is a real changing revision token at runtime): - opensearch source: inherits _version into metadata.version (matches the elasticsearch sibling) - confluence source: page version.number, increments on edit - salesforce source: SystemModstamp, changes on modification - outlook source: message change_key, changes on modification Recursion: - outlook source: has a real recursive field and _list_messages descends child folders, so supports_recursion=True location_identity completions (real fields that select the physical target but were omitted, so distinct targets could collide): - kafka cloud source/destination: +connector_config.port (broker endpoint is host:port, assembled at runtime) - redis destination: +connector_config.port - motherduck destination: +connector_config.table (write target table) - vectara destination: +connector_config.corpus_key (actual write target; corpus_name is None when only the key is given) - onedrive source/destination: +connector_config.user_pname (selects the drive) - vastdb/singlestore source: +indexer_config.table_name, with the x-runtime-eligible override matching the postgres/snowflake/teradata SQL-source pattern Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 11 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 12 unresolved issues from previous reviews.
Re-trigger cubic
…rsarially verified)
Second batch of valid cubic findings, each verified against runtime code and,
where it reversed a marker-test assertion, updated with the matching test edit.
The enforced identity-leaf invariant test stays green.
Capability corrections that reverse an earlier deliberate marker:
- box source: drop emits_record_version — BoxIndexer.get_metadata sets
metadata.version to the immutable Box file id (reused as file_id), which does
not change on edit; unlike s3/azure/gcs/dropbox (ETag/etag/content_hash). Move
box out of the test's VERSION_EMITTING_FSSPEC set.
- local source: supports_recursion=True — LocalIndexer.list_files() uses
rglob("*") when the recursive config field is set.
location_identity completions (real fields that select the physical target):
- sharepoint source: +connector_config.library (chooses the drive in
_get_drive_item)
- milvus destination: +uploader_config.db_name (overrides the write DB via
using_database at write time), marked x-runtime-eligible
- databricks volumes source/destination: +volume_path (sub-path within the
volume), marked x-runtime-eligible on the shared DatabricksPathMixin
- chroma destination: +connection endpoint/tenant/database fields (get_client
targets a distinct store even for the same collection name)
- confluence/jira source, elasticsearch destination: +connector_config.cloud_id
(Atlassian OAuth tenant / Elastic Cloud endpoint)
Declined (false positives, no change): zendesk connector_config.subdomain (that
is the correct convention; a real field on the connection config) and the
elasticsearch source cloud_id (the indexer requires hosts, so the guarded
collision is unreachable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 11 files (changes from recent commits).
Shadow auto-approve: would require human review. Adds capability markers (LocationShape enum, new fields) to the connector registry and annotates 10+ connectors. Modifies core data structures across many files; blast radius is non-trivial. Requires human review to ensure backward compatibility and correct consumption by downstream PRs.
Re-trigger cubic
| def _fields(config) -> set[str]: | ||
| return set(config.model_json_schema().get("properties", {}).keys()) | ||
|
|
||
|
|
Contributor
There was a problem hiding this comment.
This test only checks entries that are annotated. It'd be worth adding a test for the opposite case too: that entries with location_shape=None are treated as not parameterizable. That locks in the "unannotated = ignored" behavior so it can't silently break later.
potter-potter
approved these changes
Jul 10, 2026
potter-potter
left a comment
potter-potter
left a comment
Contributor
There was a problem hiding this comment.
Looks good. If you can just add the extra tests from PR comment
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Connector capability markers (location shape / identity / record-version)
Adds capability markers to the connector registry so the control plane (platform-api / platform-etl) can derive per-connector parameterization behavior instead of hardcoding it.
What's here
LocationShapeenum + kw-only markers onSourceRegistryEntry/DestinationRegistryEntry:location_shape,location_identity,supports_recursion,emits_record_versionNone(unannotated) so consumers fall back to their hand-seeded table for connectors not yet annotated — no behavior change until each entry is explicitly markedPart of the parameterized-connectors generalization (consumed by platform-api #1210/its follow-up and platform-etl).
🤖 Generated with Claude Code