weaviate output connector (#10464)

GitOrigin-RevId: 3c8b608d43c5c2a512f645f3642ca34a92a7f5fc

Author: zxqfd555

CHANGELOG.md

@@ -6,6 +6,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ## [Unreleased]
 
 ### Added
+- `pw.io.weaviate.write` writes a Pathway table to a [Weaviate](https://weaviate.io/) collection, keeping it in sync with the table: additions and updates upsert objects (keyed by a UUID derived from the required `primary_key`), and deletions remove them. An optional `vector` column is stored as the object's embedding; the remaining columns become object properties. The target collection must already exist. The connector writes in parallel across workers (`pathway spawn -n`), with `batch_size` and `concurrency` to tune throughput, and supports `api_key`/`headers` authentication for self-hosted and Weaviate Cloud deployments.
 - `pw.io.pinecone.write` writes a Pathway table to a [Pinecone](https://www.pinecone.io/) index for use as a vector store in RAG pipelines. It keeps the index in sync with the current state of the table: a row is upserted under its record id and a row removed from the table is deleted from the index. By default the id is the table's internal row key (unique, and written in parallel across workers); pass a `primary_key` column to use your own ids instead, in which case the values must uniquely identify rows (a collision raises an error) and writing runs on a single worker. The `vector` column holds the dense embedding (`list[float]` or a 1-D `numpy.ndarray`); the remaining columns — or just those listed in `metadata_columns` — are stored as record metadata. Targets both Pinecone cloud and the local Pinecone emulator via the `host` parameter; the index must already exist with a matching dimension.
 
 ### Changed

Cargo.lock

@@ -120,7 +120,7 @@ tokio-util = { version = "0.7", features = ["compat"] }
 tonic = { version = "0.13.1", features = ["tls-native-roots"] }
 typetag = "0.2.21"
 usearch = "2.15.3"
-uuid = { version = "1.17.0", features = ["v4"] }
+uuid = { version = "1.17.0", features = ["v4", "v5"] }
 xxhash-rust = { version = "0.8.15", features = ["xxh3"] }
 
 [features]

Cargo.toml

@@ -101,6 +101,7 @@ Before going into more details about the different connectors and how they work,
                 <span class="block"><a href="/developers/user-guide/connect/connectors/switching-to-redpanda">Redpanda</a></span>
                 <span class="block"><a href="/developers/user-guide/connect/connectors/slack_send_alerts">Slack</a></span>
                 <span class="block"><a href="/developers/api-docs/pathway-io/sqlite">SQLite</a></span>
+                <span class="block"><a href="/developers/api-docs/pathway-io/weaviate">Weaviate</a></span>
             </td>
             <td class="text-center !align-middle">
                 <span class="block"><a href="/developers/user-guide/connect/connectors/csv_connectors">CSV</a></span>

docs/2.developers/4.user-guide/20.connect/30.connectors-overview.md

@@ -20,6 +20,7 @@
     PostgresContext,
     PostgresWithTlsContext,
     QuestDBContext,
+    WeaviateContext,
     clickhouse_concurrency_slot,
     elasticsearch_concurrency_slot,
     mongodb_concurrency_slot,
@@ -279,3 +280,10 @@ def milvus(tmp_path):
     ctx = MilvusContext(str(tmp_path / "milvus.db"))
     yield ctx
     ctx.close()
+
+
+@pytest.fixture
+def weaviate():
+    ctx = WeaviateContext()
+    yield ctx
+    ctx.cleanup()