Skip to content

Commit a4c9513

Browse files
tsmith023rlmanriquerobbespo00dirkkulkavirajk
authored
Add support for new features of Weaviate in v1.32 (#1727)
* Implement `client.replication` namespace for all replicate ops * Remove default kwargs from executor * Remove `TypedDict` classes mapping to API responses * Add include history true default to list all method * Add `shard` option to `cluster.nodes()` method * Allow `query_sharding_state` to return `None` if collection/shard doesn't exist * Fix stubs for `list_all()` and `query()` * Fix `resp_rest` in `tenants.get_by_name` * Simplify `cluster.nodes()` output types * Add support for RQ Add Rotational quantization (RQ) support in the client to configure collections using that quantization technique * Add coverage for RQ * Fix ruff linter * Fix comment * ci: Add Weaviate 1.32 version pointing to main Preliminary commit. Update once stable v1.32 exists * ci: Enable run in 1.32 * Add RBAC permissions for replicate operations * Run `stubs_regen.sh` * Add `REPLICA_MOVEMENT_ENABLED: 'true'` to env var in ci * Skip replicate tests in <1.31.0 * Move all replicate ops under the cluster namespace * Skip replicate tests in <1.31.0 * Only test RBAC replicate with >1.31.0 * Ensure no test collisions using xdist and cleanup of repl ops * update weaviate version for 1.32 * Update to use latest API on `stable/v1.32` and forbid tests in <1.32.0 * Make types for `ReplicateOperation` with/without history * Add missing `ReplicateOperations` to `weaviate.outputs` * Ensure list is always returned for `status.errors` * Add `ShardingState` and `ShardReplicas` to `weaviate.outputs` * Fix name of `targetNode` param in query by target node * Add weaviate aliases * Add support for alias actions * Add support for alias * Fix test and update to latest version in CI * Skip tests on lower version * Fix min version in rbac test * Add min version to all tests * Fix parametrization * Disable vectorizer for alias test * None in more tests * Fix alias names in concurrent tests * Prefix alias names with alias * Fix more concurrency issues * Fix type hint * Fix test with concurrent tests * Fix test * Remove duplicated test * Remove missed docstring from `WeaviateAsyncClient` * chore: Fix the version check for alias to v1.32 Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> * Vectors/deprecate legacy introduce named vectors only syntax (#1659) * Introduce `Configure.Vectors` and `vector_config` in `.create`/`.update` * Update CI images to latest build on each branch * Change prop descr test to handle props sorted by name * Make changes to pass unit tests of legacy vectorization * Alter specific tests for old versions with new syntax * Add skip to test with resolved issue absent from previous vers * - Add `MultiVectors` factory - Allow `quantizer` at root of all `Vectors` and `MultiVectors` methods - Allow `encoding` also at root of all `MultiVectors` methods * Update CI tags to latest patch * Rename `.text2colbert_jinaa()` to `text2vec_jinaai()` * Add warnings for all deprecated syntax * Response to review comments - Add `multi_vector_config` arg to `MultiVectors` methods - Add deprs warnings if `multi_vector` - Update tests to use new syntax * Update tests to reflect single vectors usage * Fix vector config name passing when creating collections * Skip parsing vector to obj if it's empty * Update CI images to latest patches * Remove unneeded added parsing * Add changelog for 4.16.0 --------- Signed-off-by: Kaviraj <kavirajkanagaraj@gmail.com> Co-authored-by: Rodrigo Lopez <rodrigo.lopez@weaviate.io> Co-authored-by: Roberto Esposito <robyesposito2000@gmail.com> Co-authored-by: Roberto Esposito <73891662+robbespo00@users.noreply.github.com> Co-authored-by: Dirk Kulawiak <dirk@semi.technology> Co-authored-by: Dirk Kulawiak <dirk@weaviate.io> Co-authored-by: Kaviraj <kavirajkanagaraj@gmail.com>
1 parent bb6d90d commit a4c9513

72 files changed

Lines changed: 4146 additions & 761 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/main.yaml

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,12 @@ concurrency:
2020
env:
2121
WEAVIATE_125: 1.25.34
2222
WEAVIATE_126: 1.26.17
23-
WEAVIATE_127: 1.27.14
24-
WEAVIATE_128: 1.28.8
25-
WEAVIATE_129: 1.29.1
26-
WEAVIATE_130: 1.30.1
27-
WEAVIATE_131: stable-v1.31-81f8efb
23+
WEAVIATE_127: 1.27.27
24+
WEAVIATE_128: 1.28.16
25+
WEAVIATE_129: 1.29.9
26+
WEAVIATE_130: 1.30.11
27+
WEAVIATE_131: 1.31.5
28+
WEAVIATE_132: 1.32.0-rc.1-098c594
2829

2930
jobs:
3031
lint-and-format:
@@ -132,11 +133,11 @@ jobs:
132133
fail-fast: false
133134
matrix:
134135
versions: [
135-
{ py: "3.9", weaviate: $WEAVIATE_130},
136-
{ py: "3.10", weaviate: $WEAVIATE_130},
137-
{ py: "3.11", weaviate: $WEAVIATE_130},
138-
{ py: "3.12", weaviate: $WEAVIATE_130},
139-
{ py: "3.13", weaviate: $WEAVIATE_130}
136+
{ py: "3.9", weaviate: $WEAVIATE_131},
137+
{ py: "3.10", weaviate: $WEAVIATE_131},
138+
{ py: "3.11", weaviate: $WEAVIATE_131},
139+
{ py: "3.12", weaviate: $WEAVIATE_131},
140+
{ py: "3.13", weaviate: $WEAVIATE_131}
140141
]
141142
optional_dependencies: [false]
142143
steps:
@@ -187,11 +188,11 @@ jobs:
187188
fail-fast: false
188189
matrix:
189190
versions: [
190-
{ py: "3.9", weaviate: $WEAVIATE_129},
191-
{ py: "3.10", weaviate: $WEAVIATE_129},
192-
{ py: "3.11", weaviate: $WEAVIATE_129},
193-
{ py: "3.12", weaviate: $WEAVIATE_129},
194-
{ py: "3.13", weaviate: $WEAVIATE_129}
191+
{ py: "3.9", weaviate: $WEAVIATE_131},
192+
{ py: "3.10", weaviate: $WEAVIATE_131},
193+
{ py: "3.11", weaviate: $WEAVIATE_131},
194+
{ py: "3.12", weaviate: $WEAVIATE_131},
195+
{ py: "3.13", weaviate: $WEAVIATE_131}
195196
]
196197
optional_dependencies: [false]
197198
steps:
@@ -282,7 +283,8 @@ jobs:
282283
$WEAVIATE_128,
283284
$WEAVIATE_129,
284285
$WEAVIATE_130,
285-
$WEAVIATE_131
286+
$WEAVIATE_131,
287+
$WEAVIATE_132
286288
]
287289
steps:
288290
- name: Checkout

.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,4 +49,4 @@ repos:
4949
language: node
5050
pass_filenames: false
5151
types: [python]
52-
additional_dependencies: [pyright@1.1.398]
52+
additional_dependencies: [pyright@1.1.400]

ci/docker-compose-cluster.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ services:
2323
DEFAULT_VECTORIZER_MODULE: text2vec-contextionary
2424
ENABLE_MODULES: text2vec-contextionary
2525
AUTOSCHEMA_ENABLED: 'false'
26+
REPLICA_MOVEMENT_ENABLED: 'true'
2627

2728
weaviate-node-2:
2829
init: true
@@ -56,6 +57,7 @@ services:
5657
DEFAULT_VECTORIZER_MODULE: text2vec-contextionary
5758
ENABLE_MODULES: text2vec-contextionary
5859
AUTOSCHEMA_ENABLED: 'false'
60+
REPLICA_MOVEMENT_ENABLED: 'true'
5961

6062
weaviate-node-3:
6163
init: true
@@ -89,4 +91,5 @@ services:
8991
DEFAULT_VECTORIZER_MODULE: text2vec-contextionary
9092
ENABLE_MODULES: text2vec-contextionary
9193
AUTOSCHEMA_ENABLED: 'false'
94+
REPLICA_MOVEMENT_ENABLED: 'true'
9295
...

docs/changelog.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,18 @@
11
Changelog
22
=========
33

4+
Version 4.16.0
5+
--------------
6+
This minor version includes:
7+
- Support for new 1.32 features:
8+
- collection aliasing
9+
- replica movement
10+
- RQ configuration
11+
- Introduction of ``Configure.Vectors`` and ``Configure.MultiVectors`` and deprecation of legacy ``Configure.Vectorizer`` and ``Configure.NamedVectors``
12+
- Usage of ``Configure.MultiVectors`` is reserved for modules like ``text2multivec-jinaai``, which was previously known as ``text2colbert-jinaai``
13+
- Addition of helper methods for updating tenant activity statuses: ``collection.tenants.activate()``, ``collection.tenants.deactivate()`` and ``collection.tenants.offload()``
14+
15+
416
Version 4.15.4
517
--------------
618
This patch version includes:

docs/conf.py

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@
66
import contextlib
77
import os
88
import sys
9-
109
from importlib.metadata import PackageNotFoundError
1110
from importlib.metadata import version as version_func
1211

integration/conftest.py

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
_ReferencePropertyBase,
2929
_ReplicationConfigCreate,
3030
_RerankerProvider,
31+
_VectorConfigCreate,
3132
_VectorIndexConfigCreate,
3233
_VectorizerConfigCreate,
3334
)
@@ -58,6 +59,9 @@ def __call__(
5859
vector_index_config: Optional[_VectorIndexConfigCreate] = None,
5960
description: Optional[str] = None,
6061
reranker_config: Optional[_RerankerProvider] = None,
62+
vector_config: Optional[
63+
Optional[Union[_VectorConfigCreate, List[_VectorConfigCreate]]]
64+
] = None,
6165
) -> Collection[Any, Any]:
6266
"""Typing for fixture."""
6367
...
@@ -129,6 +133,9 @@ def _factory(
129133
vector_index_config: Optional[_VectorIndexConfigCreate] = None,
130134
description: Optional[str] = None,
131135
reranker_config: Optional[_RerankerProvider] = None,
136+
vector_config: Optional[
137+
Optional[Union[_VectorConfigCreate, List[_VectorConfigCreate]]]
138+
] = None,
132139
) -> Collection[Any, Any]:
133140
try:
134141
nonlocal client_fixture, name_fixtures, call_counter # noqa: F824
@@ -148,7 +155,8 @@ def _factory(
148155
collection: Collection[Any, Any] = client_fixture.collections.create(
149156
name=name_fixture,
150157
description=description,
151-
vectorizer_config=vectorizer_config or Configure.Vectorizer.none(),
158+
vectorizer_config=vectorizer_config
159+
or (Configure.Vectorizer.none() if vector_config is None else None),
152160
properties=properties,
153161
references=references,
154162
inverted_index_config=inverted_index_config,
@@ -159,6 +167,7 @@ def _factory(
159167
replication_config=replication_config,
160168
vector_index_config=vector_index_config,
161169
reranker_config=reranker_config,
170+
vector_config=vector_config,
162171
)
163172
return collection
164173
except Exception as e:
@@ -315,6 +324,9 @@ def __call__(
315324
vectorizer_config: Optional[
316325
Union[_VectorizerConfigCreate, List[_NamedVectorConfigCreate]]
317326
] = None,
327+
vector_config: Optional[
328+
Optional[Union[_VectorConfigCreate, List[_VectorConfigCreate]]]
329+
] = None,
318330
) -> Collection[Any, Any]:
319331
"""Typing for fixture."""
320332
...
@@ -329,17 +341,18 @@ def _factory(
329341
vectorizer_config: Optional[
330342
Union[_VectorizerConfigCreate, List[_NamedVectorConfigCreate]]
331343
] = None,
344+
vector_config: Optional[
345+
Optional[Union[_VectorConfigCreate, List[_VectorConfigCreate]]]
346+
] = None,
332347
) -> Collection[Any, Any]:
333348
api_key = os.environ.get("OPENAI_APIKEY")
334349
if api_key is None:
335350
pytest.skip("No OpenAI API key found.")
336351

337-
if vectorizer_config is None:
338-
vectorizer_config = Configure.Vectorizer.none()
339-
340352
collection = collection_factory(
341353
name=name,
342-
vectorizer_config=vectorizer_config or Configure.Vectorizer.none(),
354+
vectorizer_config=vectorizer_config,
355+
vector_config=vector_config or Configure.Vectors.self_provided(),
343356
properties=[
344357
Property(name="text", data_type=DataType.TEXT),
345358
Property(name="content", data_type=DataType.TEXT),

integration/test_alias.py

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
from typing import Callable, Generator, Optional
2+
from _pytest.fixtures import SubRequest
3+
4+
import pytest
5+
import weaviate
6+
import weaviate.classes as wvc
7+
from .conftest import _sanitize_collection_name
8+
9+
10+
ClientFactory = Callable[[int, int], weaviate.WeaviateClient]
11+
12+
13+
@pytest.fixture(scope="module")
14+
def client_factory() -> Generator[Callable[[int, int], weaviate.WeaviateClient], None, None]:
15+
client: Optional[weaviate.WeaviateClient] = None
16+
17+
def maker(http: int, grpc: int) -> weaviate.WeaviateClient:
18+
nonlocal client
19+
client = weaviate.WeaviateClient(
20+
connection_params=weaviate.connect.ConnectionParams.from_url(
21+
f"http://localhost:{http}", grpc
22+
),
23+
skip_init_checks=False,
24+
)
25+
client.connect()
26+
return client
27+
28+
try:
29+
yield maker
30+
finally:
31+
assert client is not None
32+
client.close()
33+
34+
35+
@pytest.fixture(scope="module")
36+
def client(client_factory: ClientFactory) -> Generator[weaviate.WeaviateClient, None, None]:
37+
yield client_factory(8080, 50051)
38+
39+
40+
def test_alias_creation_and_deletion(client: weaviate.WeaviateClient, request: SubRequest) -> None:
41+
if client._connection._weaviate_version.is_lower_than(1, 32, 0):
42+
pytest.skip("Aliases are not supported in Weaviate versions < 1.32.0")
43+
name = _sanitize_collection_name(request.node.name)
44+
name2 = _sanitize_collection_name(request.node.name + "_2")
45+
alias1: str = "Alias" + _sanitize_collection_name(request.node.name)
46+
alias2: str = "Alias" + _sanitize_collection_name(request.node.name + "_2")
47+
48+
client.collections.delete(name)
49+
client.collections.delete(name2)
50+
client.alias.delete(alias_name=alias1)
51+
client.alias.delete(alias_name=alias2)
52+
53+
try:
54+
client.collections.create(
55+
name=name, vectorizer_config=wvc.config.Configure.Vectorizer.none()
56+
)
57+
client.collections.create(
58+
name=name2, vectorizer_config=wvc.config.Configure.Vectorizer.none()
59+
)
60+
61+
client.alias.create(alias_name=alias1, target_collection=name)
62+
client.alias.create(alias_name=alias2, target_collection=name2)
63+
all_alias = client.alias.list_all()
64+
all_alias = {
65+
alias[0]: alias[1]
66+
for alias in all_alias.items()
67+
if alias[1].collection in [name, name2]
68+
}
69+
assert len(all_alias) == 2
70+
assert all_alias[alias1].alias == alias1
71+
assert all_alias[alias1].collection == name
72+
73+
collection_alias = client.alias.list_all(collection=name2)
74+
assert len(collection_alias) == 1
75+
assert collection_alias[alias2].alias == alias2
76+
77+
assert all_alias[alias2].alias == alias2
78+
assert all_alias[alias2].collection == name2
79+
80+
# Delete existing aliases
81+
assert client.alias.delete(alias_name=alias1)
82+
assert client.alias.delete(alias_name=alias2)
83+
all_alias = client.alias.list_all(collection=name2)
84+
all_alias = {
85+
alias[0]: alias[1]
86+
for alias in all_alias.items()
87+
if alias[1].collection in [name, name2]
88+
}
89+
assert len(all_alias) == 0
90+
91+
assert not client.alias.delete(alias_name=alias1)
92+
93+
finally:
94+
client.collections.delete(name)
95+
client.collections.delete(name2)
96+
97+
98+
def test_alias_creation_and_update(client: weaviate.WeaviateClient, request: SubRequest) -> None:
99+
if client._connection._weaviate_version.is_lower_than(1, 32, 0):
100+
pytest.skip("Aliases are not supported in Weaviate versions < 1.32.0")
101+
102+
name = _sanitize_collection_name(request.node.name)
103+
name2 = _sanitize_collection_name(request.node.name + "_2")
104+
alias1: str = "Alias" + _sanitize_collection_name(request.node.name)
105+
106+
client.collections.delete(name)
107+
client.collections.delete(name2)
108+
client.alias.delete(alias_name=alias1)
109+
110+
try:
111+
client.collections.create(
112+
name=name, vectorizer_config=wvc.config.Configure.Vectorizer.none()
113+
)
114+
client.collections.create(
115+
name=name2, vectorizer_config=wvc.config.Configure.Vectorizer.none()
116+
)
117+
118+
client.alias.create(alias_name=alias1, target_collection=name)
119+
alias = client.alias.get(alias_name=alias1)
120+
assert alias is not None
121+
assert alias.alias == alias1
122+
assert alias.collection == name
123+
124+
assert client.alias.update(alias_name=alias1, new_target_collection=name2)
125+
alias = client.alias.get(alias_name=alias1)
126+
assert alias is not None
127+
assert alias.alias == alias1
128+
assert alias.collection == name2
129+
130+
# return status code not yet correct
131+
assert not client.alias.update(alias_name="does_not_exist", new_target_collection=name2)
132+
finally:
133+
client.collections.delete(name)
134+
client.collections.delete(name2)
135+
client.alias.delete(alias_name=alias1)
136+
137+
138+
def test_alias_get(client: weaviate.WeaviateClient, request: SubRequest) -> None:
139+
if client._connection._weaviate_version.is_lower_than(1, 32, 0):
140+
pytest.skip("Aliases are not supported in Weaviate versions < 1.32.0")
141+
142+
name = _sanitize_collection_name(request.node.name)
143+
alias1: str = "Alias" + _sanitize_collection_name(request.node.name)
144+
145+
client.collections.delete(name)
146+
client.alias.delete(alias_name=alias1)
147+
try:
148+
client.collections.create(
149+
name=name, vectorizer_config=wvc.config.Configure.Vectorizer.none()
150+
)
151+
152+
client.alias.create(alias_name=alias1, target_collection=name)
153+
alias = client.alias.get(alias_name=alias1)
154+
assert alias is not None
155+
assert alias.alias == alias1
156+
assert alias.collection == name
157+
finally:
158+
client.collections.delete(name)
159+
client.alias.delete(alias_name=alias1)

integration/test_client.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
from typing import Callable, Generator, Tuple, Union
1+
from typing import Callable, Generator, Optional, Tuple, Union
22

33
import pytest
44
from _pytest.fixtures import SubRequest
@@ -28,7 +28,7 @@
2828

2929
@pytest.fixture(scope="module")
3030
def client_factory() -> Generator[Callable[[int, int], weaviate.WeaviateClient], None, None]:
31-
client: weaviate.WeaviateClient = None
31+
client: Optional[weaviate.WeaviateClient] = None
3232

3333
def maker(http: int, grpc: int) -> weaviate.WeaviateClient:
3434
nonlocal client
@@ -593,7 +593,7 @@ async def test_async_client_with_extra_options() -> None:
593593

594594
def test_client_error_for_wcs_without_auth() -> None:
595595
with pytest.raises(weaviate.exceptions.AuthenticationFailedError) as e:
596-
weaviate.connect_to_wcs(cluster_url=WCS_URL, auth_credentials=None)
596+
weaviate.connect_to_wcs(cluster_url=WCS_URL, auth_credentials=None) # pyright: ignore
597597
assert "wvc.init.Auth.api_key" in e.value.message
598598

599599

0 commit comments

Comments
 (0)