Skip to content

Commit fe9cdc8

Browse files
authored
Merge pull request #164 from weaviate/jose/async-replication-collection-params
Add --async_replication_config option to create/update collection
2 parents 7ebeeae + 9d530ac commit fe9cdc8

8 files changed

Lines changed: 530 additions & 9 deletions

File tree

.claude/skills/operating-weaviate-cli/SKILL.md

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -139,9 +139,26 @@ weaviate-cli delete collection --collection MyCollection --json
139139
weaviate-cli delete collection --all --json
140140
```
141141

142-
Key create options: `--multitenant`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--shards N`, `--vectorizer <type>`, `--named_vector`, `--replication_deletion_strategy`, `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
142+
Key create options: `--multitenant`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--shards N`, `--vectorizer <type>`, `--named_vector`, `--replication_deletion_strategy`, `--async_replication_config key=value` (repeatable; requires `--async_enabled` and Weaviate >= v1.36.0), `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
143143

144-
Mutable fields: `--async_enabled`, `--replication_factor`, `--vector_index`, `--description`, `--training_limit`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--replication_deletion_strategy`, `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
144+
Mutable fields: `--async_enabled`, `--replication_factor`, `--vector_index`, `--description`, `--training_limit`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--replication_deletion_strategy`, `--async_replication_config key=value` (repeatable), `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
145+
146+
#### Async Replication Config
147+
148+
```bash
149+
# Create with async replication tuning (requires --async_enabled and Weaviate >= v1.36.0)
150+
weaviate-cli create collection --collection MyCol --async_enabled \
151+
--async_replication_config max_workers=10 \
152+
--async_replication_config frequency=60 \
153+
--async_replication_config propagation_concurrency=4 --json
154+
155+
# Update async replication config on existing collection
156+
weaviate-cli update collection --collection MyCol \
157+
--async_replication_config max_workers=20 \
158+
--async_replication_config propagation_batch_size=100 --json
159+
```
160+
161+
Valid keys (all integers): `max_workers`, `hashtree_height`, `frequency`, `frequency_while_propagating`, `alive_nodes_checking_frequency`, `logging_frequency`, `diff_batch_size`, `diff_per_node_timeout`, `pre_propagation_timeout`, `propagation_timeout`, `propagation_limit`, `propagation_delay`, `propagation_concurrency`, `propagation_batch_size`
145162

146163
#### Object TTL
147164

.claude/skills/operating-weaviate-cli/references/collections.md

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ weaviate-cli create collection \
4848
- `--hfresh_search_probe` -- (hfresh only) Search probe size (default: None, uses server default)
4949
- `--distance_metric` -- Distance metric: cosine, dot, l2-squared, hamming, manhattan (default: None, uses server default). Applies to all vector index types.
5050
- `--rescore_limit` -- Rescore limit for quantized indexes (default: None, uses server default)
51+
- `--async_replication_config` -- Async replication tuning as `key=value` pairs (repeatable). Valid keys: `max_workers`, `hashtree_height`, `frequency`, `frequency_while_propagating`, `alive_nodes_checking_frequency`, `logging_frequency`, `diff_batch_size`, `diff_per_node_timeout`, `pre_propagation_timeout`, `propagation_timeout`, `propagation_limit`, `propagation_delay`, `propagation_concurrency`, `propagation_batch_size`. All values must be integers. Use `reset` to revert all to server defaults. Requires `--async_enabled` on create and Weaviate >= v1.36.0.
5152

5253
**hfresh examples:**
5354
```bash
@@ -66,6 +67,19 @@ weaviate-cli create collection \
6667
--json
6768
```
6869

70+
**Async replication config examples:**
71+
```bash
72+
# Create with custom async replication tuning
73+
weaviate-cli create collection --collection MyCol --async_enabled \
74+
--async_replication_config max_workers=10 \
75+
--async_replication_config frequency=60 \
76+
--async_replication_config propagation_concurrency=4
77+
78+
# Set a single parameter
79+
weaviate-cli create collection --collection MyCol --async_enabled \
80+
--async_replication_config propagation_batch_size=200
81+
```
82+
6983
**Object TTL examples:**
7084
```bash
7185
# Delete objects 1 hour after creation
@@ -90,10 +104,18 @@ weaviate-cli update collection \
90104
--json
91105
```
92106

93-
Mutable fields: `--async_enabled`, `--replication_factor`, `--vector_index`, `--description`, `--training_limit`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--replication_deletion_strategy`, `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
107+
Mutable fields: `--async_enabled`, `--replication_factor`, `--vector_index`, `--description`, `--training_limit`, `--auto_tenant_creation`, `--auto_tenant_activation`, `--replication_deletion_strategy`, `--async_replication_config`, `--object_ttl_type`, `--object_ttl_time`, `--object_ttl_filter_expired`, `--object_ttl_property_name` (only when `object_ttl_type=property`)
94108

95109
**Immutable (cannot change after creation):** multitenant, vectorizer, named_vector, shards
96110

111+
**Async replication config examples (update):**
112+
```bash
113+
# Update async replication tuning on existing collection
114+
weaviate-cli update collection --collection MyCol \
115+
--async_replication_config max_workers=20 \
116+
--async_replication_config propagation_batch_size=100
117+
```
118+
97119
**Object TTL options for update:**
98120
- `--object_ttl_type` -- TTL event type: create, update, property, **disable** (default: "create")
99121
- `--object_ttl_time` -- Time to live in seconds (set together with type to enable TTL)

test/unittests/test_managers/test_collection_manager.py

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -779,6 +779,146 @@ def test_update_collection_with_ttl_disable_type(mock_client, mock_wvc_object_tt
779779
mock_wvc_object_ttl["reconfigure"].disable.assert_called_once()
780780

781781

782+
def test_create_collection_with_async_replication_config(
783+
mock_client, mock_wvc_object_ttl
784+
):
785+
"""Test creating a collection with async replication config parameters."""
786+
mock_collections = MagicMock()
787+
mock_client.collections = mock_collections
788+
mock_collections.exists.side_effect = [False, True]
789+
mock_client.get_meta.return_value = {"version": "1.36.0"}
790+
791+
manager = CollectionManager(mock_client)
792+
793+
async_config = {"max_workers": 10, "frequency": 60, "propagation_concurrency": 4}
794+
795+
manager.create_collection(
796+
collection="TestCollection",
797+
replication_factor=3,
798+
vector_index="hnsw",
799+
async_enabled=True,
800+
async_replication_config=async_config,
801+
)
802+
803+
mock_collections.create.assert_called_once()
804+
create_call_kwargs = mock_collections.create.call_args.kwargs
805+
assert create_call_kwargs["name"] == "TestCollection"
806+
assert create_call_kwargs["replication_config"].asyncEnabled is True
807+
# Verify async_config is set on the replication config
808+
repl_config = create_call_kwargs["replication_config"]
809+
assert repl_config.asyncConfig is not None
810+
assert repl_config.asyncConfig.maxWorkers == 10
811+
assert repl_config.asyncConfig.frequency == 60
812+
assert repl_config.asyncConfig.propagationConcurrency == 4
813+
814+
815+
def test_create_collection_without_async_replication_config(
816+
mock_client, mock_wvc_object_ttl
817+
):
818+
"""Test creating a collection without async replication config passes None."""
819+
mock_collections = MagicMock()
820+
mock_client.collections = mock_collections
821+
mock_collections.exists.side_effect = [False, True]
822+
823+
manager = CollectionManager(mock_client)
824+
825+
manager.create_collection(
826+
collection="TestCollection",
827+
replication_factor=3,
828+
vector_index="hnsw",
829+
async_enabled=True,
830+
)
831+
832+
mock_collections.create.assert_called_once()
833+
repl_config = mock_collections.create.call_args.kwargs["replication_config"]
834+
assert repl_config.asyncConfig is None
835+
836+
837+
def test_create_collection_async_replication_config_requires_async_enabled(
838+
mock_client,
839+
):
840+
"""Test that async_replication_config is rejected when async_enabled is False."""
841+
mock_collections = MagicMock()
842+
mock_client.collections = mock_collections
843+
mock_collections.exists.return_value = False
844+
845+
manager = CollectionManager(mock_client)
846+
847+
with pytest.raises(Exception, match="requires --async_enabled"):
848+
manager.create_collection(
849+
collection="TestCollection",
850+
replication_factor=3,
851+
vector_index="hnsw",
852+
async_enabled=False,
853+
async_replication_config={"max_workers": 10},
854+
)
855+
856+
mock_collections.create.assert_not_called()
857+
858+
859+
def test_update_collection_with_async_replication_config(
860+
mock_client, mock_wvc_object_ttl
861+
):
862+
"""Test updating a collection with async replication config parameters."""
863+
mock_collections = MagicMock()
864+
mock_client.collections = mock_collections
865+
mock_client.collections.exists.side_effect = [True, True]
866+
mock_client.get_meta.return_value = {"version": "1.36.0"}
867+
868+
mock_collection = MagicMock()
869+
mock_client.collections.get.return_value = mock_collection
870+
mock_collection.config.get.return_value = MagicMock(
871+
replication_config=MagicMock(factor=3),
872+
multi_tenancy_config=MagicMock(
873+
enabled=False, auto_tenant_creation=False, auto_tenant_activation=False
874+
),
875+
)
876+
877+
manager = CollectionManager(mock_client)
878+
879+
async_config = {"max_workers": 20, "propagation_batch_size": 100}
880+
881+
manager.update_collection(
882+
collection="TestCollection",
883+
async_replication_config=async_config,
884+
)
885+
886+
mock_collection.config.update.assert_called_once()
887+
repl_config = mock_collection.config.update.call_args.kwargs["replication_config"]
888+
assert repl_config.asyncConfig is not None
889+
assert repl_config.asyncConfig.maxWorkers == 20
890+
assert repl_config.asyncConfig.propagationBatchSize == 100
891+
892+
893+
def test_update_collection_without_async_replication_config(
894+
mock_client, mock_wvc_object_ttl
895+
):
896+
"""Test updating a collection without async replication config passes None."""
897+
mock_collections = MagicMock()
898+
mock_client.collections = mock_collections
899+
mock_client.collections.exists.side_effect = [True, True]
900+
901+
mock_collection = MagicMock()
902+
mock_client.collections.get.return_value = mock_collection
903+
mock_collection.config.get.return_value = MagicMock(
904+
replication_config=MagicMock(factor=3),
905+
multi_tenancy_config=MagicMock(
906+
enabled=False, auto_tenant_creation=False, auto_tenant_activation=False
907+
),
908+
)
909+
910+
manager = CollectionManager(mock_client)
911+
912+
manager.update_collection(
913+
collection="TestCollection",
914+
description="Updated",
915+
)
916+
917+
mock_collection.config.update.assert_called_once()
918+
repl_config = mock_collection.config.update.call_args.kwargs["replication_config"]
919+
assert repl_config.asyncConfig is None
920+
921+
782922
def test_create_collection_with_hfresh_defaults(mock_client, mock_wvc_object_ttl):
783923
"""Test creating a collection with hfresh vector index using default parameters."""
784924
mock_collections = MagicMock()
@@ -860,3 +1000,117 @@ def test_create_collection_with_hfresh_invalid_distance_metric(
8601000

8611001
assert "Invalid distance_metric: 'invalid_metric'" in str(exc_info.value)
8621002
mock_collections.create.assert_not_called()
1003+
1004+
1005+
def test_update_collection_async_replication_config_rejected_when_async_false(
1006+
mock_client,
1007+
):
1008+
"""Test that async_replication_config is rejected when async_enabled is explicitly False."""
1009+
mock_collections = MagicMock()
1010+
mock_client.collections = mock_collections
1011+
mock_client.collections.exists.return_value = True
1012+
1013+
manager = CollectionManager(mock_client)
1014+
1015+
with pytest.raises(Exception, match="cannot be used when --async_enabled is False"):
1016+
manager.update_collection(
1017+
collection="TestCollection",
1018+
async_enabled=False,
1019+
async_replication_config={"max_workers": 10},
1020+
)
1021+
1022+
mock_collections.get.return_value.config.update.assert_not_called()
1023+
1024+
1025+
def test_create_collection_async_replication_config_warns_on_old_version(
1026+
mock_client, mock_wvc_object_ttl, capsys
1027+
):
1028+
"""Warn when async_replication_config is used against a server older than v1.36.0."""
1029+
mock_collections = MagicMock()
1030+
mock_client.collections = mock_collections
1031+
mock_collections.exists.side_effect = [False, True]
1032+
mock_client.get_meta.return_value = {"version": "1.34.0"}
1033+
1034+
manager = CollectionManager(mock_client)
1035+
1036+
manager.create_collection(
1037+
collection="TestCollection",
1038+
replication_factor=3,
1039+
vector_index="hnsw",
1040+
async_enabled=True,
1041+
async_replication_config={"max_workers": 10},
1042+
)
1043+
1044+
captured = capsys.readouterr()
1045+
assert "Warning: --async_replication_config requires Weaviate >= v1.36.0" in (
1046+
captured.out + captured.err
1047+
)
1048+
mock_collections.create.assert_called_once()
1049+
1050+
1051+
def test_update_collection_async_replication_config_reset(
1052+
mock_client, mock_wvc_object_ttl
1053+
):
1054+
"""Reset (empty dict) calls Reconfigure.Replication.async_config() with no kwargs."""
1055+
mock_collections = MagicMock()
1056+
mock_client.collections = mock_collections
1057+
mock_client.collections.exists.side_effect = [True, True]
1058+
mock_client.get_meta.return_value = {"version": "1.36.0"}
1059+
1060+
mock_collection = MagicMock()
1061+
mock_client.collections.get.return_value = mock_collection
1062+
mock_collection.config.get.return_value = MagicMock(
1063+
replication_config=MagicMock(factor=3),
1064+
multi_tenancy_config=MagicMock(
1065+
enabled=False, auto_tenant_creation=False, auto_tenant_activation=False
1066+
),
1067+
)
1068+
1069+
manager = CollectionManager(mock_client)
1070+
1071+
with patch.object(
1072+
wvc.Reconfigure.Replication,
1073+
"async_config",
1074+
wraps=wvc.Reconfigure.Replication.async_config,
1075+
) as mock_async_config:
1076+
manager.update_collection(
1077+
collection="TestCollection",
1078+
async_replication_config={},
1079+
)
1080+
1081+
mock_async_config.assert_called_once_with()
1082+
mock_collection.config.update.assert_called_once()
1083+
repl_config = mock_collection.config.update.call_args.kwargs["replication_config"]
1084+
assert repl_config.asyncConfig is not None
1085+
1086+
1087+
def test_update_collection_async_replication_config_warns_on_old_version(
1088+
mock_client, mock_wvc_object_ttl, capsys
1089+
):
1090+
"""Warn when async_replication_config is used against a server older than v1.36.0."""
1091+
mock_collections = MagicMock()
1092+
mock_client.collections = mock_collections
1093+
mock_client.collections.exists.side_effect = [True, True]
1094+
mock_client.get_meta.return_value = {"version": "1.34.0"}
1095+
1096+
mock_collection = MagicMock()
1097+
mock_client.collections.get.return_value = mock_collection
1098+
mock_collection.config.get.return_value = MagicMock(
1099+
replication_config=MagicMock(factor=3),
1100+
multi_tenancy_config=MagicMock(
1101+
enabled=False, auto_tenant_creation=False, auto_tenant_activation=False
1102+
),
1103+
)
1104+
1105+
manager = CollectionManager(mock_client)
1106+
1107+
manager.update_collection(
1108+
collection="TestCollection",
1109+
async_replication_config={"max_workers": 10},
1110+
)
1111+
1112+
captured = capsys.readouterr()
1113+
assert "Warning: --async_replication_config requires Weaviate >= v1.36.0" in (
1114+
captured.out + captured.err
1115+
)
1116+
mock_collection.config.update.assert_called_once()

0 commit comments

Comments
 (0)