Merge branch 'main' into dependabot/pip/pytest-asyncio-1.3.0

Author: dirkkul

CONTRIBUTING.md

@@ -1,18 +1,12 @@
 ### Setup
 
-We recommend that you use a virtual environment to contribute to the client.
-
-To create a virtual environment, activate it, and install dependencies, run the following shell code:
+The development requirements are listed in the `requirements-devel.txt` file. Install them to your virtual environment with:
 
 ```shell
-python3 -m venv .venv
-source .venv/bin/activate
 pip install -r requirements-devel.txt
 ```
 
-To activate your virtual environment, run `source .venv/bin/activate`.
-
-The newest client versions sometimes require upcoming Weaviate core features. We recommend using Docker (see https://weaviate.io/developers/weaviate/installation/docker-compose) to run a local instance of the `latest Weaviate core <https://hub.docker.com/r/semitechnologies/weaviate/tags>`_ for client development.
+The newest client versions sometimes require upcoming Weaviate core features. We recommend using Docker (see https://docs.weaviate.io/deploy/installation-guides/docker-installation) to run a local instance of the `latest Weaviate core <https://hub.docker.com/r/semitechnologies/weaviate/tags>`_ for client development.
 
 #### Installation
 
@@ -34,10 +28,16 @@ You can install a particular branch directly from GitHub with:
 pip install git+https://github.com/weaviate/weaviate-python-client.git@BRANCH_NAME
 ```
 
+If any static analysis tools such as Pylance fail, try installing the package with:
+`--config-settings editable_mode=compat` suffix. (e.g. `pip install -e . --config-settings editable_mode=compat`)
 
 ### Testing
 
-> Note: We use [pytest](https://docs.pytest.org) to write tests for new client code. However, many older tests use [unittest](https://docs.python.org/3/library/unittest.html). These commands run the `pytest` and `unittest` tests.
+To set up the testing environment, install the test requirements with:
+
+```shell
+pip install -r requirements-test.txt
+```
 
 There are three kinds of tests:
 - Unit tests test individual client components.
@@ -67,9 +67,10 @@ pytest test
 > We strongly recommend using [pre-commit](https://pre-commit.com/) to automatically run all linters locally on each commit. Install `pre-commit` on your system, and then enable it with `pre-commit install`.
 
 We use the following tools to ensure a high code quality:
-- black (formatter), run with `black $FOLDER_WITH_CHANGES`
-- flake8 with plugins. Run with `flake8 $FOLDER_WITH_CHANGES`. Note that all plugins are listed in the `requirements.txt` file and are installed in the first step.
+- ruff (formatter), run with `ruff format $FOLDER_WITH_CHANGES`
+- flake8 with plugins. Run with `flake8 $FOLDER_WITH_CHANGES`.
 
+Note that all plugins are listed in the `requirements-devel.txt` file and are installed in the first step.
 
 ### Creating a Pull Request
 

RELEASING.md

@@ -0,0 +1,30 @@
+## How to create releases
+
+### Step 1: Prepare for release
+1. Pull latest main
+	- Merge any pending PRs
+2. Determine what the next version should be, following semantic versioning
+3. Create a new branch for the changelog
+4. Add a changelog entry to `docs/changelog.rst`
+	- `gh pr list --repo weaviate/weaviate-python-client --state merged --search "merged:>=YYYY-MM-DD"` where `YYYY-MM-DD` is the last release date
+	- Only add the relevant entries to the changelog
+	- Examples - [minor release](https://github.com/weaviate/weaviate-python-client/releases/tag/v4.18.0), [patch release](https://github.com/weaviate/weaviate-python-client/releases/tag/v4.16.10)
+5. Merge back to main
+6. Pull main
+
+### Step 2: Create release
+Option 1: With GH CLI
+1. `gh release create <VERSION_TAG>` (e.g. `gh release create v4.18.1`)
+
+Option 2: With git + GH web UI
+1. git tag VERSION
+	- `git tag -a v4.18.1 -m "v4.18.1"
+2. `git push --tags`
+3. Create a release [from the GH repo](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository?tool=webui#creating-a-release)
+
+### Step 3: Monitor pipeline
+1. Monitor the CICD pipeline
+	1. When all tests pass, the release will be pushed to PyPI automatically
+
+### Notes:
+- The package version is updated automatically (see setup.cfg)

docs/changelog.rst

@@ -1,6 +1,22 @@
 Changelog
 =========
 
+
+Version 4.19.2
+--------------
+This patch version includes:
+- Revert merging #1915; not yet merged into Weaviate server
+
+
+Version 4.19.1
+--------------
+This patch version includes:
+
+- Voyage-multimodal-3.5 support in #1915 by @fzowl
+- multi2multivec-weaviate support in #1859 by @augustas1
+- Internal repo documentation improvements in #1879 by @databyjp
+
+
 Version 4.19.0
 --------------
 This minor version includes:

publishing.md

@@ -5,7 +5,7 @@ pytest-benchmark==5.1.0
 pytest-profiling==1.8.1
 coverage==7.10.7
 pytest-xdist==3.7.0
-werkzeug==3.1.4
+werkzeug==3.1.5
 pytest-httpserver==1.1.3
 py-spy==0.4.1
 

requirements-test.txt

@@ -1,64 +1 @@
-# Unit tests for Weaviate Python client
----
-Unit tests for weaviate package. Each module should have its own sub-directory in the `test` directory. Each test file should begin with `test_<fine_name>.py` and `test` directory should not be renamed, these are mandatory requirements for the `unittest` to parse and run all unittests.
-
-The `util.py` contains helper functions for unit testing.
-
----
-## Run one file unittest
-In order to unit test a single file, you can run the following command:
-```
-python -m unittest path_to_file_dir.file
-```
-E.g. if you run it from repo root folder:
-```bash
-python -m unittest test.gql.test_get -v # -v is optional, -v = verbose
-```
-## Run whole package unittest
-In order to unit test the whole package, you can run the following command:
-```bash
-python -m unittest -v # -v is optional, -v = verbose
-```
-
-# Coverage test
----
-Coverage test for weaviate package. Coverage test can be performed using the existing unit test. It runs all the unit tests in order to find which parts of the code have been executed, thus it can be used instead of the Unit test.
-Coverage test is performed by the `coverage` package that should be installed with the `development-requirements.txt`. For more information on what and how to run coverage tests visit this [link](https://coverage.readthedocs.io/en/coverage-5.3.1/ "coverage.readthedocs.io").
-
----
-
-## Run coverage test for one file
-Coverage test for one file can be performed using the following command:
-```bash
-coverage run -m unittest path_to_the_file_dir.file -v # -v is optional, -v = verbose
-```
-E.g. if you run it from repo root folder:
-```bash
-coverage run -m unittest test.gql.test_get -v # -v is optional, -v = verbose
-```
-
-## Run whole package coverage test
-In order to unit test the whole package, you can run the following command:
-```bash
-coverage run -m unittest -v # -v is optional, -v = verbose
-```
-## Show coverage report
-To get the coverage report run the following command.
-```bash
-coverage report -m --skip-covered # --skip-covered = skip 100% covered files, -m = show missing lines
-```
-
-# Linting
----
-Lint the files that are modified before commiting. The linting is done by `pylint`.
-
-To lint a file/module/package run the following command:
-```bash
-pylint path_to_the_file_or_module
-```
-E.g. if you run it from repo root folder:
-```bash
-pylint weaviate # for the whole package
-pylint weaviate/batch # for the module batch
-pylint weaviate/connect/connection.py # for the connection.py file
-```
+Refer to [CONTRIBUTING.md](CONTRIBUTING.md) file for instructions on how to run the tests.

test/README.md

@@ -1,3 +1,4 @@
+from datetime import timedelta
 from typing import List, Union
 
 import pytest
@@ -11,6 +12,7 @@
     Vectorizers,
     _CollectionConfigCreate,
     _GenerativeProvider,
+    _ObjectTTLConfig,
     _RerankerProvider,
     _VectorizerConfigCreate,
 )
@@ -19,7 +21,6 @@
     Multi2VecField,
     VectorDistances,
 )
-
 from weaviate.collections.classes.config_vectors import _VectorConfigCreate
 
 DEFAULTS = {
@@ -2118,6 +2119,24 @@ def test_config_with_named_vectors(
             }
         },
     ),
+    (
+        [Configure.MultiVectors.multi2vec_weaviate(name="test", image_field="prop")],
+        {
+            "test": {
+                "vectorizer": {
+                    "multi2multivec-weaviate": {
+                        "imageFields": ["prop"],
+                    }
+                },
+                "vectorIndexConfig": {
+                    "multivector": {
+                        "enabled": True,
+                    },
+                },
+                "vectorIndexType": "hnsw",
+            }
+        },
+    ),
     (
         [Configure.Vectors.text2vec_gpt4all(name="test", source_properties=["prop"])],
         {
@@ -2502,3 +2521,87 @@ def test_config_with_vectors(vector_config: List[_VectorConfigCreate], expected:
         "class": "Test",
         "vectorConfig": expected,
     }
+
+
+TEST_OBJECT_TTL_CONFIG_TO_DICT_PARAMETERS = [
+    # delete_by_creation_time
+    (
+        _ObjectTTLConfig(
+            enabled=True,
+            time_to_live=timedelta(hours=24),
+            filter_expired_objects=True,
+            delete_on="creationTime",
+        ),
+        {
+            "enabled": True,
+            "timeToLive": 86400,
+            "filterExpiredObjects": True,
+            "deleteOn": "creationTime",
+        },
+    ),
+    # delete_by_update_time
+    (
+        _ObjectTTLConfig(
+            enabled=True,
+            time_to_live=timedelta(days=7),
+            filter_expired_objects=False,
+            delete_on="updateTime",
+        ),
+        {
+            "enabled": True,
+            "timeToLive": 604800,
+            "filterExpiredObjects": False,
+            "deleteOn": "updateTime",
+        },
+    ),
+    # delete_by_date_property
+    (
+        _ObjectTTLConfig(
+            enabled=True,
+            time_to_live=timedelta(hours=1, minutes=30),
+            filter_expired_objects=True,
+            delete_on="releaseDate",
+        ),
+        {
+            "enabled": True,
+            "timeToLive": 5400,
+            "filterExpiredObjects": True,
+            "deleteOn": "releaseDate",
+        },
+    ),
+    # None time_to_live
+    (
+        _ObjectTTLConfig(
+            enabled=True,
+            time_to_live=None,
+            filter_expired_objects=False,
+            delete_on="creationTime",
+        ),
+        {
+            "enabled": True,
+            "filterExpiredObjects": False,
+            "deleteOn": "creationTime",
+        },
+    ),
+    # negative offset (delete_by_date_property with offset before date)
+    (
+        _ObjectTTLConfig(
+            enabled=True,
+            time_to_live=timedelta(seconds=-3600),
+            filter_expired_objects=True,
+            delete_on="eventDate",
+        ),
+        {
+            "enabled": True,
+            "timeToLive": -3600,
+            "filterExpiredObjects": True,
+            "deleteOn": "eventDate",
+        },
+    ),
+]
+
+
+@pytest.mark.parametrize("ttl_config,expected", TEST_OBJECT_TTL_CONFIG_TO_DICT_PARAMETERS)
+def test_object_ttl_config_to_dict(ttl_config: _ObjectTTLConfig, expected: dict) -> None:
+    """Test that _ObjectTTLConfig.to_dict() properly converts timedelta to seconds."""
+    assert ttl_config.to_dict() == expected

test/collection/test_config.py

@@ -84,6 +84,7 @@ class BackupStatusReturn(BaseModel):
     status: BackupStatus
     path: str
     backup_id: str = Field(alias="id")
+    size: float = Field(default=0)
 
 
 class BackupReturn(BackupStatusReturn):

weaviate/backup/backup.py

@@ -1,5 +1,6 @@
 from abc import abstractmethod
 from dataclasses import dataclass
+from datetime import timedelta
 from enum import Enum
 from typing import Any, Dict, cast
 
@@ -59,6 +60,9 @@ def to_dict(self) -> dict:
             if isinstance(v, Enum):
                 out[key] = v.value
                 continue
+            if isinstance(v, timedelta):
+                out[key] = int(v.total_seconds())
+                continue
             if isinstance(v, dict):
                 out[key] = {
                     k: v.to_dict() if isinstance(v, _ConfigBase) else v for k, v in v.items()

weaviate/collections/classes/config_base.py

@@ -75,6 +75,7 @@
 WeaviateModel: TypeAlias = Literal[
     "Snowflake/snowflake-arctic-embed-l-v2.0", "Snowflake/snowflake-arctic-embed-m-v1.5"
 ]
+WeaviateMultimodalModel: TypeAlias = Literal["ModernVBERT/colmodernvbert"]
 
 
 class Vectorizers(str, Enum):
@@ -131,6 +132,7 @@ class Vectorizers(str, Enum):
     MULTI2VEC_COHERE = "multi2vec-cohere"
     MULTI2VEC_JINAAI = "multi2vec-jinaai"
     MULTI2MULTI_JINAAI = "multi2multivec-jinaai"
+    MULTI2MULTI_WEAVIATE = "multi2multivec-weaviate"
     MULTI2VEC_BIND = "multi2vec-bind"
     MULTI2VEC_PALM = "multi2vec-palm"  # change to google once 1.27 is the lowest supported version
     MULTI2VEC_VOYAGEAI = "multi2vec-voyageai"
@@ -513,6 +515,20 @@ def _to_dict(self) -> Dict[str, Any]:
         return ret_dict
 
 
+class _Multi2MultiVecWeaviateConfig(_Multi2VecBase):
+    vectorizer: Union[Vectorizers, _EnumLikeStr] = Field(
+        default=Vectorizers.MULTI2MULTI_WEAVIATE, frozen=True, exclude=True
+    )
+    baseURL: Optional[AnyHttpUrl]
+    model: Optional[str]
+
+    def _to_dict(self) -> Dict[str, Any]:
+        ret_dict = super()._to_dict()
+        if self.baseURL is not None:
+            ret_dict["baseURL"] = self.baseURL.unicode_string()
+        return ret_dict
+
+
 class _Multi2VecClipConfig(_Multi2VecBase):
     vectorizer: Union[Vectorizers, _EnumLikeStr] = Field(
         default=Vectorizers.MULTI2VEC_CLIP, frozen=True, exclude=True