docs(database): Document the database connector API

Author: Zhe Yu

docs/CONTRIBUTING.md

@@ -40,6 +40,12 @@ You may also find it helpful to
 [enable logging](https://github.com/Davidyz/VectorCode/blob/main/docs/cli.md#debugging-and-diagnosing) 
 for the CLI when developing new features or working on fixes.
 
+### Database Connectors
+
+Please take a look at [the database documentation](../src/vectorcode/database/README.md), 
+which contains a brief introduction on the API design that explains what you'd need 
+to do to add support for a new database.
+
 ## Neovim Plugin
 
 At the moment, there isn't much to cover on here. As long as the code is 

src/vectorcode/database/README.md

@@ -2,88 +2,70 @@
 
 A database connector is a compatibility layer that converts data structures that a 
 database natively works with to the ones that VectorCode works with. The connector 
-classes provides abstractions for VectorCode operations (`vectorise`, `query`, etc.), 
+classes provide abstractions for VectorCode operations (`vectorise`, `query`, etc.), 
 which enables the use of different database backends.
 
 <!-- mtoc-start -->
 
-* [Creating Database Connectors](#creating-database-connectors)
-* [Implementation Details](#implementation-details)
-  * [Connector Configuration](#connector-configuration)
-  * [Database Settings](#database-settings)
-    * [Documenting the Database Settings](#documenting-the-database-settings)
-  * [CRUD Operations](#crud-operations)
+* [Adding a New Database Connector](#adding-a-new-database-connector)
+* [Key Implementation Details](#key-implementation-details)
+  * [The `Config` Object](#the-config-object)
+  * [Implementing Abstract Methods](#implementing-abstract-methods)
+  * [Error Handling](#error-handling)
+* [Testing](#testing)
 
 <!-- mtoc-end -->
 
-# Creating Database Connectors
-
-To add support for a new database backend, you'd need to: 
-
-1. Implement a child class of `vectorcode.database.base.DatabaseConnectorBase` and all 
-   of its abstract methods, and put it under this directory.
-2. Add a new entry in the [`get_database_connector`](./__init__.py) function that 
-   initialises your new database connector when the `configs.db_type` points to the new 
-   database.
-3. Add tests for your new database connector. The new tests should verify that your 
-   connector correctly converts between the native data structures from the database and
-   the VectorCode data structures that the rest of the codebase (embedding function, 
-   reranker, etc.)can work with.
-
-# Implementation Details
-
-> Apart from this document, you may refer to [the `DatabaseConnectorBase`](./base.py) 
-> and [the `ChromaDB0Connector`](./chroma0.py) implementations as reference designs of 
-> a new database connector.
-
-In the following sections, I'll use the term _database_ to refer to the actual database 
-backends (chromadb, pgvector, etc.) that holds the data and performs the CRUD operations, 
-and the term _connector_ to refer to our compatibility layer (child classes of 
-`vectorcode.database.base.DatabaseConnectorBase`).
-
-## Connector Configuration
-
-The connector has a private attribute (that is, the attribute name is prefixed by a `_`) 
-`self._configs`. This is a `vectorcode.cli_utils.Config` object that holds various 
-configuration options, including the database settings used to initialise the 
-connections to the database and the parameters used for the CRUD operations with the 
-database. This attribute is **mutable** and _should_ be updated before calling a CRUD 
-method using the `self.update_config(new_config)` or the `self.replace_config(new_config)` 
-methods. However, the database-related settings shouldn't be changed. A new connector
-instance should be created for that purpose.
-
-## Database Settings
-
-The database settings are configured in the JSON configuration file, and will be parsed 
-and stored in the `config.db_type` and `config.db_params` attributes of the 
-`self._configs` object.
-
-The `db_type` attribute is a string that indicates the type of the database backend 
-(for example, `ChromaDB0` for Chromadb 0.6.3). 
-
-The `db_params` attribute is a dictionary that holds some database-specific settings 
-(for example, the database API endpoint URL and/or database directory).
-
-### Documenting the Database Settings
-
-Please document about the database-specific settings (`db_params`) in the doc-string 
-of your database connector. This doc-string will be presented in the error message when 
-the database fails to initialise, and should provide instructions to help the user 
-debug their configuration.
-
-## CRUD Operations
-
-Historically, the parameters of VectorCode operations have been stored and propagated 
-in a `vectorcode.cli_utils.Config` object. The database connectors continue to follow 
-this pattern. That is, each of the abstract methods that represent an abstracted 
-database operation (`query()`, `vectorise()`, `list()`, etc.) should read the necessary 
-parameters (`project_root`, file paths, query keywords, etc.) from the `self._configs` 
-attribute. Note that the `self._configs` attribute is mutable, so you should always read 
-the parameters from it directly for each of the operations.
-
-> Some methods support keyword arguments that allows temporarily overriding some 
-> parameters. For example, the `list_collection_content` method supports overriding 
-> `self._configs` by passing `_collection_id` and `collection_path`. The idea is that 
-> these methods can usually be used by the implementation of other methods or subcommands 
-> (for example, `list_collection_content` is used in `count` and `check_orphanes`),
-> and being able to pass such parameters are convenient when writing those implementations.
+# Adding a New Database Connector
+
+To add support for a new database backend, you will need to:
+
+1.  **Implement a connector class**: Create a new file in this directory and implement a child class of `vectorcode.database.base.DatabaseConnectorBase`. You must implement all of its abstract methods.
+2.  **Write tests**: Add tests for your new connector in the `tests/database/` directory. The tests should mock the database's API and verify that your connector correctly converts data between the database's native format and VectorCode's data structures.
+3.  **Register your connector**: Add a new entry in the `get_database_connector` function in `src/vectorcode/database/__init__.py` to initialize your new connector.
+
+For a concrete example, refer to the implementation of `DatabaseConnectorBase` and the `ChromaDB0Connector`.
+
+# Key Implementation Details
+
+## The `Config` Object
+
+All settings for a connector are passed through a single `vectorcode.cli_utils.Config` object, which is available as `self._configs`. This includes:
+
+-   **Database Settings**: The `db_type` string and `db_params` dictionary are used to configure the connection to the database backend. As a contributor, you should document the specific `db_params` your connector requires in the class's docstring.
+-   **Operation Parameters**: Parameters for operations like `query` or `vectorise` are also present in this object.
+
+The `self._configs` attribute is mutable and can be updated for subsequent operations, but the database connection settings (`db_type`, `db_params`) should not be changed after initialization.
+
+## Implementing Abstract Methods
+
+When implementing the abstract methods from `DatabaseConnectorBase`, you should:
+
+-   Read the necessary parameters from the `self._configs` object.
+-   Perform the corresponding operation against the database.
+-   Return data in the format specified by the method's type hints (e.g., `QueryResult`, `CollectionInfo`).
+
+**Please refer to the docstrings in `DatabaseConnectorBase` for the specific API contract of each method.** They contain detailed information about what each method is expected to do and what parameters it uses from the `Config` object.
+
+## Error Handling
+
+If the underlying database library raises a specific exception (e.g., for a collection not being found), you should consider catching it and re-raise it as one of VectorCode's custom database exceptions from `vectorcode.database.errors`. This ensures consistent error handling in the CLI and other clients.
+
+For example:
+```python
+from vectorcode.database.errors import CollectionNotFoundError
+
+try:
+    some_action_here()
+except SomeCustomException as e:
+    raise CollectionNotFoundError("The collection was not found.") from e
+```
+
+# Testing
+
+The unit tests for database backends should go under [`tests/database/`](../../../tests/database/). 
+The tests should mock the request body and return values of the database. Integration 
+tests that interact with an actual database are out of scope for now.
+
+> The tests for the subcommands currently use mocked database connectors. They're not 
+> supposed to interact with live databases.

src/vectorcode/database/base.py

@@ -44,6 +44,10 @@
 class DatabaseConnectorBase(ABC):  # pragma: nocover
     @classmethod
     def create(cls, configs: Config):
+        """
+        Create a new instance of the database connector.
+        This classmethod will add the docstring of the child class to the exception if the initialisation fails.
+        """
         try:
             return cls(configs)
         except Exception as e:  # pragma: nocover
@@ -54,14 +58,17 @@ def create(cls, configs: Config):
 
     def __init__(self, configs: Config):
         """
-        Use the `create` classmethod so that you get docs
-        when something's wrong during the database initialisation.
+        Initialises the database connector with the given configs.
+        It is recommended to use the `create` classmethod instead of calling this directly,
+        as it provides better error handling during initialisation.
         """
         self._configs = configs
 
     async def count(self, what: ResultType = ResultType.chunk) -> int:
         """
         Returns the chunk count or file count of the given collection, depending on the value passed for `what`.
+        This method is implemented in the base class and relies on `list_collection_content`.
+        Child classes should not need to override this method if `list_collection_content` is implemented correctly.
         """
         collection_content = await self.list_collection_content(what=what)
         match what:
@@ -74,6 +81,11 @@ async def count(self, what: ResultType = ResultType.chunk) -> int:
     async def query(
         self,
     ) -> list[QueryResult]:
+        """
+        Query the database for similar chunks.
+        The query keywords are stored in `self._configs.query`.
+        The implementation of this method should handle the conversion from the native database query result to a list of `vectorcode.database.types.QueryResult` objects.
+        """
         pass
 
     @abstractmethod
@@ -85,13 +97,21 @@ async def vectorise(
         """
         Vectorise the given file and add it to the database.
         The duplicate checking (using file hash) should be done outside of this function.
+
+        For developers:
+        The implementation should chunk the file, generate embeddings for the chunks, and store them in the database.
+        It should return a `VectoriseStats` object to report the outcome.
         """
         pass
 
     @abstractmethod
     async def list_collections(self) -> Sequence[CollectionInfo]:
         """
-        List collections in the database.
+        List all collections available in the database.
+
+        For developers:
+        The implementation should retrieve all collections and return them as a sequence of `CollectionInfo` objects.
+        This includes metadata about each collection like its ID, path, and size.
         """
         pass
 
@@ -119,6 +139,10 @@ async def delete(
         """
         Delete files from the database (doesn't remove files on disk).
         Returns the actual number of files deleted.
+
+        For developers:
+        The file paths to be deleted are stored in `self._configs.rm_paths`.
+        The implementation should remove all chunks associated with these files from the database.
         """
         pass
 
@@ -127,20 +151,29 @@ async def drop(
         self, *, collection_id: str | None = None, collection_path: str | None = None
     ):
         """
-        Delete a collection (`self._configs.project_root`) from the database.
+        Delete a collection from the database.
+        The collection to be dropped is specified by `collection_id` or `collection_path`.
+        If not provided, it defaults to `self._configs.project_root`.
         """
         pass
 
     def _check_new_config(self, new_config: Config) -> bool:
         """
-        Cleanup the `new_config` so that the database config matches the existing one.
+        Ensures that the new config does not attempt to change database-specific settings.
+        It copies the `db_type` and `db_params` from the existing config to the new one.
+        This is a helper method for `update_config` and `replace_config`.
         """
         assert isinstance(new_config, Config), "`new_config` is not a `Config` object."
         new_config.db_type = self._configs.db_type
         new_config.db_params = self._configs.db_params
         return True
 
     async def update_config(self, new_config: Config) -> Self:
+        """
+        Merge the new config with the existing one.
+        This method will not change the database configs.
+        Child classes should not need to override this method.
+        """
         assert self._check_new_config(new_config), (
             "The new config has different database configs."
         )
@@ -150,6 +183,11 @@ async def update_config(self, new_config: Config) -> Self:
         return self
 
     async def replace_config(self, new_config: Config) -> Self:
+        """
+        Replace the existing config with the new one.
+        This method will not change the database configs.
+        Child classes should not need to override this method.
+        """
         assert self._check_new_config(new_config), (
             "The new config has different database configs."
         )
@@ -158,7 +196,10 @@ async def replace_config(self, new_config: Config) -> Self:
 
     async def check_orphanes(self) -> int:
         """
-        Check for files that are in the database, but no longer on the disk, and remove them.
+        Check for files that are in the database but no longer on disk, and remove them.
+        Returns the number of orphaned files removed.
+        This method relies on `list_collection_content` and `delete`.
+        Child classes should not need to override this.
         """
 
         orphanes: list[str] = []
@@ -178,7 +219,10 @@ async def check_orphanes(self) -> int:
 
     def get_embedding(self, texts: str | list[str]) -> list[NDArray]:
         """
-        Generate embeddings and truncate them to `self._configs.embedding_dims` if needed.
+        Generate embeddings for the given texts.
+        It uses the embedding function specified in `self._configs.embedding_function`.
+        If `self._configs.embedding_dims` is set, it truncates the embeddings.
+        Child classes should use this method to get embeddings.
         """
         if isinstance(texts, str):
             texts = [texts]
@@ -194,14 +238,20 @@ def get_embedding(self, texts: str | list[str]) -> list[NDArray]:
     @abstractmethod
     async def get_chunks(self, file_path) -> list[Chunk]:
         """
-        Return chunks for the provided file, if any.
-        If not found, return an empty list.
+        Retrieve all chunks for a given file from the database.
+        If the file is not found in the database, it should return an empty list.
+
+        For developers:
+        This is useful for operations that need to inspect the chunked content of a file, for example, for debugging or analysis.
         """
         pass
 
     async def cleanup(self) -> list[str]:
         """
         Remove empty collections from the database.
+        Returns a list of paths of the removed collections.
+        This method relies on `list_collections` and `drop`.
+        Child classes should not need to override this.
         """
         removed: list[str] = []
         for collection in await self.list_collections():