Refactor vector index creation and querying in Python bindings

- Updated vector index creation to use SQL commands for LSM_VECTOR with metadata.
- Enhanced vector search functionality to support key-based lookups and improved error handling.
- Added metadata retrieval for vector indices, including dimensions, similarity function, and id property.
- Introduced helper methods for querying vectors by key and retrieving index metadata.
- Improved example scripts to reflect changes in vector index creation and querying.
- Updated tests to cover new functionalities, including key-based searches and metadata validation.
- Adjusted example datasets and configurations for better performance and clarity.

Author: tae898

bindings/python/docs/api/database.md

@@ -717,6 +717,17 @@ with db.transaction():
 # Search
 query_vector = np.random.rand(384)
 results = index.find_nearest(query_vector, k=5)
+
+# Preferred when you want richer query composition
+qvec_literal = "[" + ", ".join(str(float(x)) for x in query_vector.tolist()) + "]"
+rows = db.query(
+    "sql",
+    (
+        "SELECT id, distance, (1 - distance) AS score "
+        "FROM (SELECT expand(`vector.neighbors`('Document[embedding]', "
+        f"{qvec_literal}, 5))) ORDER BY distance"
+    ),
+).to_list()
 ```
 
 See [Vector Search Guide](../guide/vectors.md) for details.

bindings/python/docs/api/vector.md

@@ -106,6 +106,11 @@ print(type(py_list))  # <class 'list'>
 
 Wrapper for ArcadeDB's vector index, providing similarity search capabilities.
 
+Creation and configuration fit well in the Python object API. For search, prefer SQL
+or Cypher when you need filtering, projection, self-exclusion, or custom score
+shaping. The Python search methods below are convenience helpers for simple
+embedded-mode workflows.
+
 ### Creation via Database
 
 Vector indexes are created using the `Database.create_vector_index()` method:
@@ -117,6 +122,7 @@ db.create_vector_index(
     vertex_type: str,
     vector_property: str,
     dimensions: int,
+    id_property: str | None = None,
     distance_function: str = "cosine",
     max_connections: int = 16,
     beam_width: int = 100,
@@ -139,6 +145,8 @@ db.create_vector_index(
 - `vertex_type` (str): Vertex type containing vectors
 - `vector_property` (str): Property name storing vector arrays
 - `dimensions` (int): Vector dimensionality (must match your embeddings)
+- `id_property` (str | None): Optional property used for key-based lookup with
+    `find_nearest_by_key()`. Defaults to the engine default (`"id"`) when omitted.
 - `distance_function` (str): Distance metric (default: `"cosine"`)
     - `"cosine"`: Cosine distance (1 - cosine similarity)
     - `"euclidean"`: Euclidean distance (L2 norm)
@@ -186,6 +194,7 @@ index = db.create_vector_index(
     vertex_type="Document",
     vector_property="embedding",
     dimensions=384,  # Match your embedding model
+    id_property="id",
     distance_function="cosine",
     max_connections=16,
     beam_width=100
@@ -247,6 +256,21 @@ for record, distance in neighbors:
     print(f"  Text: {text[:100]}...")
 ```
 
+Preferred for richer query behavior:
+
+```python
+qvec_literal = "[" + ", ".join(str(float(x)) for x in query_vector.tolist()) + "]"
+rows = db.query(
+    "sql",
+    (
+        "SELECT id, distance, (1 - distance) AS score "
+        "FROM (SELECT expand(`vector.neighbors`('Document[embedding]', "
+        f"{qvec_literal}, 10))) WHERE id <> ? ORDER BY distance LIMIT 5"
+    ),
+    "doc-42",
+).to_list()
+```
+
 **Distance Interpretation:**
 
 | Function | Range | Similarity direction |
@@ -261,6 +285,67 @@ for record, distance in neighbors:
 
 ---
 
+### `VectorIndex.find_nearest_by_key(key, k=10, ef_search=None, allowed_rids=None)`
+
+Find nearest neighbors by reusing the vector stored on an existing record.
+
+This is the Python wrapper for the common "search from an existing record" workflow,
+using the index's configured `id_property` to look up the source vector first.
+
+**Parameters:**
+
+- `key`: Value of the configured ID property
+- `k` (int): Number of neighbors to return (default: 10)
+- `ef_search` (int | None): Optional exact-search beam width override
+- `allowed_rids` (List[str] | None): Optional RID whitelist to restrict search
+
+**Returns:**
+
+- `List[Tuple[record, float]]`: Same shape as `find_nearest()`
+
+**Example:**
+
+```python
+neighbors = index.find_nearest_by_key("doc-42", k=5)
+
+for record, distance in neighbors:
+    print(record.get("id"), distance)
+```
+
+The helper keeps current nearest-neighbor semantics, so the source record may also be
+returned. If you want to exclude it, do that in SQL/Cypher with a `WHERE` clause.
+
+---
+
+### `VectorIndex.get_metadata()`
+
+Return stable vector index metadata as a Python dictionary.
+
+**Returns:**
+
+- `dict` with keys such as:
+    - `index_name`
+    - `bucket_index_name`
+    - `type_name`
+    - `vector_property`
+    - `dimensions`
+    - `similarity_function`
+    - `id_property`
+    - `quantization`
+    - `max_connections`
+    - `beam_width`
+    - `store_vectors_in_graph`
+    - `build_state`
+
+**Example:**
+
+```python
+meta = index.get_metadata()
+print(meta["dimensions"], meta["similarity_function"], meta["id_property"])
+```
+
+---
+
 ### `VectorIndex.build_graph_now()`
 
 Force an immediate rebuild/preparation of the vector graph.

bindings/python/docs/examples/03_vector_search.md

@@ -78,24 +78,39 @@ index = db.create_vector_index(
 
 ### 5. Semantic Search
 
-Find the k most similar documents to a query embedding:
+Find the k most similar documents to a query embedding with SQL nearest-neighbor
+queries. This keeps search in the query layer, where filtering and score shaping are
+easy to express.
 
 ```python
 query_embedding = create_mock_embedding(category, "query")
 qvec_literal = "[" + ", ".join(str(float(x)) for x in query_embedding.tolist()) + "]"
 rows = db.query(
-    "sql",
-    f"SELECT vectorNeighbors('Article[embedding]', {qvec_literal}, 5) as res",
+   "sql",
+   (
+      "SELECT title, category, distance, (1 - distance) AS score "
+      "FROM (SELECT expand(`vector.neighbors`('Article[embedding]', "
+      f"{qvec_literal}, 5))) ORDER BY distance"
+   ),
 ).to_list()
 
-for hit in rows[0].get("res", []):
-    vertex = hit.get("record")
-    distance = hit.get("distance")
-    if vertex is not None:
-        print(f"{vertex.get('title')}: {distance:.4f}")
+for hit in rows:
+   print(f"{hit.get('title')}: {hit.get('distance'):.4f}")
 ```
 
-The `find_nearest()` method returns (vertex, distance) pairs sorted by distance.
+The example also shows a filtered query in the same category:
+
+```python
+filtered_rows = db.query(
+   "sql",
+   (
+      "SELECT title, category, distance, (1 - distance) AS score "
+      "FROM (SELECT expand(`vector.neighbors`('Article[embedding]', "
+      f"{qvec_literal}, 50))) WHERE category = ? ORDER BY distance LIMIT 5"
+   ),
+   category,
+).to_list()
+```
 
 ## Example Output
 

bindings/python/docs/examples/vectors.md

@@ -115,7 +115,7 @@ with arcadedb.open_database("./vector_demo") as db:
         print(f"{record.get('name')}: {record.get('distance'):.4f}")
 ```
 
-#### SQL nearest-neighbor (preferred for DSL-first code):
+#### SQL nearest-neighbor (preferred for query-first code):
 
 ```python
 import arcadedb_embedded as arcadedb
@@ -135,6 +135,20 @@ with arcadedb.open_database("./vector_demo") as db:
             print(f"{record.get('name')}: {distance:.4f}")
 ```
 
+#### SQL filtered search with score shaping:
+
+```python
+rows = db.query(
+    "sql",
+    (
+        "SELECT name, description, distance, (1 - distance) AS score "
+        "FROM (SELECT expand(`vector.neighbors`('Product[embedding]', "
+        f"{qvec_literal}, 20))) WHERE name <> ? ORDER BY distance LIMIT 5"
+    ),
+    "Laptop",
+).to_list()
+```
+
 ## Vector Functions
 
 ArcadeDB provides several vector functions:

bindings/python/docs/guide/vectors.md

@@ -39,16 +39,30 @@ with arcadedb.create_database("./vector_demo") as db:
                 to_java_float_array(embedding),
             )
 
-    results = index.find_nearest([0.9, 0.1, 0.0], k=2)
-    for vertex, score in results:
-        print(vertex.get("text"), score)
+    rows = db.query(
+        "sql",
+        "SELECT vectorNeighbors('Doc[embedding]', [0.9, 0.1, 0.0], 2) as res",
+    ).to_list()
+    for hit in rows[0].get("res", []):
+        record = hit.get("record")
+        if record is not None:
+            print(record.get("text"), hit.get("distance"))
 ```
 
 ## API Essentials
 
+Preferred split:
+
+- Use Python object API for vector index creation and configuration.
+- Prefer SQL or Cypher for vector retrieval/search, because search composes naturally
+  with filters, projections, and graph traversal.
+- Treat `find_nearest()` and `find_nearest_by_key()` as convenience wrappers for
+  simple embedded-mode workflows.
+
 - Vector property type must be `ARRAY_OF_FLOATS`.
 - `create_vector_index(vertex_type, vector_property, dimensions,
-  distance_function="cosine", max_connections=16, beam_width=100, quantization="INT8",
+  id_property=None, distance_function="cosine", max_connections=16,
+  beam_width=100, quantization="INT8",
   location_cache_size=None, graph_build_cache_size=None, mutations_before_rebuild=None,
   store_vectors_in_graph=False, add_hierarchy=True, pq_subspaces=None, pq_clusters=None,
   pq_center_globally=None, pq_training_limit=None, build_graph_now=True)`
@@ -60,6 +74,11 @@ with arcadedb.create_database("./vector_demo") as db:
     - `ef_search` optionally overrides the exact-search beam width.
     - Leave it as `None` to use ArcadeDB's default/adaptive behavior.
     - `allowed_rids` filters candidates server-side (useful for metadata-prefilter).
+- `find_nearest_by_key(key, k=10, ef_search=None, allowed_rids=None)`
+    - Looks up the source vector by the index `id_property` and then runs the same
+      Python search path as `find_nearest()`.
+- `get_metadata()` returns stable index metadata such as dimensions, similarity
+  function, configured `id_property`, quantization, and cache/build settings.
 
 ## Distance Functions (scoring behavior)
 
@@ -161,6 +180,93 @@ rids = [row.get_rid() for row in db.query("sql", "SELECT @rid FROM Doc WHERE top
 results = index.find_nearest(query_vec, k=5, allowed_rids=rids)
 ```
 
+## Preferred Search Surface: SQL / Cypher
+
+For new code, prefer query APIs for search.
+
+### SQL filtered vector search with score shaping
+
+```python
+qvec_literal = "[" + ", ".join(str(float(x)) for x in query_vec) + "]"
+
+rows = db.query(
+  "sql",
+  (
+    "SELECT title, category, distance, (1 - distance) AS score "
+    "FROM (SELECT expand(`vector.neighbors`('Article[embedding]', "
+    f"{qvec_literal}, 50))) WHERE category = ? ORDER BY distance LIMIT 5"
+  ),
+  "category_42",
+).to_list()
+```
+
+### SQL self-exclusion
+
+```python
+rows = db.query(
+  "sql",
+  (
+    "SELECT title, distance, (1 - distance) AS score "
+    "FROM (SELECT expand(`vector.neighbors`('Movie[embedding]', "
+    f"{qvec_literal}, 20))) WHERE title <> ? ORDER BY distance LIMIT 10"
+  ),
+  movie_title,
+).to_list()
+```
+
+### Cypher search with score shaping
+
+```python
+rows = db.query(
+  "opencypher",
+  (
+    "CALL vector.neighbors('Doc[embedding]', $vec, $k) "
+    "YIELD name, distance RETURN name, (1 - distance) AS score ORDER BY score DESC"
+  ),
+  {"vec": query_vec, "k": 5},
+).to_list()
+```
+
+## Search from an Existing Record
+
+```python
+with arcadedb.create_database("./vector_demo") as db:
+  db.command("sql", "CREATE VERTEX TYPE Doc")
+  db.command("sql", "CREATE PROPERTY Doc.slug STRING")
+  db.command("sql", "CREATE PROPERTY Doc.embedding ARRAY_OF_FLOATS")
+
+  index = db.create_vector_index(
+    vertex_type="Doc",
+    vector_property="embedding",
+    dimensions=3,
+    id_property="slug",
+  )
+
+  with db.transaction():
+    db.command(
+      "sql",
+      "INSERT INTO Doc SET slug = ?, embedding = ?",
+      "doc-a",
+      to_java_float_array([1.0, 0.0, 0.0]),
+    )
+    db.command(
+      "sql",
+      "INSERT INTO Doc SET slug = ?, embedding = ?",
+      "doc-b",
+      to_java_float_array([0.9, 0.1, 0.0]),
+    )
+
+  neighbors = index.find_nearest_by_key("doc-a", k=2)
+  metadata = index.get_metadata()
+
+  print(metadata["dimensions"], metadata["id_property"])
+  for record, distance in neighbors:
+    print(record.get("slug"), distance)
+```
+
+Use this helper when you want a small embedded-mode shortcut. For richer filtering,
+projection, self-exclusion, or score shaping, prefer SQL/Cypher queries.
+
 ## Quantization
 
 - `quantization` accepts `"INT8"`, `"BINARY"`, `"PRODUCT"` (PQ), or `None` (full precision).