Skip to content

Commit f9ea10d

Browse files
committed
feat(vector): add PgVectorStore implementation with async support
1 parent 0218c3a commit f9ea10d

3 files changed

Lines changed: 79 additions & 108 deletions

File tree

docs/zh/guide/vector_store_guide.md

Lines changed: 65 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ FlowLLM 提供了多种 Vector Store 实现,适用于不同的使用场景:
1616

1717
- **LocalVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/local_vector_store.py)):基于本地文件的实现,使用 JSONL 格式持久化存储,适合单机部署和小规模数据
1818
- **MemoryVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/memory_vector_store.py)):内存实现,数据存储在内存中,访问速度快,适合临时数据或测试场景
19+
- **PgVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/pgvector_vector_store.py)):基于 PostgreSQL 和 pgvector 扩展的实现,支持同步和异步操作,适合已有 PostgreSQL 基础设施的场景
1920
- **QdrantVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/qdrant_vector_store.py)):基于 Qdrant 向量数据库,支持高性能向量搜索,适合大规模生产环境
2021
- **ChromaVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/chroma_vector_store.py)):基于 ChromaDB 的实现,提供持久化存储和元数据过滤能力
2122
- **EsVectorStore**[代码路径](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/es_vector_store.py)):基于 Elasticsearch 的实现,支持强大的全文搜索和向量搜索组合
@@ -77,6 +78,12 @@ FlowLLM 提供了多种 Vector Store 实现,适用于不同的使用场景:
7778

7879
- **store_dir**:持久化存储目录,默认 `./memory_vector_store`
7980

81+
### PgVectorStore 配置
82+
83+
- **connection_string**:PostgreSQL 连接字符串(同步操作),默认从环境变量 `FLOW_PGVECTOR_CONNECTION_STRING` 读取,或使用 `postgresql://localhost/postgres`
84+
- **async_connection_string**:PostgreSQL 连接字符串(异步操作,可选),默认从环境变量 `FLOW_PGVECTOR_ASYNC_CONNECTION_STRING` 读取,如果未提供则基于 `connection_string` 自动转换
85+
- **batch_size**:批量操作大小,默认 `1024`
86+
8087
### QdrantVectorStore 配置
8188

8289
- **url**:Qdrant 服务地址(可选,用于 Qdrant Cloud 或自定义部署)
@@ -109,7 +116,7 @@ vector_store:
109116

110117
### 配置字段说明
111118

112-
- **`backend`**(必需):向量库后端类型,可选值:`local``memory``chroma``qdrant``elasticsearch`
119+
- **`backend`**(必需):向量库后端类型,可选值:`local``memory``pgvector``chroma``qdrant``elasticsearch`
113120
- **`embedding_model`**(必需):嵌入模型配置名称,引用 `embedding_model` 部分的配置
114121
- **`params`**(可选):后端特定参数字典,将传递给向量库构造函数
115122

@@ -160,7 +167,54 @@ vector_store:
160167
store_dir: "./chroma_vector_store" # ChromaDB 数据目录(可选,默认:"./chroma_vector_store")
161168
```
162169

163-
#### 4. QdrantVectorStore 配置
170+
#### 4. PgVectorStore 配置
171+
172+
基于 PostgreSQL 和 pgvector 扩展的向量存储,支持同步和异步操作,适合已有 PostgreSQL 基础设施的场景。
173+
174+
**代码实现**:[`flowllm/core/vector_store/pgvector_vector_store.py`](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/pgvector_vector_store.py)
175+
176+
**前置要求**:需要安装 PostgreSQL 并启用 pgvector 扩展。
177+
178+
**基本配置(使用环境变量):**
179+
180+
```yaml
181+
vector_store:
182+
default:
183+
backend: pgvector
184+
embedding_model: default
185+
params:
186+
# connection_string 和 async_connection_string 可通过环境变量配置
187+
# FLOW_PGVECTOR_CONNECTION_STRING=postgresql://user:password@localhost/dbname
188+
# FLOW_PGVECTOR_ASYNC_CONNECTION_STRING=postgresql://user:password@localhost/dbname
189+
batch_size: 1024 # 批量操作大小(可选,默认:1024)
190+
```
191+
192+
**显式配置连接字符串:**
193+
194+
```yaml
195+
vector_store:
196+
default:
197+
backend: pgvector
198+
embedding_model: default
199+
params:
200+
connection_string: "postgresql://user:password@localhost:5432/dbname" # 同步连接字符串
201+
async_connection_string: "postgresql://user:password@localhost:5432/dbname" # 异步连接字符串(可选)
202+
batch_size: 1024
203+
```
204+
205+
**仅配置同步连接(异步将自动转换):**
206+
207+
```yaml
208+
vector_store:
209+
default:
210+
backend: pgvector
211+
embedding_model: default
212+
params:
213+
connection_string: "postgresql://user:password@localhost:5432/dbname"
214+
batch_size: 1024
215+
```
216+
217+
#### 5. QdrantVectorStore 配置
164218

165219
**代码实现**:[`flowllm/core/vector_store/qdrant_vector_store.py`](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/qdrant_vector_store.py)
166220

@@ -190,7 +244,7 @@ vector_store:
190244
distance: "COSINE"
191245
```
192246

193-
#### 5. EsVectorStore 配置
247+
#### 6. EsVectorStore 配置
194248

195249
**代码实现**:[`flowllm/core/vector_store/es_vector_store.py`](https://github.com/flowllm-ai/flowllm/blob/main/flowllm/core/vector_store/es_vector_store.py)
196250

@@ -257,6 +311,9 @@ vector_store:
257311

258312
部分 Vector Store 支持通过环境变量进行配置,作为 YAML 配置的补充:
259313

314+
- **PgVectorStore**:
315+
- `FLOW_PGVECTOR_CONNECTION_STRING` - PostgreSQL 同步连接字符串(默认:`postgresql://localhost/postgres`)
316+
- `FLOW_PGVECTOR_ASYNC_CONNECTION_STRING` - PostgreSQL 异步连接字符串(可选,未提供时基于同步连接字符串自动转换)
260317
- **Elasticsearch**:`FLOW_ES_HOSTS` - Elasticsearch 主机地址
261318
- **Qdrant**:
262319
- `FLOW_QDRANT_HOST` - Qdrant 主机(默认:`localhost`)
@@ -277,14 +334,17 @@ vector_store:
277334

278335
- **开发测试**:使用 MemoryVectorStore 或 LocalVectorStore,无需额外服务
279336
- **小规模应用**:使用 LocalVectorStore 或 ChromaVectorStore,简单易用
280-
- **生产环境**:使用 QdrantVectorStore 或 EsVectorStore,提供高性能和可扩展性
337+
- **已有 PostgreSQL 基础设施**:使用 PgVectorStore,利用现有数据库资源,支持同步和异步操作
338+
- **生产环境**:使用 PgVectorStore、QdrantVectorStore 或 EsVectorStore,提供高性能和可扩展性
281339
- **混合搜索**:使用 EsVectorStore,结合向量搜索和全文搜索能力
282340

283341
## 注意事项
284342

285343
- 确保嵌入模型的维度与 Vector Store 配置一致
286-
- 大规模数据建议使用专业的向量数据库(Qdrant、Elasticsearch)
344+
- 使用 PgVectorStore 前需要确保 PostgreSQL 已安装并启用 pgvector 扩展(`CREATE EXTENSION vector`)
345+
- 大规模数据建议使用专业的向量数据库(PgVectorStore、Qdrant、Elasticsearch)
287346
- 异步接口在异步环境中能提供更好的性能
288347
- 定期备份重要数据,特别是使用内存存储时
289348
- 根据数据规模选择合适的批量大小以优化性能
349+
- PgVectorStore 会自动为每个工作空间创建向量索引(IVFFlat)和元数据索引(GIN),以优化查询性能
290350

docs/zh/todo.md

Lines changed: 0 additions & 20 deletions
This file was deleted.

flowllm/core/vector_store/pgvector_vector_store.py

Lines changed: 14 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -267,22 +267,11 @@ def _row2node(row: Tuple, workspace_id: str) -> VectorNode:
267267
import json
268268

269269
# pgvector returns vector as string like '[0.1,0.2,0.3]'
270-
# Try JSON parsing first, if fails, parse manually
271-
try:
272-
vector = json.loads(vector_str)
273-
except (json.JSONDecodeError, TypeError):
274-
# Fallback: parse manually if format is '[0.1,0.2,0.3]'
275-
if isinstance(vector_str, str) and vector_str.startswith("[") and vector_str.endswith("]"):
276-
vector = [float(x.strip()) for x in vector_str[1:-1].split(",")]
277-
else:
278-
vector = vector_str
270+
vector = json.loads(vector_str)
279271

280272
# Parse metadata if it's a string (psycopg may return JSONB as string in some cases)
281273
if isinstance(metadata, str):
282-
try:
283-
metadata = json.loads(metadata)
284-
except (json.JSONDecodeError, TypeError):
285-
metadata = {}
274+
metadata = json.loads(metadata)
286275
elif metadata is None:
287276
metadata = {}
288277

@@ -428,21 +417,11 @@ def search(
428417
import json
429418

430419
# pgvector returns vector as string like '[0.1,0.2,0.3]'
431-
try:
432-
vector = json.loads(vector_str)
433-
except (json.JSONDecodeError, TypeError):
434-
# Fallback: parse manually if format is '[0.1,0.2,0.3]'
435-
if isinstance(vector_str, str) and vector_str.startswith("[") and vector_str.endswith("]"):
436-
vector = [float(x.strip()) for x in vector_str[1:-1].split(",")]
437-
else:
438-
vector = vector_str
420+
vector = json.loads(vector_str)
439421

440422
# Parse metadata if it's a string (psycopg may return JSONB as string in some cases)
441423
if isinstance(metadata, str):
442-
try:
443-
metadata = json.loads(metadata)
444-
except (json.JSONDecodeError, TypeError):
445-
metadata = {}
424+
metadata = json.loads(metadata)
446425
elif metadata is None:
447426
metadata = {}
448427

@@ -562,15 +541,9 @@ async def _get_async_conn(self):
562541
# If it doesn't start with postgresql://, assume it's a standard connection string
563542
pass
564543

565-
try:
566-
logger.debug(f"Establishing async PostgreSQL connection: {conn_str}")
567-
self._async_conn = await asyncpg.connect(conn_str)
568-
logger.debug(f"Async PostgreSQL connection established successfully")
569-
except Exception as e:
570-
logger.error(f"Failed to establish async PostgreSQL connection to {conn_str}: {e}")
571-
import traceback
572-
logger.error(traceback.format_exc())
573-
raise
544+
logger.debug(f"Establishing async PostgreSQL connection: {conn_str}")
545+
self._async_conn = await asyncpg.connect(conn_str)
546+
logger.debug(f"Async PostgreSQL connection established successfully")
574547
return self._async_conn
575548

576549
async def async_exist_workspace(self, workspace_id: str, **kwargs) -> bool:
@@ -604,44 +577,12 @@ async def async_delete_workspace(self, workspace_id: str, **kwargs):
604577
workspace_id: The identifier of the workspace/table to delete.
605578
**kwargs: Additional keyword arguments (unused).
606579
"""
607-
import asyncio
608-
609580
table_name = self._get_table_name(workspace_id)
610-
try:
611-
logger.debug(f"Attempting to delete workspace table: {table_name}")
612-
conn = await self._get_async_conn()
613-
logger.debug(f"Connection established, executing DROP TABLE for {table_name}")
614-
615-
# Try to execute DROP TABLE with a short timeout
616-
# If it fails due to locks, we'll fall back to sync connection
617-
await asyncio.wait_for(
618-
conn.execute(f'DROP TABLE IF EXISTS "{table_name}" CASCADE'),
619-
timeout=5.0 # Short timeout, will fallback to sync if needed
620-
)
621-
logger.info(f"Successfully deleted workspace table: {table_name}")
622-
except (asyncio.TimeoutError, Exception) as e:
623-
# If async delete fails or times out, try using sync connection instead
624-
# This often works because the sync connection can delete tables it has access to
625-
# and won't conflict with itself
626-
if isinstance(e, asyncio.TimeoutError):
627-
logger.warning(f"Async delete timed out for {table_name}, trying sync delete as fallback")
628-
else:
629-
logger.warning(f"Async delete failed for {table_name}: {e}, trying sync delete as fallback")
630-
try:
631-
# Use sync connection to delete (it may have better access and won't conflict)
632-
with self._conn.cursor() as cur:
633-
cur.execute(f'DROP TABLE IF EXISTS "{table_name}" CASCADE')
634-
self._conn.commit()
635-
logger.info(f"Successfully deleted workspace table {table_name} using sync connection")
636-
except Exception as sync_error:
637-
error_msg = f"Both async and sync delete failed for workspace {workspace_id} (table: {table_name}). The table may be locked by another connection. Async error: {e}, Sync error: {sync_error}"
638-
logger.error(error_msg)
639-
raise TimeoutError(error_msg) from e
640-
except Exception as e:
641-
logger.error(f"Failed to delete workspace {workspace_id} (table: {table_name}): {e}")
642-
import traceback
643-
logger.error(traceback.format_exc())
644-
raise
581+
logger.debug(f"Attempting to delete workspace table: {table_name}")
582+
conn = await self._get_async_conn()
583+
logger.debug(f"Connection established, executing DROP TABLE for {table_name}")
584+
await conn.execute(f'DROP TABLE IF EXISTS "{table_name}" CASCADE')
585+
logger.info(f"Successfully deleted workspace table: {table_name}")
645586

646587
async def async_create_workspace(self, workspace_id: str, **kwargs):
647588
"""Create a new PostgreSQL table (workspace) with vector field (async).
@@ -776,21 +717,11 @@ async def async_search(
776717
import json
777718

778719
# pgvector returns vector as string like '[0.1,0.2,0.3]'
779-
try:
780-
vector = json.loads(vector_str)
781-
except (json.JSONDecodeError, TypeError):
782-
# Fallback: parse manually if format is '[0.1,0.2,0.3]'
783-
if isinstance(vector_str, str) and vector_str.startswith("[") and vector_str.endswith("]"):
784-
vector = [float(x.strip()) for x in vector_str[1:-1].split(",")]
785-
else:
786-
vector = vector_str
720+
vector = json.loads(vector_str)
787721

788722
# Parse metadata if it's a string (asyncpg may return JSONB as string)
789723
if isinstance(metadata, str):
790-
try:
791-
metadata = json.loads(metadata)
792-
except (json.JSONDecodeError, TypeError):
793-
metadata = {}
724+
metadata = json.loads(metadata)
794725
elif metadata is None:
795726
metadata = {}
796727

0 commit comments

Comments
 (0)