Skip to content

fix(storage): 优化路径锁与语义刷新并发#2029

Merged
qin-ctx merged 10 commits into
mainfrom
fix/path-lock-semantic-refresh
May 14, 2026
Merged

fix(storage): 优化路径锁与语义刷新并发#2029
qin-ctx merged 10 commits into
mainfrom
fix/path-lock-semantic-refresh

Conversation

@qin-ctx

@qin-ctx qin-ctx commented May 14, 2026

Copy link
Copy Markdown
Collaborator

Description

这次 PR 解决的是本地压测中暴露出来的路径锁粒度问题:同一个目录下并发添加不同资源时,过去会因为父目录级锁过粗而互相阻塞,极端情况下返回 500。核心调整是把“源数据提交”和“语义派生刷新”拆成两件事:

  • 源数据提交只保护最终路径本身,例如 docs/a.md 只拿 ExactPathLock(docs/a.md),不再因为同目录还有 docs/b.md 就互斥。
  • 目录删除、目录移动、资源生命周期、memory 变量文件名 schema 这类确实需要子树一致性的场景,继续使用 TreeLock
  • .overview.md / .abstract.md 这类派生文件不参与前台源数据提交锁,后台刷新通过 coalesce version 淘汰旧任务,最终写回时再短暂拿派生文件自己的 ExactPathLock

举例:并发添加 viking://resources/docs/a.mdviking://resources/docs/b.mdviking://resources/docs/c.md 时,三个请求分别锁 a.mdb.mdc.md,可以并发提交;后台可能产生多个 docs/ 摘要刷新任务,但只有最新版本会写回 docs/.overview.mddocs/.abstract.md,旧任务写回前发现过期会丢弃结果。

同时补齐了 memory / agent memory 的锁边界。固定文件名 schema,例如 profile.mdidentity.md,使用具体文件的 ExactPathLock;变量文件名 schema,例如 events/{{ name }}.mdexperiences/{{ experience_name }}.md,使用 schema 目录的 TreeLock。agent experience 的 source_trajectories 回填也被纳入 experience schema 锁作用域,避免出现“experience 内容已经更新,但来源轨迹 metadata 还没补上”的中间状态被另一个 extraction 读取。

Related Issue

N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • 重构路径锁语义:引入 PathLockEngineExactPathLock(path)TreeLock(path),移除业务层对 point/source/subtree 命名的依赖,并统一 LockContext / LockManager 的 exact/tree/mv 调用方式。
  • 优化 add_resource 提交流程:TreeBuilder 只构建 final/temp URI 元信息,ResourceProcessor 在短 ExactPathLock(final_path) 内提交源数据;首次落盘后再交给生命周期 TreeLock(final_path) 保护后台解析、摘要、向量化期间的资源不被删除或移动。
  • 优化派生摘要刷新:SemanticMsg 增加 coalesce 信息,SemanticQueue 维护运行时版本;资源父目录摘要、memory 目录摘要、直接 write 后刷新都会淘汰旧任务,并在写 .overview.md / .abstract.md 时使用派生文件自己的 ExactPathLock
  • 收敛 memory / agent memory 锁模型:固定文件 schema 用具体文件 exact lock,变量文件 schema 用 schema 目录 tree lock;agent experience 的 source_trajectories 系统 metadata 回填在 schema 锁释放前完成,fallback 路径也用短 exact lock 保护 read-modify-write。
  • 补充结构化冲突错误、SDK/CLI 文档说明、事务中文/英文文档、技术方案文档,并新增中文压测框架,覆盖 SDK、CLI HTTP、CLI subprocess 的资源添加、检索、session message、commit 和混合请求场景。
  • 增加和更新锁、资源处理、semantic queue、memory extraction、错误映射、压测脚本相关测试,覆盖 exact lock 与 tree lock 的冲突关系、语义刷新去重、agent memory 锁内回填等行为。

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

本地执行过的验证:

.venv/bin/ruff check openviking/session/compressor_v2.py tests/session/memory/test_compressor_v2.py
.venv/bin/python -m pytest tests/session/memory/test_compressor_v2.py -q
.venv/bin/python -m pytest tests/transaction/test_exact_path_lock.py tests/transaction/test_lock_reentrancy_unit.py tests/storage/test_semantic_queue_memory_dedupe.py tests/storage/test_semantic_processor_lifecycle_lock.py tests/session/memory/test_compressor_v2.py tests/misc/test_resource_processor_mv.py tests/server/test_content_write_service.py tests/server/test_request_wait_tracking.py -q
git diff --cached --check

提交时 pre-commit 也通过了:

ruff (legacy alias)
ruff format

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

N/A

Additional Notes

这次 PR 里的“最新版本判断”先使用运行时 coalesce version,没有引入持久化 dir_revision / exact_revision。这样已经可以解决同进程并发刷新中旧 .overview.md 覆盖新 .overview.md 的问题;跨重启的持久化版本控制可以后续单独设计。

压测脚本默认指向 http://127.0.0.1:1935,用于配合本地 ov.conf 启动的 OpenViking server。脚本会输出中文 Markdown 报告、JSONL 明细和 CSV 汇总,便于比较 SDK 与 CLI 路径的并发表现。

Use exact path locks for source commits, tree locks only for lifecycle and schema scopes, and coalesce derived semantic writes to avoid stale summary overwrites under concurrent resource and memory updates.
@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🏅 Score: 85
🧪 PR contains tests
🔒 No security concerns identified
✅ No TODO sections
🔀 Multiple PR themes

Sub-PR theme: Add mixed SDK/CLI load benchmark

Relevant files:

  • benchmark/custom/openviking_server_load_benchmark.py
  • benchmark/custom/README.md
  • tests/unit/test_openviking_server_load_benchmark.py

Sub-PR theme: Refactor path lock engine (rename methods, add exact path lock)

Relevant files:

  • openviking/storage/transaction/lock_manager.py
  • openviking/storage/transaction/path_lock.py
  • openviking/storage/transaction/lock_context.py
  • tests/transaction/test_path_lock.py
  • tests/transaction/test_exact_path_lock.py
  • tests/transaction/test_concurrent_lock.py
  • tests/transaction/test_lock_context.py
  • tests/transaction/test_e2e.py
  • tests/transaction/test_lock_manager.py
  • tests/transaction/test_lock_reentrancy_unit.py

Sub-PR theme: Optimize semantic refresh with coalescing and parent refresh

Relevant files:

  • openviking/storage/queuefs/semantic_processor.py
  • openviking/storage/queuefs/semantic_queue.py
  • openviking/storage/queuefs/semantic_msg.py
  • openviking/storage/queuefs/semantic_dag.py
  • tests/storage/test_semantic_queue_memory_dedupe.py
  • tests/storage/test_semantic_processor_lifecycle_lock.py

⚡ Recommended focus areas for review

Missing logging on exception

The except block catching all exceptions when importing lock manager doesn't log the error, which could hide issues in production.

try:
    from openviking.storage.transaction import LockContext, get_lock_manager

    lock_manager = get_lock_manager()
except Exception:
    await viking_fs.write_file(f"{dir_uri}/.overview.md", overview, ctx=ctx)
    await viking_fs.write_file(f"{dir_uri}/.abstract.md", abstract, ctx=ctx)
    return True

@github-actions

Copy link
Copy Markdown

PR Code Suggestions ✨

No code suggestions found for the PR.

@qin-ctx qin-ctx changed the title fix(storage): 优化路径锁与语义刷新并发 WIP fix(storage): 优化路径锁与语义刷新并发 May 14, 2026
@qin-ctx qin-ctx changed the title WIP fix(storage): 优化路径锁与语义刷新并发 fix(storage): 优化路径锁与语义刷新并发 May 14, 2026
@qin-ctx qin-ctx merged commit 77b604a into main May 14, 2026
7 checks passed
@qin-ctx qin-ctx deleted the fix/path-lock-semantic-refresh branch May 14, 2026 12:44
@github-project-automation github-project-automation Bot moved this from Backlog to Done in OpenViking project May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants