Skip to content

Commit dd83be4

Browse files
talorickylewanginchina
authored andcommitted
docs: add ai related context
1 parent 54b400e commit dd83be4

6 files changed

Lines changed: 336 additions & 0 deletions

File tree

.ai/agents.md

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# deepflow-app 项目
2+
3+
## 项目简介
4+
5+
deepflow-app 是 DeepFlow 的后端应用服务,核心功能是**分布式追踪火焰图计算**:将 DeepFlow 采集到的原始 L7 Flow 数据,通过多阶段流水线处理,输出一棵完整的 Span 树(火焰图)。
6+
7+
主要入口:[app/app/application/l7_flow_tracing.py](app/app/application/l7_flow_tracing.py)
8+
9+
---
10+
11+
## 技术约束
12+
13+
### 运行时
14+
- **Python 3.10**
15+
- 使用 `pandas.DataFrame` 承载原始 Flow 数据;排序阶段转为 `SpanNode` 对象图
16+
- 异步框架:Sanic(HTTP 服务层);业务逻辑为同步计算
17+
18+
### 配置
19+
- 所有可调参数集中在 [app/app.yaml](app/app.yaml),运行时通过 [app/app/config.py](app/app/config.py)`config` 单例读取
20+
- 新增可配置行为时,**必须**同步在 `app.yaml` 中添加注释说明,且默认值应保持向后兼容(opt-in 原则)
21+
22+
### 向后兼容原则
23+
- **不得破坏已有追踪结果**:新增连接策略时,如果会影响到已有结果,必须默认关闭
24+
- 修改现有连接条件时,需评估对存量用户火焰图结果的影响
25+
26+
---
27+
28+
## 核心架构:追踪流水线
29+
30+
```
31+
Search → Merge → Sort → Prune → Statistics
32+
搜索 合并 排序 裁剪 统计
33+
```
34+
35+
### 1. 搜索(Search)
36+
- 以「入口 Flow」为起点,迭代扩展,基于 `trace_id` / `tcp_seq` / `syscall_trace_id` / `x_request_id` 拉取关联 Flow
37+
- 上限:`config.max_iteration`(默认 30)、`config.l7_tracing_limit`(默认 1000)
38+
- 控制参数:`config.tracing_source`(列表,控制启用哪些扩展维度)
39+
40+
### 2. 合并(Merge)
41+
- 将单向 Flow(只有请求或只有响应)合并成完整会话
42+
-`start_time` 升序,遇到 Response 合并到前置 Request
43+
44+
### 3. 排序(Sort)—— 最复杂阶段
45+
46+
#### SpanSet 构建
47+
48+
| 类型 | 组成 | 关键内部约束 |
49+
|---|---|---|
50+
| **NetworkSpanSet** | 0-1 个 c-p + N 个网络 Span + 0-1 个 s-p | ① 所有 Span 的 `tcp_seq` 必须相等;② 流信息(五元组等)必须相等 |
51+
| **ProcessSpanSet** | 0-1 个 s-p + N 个 App Span + M 个 c-p | ① 所有 Span 的**进程信息**必须相等;② `s-p` 时间必须**完全覆盖** `c-p` |
52+
53+
#### SpanSet 连接(`_connect_process_and_networks`
54+
55+
连接分两阶段执行:
56+
57+
**准确连接阶段(场景 1-6)**:基于强关联证据,依次执行,结果写入 `network_match_parent`
58+
59+
| 场景 | 连接方向 | 核心条件 |
60+
|---|---|---|
61+
| 1 | Process 叶 → NetworkSpanSet 根 | 共享同一 `c-p`,或叶 `span_id` = 网络首 `span_id` |
62+
| 2 | NetworkSpanSet 尾 → Process 根 | 共享同一 `s-p`,或根 `span_id` = 网络尾 `parent_span_id`/`span_id` |
63+
| 3 | Process 叶 → Process 根 | 共享同一 `c-p`,或叶 `span_id` = 根 `parent_span_id`/`span_id` |
64+
| 4 | Process 根 → Process 内任意 |`parent_span_id` = 目标 `span_id` |
65+
| 5 | NetworkSpanSet → NetworkSpanSet | `x_request_id` 匹配 / `span_id` 相同 / gRPC `stream_id` 相同;前者 `response_duration` ≥ 后者 |
66+
| 6 | NetworkSpanSet → NetworkSpanSet(异步 MQ) | `trace_id` 有交集;仅限 `is_async` 的 WebSphereMQ;client 开始时间早于 server |
67+
68+
**通用约束(场景 1-5)**:两 Span 不能属于同一 SpanSet;当在同一 Agent 下时,Parent 的时延必须大于 Child。
69+
70+
**弱关联阶段(场景 7)**:独立 pass,结果写入 `weak_match_parent`,不影响准确连接
71+
72+
| 场景 | 条件 | 配置开关 |
73+
|---|---|---|
74+
| 7 | client 侧叶(无子)→ server 侧根(无父,`is_net_root=True`);`trace_id` 有交集且 `tcp_seq` 不同;叶 `response_duration` ≥ 根 | `span_set_connection_strategies: [net_span_c_to_s_via_trace_id]` |
75+
| 8 | client 侧叶(无子)→ server 侧根(无父,`is_ps_root=True`);`trace_id` 有交集且 `auto_instance``auto_instance_type` 不同;叶子和根的 `agent_id` 是同一个,且叶子时间覆盖根 | `span_set_connection_strategies: [sys_span_s_to_c_via_trace_id]` |
76+
77+
### 4. 裁剪(Prune)
78+
- 存在多棵树时,以「入口 Span 所在的树」为基准,裁剪时钟偏差超出 `host_clock_offset_us` 的树
79+
-`trace_id` 的 Span 不裁剪;通过强关联(`x_request_id` 等)连接的不裁剪
80+
81+
### 5. 统计(Statistics)
82+
- 自顶向下计算每个 Span 的自身时延(Parent 减去一级子节点时延之和)
83+
- 按 AutoService 分组统计服务总时延
84+
85+
---
86+
87+
## Review / Lint 规范
88+
89+
### 添加新连接关系时的必检清单
90+
91+
添加新的 SpanSet 连接场景时(参考 `_connect_process_and_networks`),必须覆盖以下所有开发点:
92+
93+
#### 1. 核心逻辑
94+
- [ ] 连接条件是否清晰定义(方向、字段、阈值)?
95+
- [ ] 是否正确判断 `is_net_root` / `is_net_leaf` / `children_count` / `get_parent_id()` 等 SpanNode 状态?
96+
- [ ] 是否调用 `_same_span_set()` 防止同组首尾互连?
97+
- [ ] 是否检查 `response_duration` 约束(Parent 时延必须 ≥ Child 时延)?
98+
- [ ] `set_parent()` 调用时是否传入了合理的 `reason` 字符串(用于调试日志)?
99+
100+
#### 2. 准确 vs 弱关联判断
101+
- [ ] 新场景属于**准确连接**(强证据:`tcp_seq` / `span_id` / `x_request_id`)还是**弱关联**(推断性)?
102+
- [ ] 弱关联场景必须:
103+
- 写入独立的 `weak_match_parent` dict,在准确连接阶段结束后单独执行
104+
- 通过 `config.span_set_connection_strategies` 配置 opt-in,默认不生效
105+
106+
#### 3. 配置
107+
- [ ] 若新场景需要配置开关,是否在 `app/app/config.py``parse_spec()` 中读取并赋值到 `config` 对象?
108+
- [ ] 是否在 `app/app.yaml``spec` 节添加了带注释的配置项(说明用途、默认值、可选值)?
109+
- [ ] 默认值是否保持向后兼容(不改变现有用户的追踪结果)?
110+
111+
#### 4. 文档
112+
- [ ] 是否更新了 [HOW-TO-GET-SPAN-LIKE-DATA.md](HOW-TO-GET-SPAN-LIKE-DATA.md) 的「SpanSet 连接」章节,新增场景的说明(连接方向、条件、配置项)?
113+
- [ ] 是否更新了 [FlowTracingIssue.md](FlowTracingIssue.md) 中「SpanSet 连接」速查表?
114+
115+
---
116+
117+
### 性能检查
118+
119+
`_connect_process_and_networks` 是 O(N²) 的双层循环,N = 所有 SpanNode 数量(上限受 `l7_tracing_limit` 控制)。添加新场景时:
120+
121+
- [ ] **避免在内层循环中重复计算**:将不变量(如 `get_req_tcp_seq()``get_trace_id_set()`)提前缓存到局部变量
122+
- [ ] **提前剪枝**:将最能过滤候选的条件放在内层循环最前面(fail-fast)
123+
- [ ] **避免额外数据结构**:不在循环内部创建新 list/dict,若需要,用已有的 `flow_index_to_span` / `related_flow_index_map` 等结构
124+
- [ ] **评估最坏情况**:在 `l7_tracing_limit=1000` 规模下,新场景的额外循环次数是否可接受?
125+
126+
---
127+
128+
## Python 代码审查
129+
130+
> 本项目为 Python,建议在完成较大改动后做专项审查。
131+
132+
检查要点:PEP 8 合规性、类型注解、Pythonic 惯用法、潜在的性能问题和安全风险。
133+
134+
---
135+
136+
## 关键文件索引
137+
138+
| 文件 | 作用 |
139+
|---|---|
140+
| [app/app/application/l7_flow_tracing.py](app/app/application/l7_flow_tracing.py) | 追踪流水线全部核心逻辑 |
141+
| [app/app/config.py](app/app/config.py) | 配置解析,`config` 单例 |
142+
| [app/app.yaml](app/app.yaml) | 配置文件模板及注释 |
143+
| [HOW-TO-GET-SPAN-LIKE-DATA.md](HOW-TO-GET-SPAN-LIKE-DATA.md) | 追踪计算原理的完整设计文档 |
144+
| [FlowTracingIssue.md](FlowTracingIssue.md) | 断链诊断知识库与速查表 |
145+
146+
## 关键函数索引
147+
148+
| 函数 | 位置 | 说明 |
149+
|---|---|---|
150+
| `_connect_process_and_networks` | l7_flow_tracing.py ~L3242 | SpanSet 连接,7 个场景 |
151+
| `_same_span_set` | l7_flow_tracing.py ~L3235 | 防止同组首尾互连的工具函数 |
152+
| `merge_flow` | l7_flow_tracing.py ~L1845 | 合并单向 Flow 为会话 |
153+
| `Config.parse_spec` | config.py L17 | 解析 `spec` 配置节 |

.claude/CLAUDE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../.ai/agents.md

.codex/agents.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../.ai/agents.md

.serena/.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
/cache

.serena/project.yml

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# list of languages for which language servers are started; choose from:
2+
# al bash clojure cpp csharp
3+
# csharp_omnisharp dart elixir elm erlang
4+
# fortran fsharp go groovy haskell
5+
# java julia kotlin lua markdown
6+
# matlab nix pascal perl php
7+
# powershell python python_jedi r rego
8+
# ruby ruby_solargraph rust scala swift
9+
# terraform toml typescript typescript_vts vue
10+
# yaml zig
11+
# (This list may be outdated. For the current list, see values of Language enum here:
12+
# https://github.com/oraios/serena/blob/main/src/solidlsp/ls_config.py
13+
# For some languages, there are alternative language servers, e.g. csharp_omnisharp, ruby_solargraph.)
14+
# Note:
15+
# - For C, use cpp
16+
# - For JavaScript, use typescript
17+
# - For Free Pascal/Lazarus, use pascal
18+
# Special requirements:
19+
# - csharp: Requires the presence of a .sln file in the project folder.
20+
# - pascal: Requires Free Pascal Compiler (fpc) and optionally Lazarus.
21+
# When using multiple languages, the first language server that supports a given file will be used for that file.
22+
# The first language is the default language and the respective language server will be used as a fallback.
23+
# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
24+
languages:
25+
- python
26+
27+
# the encoding used by text files in the project
28+
# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
29+
encoding: "utf-8"
30+
31+
# whether to use project's .gitignore files to ignore files
32+
ignore_all_files_in_gitignore: true
33+
34+
# list of additional paths to ignore in all projects
35+
# same syntax as gitignore, so you can use * and **
36+
ignored_paths: []
37+
38+
# whether the project is in read-only mode
39+
# If set to true, all editing tools will be disabled and attempts to use them will result in an error
40+
# Added on 2025-04-18
41+
read_only: false
42+
43+
# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details.
44+
# Below is the complete list of tools for convenience.
45+
# To make sure you have the latest list of tools, and to view their descriptions,
46+
# execute `uv run scripts/print_tool_overview.py`.
47+
#
48+
# * `activate_project`: Activates a project by name.
49+
# * `check_onboarding_performed`: Checks whether project onboarding was already performed.
50+
# * `create_text_file`: Creates/overwrites a file in the project directory.
51+
# * `delete_lines`: Deletes a range of lines within a file.
52+
# * `delete_memory`: Deletes a memory from Serena's project-specific memory store.
53+
# * `execute_shell_command`: Executes a shell command.
54+
# * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced.
55+
# * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type).
56+
# * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
57+
# * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes.
58+
# * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file.
59+
# * `initial_instructions`: Gets the initial instructions for the current project.
60+
# Should only be used in settings where the system prompt cannot be set,
61+
# e.g. in clients you have no control over, like Claude Desktop.
62+
# * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol.
63+
# * `insert_at_line`: Inserts content at a given line in a file.
64+
# * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol.
65+
# * `list_dir`: Lists files and directories in the given directory (optionally with recursion).
66+
# * `list_memories`: Lists memories in Serena's project-specific memory store.
67+
# * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building).
68+
# * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context).
69+
# * `read_file`: Reads a file within the project directory.
70+
# * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store.
71+
# * `remove_project`: Removes a project from the Serena configuration.
72+
# * `replace_lines`: Replaces a range of lines within a file with new content.
73+
# * `replace_symbol_body`: Replaces the full definition of a symbol.
74+
# * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen.
75+
# * `search_for_pattern`: Performs a search for a pattern in the project.
76+
# * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase.
77+
# * `switch_modes`: Activates modes by providing a list of their names
78+
# * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information.
79+
# * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task.
80+
# * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed.
81+
# * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store.
82+
excluded_tools: []
83+
84+
# initial prompt for the project. It will always be given to the LLM upon activating the project
85+
# (contrary to the memories, which are loaded on demand).
86+
initial_prompt: ""
87+
# the name by which the project can be referenced within Serena
88+
project_name: "deepflow-app"
89+
90+
# list of tools to include that would otherwise be disabled (particularly optional tools that are disabled by default)
91+
included_optional_tools: []
92+
93+
# list of mode names to that are always to be included in the set of active modes
94+
# The full set of modes to be activated is base_modes + default_modes.
95+
# If the setting is undefined, the base_modes from the global configuration (serena_config.yml) apply.
96+
# Otherwise, this setting overrides the global configuration.
97+
# Set this to [] to disable base modes for this project.
98+
# Set this to a list of mode names to always include the respective modes for this project.
99+
base_modes:
100+
101+
# list of mode names that are to be activated by default.
102+
# The full set of modes to be activated is base_modes + default_modes.
103+
# If the setting is undefined, the default_modes from the global configuration (serena_config.yml) apply.
104+
# Otherwise, this overrides the setting from the global configuration (serena_config.yml).
105+
# This setting can, in turn, be overridden by CLI parameters (--mode).
106+
default_modes:
107+
108+
# fixed set of tools to use as the base tool set (if non-empty), replacing Serena's default set of tools.
109+
# This cannot be combined with non-empty excluded_tools or included_optional_tools.
110+
fixed_tools: []
111+
112+
# time budget (seconds) per tool call for the retrieval of additional symbol information
113+
# such as docstrings or parameter information.
114+
# This overrides the corresponding setting in the global configuration; see the documentation there.
115+
# If null or missing, use the setting from the global configuration.
116+
symbol_info_budget:
117+
118+
# The language backend to use for this project.
119+
# If not set, the global setting from serena_config.yml is used.
120+
# Valid values: LSP, JetBrains
121+
# Note: the backend is fixed at startup. If a project with a different backend
122+
# is activated post-init, an error will be returned.
123+
language_backend:
124+
125+
# list of regex patterns which, when matched, mark a memory entry as read‑only.
126+
# Extends the list from the global configuration, merging the two lists.
127+
read_only_memory_patterns: []
128+
129+
# line ending convention to use when writing source files.
130+
# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
131+
# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
132+
line_ending:
133+
134+
# list of regex patterns for memories to completely ignore.
135+
# Matching memories will not appear in list_memories or activate_project output
136+
# and cannot be accessed via read_memory or write_memory.
137+
# To access ignored memory files, use the read_file tool on the raw file path.
138+
# Extends the list from the global configuration, merging the two lists.
139+
# Example: ["_archive/.*", "_episodes/.*"]
140+
ignored_memory_patterns: []
141+
142+
# advanced configuration option allowing to configure language server-specific options.
143+
# Maps the language key to the options.
144+
# Have a look at the docstring of the constructors of the LS implementations within solidlsp (e.g., for C# or PHP) to see which options are available.
145+
# No documentation on options means no options are available.
146+
ls_specific_settings: {}

requirements3.txt

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
aiofiles==23.2.1
2+
aiohttp==3.10.11
3+
aiosignal==1.3.1
4+
async-timeout==4.0.3
5+
attrs==23.1.0
6+
Deprecated==1.2.14
7+
frozenlist==1.4.0
8+
html5tagger==1.3.0
9+
httptools==0.6.1
10+
idna==3.7
11+
importlib-metadata==6.8.0
12+
multidict==6.0.4
13+
numpy==1.24.4
14+
opentelemetry-api==1.21.0
15+
opentelemetry-sdk==1.21.0
16+
opentelemetry-semantic-conventions==0.42b0
17+
pandas==2.0.3
18+
python-dateutil==2.8.2
19+
pytz==2023.3.post1
20+
PyYAML==6.0.1
21+
sanic==23.6.0
22+
sanic-routing==23.6.0
23+
schematics==2.1.1
24+
six==1.16.0
25+
tracerite==1.1.1
26+
typing_extensions==4.8.0
27+
tzdata==2023.3
28+
ujson==5.8.0
29+
uvloop==0.19.0
30+
websockets==12.0
31+
wrapt==1.16.0
32+
yarl==1.12.1
33+
zipp==3.19.2
34+
setuptools==68.1.2

0 commit comments

Comments
 (0)