Skip to content

Commit 003d6a8

Browse files
kevincheng2claude
andcommitted
[Feature][KVCache] update cache_manager_v1 modules
## Motivation 更新 Cache Manager V1 相关模块,完善版权信息、改进模块结构与可维护性。 ## Modifications - `fastdeploy/cache_manager/v1/` 系列模块:补充版权 header,优化代码结构 - `fastdeploy/config.py`:配置项更新 - `fastdeploy/engine/sched/resource_manager_v1.py`:调度相关更新 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 04d281b commit 003d6a8

23 files changed

Lines changed: 284 additions & 95 deletions

fastdeploy/cache_manager/v1/__init__.py

Lines changed: 13 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,17 @@
11
"""
2-
Cache Manager V1 - Multi-level KV Cache Management System
3-
4-
This module provides a three-level cache hierarchy:
5-
- Device (GPU) → Host (CPU) → Storage
6-
7-
Key components:
8-
- KVCacheBase: Abstract base class defining common interface
9-
- CacheManager: Scheduler-side cache management with block pools
10-
- CacheController: Worker-side cache control for transfer operations
11-
- CacheTransferManager: Manages cache transfer operations
12-
- LayerDoneCounter: Tracks layer-by-layer transfer completion
13-
- create_storage_scheduler: Factory function to create StorageScheduler
14-
- create_storage_connector: Factory function to create StorageConnector
15-
- create_transfer_connector: Factory function to create TransferConnector
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
1615
"""
1716

1817
from .base import KVCacheBase

fastdeploy/cache_manager/v1/base.py

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,17 @@
11
"""
2-
KVCacheBase - Abstract base class for KV cache management
3-
4-
Defines the common interface that both CacheManager (Scheduler) and
5-
CacheController (Worker) must implement.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
615
"""
716

817
from abc import ABC, abstractmethod

fastdeploy/cache_manager/v1/block_pool.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
"""
2-
BlockPool implementations for GPU and CPU memory management.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
315
"""
416

517
import threading

fastdeploy/cache_manager/v1/cache_controller.py

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
11
"""
2-
CacheController - Worker-side cache control.
3-
4-
Responsible for:
5-
- Managing cache transfer operations
6-
- Layer-by-layer transfer synchronization
7-
- Cross-node transfer via TransferConnector
8-
9-
Note: CacheController does NOT manage BlockPool. BlockPool is managed
10-
by CacheManager in the Scheduler process. CacheController only handles
11-
data transfer operations based on block IDs provided by Scheduler.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
1215
"""
1316

1417
import threading

fastdeploy/cache_manager/v1/cache_manager.py

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
"""
2-
CacheManager - Scheduler-side cache management.
3-
4-
Responsible for:
5-
- Managing DeviceBlockPool and HostBlockPool
6-
- Block allocation and release
7-
- RadixTree for prefix matching
8-
- Storage operations coordination
9-
- Three-level cache matching (Device → Host → Storage)
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
1015
"""
1116

1217
from __future__ import annotations

fastdeploy/cache_manager/v1/cache_utils.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
"""
2-
Utility classes and functions for cache management.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
315
"""
416

517
import hashlib

fastdeploy/cache_manager/v1/metadata.py

Lines changed: 14 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,17 @@
11
"""
2-
Metadata definitions for cache management.
3-
4-
This module contains data structures and configurations used across
5-
the cache management system.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
615
"""
716

817
import time
@@ -250,8 +259,7 @@ class BlockNode:
250259
# Backup-related fields
251260
backuped: bool = False # Whether a backup exists on host memory
252261
host_block_id: Optional[int] = None # Host block ID where the backup is stored
253-
# write_through_selective policy fields
254-
hit_count: int = 0 # Access count; triggers backup when reaching the threshold
262+
hit_count: int = 1 # triggers backup when reaching the threshold
255263

256264
def __post_init__(self):
257265
"""Initialize instance with current time if last_access_time not set."""

fastdeploy/cache_manager/v1/radix_tree.py

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
"""
2-
RadixTree implementation for prefix matching in KV cache.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
315
"""
416

517
import heapq

fastdeploy/cache_manager/v1/storage/__init__.py

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
"""
2-
Storage module for cache offloading and loading.
3-
4-
This module provides storage backends for KV cache persistence
5-
and retrieval across different storage systems.
6-
7-
Factory functions:
8-
- create_storage_scheduler: Create a StorageScheduler instance based on config
9-
- create_storage_connector: Create a StorageConnector instance based on config
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
1015
"""
1116

1217
from typing import TYPE_CHECKING, Any, Dict, Optional

fastdeploy/cache_manager/v1/storage/attnstore/__init__.py

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,17 @@
11
"""
2-
AttnStore storage implementation.
3-
4-
AttnStore is an attention-aware storage system for KV cache.
2+
# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License"
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
515
"""
616

717
from .connector import AttnStoreConnector, AttnStoreScheduler

0 commit comments

Comments
 (0)