Skip to content

Commit b2263fb

Browse files
xliuqqxliuqq
authored
add dataload docs for cache runtime (#5807)
* add dataload docs for cache runtime Signed-off-by: xliuqq <xlzq1992@gmail.com> * fix typo Signed-off-by: xliuqq <xlzq1992@gmail.com> * add trailing newline Signed-off-by: xliuqq <xlzq1992@gmail.com> --------- Signed-off-by: xliuqq <xlzq1992@gmail.com>
1 parent 06145aa commit b2263fb

4 files changed

Lines changed: 198 additions & 0 deletions

File tree

‎docs/en/TOC.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
- [Share data across namespace (Sidecar mode)](samples/dataset_across_namespace_with_sidecar.md)
2121
+ Operation
2222
- [Data Preloading](samples/data_warmup.md)
23+
- [CacheRuntime Data Operations](samples/cacheruntime_data_operations.md)
2324
- [Cache Runtime Manually Scaling](samples/dataset_scaling.md)
2425
- [Automatic Cleanup Data Operation](samples/automatic_clean_up_data_operation.md)
2526
+ Security

‎docs/en/samples/cacheruntime_data_operations.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Example - CacheRuntime Data Operations
2+
3+
## Prerequisites
4+
5+
This document is an extension of the [CacheRuntime Integration Guide](../dev/generic_cache_runtime_integration.md) and **assumes you have already completed the basic CacheRuntime integration** (including defining topology, configuring components, etc.).
6+
7+
This document only explains how to **add data operation support** to an existing `CacheRuntimeClass`. The core change involves just one field: `dataOperationSpecs`.
8+
9+
## Background
10+
11+
Fluid's CacheRuntime provides a generic cache runtime abstraction that allows users to define implementation details for different caching systems through `CacheRuntimeClass`. Starting from the latest version of Fluid, CacheRuntime natively supports data operations, including DataLoad (data preloading) and DataProcess (data processing).
12+
13+
This document demonstrates how to configure and use the DataLoad feature for CacheRuntime.
14+
15+
> Note: The DataProcess Spec defines Pod information and mounts the Dataset as a PVC, so no modifications to the caching system are required to use the DataProcess feature.
16+
17+
## Environment Verification
18+
19+
Before running this example, please refer to the [Installation Guide](../userguide/install.md) to complete the Fluid installation and verify that all Fluid components are running properly:
20+
21+
```shell
22+
$ kubectl get pod -n fluid-system
23+
cacheruntime-controller-xxxxx 1/1 Running 0 8h
24+
csi-nodeplugin-fluid-fwgjh 2/2 Running 0 8h
25+
dataset-controller-5b7848dbbb-n44dj 1/1 Running 0 8h
26+
```
27+
28+
Ensure that your cluster has installed a CacheRuntime controller that supports data operations.
29+
30+
## Core Concepts
31+
32+
### The Only Change Compared to Basic Integration
33+
34+
Compared to the basic CacheRuntime integration, supporting data operations **only requires adding one top-level field** to the CacheRuntimeClass:
35+
36+
- **Only add** the `dataOperationSpecs` field
37+
- **No need to modify** any existing fields (topology, fileSystemType, extraResources, etc.)
38+
- **Backward compatible**: CacheRuntimeClass without this field configured can still use basic caching functionality normally
39+
40+
```yaml
41+
apiVersion: data.fluid.io/v1alpha1
42+
kind: CacheRuntimeClass
43+
metadata:
44+
name: curvine-demo
45+
fileSystemType: curvinefs
46+
47+
# [NEW] Only this field is added; other configurations (topology, etc.) remain unchanged
48+
dataOperationSpecs:
49+
- name: DataLoad
50+
command: ["/bin/bash", "-c"]
51+
args: ["..."]
52+
53+
# [ORIGINAL CONFIGURATION] Fields like topology and extraResources require no modifications
54+
topology:
55+
master:
56+
# ... Exactly the same as basic integration
57+
worker:
58+
# ... Exactly the same as basic integration
59+
client:
60+
# ... Exactly the same as basic integration
61+
```
62+
63+
### Detailed Explanation of dataOperationSpecs Field
64+
65+
`dataOperationSpecs` is an array where each element defines the execution specification for a type of data operation.
66+
67+
#### Field Structure
68+
69+
```yaml
70+
dataOperationSpecs:
71+
- name: <operation type>
72+
command: [<command>, <parameters>]
73+
args: [<script or parameters>]
74+
image: <optional: dedicated image>
75+
```
76+
77+
#### Field Description
78+
79+
| Field Name | Type | Required | Description |
80+
|--------|------|----|---------------------------------------------------------------------------------------------------------------------------|
81+
| `name` | string | Yes | Operation type identifier. Currently supported values:<br>• `DataLoad`: Data preloading operation<br>• `DataMigrate`: Data migration operation (not yet supported)<br>• `DataBackup`: Data backup operation (not yet supported) |
82+
| `command` | []string | Yes | Command to execute in the container (entrypoint), typically set to `["/bin/bash", "-c"]` to support script execution |
83+
| `args` | []string | Yes | Arguments for the command, usually containing the complete execution script. The script can use environment variables injected by Fluid (see below) |
84+
| `image` | string | No | Container image used for the operation.<br>• **If not specified**: Defaults to using the `worker` component image from `CacheRuntimeClass`<br>• **If specified**: Uses a custom dedicated image (suitable for scenarios requiring special tools) |
85+
86+
### Available Environment Variables
87+
88+
During data operation execution, Fluid automatically injects the following environment variables into the container:
89+
90+
#### DataLoad-Specific Environment Variables
91+
92+
| Environment Variable Name | Description | Example Value |
93+
|-----------|--------------------------------|--------|
94+
| `FLUID_DATALOAD_METADATA` | Whether to load metadata | `"true"` or `"false"` |
95+
| `FLUID_DATALOAD_DATA_PATH` | Data paths to be loaded (multiple paths separated by colons) | `/spark/spark-3.0.1:/spark/spark-2.4.7` |
96+
| `FLUID_DATALOAD_PATH_REPLICAS` | Number of replicas for each path (separated by colons, corresponding one-to-one with DATA_PATH) | `1:2` |
97+
98+
The underlying caching system writes data preloading scripts based on the above environment variables and packages them into the image. When users define DataLoad operations, they can specify the script through the `command` and `args` fields.

‎docs/zh/TOC.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
- [跨namespace共享数据(sidecar模式)](samples/dataset_across_namespace_with_sidecar.md)
2626
+ 操作
2727
- [数据预加载](samples/data_warmup.md)
28+
- [CacheRuntime 数据操作](samples/cacheruntime_data_operations.md)
2829
- [Cache Runtime手动扩缩容](samples/dataset_scaling.md)
2930
- [数据操作自动清理](samples/automatic_clean_up_data_operation.md)
3031
+ 安全

‎docs/zh/samples/cacheruntime_data_operations.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# 示例 - CacheRuntime 数据操作
2+
3+
## 前提说明
4+
5+
本文档是 [CacheRuntime 对接社区文档](../dev/generic_cache_runtime_integration.md) 的扩展，**假设您已经完成了基础的 CacheRuntime 集成**（包括定义 topology、配置组件等）。
6+
7+
本文档仅说明如何在已有的 `CacheRuntimeClass` 基础上**新增数据操作支持**，核心改动只有一个字段：`dataOperationSpecs`。
8+
9+
## 背景介绍
10+
11+
Fluid 的 CacheRuntime 提供了一种通用的缓存运行时抽象，允许用户通过 `CacheRuntimeClass` 定义不同缓存系统的实现细节。从 Fluid 最新版本开始，CacheRuntime 原生支持数据操作（Data Operations），包括数据预热（DataLoad）和数据处理（DataProcess）。
12+
13+
本文档将阐述如何为 CacheRuntime 配置和使用 DataLoad 功能。
14+
15+
> 注意： DataProcess Spec 中定义了 Pod 信息，是把 DataSet 当作 PVC 进行挂载使用，因此不需要缓存系统做改动即可使用 DataProcess 功能。
16+
17+
## 前提条件
18+
19+
在运行该示例之前，请参考[安装文档](../userguide/install.md)完成 Fluid 安装，并检查 Fluid 各组件正常运行：
20+
21+
```shell
22+
$ kubectl get pod -n fluid-system
23+
cacheruntime-controller-xxxxx 1/1 Running 0 8h
24+
csi-nodeplugin-fluid-fwgjh 2/2 Running 0 8h
25+
dataset-controller-5b7848dbbb-n44dj 1/1 Running 0 8h
26+
```
27+
28+
确保你的集群中已安装了支持数据操作的 CacheRuntime 控制器。
29+
30+
## 核心概念
31+
32+
### 相对于基础集成的唯一改动
33+
34+
与基础的 CacheRuntime 集成相比，支持数据操作**仅需在 CacheRuntimeClass 中添加一个顶层字段**：
35+
36+
- **只需添加** `dataOperationSpecs` 字段
37+
- **无需修改** 任何现有字段（topology、fileSystemType、extraResources 等）
38+
- **向后兼容**：未配置此字段的 CacheRuntimeClass 仍可正常使用基础缓存功能
39+
40+
```yaml
41+
apiVersion: data.fluid.io/v1alpha1
42+
kind: CacheRuntimeClass
43+
metadata:
44+
name: curvine-demo
45+
fileSystemType: curvinefs
46+
47+
# 【新增】仅此一个字段，其他配置（topology等）完全保持不变
48+
dataOperationSpecs:
49+
- name: DataLoad
50+
command: ["/bin/bash", "-c"]
51+
args: ["..."]
52+
53+
# 【原有配置】topology、extraResources 等字段无需任何修改
54+
topology:
55+
master:
56+
# ... 与基础集成完全一致
57+
worker:
58+
# ... 与基础集成完全一致
59+
client:
60+
# ... 与基础集成完全一致
61+
```
62+
63+
### dataOperationSpecs 字段详解
64+
65+
`dataOperationSpecs` 是一个数组，每个元素定义一种数据操作的执行规范。
66+
67+
#### 字段结构
68+
69+
```yaml
70+
dataOperationSpecs:
71+
- name: <操作类型>
72+
command: [<命令>, <参数>]
73+
args: [<脚本或参数>]
74+
image: <可选：专用镜像>
75+
```
76+
77+
#### 字段说明
78+
79+
| 字段名 | 类型 | 必填 | 说明 |
80+
|--------|------|----|---------------------------------------------------------------------------------------------------------------------------|
81+
| `name` | string | 是 | 操作类型标识符，当前支持的值：<br>• `DataLoad`：数据预热操作<br>• `DataMigrate`：数据迁移操作（暂未支持)<br>• `DataBackup`：数据备份操作（暂未支持) |
82+
| `command` | []string | 是 | 容器中执行的命令（entrypoint），通常设置为 `["/bin/bash", "-c"]` 以支持脚本执行 |
83+
| `args` | []string | 是 | 命令的参数，通常包含完整的执行脚本。脚本中可使用 Fluid 注入的环境变量（见下文） |
84+
| `image` | string | 否 | 操作使用的容器镜像。<br>• **如果不指定**：默认使用 `CacheRuntimeClass` 中 `worker` 组件的镜像<br>• **如果指定**：使用自定义的专用镜像（适用于需要特殊工具的场景） |
85+
86+
### 可用环境变量
87+
88+
在数据操作执行过程中，Fluid 会自动向容器中注入以下环境变量：
89+
90+
#### DataLoad 专属环境变量
91+
92+
| 环境变量名 | 说明 | 示例值 |
93+
|-----------|--------------------------------|--------|
94+
| `FLUID_DATALOAD_METADATA` | 是否加载元数据 | `"true"` 或 `"false"` |
95+
| `FLUID_DATALOAD_DATA_PATH` | 需要加载的数据路径（多个路径用冒号分隔） | `/spark/spark-3.0.1:/spark/spark-2.4.7` |
96+
| `FLUID_DATALOAD_PATH_REPLICAS` | 每个路径的副本数（用冒号分隔，与 DATA_PATH 一一对应） | `1:2` |
97+
98+
底层的缓存系统根据上面的环境变量，编写数据预热的脚本并打包到镜像中，用户在定义 DataLoad 操作时，即可通过`command` 和 `args` 字段指定脚本。

0 commit comments

Comments
 (0)