diff --git a/docker/OVERVIEW.en.md b/docker/OVERVIEW.en.md index ef55af2c..a781e171 100644 --- a/docker/OVERVIEW.en.md +++ b/docker/OVERVIEW.en.md @@ -22,7 +22,7 @@ ### AISBench Benchmark AISBench Benchmark is a model evaluation tool built on [OpenCompass](https://github.com/open-compass/opencompass). It is compatible with OpenCompass's configuration system, dataset structure, and model backend implementation, and extends support for service-based models. -> ⚠️Note: AISBench Benchmark images only support evaluation of service-based models, not offline inference models. Evaluation scenarios that require launching independent sandbox environments (such as SWE-Bench, terminal-bench 2, etc.) are also not currently supported. +> ⚠️Note: AISBench Benchmark images focus on service-based model evaluation and do not support offline inference models. Built-in pipelines for sandbox-isolated benchmarks (SWE-Bench, terminal-bench 2, etc.) are NOT provided — but the image ships with Docker Engine (>= 20.0) and Docker Compose v2 (>= 2.0.0), so these benchmarks can be run manually inside the container. See [Running Agent / Sandbox Benchmarks](#running-agent--sandbox-benchmarks-docker-inside-the-container). ## Image Tag Convention & Dockerfile Archive Paths @@ -178,6 +178,197 @@ docker build \ docker/ ``` +## Running Agent / Sandbox Benchmarks (Docker Inside the Container) + +> ⚠️ This section only applies when running agent benchmarks that require isolated sandboxes (SWE-Bench, terminal-bench 2, etc.). It is **not** required for regular service-based model evaluation. + +The image ships with Docker Engine (≥ 20.0) and Docker Compose v2 (≥ 2.0.0). There are two modes for running Docker inside the container — pick one based on your host Docker version and isolation needs. + +### Mode A — Docker-in-Docker (recommended, true nested containers, requires host Docker ≥ 20.10 + cgroup v2) + +A standalone `dockerd` is started inside the container, so child containers are fully isolated from the host. This is the **preferred mode** for agent benchmarks: child containers inherit Docker's official default seccomp profile, so the `pthread_create` / `clone3` block triggered by openEuler / RHEL hardened profiles does not occur; and there is no stale-socket issue when the host's `dockerd` restarts. + +**Prerequisites (verify on the host before proceeding):** + +```bash +docker version --format '{{.Server.Version}}' # must be >= 20.10 +uname -r # recommended >= 5.10 +stat -fc %T /sys/fs/cgroup # must print "cgroup2fs" +``` + +**Step 1 — Start the container:** + +```bash +# --privileged + --cgroupns=host are BOTH required on cgroup v2 hosts. +# Without --cgroupns=host, nested containers fail with: +# "cannot enter cgroupv2 /sys/fs/cgroup/docker with domain controllers -- it is in an invalid state" +docker run --name ais_bench_container -it -d \ + --net=host \ + --ipc=host \ + --privileged \ + --cgroupns=host \ + -w /benchmark \ + -v /data/datasets:/datasets \ + ghcr.io/aisbench/aisbench_benchmark:v3.1-20260522-master-openeuler24.03-py311-aarch64 \ + bash +``` + +**Step 2 — Configure and start dockerd inside the container:** + +```bash +docker exec -it --privileged ais_bench_container /bin/bash + +# Force cgroupfs driver — required for DinD on cgroup v2 hosts. +mkdir -p /etc/docker +cat > /etc/docker/daemon.json <<'EOF' +{ + "exec-opts": ["native.cgroupdriver=cgroupfs"], + "storage-driver": "vfs" +} +EOF + +nohup dockerd > /tmp/dockerd.log 2>&1 & + +# Wait for the daemon socket to be ready +for i in $(seq 1 30); do + [ -S /var/run/docker.sock ] && break + sleep 1 +done + +# Verify +docker info +docker --version +docker compose version +``` + +**Notes (Mode A):** + +- `--privileged` is mandatory; without it `dockerd` will fail to start. +- `--cgroupns=host` is mandatory on cgroup v2 hosts. +- `cgroupfs` cgroup driver (set via `daemon.json`) is mandatory for DinD. Docker 27.x defaults to `systemd`, which is unavailable inside a container. +- `vfs` is the safest storage driver for DinD. Use `overlay2` only when the host kernel and the container's rootfs support it (still requires `--privileged`). +- For very long-running DinD workloads, consider adding `"default-runtime": "runc"`, `"log-driver": "json-file"`, and `"data-root"` overrides to `/etc/docker/daemon.json`. + +### Mode B — Socket Passthrough (works with any Docker version ≥ 1.0) + +Mount the host's Docker socket so that `docker run` inside the container actually creates containers on the **host** daemon. Use this mode only when the host Docker is older than 20.10, or when cgroup v2 is unavailable and Mode A cannot be used. + +**Step 1 — Start the container** + +```bash +HOST_PATH=/path/to/benchmark_wkp + +# 1. Create the working directory on the host and populate it with the image's /benchmark contents +mkdir -p $HOST_PATH +docker run -d --name tmp_extract ${image_id} bash +docker cp tmp_extract:/benchmark/. $HOST_PATH/ +docker rm -f tmp_extract + +docker run --name ais_bench_container -it -d \ + --net=host \ + --privileged \ + -w ${HOST_PATH} \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v ${HOST_PATH}:${HOST_PATH} \ + -v /data/datasets:/datasets \ + ghcr.io/aisbench/aisbench_benchmark:v3.1-20260522-master-openeuler24.03-py311-aarch64 \ + bash +``` + +That's it — no `dockerd` to start, no `daemon.json` to write. The Docker CLI inside the container talks to the host daemon. +Note: `HOST_PATH` is a working directory that must be created on the **host**. Please make sure the working directory contains no other files or sub-directories. + +**Step 2 — Enter the container and re-link ais_bench** + +```bash +# Enter the container; you are now in $HOST_PATH +docker exec -it ais_bench_container /bin/bash +# Re-install ais_bench in editable mode inside $HOST_PATH +# (only changes the link target, does not pull dependencies) +pip3 install -e ./ --use-pep517 --no-deps --no-build-isolation --break-system-packages +``` + +Pros: +- Works with **any Docker version (1.0+)** — no version requirement on the host +- No `--cgroupns=host`, no kernel version check +- Simplest setup; this is how most CI platforms (GitHub Actions, Buildkite, GitLab CI) run Docker + +Cons: +- Containers spawned inside the benchmark container appear on the host's `docker ps` +- No isolation — child containers share the host kernel, network, and PID namespace +- Image pulls must be reachable from the host +- ⚠️ **The container's socket handle goes stale whenever the host's `dockerd` restarts.** When the host daemon restarts (machine reboot, daemon upgrade, `systemctl restart docker`, etc.), the bind-mounted `/var/run/docker.sock` inside the container still points to the old (unlinked) inode, so subsequent `docker` commands fail with `Cannot connect to the Docker daemon`. In that case, run `docker restart ` to re-establish the bind mount. In CI pipelines, **start a fresh container for every task** to avoid state carried over from previous runs. + +**Common issue: child container reports `pthread_create failed: Operation not permitted`** + +Under Mode B, child containers are created by the **host dockerd** and inherit the **host daemon's default seccomp profile**. Distributions such as openEuler / RHEL ship a stricter default profile than Docker's upstream one, which blocks the `clone3` syscall used by OpenBLAS / NumPy when spawning worker threads. The symptom looks like: + +``` +OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Operation not permitted +OpenBLAS blas_thread_init: RLIMIT_NPROC 1048576 current, 1048576 max +``` + +Note that `RLIMIT_NPROC` is not exhausted (the call returns `EPERM`, not `EAGAIN`), which confirms a seccomp block rather than a resource limit. Mode A is unaffected because the inner dockerd uses Docker's official default profile. + +**Fix**: explicitly relax seccomp in the `docker-compose.yml` that launches the child container: + +```yaml +services: + main: + security_opt: + - seccomp=unconfined + # ... rest of the config unchanged +``` + +This only applies to that service; it does not affect the outer AISBench container or the host daemon. If Agent further spawns grandchild containers from `main`, those still inherit the host profile by default — add the same `security_opt` at that layer as well. + +### Run a container workload (works in both modes) + +```bash +# Inside the container +docker pull alpine:latest +docker run --rm alpine:latest echo "Hello from a nested container" + +# Or with docker compose v2 +cat > /tmp/docker-compose.yml <<'EOF' +services: + hello: + image: alpine:latest + command: echo "compose v2 works" +EOF +docker compose -f /tmp/docker-compose.yml up +``` + +### Which mode should I choose? + +- **Mode A (Docker-in-Docker)** is the recommended mode, covering terminal-bench 2, SWE-Bench, and most agent benchmarks. Child containers are isolated from the host and are not affected by the host's seccomp profile — the fewest pitfalls. Requires host Docker ≥ 20.10 + cgroup v2. +- **Mode B (Socket Passthrough)** is used only when the host Docker is older than 20.10, or when cgroup v2 is unavailable. It is the simplest to set up, but child containers inherit the host's seccomp profile (which may block `clone3`), and the container must be manually `docker restart`-ed after the host's `dockerd` restarts. + +### Security Implications of `--privileged` + +`--privileged` is a heavy flag that removes most container isolation. Use it only when needed (i.e., for Mode A / DinD). + +**Risks** + +- **Disables all Linux capability restrictions** — the container gets nearly all root capabilities (`CAP_SYS_ADMIN`, `CAP_NET_ADMIN`, `CAP_SYS_PTRACE`, `CAP_SYS_MODULE`, …). +- **Bypasses seccomp and AppArmor** — any syscall filter is removed, so rules like the ones that broke Python threading no longer apply. +- **Full device access** — `/dev/sda`, `/dev/mem`, `/dev/kvm`, etc. become accessible; a process inside the container can mount, read, or wipe host disks. +- **Writes to the host cgroup tree** — a privileged container can modify other containers' resource limits and freeze/kill them. +- **Kernel-module load/unload** — the container can load or unload host kernel modules (when `CAP_SYS_MODULE` is retained on the host). + +**Mitigations when `--privileged` is unavoidable** + +- Run the benchmark inside a dedicated VM (or a dedicated bare-metal machine) that holds no production data. +- Use a separate Docker daemon (a dedicated `dockerd` on a non-default socket) so the privileged container cannot reach production workloads. +- Audit any code that will run inside the container before granting the flag. +- If you only need to relax the seccomp filter (e.g., for `pip install` threading), prefer `--security-opt seccomp=unconfined` instead — it is a much smaller blast radius. + +**Alternatives to `--privileged` for DinD** + +- `--security-opt seccomp=unconfined` alone is **not** enough for DinD — dockerd still needs cgroup and device access, which only `--privileged` (or a custom runtime) provides. +- [Sysbox](https://github.com/nestybox/sysbox) — a container runtime that supports nested containers without `--privileged`, at the cost of installing a custom runtime on the host. +- Rootless Docker — runs `dockerd` as a non-root user; has its own limitations (no `overlay2` on most distros, network restrictions, etc.). + ## License / Disclaimer This project's images and build scripts are licensed under the [LICENSE file](https://github.com/AISBench/benchmark/blob/master/LICENSE) in the repository root. diff --git a/docker/OVERVIEW.zh.md b/docker/OVERVIEW.zh.md index e66d70a6..ccc26dba 100644 --- a/docker/OVERVIEW.zh.md +++ b/docker/OVERVIEW.zh.md @@ -22,7 +22,7 @@ ### AISBench Benchmark AISBench Benchmark 是基于 [OpenCompass](https://github.com/open-compass/opencompass) 构建的模型评测工具,兼容 OpenCompass 的配置体系、数据集结构与模型后端实现,并在此基础上扩展了对服务化模型的支持能力。 -> ⚠️注意:AISBench Benchmark的镜像只支持服务化模型的评测,不支持离线推理模型的评测。当前也不支持SWE-Bench、terminal-bench 2等测评过程中需要启动独立沙箱环境的测评常见 +> ⚠️注意:AISBench Benchmark 镜像主要用于服务化模型评测,不支持离线推理模型评测。镜像内未内置 SWE-Bench、terminal-bench 2 等需要独立沙箱环境的测评流程,但已预装 Docker Engine(>= 20.0)与 Docker Compose v2(>= 2.0.0),用户可手动启动嵌套容器运行这些测评。详见[运行 Agent / 沙箱类测评](#运行-agent--沙箱类测评在容器内使用-docker)。 ## 镜像 Tag 说明及 Dockerfile 归档路径 @@ -164,6 +164,196 @@ docker build \ docker/ ``` +## 运行 Agent / 沙箱类测评(在容器内使用 Docker) + +> ⚠️ 本节内容仅在需要运行 SWE-Bench、terminal-bench 2 等需要在独立沙箱中执行任务的 agent 测评时使用。常规服务化模型评测无需阅读本节。 + +镜像预装了 Docker Engine(≥ 20.0)与 Docker Compose v2(≥ 2.0.0)。在容器内启动 Docker 有两种模式,请根据宿主 Docker 版本与隔离需求二选一。 + +### 模式 A:Docker-in-Docker(推荐,真嵌套容器,要求宿主 Docker ≥ 20.10 + cgroup v2) + +容器内自起一个独立的 `dockerd`,子容器与宿主机完全隔离。这是 agent 测评的**首选模式**:子容器继承的是 Docker 官方默认 seccomp profile,不会触发 openEuler / RHEL 加固 profile 导致的 `pthread_create` / `clone3` 拦截问题;也不存在宿主 dockerd 重启后 socket 句柄失效的问题。 + +**前置检查(在宿主机执行)** + +```bash +docker version --format '{{.Server.Version}}' # 必须 >= 20.10 +uname -r # 建议 >= 5.10 +stat -fc %T /sys/fs/cgroup # 必须输出 cgroup2fs +``` + +**步骤一:启动容器** + +```bash +# cgroup v2 宿主机上 --privileged + --cgroupns=host 必须同时使用 +# 缺少 --cgroupns=host 时嵌套容器会报: +# "cannot enter cgroupv2 /sys/fs/cgroup/docker with domain controllers -- it is in an invalid state" +docker run --name ais_bench_container -it -d \ + --net=host \ + --ipc=host \ + --privileged \ + --cgroupns=host \ + -w /benchmark \ + -v /data/datasets:/datasets \ + ghcr.io/aisbench/aisbench_benchmark:v3.1-20260522-master-openeuler24.03-py311-aarch64 \ + bash +``` + +**步骤二:配置并启动 dockerd** + +```bash +docker exec -it --privileged ais_bench_container /bin/bash + +# 强制使用 cgroupfs driver —— DinD 在 cgroup v2 宿主机上的必需配置 +mkdir -p /etc/docker +cat > /etc/docker/daemon.json <<'EOF' +{ + "exec-opts": ["native.cgroupdriver=cgroupfs"], + "storage-driver": "vfs" +} +EOF + +nohup dockerd > /tmp/dockerd.log 2>&1 & + +# 等待 daemon socket 就绪 +for i in $(seq 1 30); do + [ -S /var/run/docker.sock ] && break + sleep 1 +done + +# 验证 +docker info +docker --version +docker compose version +``` + +**模式 A 注意事项** + +- `--privileged` 是必需的,否则 `dockerd` 启动会失败。 +- `--cgroupns=host` 在 cgroup v2 宿主机上是必需的。 +- `cgroupfs` cgroup driver(通过 `daemon.json` 设置)是 DinD 必需的。Docker 27.x 默认是 `systemd`,容器内无 systemd 可用。 +- `vfs` 是 DinD 通用性最高的存储驱动;若宿主内核与容器根文件系统支持,使用 `overlay2` 性能更好,但仍需 `--privileged`。 +- 对于长时间运行的 DinD 场景,建议在 `/etc/docker/daemon.json` 中加入 `"default-runtime": "runc"`、`"log-driver": "json-file"`、`"data-root"` 等调优项。 + +### 模式 B:Socket 代理(兼容任意 Docker 版本 ≥ 1.0) + +挂载宿主的 Docker socket,使容器内的 `docker run` 实际上在**宿主机** daemon 上创建容器。仅当宿主 Docker 版本低于 20.10、或不支持 cgroup v2 无法使用模式 A 时再选用本模式。 + +**步骤一:启动容器** + +```bash +HOST_PATH=/path/to/benchmark_wkp + +# 1. 创建承载目录并填充容器内容 +mkdir -p $HOST_PATH +docker run -d --name tmp_extract ${image_id} bash +docker cp tmp_extract:/benchmark/. $HOST_PATH/ +docker rm -f tmp_extract + +docker run --name ais_bench_container -it -d \ + --net=host \ + --privileged \ + -w ${HOST_PATH} \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v ${HOST_PATH}:${HOST_PATH} \ + -v /data/datasets:/datasets \ + ghcr.io/aisbench/aisbench_benchmark:v3.1-20260522-master-openeuler24.03-py311-aarch64 \ + bash +``` + +无需启动 `dockerd`,也无需写 `daemon.json`,容器内的 Docker CLI 直接与宿主 daemon 通信。 +注:`HOST_PATH`是一个需要在物理机创建的工作路径, 请确保工作路径内没有其他文件或目录。 + +**步骤二:进入容器并重新链接ais_bench** + +```bash +# 进入容器,处于HOST_PATH +docker exec -it ais_bench_container /bin/bash +# 在HOST_PATH内重新编辑模式安装ais_bench(只是链接变更,不会装依赖) +pip3 install -e ./ --use-pep517 --no-deps --no-build-isolation --break-system-packages +``` + +**优势** +- 兼容**任意 Docker 版本(1.0+)**,对宿主无版本要求 +- 无需`--cgroupns=host`,不检查内核版本 +- 配置最简单;与 GitHub Actions、Buildkite、GitLab CI 等 CI 平台的实现一致 + +**代价** +- 容器内启动的子容器会出现在宿主的 `docker ps` 列表中 +- 无隔离——子容器共享宿主的内核、网络、PID 命名空间 +- 镜像拉取必须由宿主可达(pull 通过宿主 daemon 完成) +- ⚠️ **宿主机 dockerd 重启后容器内的 socket 句柄会失效**。宿主机 dockerd 一旦重启(机器重启、daemon 升级、`systemctl restart docker` 等),容器内 bind mount 的 `/var/run/docker.sock` 仍指向已被 unlink 的旧 inode,再次执行 `docker` 命令会报 `Cannot connect to the Docker daemon`。遇到这种情况执行 `docker restart ` 重新建立 bind mount 即可恢复。CI 流水线场景下**推荐每个任务都重新起一个新容器**,避免复用带来的状态问题。 + +**常见问题:子容器报 `pthread_create failed: Operation not permitted`** + +Mode B 下子容器由**宿主机 dockerd** 创建,会继承**宿主 daemon 的 default seccomp profile**。openEuler / RHEL 等发行版的默认 profile 比 Docker 官方更严格,会拦截 OpenBLAS / NumPy 初始化线程时使用的 `clone3` 调用,表现为: + +``` +OpenBLAS blas_thread_init: pthread_create failed for thread 52 of 64: Operation not permitted +OpenBLAS blas_thread_init: RLIMIT_NPROC 1048576 current, 1048576 max +``` + +注意 `RLIMIT_NPROC` 并未触顶(返回 EPERM 而非 EAGAIN),说明是 seccomp 拦截而非资源限制。模式 A 不会触发,因为内层 dockerd 用的是 Docker 官方默认 profile。 + +**解决方案**:在拉起子容器的 `docker-compose.yml` 中显式放宽 seccomp: + +```yaml +services: + main: + security_opt: + - seccomp=unconfined + # ... 其余配置不变 +``` + +只作用于该 service,不影响外层 AISBench 容器和宿主 daemon。如果 Agent 还会从 `main` 里再起孙容器,孙容器默认仍套宿主 profile,需在那一层 compose 里同样加一次。 + +### 运行容器工作负载(两种模式通用) + +```bash +# 在容器内执行 +docker pull alpine:latest +docker run --rm alpine:latest echo "Hello from a nested container" + +# 或运行 docker compose v2 +cat > /tmp/docker-compose.yml <<'EOF' +services: + hello: + image: alpine:latest + command: echo "compose v2 works" +EOF +docker compose -f /tmp/docker-compose.yml up +``` + +### 如何选择 + +- **模式 A(Docker-in-Docker)** 为推荐模式,覆盖 terminal-bench 2、SWE-Bench 及绝大多数 agent 测评。子容器与宿主隔离,且不受宿主 seccomp profile 影响,坑最少。要求宿主 Docker ≥ 20.10 + cgroup v2。 +- **模式 B(Socket 代理)** 仅在宿主 Docker 版本低于 20.10、或不支持 cgroup v2 时使用。配置最简单,但子容器会继承宿主 seccomp profile(可能拦截 `clone3`),且宿主 dockerd 重启后需要手动 `docker restart` 恢复。 + +### 关于 `--privileged` 的安全说明 + +`--privileged` 是一个非常「重」的 flag,开启后会移除几乎所有容器隔离机制。仅在确实必要时使用(即模式 A / DinD 场景)。 + +**风险** + +- **关闭所有 Linux capability 限制**——容器获得几乎全部 root capability(`CAP_SYS_ADMIN`、`CAP_NET_ADMIN`、`CAP_SYS_PTRACE`、`CAP_SYS_MODULE` 等)。 +- **绕过 seccomp 与 AppArmor**——所有系统调用过滤器失效,前面卡住 Python 线程的那种规则就不再起作用。 +- **设备完全开放**——`/dev/sda`、`/dev/mem`、`/dev/kvm` 等宿主机设备可直接访问,容器内进程可挂载、读取甚至擦写宿主机磁盘。 +- **可写宿主 cgroup 树**——特权容器能修改其他容器的资源限制,甚至冻结/杀掉其他容器。 +- **可加载/卸载内核模块**——若宿主机保留了 `CAP_SYS_MODULE`,容器能加载或卸载宿主机内核模块。 + +**必须使用 `--privileged` 时的缓解措施** + +- 在专用虚拟机(或专用物理机)内运行测评容器,机器里不放任何生产数据。 +- 使用独立的 Docker daemon(在非默认 socket 上跑一个专用 `dockerd`),让特权容器无法触达生产负载。 +- 给任何要跑在容器里的代码做审计后再授予该 flag。 +- 如果只是要放宽 seccomp(例如解决 `pip install` 线程问题),优先用 `--security-opt seccomp=unconfined`,影响面小很多。 + +**DinD 场景下 `--privileged` 的替代方案** + +- `--security-opt seccomp=unconfined` 单用**不够**——dockerd 还需要 cgroup 与设备访问能力,这只有 `--privileged`(或自定义运行时)能提供。 +- [Sysbox](https://github.com/nestybox/sysbox)——支持嵌套容器而无需 `--privileged`,代价是要在宿主机装自定义运行时。 +- Rootless Docker——以非 root 用户运行 `dockerd`,但有限制(多数发行版不支持 `overlay2`、网络限制等)。 + ## 许可证 / 免责声明 本项目镜像及其构建脚本按仓库根目录的 [LICENSE 文件](https://github.com/AISBench/benchmark/blob/master/LICENSE) 授权。 diff --git a/docker/build_image.sh b/docker/build_image.sh index aed2ca17..8b74709d 100644 --- a/docker/build_image.sh +++ b/docker/build_image.sh @@ -195,6 +195,86 @@ fi echo "镜像验证成功!" +echo "开始验证 Docker 与 Docker Compose 基本命令..." + +docker_version_output=$(docker run --rm ${image_name} docker --version 2>&1) +docker_version_status=$? +if [ ${docker_version_status} -ne 0 ]; then + echo "错误:镜像中 docker --version 执行失败(退出码: ${docker_version_status})" + echo "输出:${docker_version_output}" + exit 1 +fi + +compose_version_output=$(docker run --rm ${image_name} docker compose version 2>&1) +compose_version_status=$? +if [ ${compose_version_status} -ne 0 ]; then + echo "错误:镜像中 docker compose version 执行失败(退出码: ${compose_version_status})" + echo "输出:${compose_version_output}" + exit 1 +fi + +dockerd_version_output=$(docker run --rm ${image_name} dockerd --version 2>&1) +dockerd_version_status=$? +if [ ${dockerd_version_status} -ne 0 ]; then + echo "错误:镜像中 dockerd --version 执行失败(退出码: ${dockerd_version_status})" + echo "输出:${dockerd_version_output}" + exit 1 +fi + +# dockerd 二进制存在性二次校验(Docker 27.x 输出与 docker --version 同格式,需确保调用的是 dockerd 而非 docker) +dockerd_which_output=$(docker run --rm ${image_name} which dockerd 2>&1) +if [ $? -ne 0 ] || [ -z "${dockerd_which_output}" ]; then + echo "错误:镜像中未找到 dockerd 可执行文件" + echo "输出:${dockerd_which_output}" + exit 1 +fi + +# 关键字符串校验 +if ! echo "${docker_version_output}" | grep -F "Docker version" > /dev/null 2>&1; then + echo "错误:Docker 验证失败,未找到预期内容:Docker version" + echo "实际输出:${docker_version_output}" + exit 1 +fi +if ! echo "${compose_version_output}" | grep -F "Docker Compose version" > /dev/null 2>&1; then + echo "错误:Docker Compose 验证失败,未找到预期内容:Docker Compose version" + echo "实际输出:${compose_version_output}" + exit 1 +fi +# Docker 27.x 的 dockerd --version 输出格式为 "Docker version X.Y.Z, build ...",与 docker --version 相同 +if ! echo "${dockerd_version_output}" | grep -F "Docker version" > /dev/null 2>&1; then + echo "错误:dockerd 验证失败,未找到预期内容:Docker version" + echo "实际输出:${dockerd_version_output}" + exit 1 +fi + +# 版本号校验:Docker >= 20.0,Docker Compose >= 2.0.0 +docker_ver=$(echo "${docker_version_output}" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1) +compose_ver=$(echo "${compose_version_output}" | grep -oE 'v?[0-9]+\.[0-9]+\.[0-9]+' | head -1 | sed 's/^v//') +dockerd_ver=$(echo "${dockerd_version_output}" | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1) + +docker_major=$(echo "${docker_ver}" | cut -d. -f1) +compose_major=$(echo "${compose_ver}" | cut -d. -f1) + +# 进一步校验:dockerd 与 docker 版本号应一致,确保不是同一二进制被误调用两次 +if [ "${docker_ver}" != "${dockerd_ver}" ]; then + echo "错误:docker 与 dockerd 版本不一致(${docker_ver} vs ${dockerd_ver}),可能 dockerd 安装异常" + exit 1 +fi + +if [ -z "${docker_ver}" ] || [ "${docker_major}" -lt 20 ]; then + echo "错误:Docker 版本 ${docker_ver} 低于要求的 20.0" + exit 1 +fi + +if [ -z "${compose_ver}" ] || [ "${compose_major}" -lt 2 ]; then + echo "错误:Docker Compose 版本 ${compose_ver} 低于要求的 2.0.0" + exit 1 +fi + +echo "Docker 验证通过:${docker_ver}" +echo "Docker Compose 验证通过:${compose_ver}" +echo "dockerd 验证通过:${dockerd_version_output}" + if [ "$push" == "1" ]; then echo "开始推送镜像到远程仓库(覆盖已有同名镜像)..." docker push ${image_name} diff --git a/docker/openeuler/Dockerfile.py310.openeuler22.03 b/docker/openeuler/Dockerfile.py310.openeuler22.03 index be30ba53..db502db1 100644 --- a/docker/openeuler/Dockerfile.py310.openeuler22.03 +++ b/docker/openeuler/Dockerfile.py310.openeuler22.03 @@ -103,7 +103,7 @@ ENV PYTHONUNBUFFERED=1 ENV NLTK_DATA=/usr/share/nltk_data ENV PATH=/usr/local/bin:$PATH -RUN dnf install -y gcc openssl-devel bzip2-devel libffi-devel zlib-devel wget make findutils unzip sqlite-devel xz-devel && dnf clean all +RUN dnf install -y git vim curl gcc openssl-devel bzip2-devel libffi-devel zlib-devel wget make findutils unzip sqlite-devel xz-devel && dnf clean all RUN ln -s /usr/local/bin/python3.10 /usr/local/bin/python3 && \ ln -s /usr/local/bin/pip3.10 /usr/local/bin/pip3 @@ -113,4 +113,25 @@ COPY --from=builder /usr/local/bin /usr/local/bin COPY --from=builder /benchmark /benchmark COPY --from=builder /usr/share/nltk_data /usr/share/nltk_data +# 安装 Docker Engine (>= 20.0) 与 Docker Compose v2 (>= 2.0.0),用于支持 terminal-bench 等 agent 测评中的嵌套容器执行 +# 依赖包已上传至华为云 OBS: +# docker-${DOCKER_VERSION}-${ARCH}.tgz +# docker-compose-linux-${ARCH} +ARG DOCKER_VERSION=27.3.1 +RUN ARCH=$(uname -m) && \ + dnf install -y iptables curl && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-${DOCKER_VERSION}-${ARCH}.tgz" \ + -o /tmp/docker.tgz && \ + tar -xzf /tmp/docker.tgz -C /tmp && \ + mv /tmp/docker/* /usr/local/bin/ && \ + rm -rf /tmp/docker /tmp/docker.tgz && \ + mkdir -p /usr/local/lib/docker/cli-plugins && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-compose-linux-${ARCH}" \ + -o /usr/local/lib/docker/cli-plugins/docker-compose && \ + chmod +x /usr/local/lib/docker/cli-plugins/docker-compose && \ + dnf clean all && \ + docker --version && \ + docker compose version && \ + dockerd --version + WORKDIR /benchmark diff --git a/docker/openeuler/Dockerfile.py311.openeuler24.03 b/docker/openeuler/Dockerfile.py311.openeuler24.03 index 991b8e0f..6cb711d6 100644 --- a/docker/openeuler/Dockerfile.py311.openeuler24.03 +++ b/docker/openeuler/Dockerfile.py311.openeuler24.03 @@ -95,7 +95,7 @@ ENV PYTHONUNBUFFERED=1 ENV NLTK_DATA=/usr/share/nltk_data ENV PATH=/usr/local/bin:$PATH -RUN dnf install -y gcc openssl-devel bzip2-devel libffi-devel zlib-devel wget make findutils unzip sqlite-devel xz-devel python3 python3-pip && dnf clean all +RUN dnf install -y git vim curl gcc openssl-devel bzip2-devel libffi-devel zlib-devel wget make findutils unzip sqlite-devel xz-devel python3 python3-pip && dnf clean all COPY --from=builder /usr/share/nltk_data /usr/share/nltk_data COPY --from=builder /usr/local/bin /usr/local/bin @@ -105,4 +105,25 @@ COPY --from=builder /usr/lib64/python3.11/site-packages /usr/lib64/python3.11/si COPY --from=builder /usr/lib/python3.11/site-packages /usr/lib/python3.11/site-packages COPY --from=builder /benchmark /benchmark +# 安装 Docker Engine (>= 20.0) 与 Docker Compose v2 (>= 2.0.0),用于支持 terminal-bench 等 agent 测评中的嵌套容器执行 +# 依赖包已上传至华为云 OBS: +# docker-${DOCKER_VERSION}-${ARCH}.tgz +# docker-compose-linux-${ARCH} +ARG DOCKER_VERSION=27.3.1 +RUN ARCH=$(uname -m) && \ + dnf install -y iptables curl && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-${DOCKER_VERSION}-${ARCH}.tgz" \ + -o /tmp/docker.tgz && \ + tar -xzf /tmp/docker.tgz -C /tmp && \ + mv /tmp/docker/* /usr/local/bin/ && \ + rm -rf /tmp/docker /tmp/docker.tgz && \ + mkdir -p /usr/local/lib/docker/cli-plugins && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-compose-linux-${ARCH}" \ + -o /usr/local/lib/docker/cli-plugins/docker-compose && \ + chmod +x /usr/local/lib/docker/cli-plugins/docker-compose && \ + dnf clean all && \ + docker --version && \ + docker compose version && \ + dockerd --version + WORKDIR /benchmark \ No newline at end of file diff --git a/docker/ubuntu/Dockerfile.py310.ubuntu22.04 b/docker/ubuntu/Dockerfile.py310.ubuntu22.04 index c72d8085..dfb5c0eb 100644 --- a/docker/ubuntu/Dockerfile.py310.ubuntu22.04 +++ b/docker/ubuntu/Dockerfile.py310.ubuntu22.04 @@ -148,4 +148,27 @@ COPY --from=builder /usr/lib/python3/dist-packages /usr/lib/python3/dist-package COPY --from=builder /benchmark /benchmark COPY --from=builder /usr/share/nltk_data /usr/share/nltk_data +# 安装 Docker Engine (>= 20.0) 与 Docker Compose v2 (>= 2.0.0),用于支持 terminal-bench 等 agent 测评中的嵌套容器执行 +# 依赖包已上传至华为云 OBS: +# docker-${DOCKER_VERSION}-${ARCH}.tgz +# docker-compose-linux-${ARCH} +ARG DOCKER_VERSION=27.3.1 +RUN ARCH=$(uname -m) && \ + apt-get update && \ + apt-get install -y --no-install-recommends iptables && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-${DOCKER_VERSION}-${ARCH}.tgz" \ + -o /tmp/docker.tgz && \ + tar -xzf /tmp/docker.tgz -C /tmp && \ + mv /tmp/docker/* /usr/local/bin/ && \ + rm -rf /tmp/docker /tmp/docker.tgz && \ + mkdir -p /usr/local/lib/docker/cli-plugins && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-compose-linux-${ARCH}" \ + -o /usr/local/lib/docker/cli-plugins/docker-compose && \ + chmod +x /usr/local/lib/docker/cli-plugins/docker-compose && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ + docker --version && \ + docker compose version && \ + dockerd --version + WORKDIR /benchmark \ No newline at end of file diff --git a/docker/ubuntu/Dockerfile.py312.ubuntu24.04 b/docker/ubuntu/Dockerfile.py312.ubuntu24.04 index eb545557..04d0317a 100644 --- a/docker/ubuntu/Dockerfile.py312.ubuntu24.04 +++ b/docker/ubuntu/Dockerfile.py312.ubuntu24.04 @@ -146,4 +146,27 @@ COPY --from=builder /usr/lib/python3/dist-packages /usr/lib/python3/dist-package COPY --from=builder /benchmark /benchmark COPY --from=builder /usr/share/nltk_data /usr/share/nltk_data +# 安装 Docker Engine (>= 20.0) 与 Docker Compose v2 (>= 2.0.0),用于支持 terminal-bench 等 agent 测评中的嵌套容器执行 +# 依赖包已上传至华为云 OBS: +# docker-${DOCKER_VERSION}-${ARCH}.tgz +# docker-compose-linux-${ARCH} +ARG DOCKER_VERSION=27.3.1 +RUN ARCH=$(uname -m) && \ + apt-get update && \ + apt-get install -y --no-install-recommends iptables && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-${DOCKER_VERSION}-${ARCH}.tgz" \ + -o /tmp/docker.tgz && \ + tar -xzf /tmp/docker.tgz -C /tmp && \ + mv /tmp/docker/* /usr/local/bin/ && \ + rm -rf /tmp/docker /tmp/docker.tgz && \ + mkdir -p /usr/local/lib/docker/cli-plugins && \ + curl -fsSL "https://aisbench.obs.cn-north-4.myhuaweicloud.com/images/resources/docker-compose-linux-${ARCH}" \ + -o /usr/local/lib/docker/cli-plugins/docker-compose && \ + chmod +x /usr/local/lib/docker/cli-plugins/docker-compose && \ + apt-get clean && \ + rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ + docker --version && \ + docker compose version && \ + dockerd --version + WORKDIR /benchmark \ No newline at end of file diff --git a/docs/source_en/extended_benchmark/agent/harbor_bench.md b/docs/source_en/extended_benchmark/agent/harbor_bench.md index 919534fb..d2e95308 100644 --- a/docs/source_en/extended_benchmark/agent/harbor_bench.md +++ b/docs/source_en/extended_benchmark/agent/harbor_bench.md @@ -44,22 +44,50 @@ Official repository: [https://github.com/harbor-framework/harbor](https://github Ensure deployment of tested inference services following OpenAI chat/completions API specification with tool call support. -### 2. Environment Preparation +### 2. Install AISBench Evaluation Tool & Harbor Dependencies -Ensure Docker version >= 20.10.0 and Docker Compose version >= 2.0.0. Also prepare a Python 3.12 runtime environment. - -### 3. Install AISBench Evaluation Tool & Harbor Dependencies - -1. In Python 3.12 environment, refer to [AISBench Installation Documentation](../../get_started/install.md) to install AISBench evaluation tool. -2. In Python 3.12 environment, install Harbor: +#### 2.1 Install from Source +> ⚠️ Environment requirements: Ensure Docker version >= 20.10.0 and Docker Compose version >= 2.0.0 (docker compose may need to be installed separately). Also prepare a Python 3.12 runtime environment. +1. In the Python 3.12 environment, refer to [AISBench Installation Documentation](../../get_started/install.md) to install AISBench evaluation tool. +2. In the Python 3.12 environment, install Harbor: ```bash pip install harbor==0.6.1 ``` +> ⚠️ Note: Installing Harbor will upgrade the datasets library to version 4.0.0 or higher, which will cause dependency conflicts for the datasets library after installation. This does not affect tests for Terminal-Bench datasets using Harbor. However, if you need to test other datasets, you will need to downgrade the datasets library. +#### 2.2 Install Inside a Docker Container +1. Refer to the "Running Agent / Sandbox Benchmarks (Docker Inside the Container)" section in the [Image Overview](https://github.com/AISBench/benchmark/blob/master/docker/OVERVIEW.en.md) to start a container based on a **Python 3.12 or above image (only images published after 2026.7.1 are supported)**. +2. Inside the container, run the following command to install Harbor: + ```bash + pip install harbor==0.6.1 --break-system-packages + ``` +3. Edit Harbor's docker compose configuration file `/usr/local/lib/python3.12/dist-packages/harbor/environments/docker/docker-compose-base.yaml`: +```yaml +services: + main: + network_mode: host # Share host network, required + security_opt: # Required when starting the container with Mode B (Socket Passthrough) + - seccomp=unconfined + volumes: + - type: bind + source: ${HOST_VERIFIER_LOGS_PATH} + target: ${ENV_VERIFIER_LOGS_PATH} + - type: bind + source: ${HOST_AGENT_LOGS_PATH} + target: ${ENV_AGENT_LOGS_PATH} + - type: bind + source: ${HOST_ARTIFACTS_PATH} + target: ${ENV_ARTIFACTS_PATH} + deploy: + resources: + limits: + cpus: ${CPUS} + memory: ${MEMORY} +``` > ⚠️ Note: Installing Harbor will upgrade the datasets library to version 4.0.0 or higher, which will cause dependency conflicts for the datasets library after installation. This does not affect tests for Terminal-Bench datasets using Harbor. However, if you need to test other datasets, you will need to downgrade the datasets library. -### 4. Prepare AISBench-modified Terminal-Bench-2 Dataset and Images +### 3. Prepare AISBench-modified Terminal-Bench-2 Dataset and Images AISBench modified dataset repository: [https://github.com/AISBench/terminal-bench-2](https://github.com/AISBench/terminal-bench-2) @@ -71,9 +99,14 @@ Terminal-Bench-2 pre-packaged images: | `terminal-bench-2-prepared-images_aarch64.tar` | [Link](https://aisbench.obs.cn-north-4.myhuaweicloud.com/terminal-bench-2-images/terminal-bench-2-prepared-images_aarch64.tar) | aarch64 | 48.50 GB | | `terminal-bench-2-prepared-images_x86_64.tar` | [Link](https://aisbench.obs.cn-north-4.myhuaweicloud.com/terminal-bench-2-images/terminal-bench-2-prepared-images_x86_64.tar) | x86_64 | 71.43 GB | -> Note: If you don't want to prepare images for all cases, you can get the terminal-bench-2-offline-mini sampled dataset from [terminal-bench-2-offline-mini](https://modelers.cn/datasets/AISBench/terminal-bench-2-offline-mini). +> Tip: If you don't want to prepare images for all cases, you can get the terminal-bench-2-offline-mini sampled dataset from [terminal-bench-2-offline-mini](https://modelers.cn/datasets/AISBench/terminal-bench-2-offline-mini). + +> ⚠️ Note: +> If you installed AISBench & Harbor dependencies from source, deploy the Terminal-Bench-2 images on the **host machine** by running `docker load -i xxxxxxx.tar`. +> If you started the AISBench container using Mode A (true Docker-in-Docker), deploy the Terminal-Bench-2 images **inside the container** by running `docker load -i xxxxxxx.tar`. +> If you started the AISBench container using Mode B (Socket Passthrough), deploy the Terminal-Bench-2 images on the **host machine** by running `docker load -i xxxxxxx.tar`. -### 5. Configure Custom Configuration File for Harbor Tasks +### 4. Configure Custom Configuration File for Harbor Tasks Modify `ais_bench/configs/agent_example/harbor_terminal_bench_2_task.py` under AISBench tool root directory: @@ -122,7 +155,7 @@ datasets.append( # ...... ``` -### 6. Execute Harbor Tasks +### 5. Execute Harbor Tasks 1. Execute the following command in AISBench tool root directory: ```bash diff --git a/docs/source_zh_cn/extended_benchmark/agent/harbor_bench.md b/docs/source_zh_cn/extended_benchmark/agent/harbor_bench.md index b172ed92..113d55f9 100644 --- a/docs/source_zh_cn/extended_benchmark/agent/harbor_bench.md +++ b/docs/source_zh_cn/extended_benchmark/agent/harbor_bench.md @@ -44,11 +44,9 @@ 确保本地或云端部署了遵循 OpenAI chat/completions API 规范且支持 tool call 的被测推理服务。 -### 2. 测评环境准备 -确保环境docker 版本 >= 20.10.0,docker compose 版本 >= 2.0.0(docker compose可能需要额外安装)。同时需要准备一个python 3.12的运行环境 - -### 3. 安装 AISBench 测评工具 & Harbor 依赖 - +### 2. 安装 AISBench 测评工具 & Harbor 依赖 +#### 2.1 源码安装 +> ⚠️环境限制: 确保环境docker 版本 >= 20.10.0,docker compose 版本 >= 2.0.0(docker compose可能需要额外安装)。同时需要准备一个python 3.12的运行环境 1. 在python 3.12的运行环境内,参考 [AISBench 安装文档](../../get_started/install.md) 安装 AISBench 测评工具。 2. python 3.12的运行环境内安装 Harbor: ```bash @@ -56,8 +54,39 @@ ``` > ⚠️注意:安装harbor会将datasets库的版本升级到4.0.0以上的版本,这会导致安装后报datasets库的依赖冲突,对于执行harbor测试terminal-bench相关数据集没有影响,但是如果你需要测试其他数据集,需要降低datasets库的版本。 +#### 2.2 在docker容器中安装 +1. 参考[镜像概览](https://github.com/AISBench/benchmark/blob/master/docker/OVERVIEW.zh.md)的“运行 Agent / 沙箱类测评(在容器内使用 Docker)”章节启动基于**python3.12及以上版本镜像(2026.7.1之后发布的镜像才支持)**的容器。 +2. 在容器内执行以下命令安装 Harbor: + ```bash + pip install harbor==0.6.1 --break-system-packages + ``` +3. 编辑harbor中的docker compose配置文件`/usr/local/lib/python3.12/dist-packages/harbor/environments/docker/docker-compose-base.yaml` +```yaml +services: + main: + network_mode: host # 共享主机网络,必须配置 + security_opt: # 模式 B 启动的容器需要配置 + - seccomp=unconfined + volumes: + - type: bind + source: ${HOST_VERIFIER_LOGS_PATH} + target: ${ENV_VERIFIER_LOGS_PATH} + - type: bind + source: ${HOST_AGENT_LOGS_PATH} + target: ${ENV_AGENT_LOGS_PATH} + - type: bind + source: ${HOST_ARTIFACTS_PATH} + target: ${ENV_ARTIFACTS_PATH} + deploy: + resources: + limits: + cpus: ${CPUS} + memory: ${MEMORY} +``` +> ⚠️注意:安装harbor会将datasets库的版本升级到4.0.0以上的版本,这会导致安装后报datasets库的依赖冲突,对于执行harbor测试terminal-bench相关数据集没有影响,但是如果你需要测试其他数据集,需要降低datasets库的版本。 + -### 4. 准备AISBench修改过的Terminal-Bench-2数据集和对应镜像 +### 3. 准备AISBench修改过的Terminal-Bench-2数据集和对应镜像 AISBench修改的数据集获取链接:https://github.com/AISBench/terminal-bench-2 > 👉注意: AISBench没有改用例内容,只是将所有环境的准备全部集中到Dockerfile中,避免反复执行还需要反复构建环境和安装依赖 @@ -67,9 +96,14 @@ Terminal-Bench-2 预制打包镜像信息: |`terminal-bench-2-prepared-images_aarch64.tar`| https://aisbench.obs.cn-north-4.myhuaweicloud.com/terminal-bench-2-images/terminal-bench-2-prepared-images_aarch64.tar | aarch64 | 48.50 GB | |`terminal-bench-2-prepared-images_x86_64.tar`| https://aisbench.obs.cn-north-4.myhuaweicloud.com/terminal-bench-2-images/terminal-bench-2-prepared-images_x86_64.tar | x86_64 | 71.43GB | -> 👉注意:如果不想准备所有case的镜像,可以从[terminal-bench-2-offline-mini](https://modelers.cn/datasets/AISBench/terminal-bench-2-offline-mini)获取基于terminal-bench-2.0小规模采样的数据集及对应打包镜像 +> 🌟提示:如果不想准备所有case的镜像,可以从[terminal-bench-2-offline-mini](https://modelers.cn/datasets/AISBench/terminal-bench-2-offline-mini)获取基于terminal-bench-2.0小规模采样的数据集及对应打包镜像 + +> ⚠️注意: +> 如果通过源码安装 AISBench 测评工具 & Harbor 依赖这种方式安装依赖的情况下,部署Terminal-Bench-2的镜像需要在**物理机**上执行`docker load -i xxxxxxx.tar` +> 如果通过模式 A(真 docker in docker)启动AISBench容器,部署Terminal-Bench-2的镜像需要在**容器内**上执行`docker load -i xxxxxxx.tar` +> 如果提供给模式 B(Socket 代理)启动AISBench容器,部署Terminal-Bench-2的镜像需要在**物理机**上执行`docker load -i xxxxxxx.tar` -### 5. 配置 Harbor 任务的自定义配置文件 +### 4. 配置 Harbor 任务的自定义配置文件 在 AISBench 工具根目录下修改 `ais_bench/configs/agent_example/harbor_terminal_bench_2_task.py`: @@ -120,7 +154,7 @@ for task in sub_tasks: # ...... ``` -### 6. 执行 Harbor 任务 +### 5. 执行 Harbor 任务 1. 在 AISBench 工具根目录下执行以下命令: ```bash