Skip to content

feat: merge extension test with neug test #367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lnfjpt wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat: merge extension test with neug test #367

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bafdac9
merge extension test with neug test
lnfjpt May 18, 2026
b359a18
fix build nightly wheel of macos arm64
lnfjpt May 18, 2026
b14ca80
add report for build nightly wheel
lnfjpt May 18, 2026
0920060
update schedule of extension test
lnfjpt May 19, 2026
70bd227
Merge branch 'main' into merge-build-wheel
lnfjpt May 25, 2026
2484804
Merge branch 'main' into merge-build-wheel
lnfjpt May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .github/workflows/build-nightly-wheel.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,36 @@ jobs:
enable_macos_x86: false
caller_event_name: ${{ github.event_name }}
secrets: inherit

# ============================================================
# Notify: Send results to DingTalk
# ============================================================
notify-nightly-wheel-result:
needs: [build_wheels]
if: always() && github.repository == 'alibaba/neug'
runs-on: ubuntu-latest
steps:
- name: Send DingTalk notification
env:
DINGTALK_TOKEN: ${{ secrets.DINGTALK_TOKEN }}
run: |
if [ -z "$DINGTALK_TOKEN" ]; then
echo "DINGTALK_TOKEN not set, skipping notification."
exit 0
fi
WEBHOOK_URL="https://oapi.dingtalk.com/robot/send?access_token=${DINGTALK_TOKEN}"
STATUS="${{ needs.build_wheels.result }}"
if [[ "$STATUS" == "success" ]]; then
TITLE="[Nightly Report] Build Nightly Wheels Passed"
else
TITLE="[Nightly Report] Build Nightly Wheels Failed"
fi
curl -s -X POST "$WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d "{
\"msgtype\": \"markdown\",
\"markdown\": {
\"title\": \"$TITLE\",
\"text\": \"## $TITLE\\n\\n- **build_wheels**: ${STATUS}\\n- **Repo**: ${{ github.repository }}\\n- **Branch**: ${{ github.ref_name }}\\n- **Commit**: ${{ github.sha }}\\n- [View Run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\"
}
}"
116 changes: 5 additions & 111 deletions .github/workflows/neug-extension-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ name: NeuG Extension Test

on:
schedule:
# Run at 08:00 UTC every Saturday
- cron: '0 8 * * 6'
# Run at 04:00 CST (20:00 UTC Friday) every Saturday
- cron: '0 20 * * 5'
pull_request:
branches: [main]
paths:
Expand All @@ -21,111 +21,6 @@ concurrency:
cancel-in-progress: true

jobs:
# ============================================================
# Job 1: Build NeuG with extensions
# ============================================================
extension_tests_default:
runs-on: [self-hosted, daily, linux]
container:
image: neug-registry.cn-hongkong.cr.aliyuncs.com/neug/neug-dev:v0.1.1
steps:
- name: checkout
uses: actions/checkout@v4
with:
submodules: true

- name: Mark directory safe
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/aliyun-oss-cpp-sdk"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/parallel-hashmap"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/pybind11"

- name: Clean the tool directory for self-hosted runner
run: |
rm -rf /__w/_tool/Python

- name: Cache CCache
uses: actions/cache@v4
with:
path: ~/.cache/ccache
key: ${{ runner.os }}-ccache-ext-${{ github.sha }}
restore-keys: |
${{ runner.os }}-ccache-ext-
${{ runner.os }}-ccache-

- name: Setup CCache and build dependencies
run: |
sudo apt-get update && sudo apt-get install -y ccache libcurl4-openssl-dev
ccache --max-size=2G
ccache --zero-stats

- name: Increase the maximum number of opened files
run: |
ulimit -n 65535
sudo chmod -R 777 /etc/security/*
echo "* soft nofile 1048576" | tee -a /etc/security/limits.conf

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.13'

# TODO: After unifying wheel build (neug-manylinux:v0.1.1-x86_64) and dev images (neug-dev:v0.1.1),
# skip the build process and use `pip3 install neug` to test the locally built json extension
# with the released wheel package.
- name: Build NeuG with extensions
run: |
. /home/neug/.neug_env
cd ${GITHUB_WORKSPACE}/
export BUILD_TYPE=RELEASE
export CMAKE_BUILD_PARALLEL_LEVEL=$(printf '%s\n' "$(nproc)" 32 | awk 'NR==1 || $0<min { min=$0 } END { print min }')
export BUILD_EXECUTABLES=ON
export BUILD_HTTP_SERVER=ON
export BUILD_TEST=ON
export USE_NINJA=OFF
export WITH_MIMALLOC=OFF
export CMAKE_INSTALL_PREFIX=/opt/neug-install/
export BUILD_EXTENSIONS="parquet;httpfs"
sudo mkdir -p /opt/neug-install
sudo chown -R $USER:$USER /opt/neug-install/
export CMAKE_C_COMPILER_LAUNCHER=ccache
export CMAKE_CXX_COMPILER_LAUNCHER=ccache
cd tools/python_bind
make clean
make requirements
make build
ccache --show-stats

- name: Run Extension C++ Test
run: |
cd ${GITHUB_WORKSPACE}/tools/python_bind/
export MODERN_GRAPH_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/modern_graph
export COMPREHENSIVE_GRAPH_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/comprehensive_graph
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/modern_graph
export TEST_PATH=${GITHUB_WORKSPACE}/tests
cd build/neug_py_bind
ctest -R parquet_extension_test -V
ctest -R httpfs_extension_test -V
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/tinysnb
ctest -R test_extension -V

- name: Run Extension Python Test
run: |
cd ${GITHUB_WORKSPACE}/tools/python_bind/
pip3 install -r requirements.txt
pip3 install -r requirements_dev.txt
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/tinysnb
GLOG_v=10 ./build/neug_py_bind/tools/utils/bulk_loader -g ../../example_dataset/tinysnb/graph.yaml -l ../../example_dataset/tinysnb/import.yaml -d /tmp/tinysnb
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/comprehensive_graph
GLOG_v=10 ./build/neug_py_bind/tools/utils/bulk_loader -g ../../example_dataset/comprehensive_graph/graph.yaml -l ../../example_dataset/comprehensive_graph/import.yaml -d /tmp/comprehensive_graph
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_load.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_export.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_db_import_export.py -k "copy_from_parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_sniffer.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_load.py -k "httpfs"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_export.py -k "httpfs"

extension_tests_wheel_linux_x86_64:
runs-on: [self-hosted, daily, linux, x64]
steps:
Expand Down Expand Up @@ -282,7 +177,7 @@ jobs:
# Notify: Send results to DingTalk
# ============================================================
notify-extension-result:
needs: [extension_tests_default, extension_tests_wheel_linux_x86_64, extension_tests_wheel_linux_arm64]
needs: [extension_tests_wheel_linux_x86_64, extension_tests_wheel_linux_arm64]
if: always() && github.repository == 'alibaba/neug'
runs-on: ubuntu-latest
steps:
Expand All @@ -295,10 +190,9 @@ jobs:
exit 0
fi
WEBHOOK_URL="https://oapi.dingtalk.com/robot/send?access_token=${DINGTALK_TOKEN}"
STATUS_DEFAULT="${{ needs.extension_tests_default.result }}"
STATUS_X86="${{ needs.extension_tests_wheel_linux_x86_64.result }}"
STATUS_ARM="${{ needs.extension_tests_wheel_linux_arm64.result }}"
if [[ "$STATUS_DEFAULT" == "success" && "$STATUS_X86" == "success" && "$STATUS_ARM" == "success" ]]; then
if [[ "$STATUS_X86" == "success" && "$STATUS_ARM" == "success" ]]; then
TITLE="[Nightly Report] NeuG Extension Test Passed"
else
TITLE="[Nightly Report] NeuG Extension Test Failed"
Expand All @@ -309,6 +203,6 @@ jobs:
\"msgtype\": \"markdown\",
\"markdown\": {
\"title\": \"$TITLE\",
\"text\": \"## $TITLE\n\n- **extension_tests_default**: ${STATUS_DEFAULT}\n- **wheel_linux_x86_64**: ${STATUS_X86}\n- **wheel_linux_arm64**: ${STATUS_ARM}\n- **Repo**: ${{ github.repository }}\n- **Branch**: ${{ github.ref_name }}\n- **Commit**: ${{ github.sha }}\n- [View Run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\"
\"text\": \"## $TITLE\n\n- **wheel_linux_x86_64**: ${STATUS_X86}\n- **wheel_linux_arm64**: ${STATUS_ARM}\n- **Repo**: ${{ github.repository }}\n- **Branch**: ${{ github.ref_name }}\n- **Commit**: ${{ github.sha }}\n- [View Run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\"
}
}"
105 changes: 105 additions & 0 deletions .github/workflows/neug-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,3 +468,108 @@ jobs:
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ${{ github.workspace }}/tools/python_bind/build/neug_py_bind/coverage_filtered.info

# ============================================================
# Job: Build NeuG with extensions (from neug-extension-test.yml)
# ============================================================
extension_tests_default:
runs-on: [self-hosted, daily, linux]
container:
image: neug-registry.cn-hongkong.cr.aliyuncs.com/neug/neug-dev:v0.1.1
steps:
- name: checkout
uses: actions/checkout@v4
with:
submodules: true

- name: Mark directory safe
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/aliyun-oss-cpp-sdk"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/parallel-hashmap"
git config --global --add safe.directory "$GITHUB_WORKSPACE/third_party/pybind11"

- name: Clean the tool directory for self-hosted runner
run: |
rm -rf /__w/_tool/Python

- name: Cache CCache
uses: actions/cache@v4
with:
path: ~/.cache/ccache
key: ${{ runner.os }}-ccache-ext-${{ github.sha }}
restore-keys: |
${{ runner.os }}-ccache-ext-
${{ runner.os }}-ccache-

- name: Setup CCache
run: |
sudo apt-get update && sudo apt-get install -y ccache libcurl4-openssl-dev
ccache --max-size=2G
ccache --zero-stats

- name: Increase the maximum number of opened files
run: |
ulimit -n 65535
sudo chmod -R 777 /etc/security/*
echo "* soft nofile 1048576" | tee -a /etc/security/limits.conf

- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: '3.13'

# TODO: After unifying wheel build (neug-manylinux:v0.1.1-x86_64) and dev images (neug-dev:v0.1.1),
# skip the build process and use `pip3 install neug` to test the locally built json extension
# with the released wheel package.
- name: Build NeuG with extensions
run: |
. /home/neug/.neug_env
cd ${GITHUB_WORKSPACE}/
export BUILD_TYPE=RELEASE
export CMAKE_BUILD_PARALLEL_LEVEL=$(printf '%s\n' "$(nproc)" 32 | awk 'NR==1 || $0<min { min=$0 } END { print min }')
export BUILD_EXECUTABLES=ON
export BUILD_HTTP_SERVER=ON
export BUILD_TEST=ON
export USE_NINJA=OFF
export WITH_MIMALLOC=OFF
export CMAKE_INSTALL_PREFIX=/opt/neug-install/
export BUILD_EXTENSIONS="parquet;httpfs"
sudo mkdir -p /opt/neug-install
sudo chown -R $USER:$USER /opt/neug-install/
export CMAKE_C_COMPILER_LAUNCHER=ccache
export CMAKE_CXX_COMPILER_LAUNCHER=ccache
cd tools/python_bind
make clean
make requirements
make build
ccache --show-stats

- name: Run Extension C++ Test
run: |
cd ${GITHUB_WORKSPACE}/tools/python_bind/
export MODERN_GRAPH_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/modern_graph
export COMPREHENSIVE_GRAPH_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/comprehensive_graph
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/modern_graph
export TEST_PATH=${GITHUB_WORKSPACE}/tests
cd build/neug_py_bind
ctest -R parquet_extension_test -V
ctest -R httpfs_extension_test -V
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/tinysnb
ctest -R test_extension -V

- name: Run Extension Python Test
run: |
cd ${GITHUB_WORKSPACE}/tools/python_bind/
pip3 install -r requirements.txt
pip3 install -r requirements_dev.txt
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/tinysnb
GLOG_v=10 ./build/neug_py_bind/tools/utils/bulk_loader -g ../../example_dataset/tinysnb/graph.yaml -l ../../example_dataset/tinysnb/import.yaml -d /tmp/tinysnb
export FLEX_DATA_DIR=${GITHUB_WORKSPACE}/example_dataset/comprehensive_graph
GLOG_v=10 ./build/neug_py_bind/tools/utils/bulk_loader -g ../../example_dataset/comprehensive_graph/graph.yaml -l ../../example_dataset/comprehensive_graph/import.yaml -d /tmp/comprehensive_graph
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_load.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_export.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_db_import_export.py -k "copy_from_parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_sniffer.py -k "parquet"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_load.py -k "httpfs"
NEUG_RUN_EXTENSION_TESTS=true python3 -m pytest -sv tests/test_export.py -k "httpfs"
2 changes: 1 addition & 1 deletion tools/python_bind/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ environment = {CMAKE_PREFIX_PATH="/opt/neug/", OPENSSL_ROOT_DIR="/opt/neug", LIB
[tool.cibuildwheel.macos]
before-all = ["git submodule update --init --recursive"]

environment = {MACOSX_DEPLOYMENT_TARGET="10.15", CMAKE_PREFIX_PATH="/opt/neug/", DYLD_LIBRARY_PATH="/opt/neug/lib:/opt/neug/lib64", OPENSSL_ROOT_DIR="/opt/neug"}
environment = {MACOSX_DEPLOYMENT_TARGET="10.15", CMAKE_PREFIX_PATH="/opt/neug/", DYLD_LIBRARY_PATH="/opt/neug/lib:/opt/neug/lib64", OPENSSL_ROOT_DIR="/opt/neug", PYBIND11_FINDPYTHON="OFF"}
Copy link
Copy Markdown
Member

@zhanglei1949 zhanglei1949 May 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to change this?

repair-wheel-command = [
'source ~/.neug_env',
'delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}'
Expand Down
Loading