Skip to content

Commit dc052ce

Browse files
MaojiaShengopenviking
MaojiaSheng
and
openviking
authored
reorg: Rewrite agfs to ragfs with rust (#1221)
* reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * reorg: rewrite agfs with rust, and named with ragfs, keep License * fix: grep level limit * fix: grep root * fix: import error * fix: rust code optimazation * fix: CI error * fix: CI go mod cache * fix: grep level limit * fix: CI --------- Co-authored-by: openviking <openviking@example.com>
1 parent 37d4e32 commit dc052ce

53 files changed

Lines changed: 12292 additions & 122 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/_build.yml‎

Lines changed: 82 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -237,12 +237,50 @@ jobs:
237237
mkdir -p openviking/bin
238238
cp target/${{ matrix.arch == 'aarch64' && 'aarch64-unknown-linux-gnu' || 'x86_64-unknown-linux-gnu' }}/release/ov openviking/bin/
239239
chmod +x openviking/bin/ov
240+
241+
- name: Build ragfs-python and extract into openviking/lib/ (Linux)
242+
shell: bash
243+
run: |
244+
uv pip install maturin
245+
TMPDIR=$(mktemp -d)
246+
cd crates/ragfs-python
247+
maturin build --release \
248+
--target ${{ matrix.arch == 'aarch64' && 'aarch64-unknown-linux-gnu' || 'x86_64-unknown-linux-gnu' }} \
249+
--out "$TMPDIR"
250+
cd ../..
251+
mkdir -p openviking/lib
252+
python3 -c "
253+
import zipfile, glob, os, sys
254+
whls = glob.glob(os.path.join('$TMPDIR', 'ragfs_python-*.whl'))
255+
assert whls, 'maturin produced no wheel'
256+
with zipfile.ZipFile(whls[0]) as zf:
257+
for name in zf.namelist():
258+
bn = os.path.basename(name)
259+
if bn.startswith('ragfs_python') and (bn.endswith('.so') or bn.endswith('.pyd')):
260+
dst = os.path.join('openviking', 'lib', bn)
261+
with zf.open(name) as src, open(dst, 'wb') as f:
262+
f.write(src.read())
263+
os.chmod(dst, 0o755)
264+
print(f'Extracted {bn} -> {dst}')
265+
sys.exit(0)
266+
print('ERROR: No ragfs_python .so/.pyd found in wheel')
267+
sys.exit(1)
268+
"
269+
rm -rf "$TMPDIR"
270+
echo "Contents of openviking/lib/:"
271+
ls -la openviking/lib/
240272
- name: Clean workspace (force ignore dirty)
241273
shell: bash
242274
run: |
275+
# Back up pre-built artifacts before cleaning
276+
cp -a openviking/bin /tmp/_ov_bin || true
277+
cp -a openviking/lib /tmp/_ov_lib || true
243278
git reset --hard HEAD
244279
git clean -fd
245280
rm -rf openviking/_version.py openviking.egg-info
281+
# Restore pre-built artifacts
282+
cp -a /tmp/_ov_bin openviking/bin || true
283+
cp -a /tmp/_ov_lib openviking/lib || true
246284
# Ignore uv.lock changes to avoid dirty state in setuptools_scm
247285
git update-index --assume-unchanged uv.lock || true
248286
@@ -257,6 +295,8 @@ jobs:
257295
git status --ignored
258296
echo "=== Check openviking/_version.py ==="
259297
if [ -f openviking/_version.py ]; then cat openviking/_version.py; else echo "Not found"; fi
298+
echo "=== Verify pre-built artifacts survived clean ==="
299+
ls -la openviking/bin/ openviking/lib/ || true
260300
261301
- name: Build package (Wheel Only)
262302
run: uv build --wheel
@@ -276,11 +316,8 @@ jobs:
276316
- name: Repair wheels (Linux)
277317
run: |
278318
uv pip install auditwheel
279-
# Repair wheels and output to a temporary directory
280319
uv run auditwheel repair dist/*.whl -w dist_fixed
281-
# Remove original non-compliant wheels
282320
rm dist/*.whl
283-
# Move repaired wheels back to dist
284321
mv dist_fixed/*.whl dist/
285322
rmdir dist_fixed
286323
@@ -405,12 +442,52 @@ jobs:
405442
cp target/release/ov openviking/bin/
406443
chmod +x openviking/bin/ov
407444
fi
445+
446+
- name: Build ragfs-python and extract into openviking/lib/ (macOS/Windows)
447+
shell: bash
448+
run: |
449+
uv pip install maturin
450+
TMPDIR=$(mktemp -d)
451+
cd crates/ragfs-python
452+
if [[ "${{ matrix.os }}" == "windows-latest" ]]; then
453+
maturin build --release --target x86_64-pc-windows-msvc --out "$TMPDIR"
454+
else
455+
maturin build --release --out "$TMPDIR"
456+
fi
457+
cd ../..
458+
mkdir -p openviking/lib
459+
python3 -c "
460+
import zipfile, glob, os, sys
461+
whls = glob.glob(os.path.join('$TMPDIR', 'ragfs_python-*.whl'))
462+
assert whls, 'maturin produced no wheel'
463+
with zipfile.ZipFile(whls[0]) as zf:
464+
for name in zf.namelist():
465+
bn = os.path.basename(name)
466+
if bn.startswith('ragfs_python') and (bn.endswith('.so') or bn.endswith('.pyd')):
467+
dst = os.path.join('openviking', 'lib', bn)
468+
with zf.open(name) as src, open(dst, 'wb') as f:
469+
f.write(src.read())
470+
os.chmod(dst, 0o755)
471+
print(f'Extracted {bn} -> {dst}')
472+
sys.exit(0)
473+
print('ERROR: No ragfs_python .so/.pyd found in wheel')
474+
sys.exit(1)
475+
"
476+
rm -rf "$TMPDIR"
477+
echo "Contents of openviking/lib/:"
478+
ls -la openviking/lib/
408479
- name: Clean workspace (force ignore dirty)
409480
shell: bash
410481
run: |
482+
# Back up pre-built artifacts before cleaning
483+
cp -a openviking/bin /tmp/_ov_bin || true
484+
cp -a openviking/lib /tmp/_ov_lib || true
411485
git reset --hard HEAD
412486
git clean -fd
413487
rm -rf openviking/_version.py openviking.egg-info
488+
# Restore pre-built artifacts
489+
cp -a /tmp/_ov_bin openviking/bin || true
490+
cp -a /tmp/_ov_lib openviking/lib || true
414491
# Ignore uv.lock changes to avoid dirty state in setuptools_scm
415492
git update-index --assume-unchanged uv.lock || true
416493
@@ -425,6 +502,8 @@ jobs:
425502
git status --ignored
426503
echo "=== Check openviking/_version.py ==="
427504
if [ -f openviking/_version.py ]; then cat openviking/_version.py; else echo "Not found"; fi
505+
echo "=== Verify pre-built artifacts survived clean ==="
506+
ls -la openviking/bin/ openviking/lib/ || true
428507
429508
- name: Build package (Wheel Only)
430509
run: uv build --wheel

‎.github/workflows/api_test.yml‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -59,20 +59,13 @@ jobs:
5959

6060
- name: Cache Go modules
6161
uses: actions/cache@v5
62+
continue-on-error: true
6263
with:
6364
path: ~/go/pkg/mod
64-
key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
65+
key: ${{ runner.os }}-go-${{ hashFiles('third_party/agfs/**/go.sum') }}
6566
restore-keys: |
6667
${{ runner.os }}-go-
6768
68-
- name: Cache C++ extensions
69-
uses: actions/cache@v5
70-
with:
71-
path: openviking/pyagfs
72-
key: ${{ runner.os }}-cpp-${{ hashFiles('**/CMakeLists.txt', '**/*.cpp', '**/*.h') }}
73-
restore-keys: |
74-
${{ runner.os }}-cpp-
75-
7669
- name: Cache Python dependencies (Unix)
7770
if: runner.os != 'Windows'
7871
uses: actions/cache@v5
@@ -94,7 +87,7 @@ jobs:
9487
- name: Set up Go
9588
uses: actions/setup-go@v6
9689
with:
97-
go-version: '1.22'
90+
go-version: '1.25.1'
9891

9992
- name: Install system dependencies (Ubuntu)
10093
if: runner.os == 'Linux'

0 commit comments

Comments
 (0)