fix: stabilize install and restore cross-platform wheel CI

[ScissorJack-ever]

Description

This PR stabilizes local installation and restores wheel/test reliability across macOS, Windows, and Linux.

It fixes Conan compiler detection during editable/uv builds so make install no longer fails when CC/CXX include flags, removes an unavailable macOS Conan libomp dependency in favor of system/Homebrew OpenMP resolution, fixes Windows-specific build issues caused by the IN macro and non-portable uint64_t formatting, and updates a recovery test annotation so older Python versions used in CI can collect the test suite correctly.

It also adds focused unit tests to improve coverage for platform.hpp, Log.hpp, and platform_fs.hpp.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Performance improvement
• Code refactoring
• Documentation update
• CI/CD changes

Changes Made

• Normalize CC/CXX handling before Conan profile detection in editable builds to fix make install
• Preserve the intended host C++ standard during Conan dependency resolution
• Remove Conan libomp usage for macOS wheel builds and rely on system/Homebrew OpenMP discovery
• Replace non-portable uint64_t hex formatting with PRIx64
• Rename internal metadata filter enum values to avoid Windows IN macro conflicts while preserving Python API names
• Update recovery test typing to stay compatible with older Python versions used in Linux wheel jobs
• Add targeted unit tests covering logging helpers, platform allocation helpers, and platform filesystem helpers

Testing

Describe how you tested your changes:

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed

make install
python scripts/conan_build/conan_install.py --project-dir . --build-type Release
cmake --build build --target log_test metadata_filter_test index_encoding_test -j2
ctest --test-dir build --output-on-failure -R '^utils_test_log$'
ctest --test-dir build --output-on-failure -R 'utils_test_(metadata_filter|index_encoding)$'
python3 -m py_compile python/tests/test_recovery.py
uv run pytest python/tests/test_recovery.py -q

Checklist

• My code follows the project's coding style
• I have run make lint and fixed any issues
• I have added tests that prove my fix/feature works
• All new and existing tests pass (make test)
• I have updated documentation if needed
• My commits follow the conventional commits (https://www.conventionalcommits.org/) format

Screenshots (if applicable)

Additional Notes

This PR includes the commits after d59e038 and keeps the platform-specific wheel fixes split into
separate conventional commits for easier review.

The GitHub Actions Node.js 20 deprecation notice is unrelated to the actual wheel build failures addressed here.

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
python/src/alayalite/schema.py	66.66%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

This PR stabilizes cross-platform builds (editable installs and wheel CI) by improving Conan environment/profile handling, removing macOS Conan OpenMP dependency assumptions, and adding portable fallbacks for AVX512/OpenMP-dependent code paths so tests and functionality run across Linux/macOS/Windows (including ARM).

Changes:

• Refactors Conan install flow to normalize CC/CXX and prefer Conan’s detected default profile with targeted host/build overrides.
• Introduces cross-platform portability layers (OpenMP wrapper + filesystem sync/atomic replace helpers) and removes platform-fragile headers/usages.
• Expands/adjusts unit tests and Python APIs (metadata filter op rename, dataset loaders, explicit resource closing) to improve reliability and coverage across platforms.

Reviewed changes

Copilot reviewed 82 out of 82 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/utils/rabitq_utils/rotator_test.cpp	Removes AVX512-only assertions so rotator behavior is validated cross-platform.
tests/utils/rabitq_utils/lut_test.cpp	Adds scalar fastscan accumulation test coverage.
tests/utils/metadata_filter_test.cpp	Updates tests for renamed filter ops to avoid Windows macro conflicts.
tests/utils/log_test.cpp	Adds tests for path extraction/logging macros and platform helpers.
tests/utils/dataset_utils_test.cpp	Adds parser + hybrid dataset tests and switches dataset root resolution logic.
tests/space/sq8_space_test.cpp	Removes non-portable includes for cross-platform builds.
tests/space/sq4_space_test.cpp	Removes non-portable includes for cross-platform builds.
tests/space/raw_space_test.cpp	Removes non-portable includes for cross-platform builds.
tests/space/CMakeLists.txt	Adds quant_sq8_test target and registers it with CTest/coverage.
tests/index/rabitq_test.cpp	Uses smaller dataset config and removes AVX512-only branches in test flow.
tests/index/rabitq_performance_test.cpp	Removes AVX512-only branches so test codepaths run portably.
tests/index/rabitq_benchmark.cpp	Makes benchmark path/exec logic more portable across OSes.
tests/index/nsg_test.cpp	Switches to tiny dataset config to reduce CI/runtime burden.
tests/index/nndescent_test.cpp	Switches to tiny dataset config to reduce CI/runtime burden.
tests/index/hybrid_quantization_performance_test.cpp	Removes AVX512 compile-time gating for bundle availability checks.
tests/index/graph_test.cpp	Removes non-portable includes for cross-platform builds.
tests/index/fusion_graph_test.cpp	Switches to tiny dataset config to reduce CI/runtime burden.
tests/index/CMakeLists.txt	Adds unit labels to index tests to improve CI/test selection.
tests/executor/update_test.cpp	Uses smaller dataset config for faster/more reliable CI.
tests/executor/search_test.cpp	Removes AVX512-only compilation guards around RaBitQ hybrid tests.
tests/executor/coro_test.cpp	Fixes Linux guard typo and refactors coroutine tests to be dataset-based + deterministic.
tests/executor/CMakeLists.txt	Adds unit labels to executor tests to improve CI/test selection.
scripts/conan_build/conan_install.py	Normalizes compiler env vars and improves Conan profile/setting resolution for CI/editables.
scripts/ci/codecov/gnu_codecoverage.sh	Expands coverage targets and removes default exclusion of dataset utils tests.
scripts/benchmarks/raw_hybrid_collection_benchmark.py	Adds benchmark to measure Collection.hybrid_query on real hybrid datasets.
scripts/benchmarks/download_qdrant_filtered_dataset.py	Adds safe downloader/extractor for benchmark datasets.
python/tests/test_vec_loader.py	Adds tests for new .npy + .jsonl loader utility.
python/tests/test_recovery.py	Adjusts typing annotation to remain compatible with older Python used in CI.
python/tests/test_rabitq_search.py	Removes platform skip logic and strengthens IP test data generation.
python/tests/test_hybrid_search.py	Reduces dataset sizes and expands coverage to include full-precision quantization.
python/tests/test_collection.py	Adds columnar Collection.fit tests and removes RaBitQ platform skips.
python/tests/test_client_with_url.py	Adds tests ensuring indices/collections are closed before disk deletion/reset.
python/src/pybind.cpp	Remaps Python enum values to renamed C++ filter ops while preserving API names.
python/src/alayalite/utils.py	Adds ColumnarDataset + load_npy_jsonl_dataset for efficient hybrid dataset loading.
python/src/alayalite/index.py	Avoids redundant dtype conversion and adds explicit close()/__del__ resource cleanup.
python/src/alayalite/collection.py	Adds columnar Collection.fit and improves close semantics.
python/src/alayalite/client.py	Ensures collections/indices are closed before deletion/reset to avoid RocksDB locks.
python/src/alayalite/init.py	Exposes new dataset utilities via package exports.
python/include/parse.hpp	Adjusts structured binding iteration style for broader compiler compatibility.
python/include/params.hpp	Removes non-portable includes for cross-platform builds.
python/include/materialized_view.hpp	Updates switch cases to renamed filter ops.
python/include/index.hpp	Adds LOG_INFO_ONCE for coroutine-search fallback path.
python/include/client.hpp	Removes non-portable includes for cross-platform builds.
pytest.ini	Sets pytest-asyncio default loop fixture scope.
pyproject.toml	Updates wheel/CI configuration, adds native-opt toggle, and adjusts cibuildwheel hooks/env.
include/utils/rabitq_utils/search_utils/buffer.hpp	Fixes uninitialized capacity + adds safer edge-case behavior.
include/utils/rabitq_utils/rotator.hpp	Adds non-AVX512 fallback implementations and removes ARM hard-fail.
include/utils/rabitq_utils/fastscan.hpp	Adds scalar fallbacks and logs once when fastscan SIMD is unavailable.
include/utils/query_utils.hpp	Removes non-portable includes for cross-platform builds.
include/utils/platform.hpp	Fixes aligned alloc for C++17 requirements (size multiple of alignment) and edge cases.
include/utils/platform_fs.hpp	Adds portable fs sync + atomic replace helpers used by recovery paths.
include/utils/parser.hpp	Adds lightweight JSONL + NPY parsing helpers for hybrid datasets/tests.
include/utils/openmp.hpp	Adds OpenMP abstraction + serial fallback with once-only logging.
include/utils/metadata_filter.hpp	Renames IN/NOT_IN to IN_SET/NOT_IN_SET to avoid Windows macro collisions.
include/utils/metadata_filter_matcher.hpp	Updates executor fast-path switch logic to renamed filter ops.
include/utils/log.hpp	Improves relative path extraction for Windows paths and adds LOG_INFO_ONCE.
include/utils/index_encoding.hpp	Fixes portable uint64_t formatting via PRIx64.
include/utils/hybrid_dataset_utils.hpp	Adds hybrid dataset config/loading + Qdrant filter parsing helpers.
include/utils/evaluate.hpp	Removes non-portable includes for cross-platform builds.
include/utils/dataset_utils.hpp	Adds resolve_data_dir, sift_tiny, metric metadata, and improved truncation/GT handling.
include/storage/sequential_storage.hpp	Removes non-portable includes for cross-platform builds.
include/storage/rocksdb_storage.hpp	Uses std::filesystem for directory creation to improve portability.
include/space/sq4_space.hpp	Removes non-portable includes for cross-platform builds.
include/space/raw_space.hpp	Removes non-portable includes for cross-platform builds.
include/space/rabitq_space.hpp	Uses OpenMP abstraction macros and logs fallback behavior when OpenMP is absent.
include/recovery/write_ahead_log.hpp	Uses new platform fs helpers for syncing instead of raw POSIX calls.
include/recovery/recovery_manager.hpp	Uses platform fs helpers for atomic publish + directory sync.
include/index/graph/qg/qg_builder.hpp	Uses OpenMP abstraction and removes AVX512-only compilation gating.
include/index/graph/overlay_graph.hpp	Removes non-portable includes for cross-platform builds.
include/index/graph/graph.hpp	Removes non-portable includes for cross-platform builds.
include/index/graph/graph_concepts.hpp	Removes non-portable includes for cross-platform builds.
include/executor/jobs/graph_search_job.hpp	Removes AVX512-only gating of RaBitQ search paths.
include/executor/jobs/graph_hybrid_search_job.hpp	Removes AVX512-only gating of RaBitQ hybrid search path.
conanfile.py	Makes libcoro Linux-only and removes Conan-managed libomp to rely on toolchain resolution.
CMakeLists.txt	Makes OpenMP optional and gates -march=native behind a CMake option.
cmake/PrintSummary.cmake	Improves build summary and explicitly reports OpenMP availability and native-opt toggle.
app/tests/test_client.py	Migrates FastAPI tests to async httpx and improves restart/reset coverage.
app/tests/conftest.py	Switches fixture to async httpx client and uses shared helper for app reload/reset.
app/tests/client_helpers.py	Adds helper to create/teardown ASGI client while clearing modules and releasing RocksDB locks.
.github/workflows/cross-platform-perf.yaml	Adds cross-platform benchmark workflow for hybrid query performance.
.github/workflows/code-checker.yaml	Removes Python unit test job from code-checker workflow.
.github/workflows/cibuildwheel.yaml	Improves wheel job matrix labeling, caching, and adds artifact-upload retry logic.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[huanglune]

/gemini review

[gemini-code-assist]

Code Review

This pull request implements cross-platform compatibility and portable fallbacks for non-AVX512 architectures, refactors filesystem operations into platform-agnostic utilities, and enhances the Python API with columnar data fitting. The review identifies several issues in the new NPY and JSON parsers, including incomplete UTF-8 encoding for 4-byte characters, locale-dependent floating-point parsing, and a lack of endianness handling. Further feedback suggests optimizing hybrid search ground-truth generation with partial sorting and securing shell command construction against potential command injection.