Enhance Python bindings and examples with new datasets and tests

tae898 · tae898 · commit 9e0b40032d5b · 2026-02-05T16:42:48.000+01:00
- Updated documentation to reflect the addition of TPC-H, LDBC SNB, and MSMARCO datasets.
- Improved dataset downloader with new options for TPC-H, LDBC SNB, and MSMARCO.
- Added functionality to download and convert TPC-H and LDBC SNB datasets using Docker.
- Enhanced the dataset downloader script to include detailed dataset descriptions and usage examples.
- Increased test coverage by adding a new OLTP mixed workload test for concurrent read/write operations.
- Updated test statistics to reflect the addition of new tests, bringing the total to 260.
- Minor documentation updates across various README files to improve clarity and usability.
diff --git a/bindings/python/README.md b/bindings/python/README.md
@@ -2,7 +2,7 @@
 
 Native Python bindings for ArcadeDB - the multi-model database that supports Graph, Document, Key/Value, Search Engine, Time Series, and Vector models.
 
-**Status**: ✅ Production Ready | **Tests**: 259 Passing | **Platforms**: 4 Supported
+**Status**: ✅ Production Ready | **Tests**: 260 Passing | **Platforms**: 4 Supported
 
 ---
 
@@ -92,7 +92,7 @@ Import: `import arcadedb_embedded as arcadedb`
 
 ## 🧪 Testing
 
-**Status**: 259 tests + example scripts passing on all 4 platforms
+**Status**: 260 tests + example scripts passing on all 4 platforms
 
 ```bash
 # Run all tests
diff --git a/bindings/python/docs/development/build-architecture.md b/bindings/python/docs/development/build-architecture.md
@@ -19,7 +19,7 @@ This document describes the build architecture for creating platform-specific Py
 
 **All supported platforms:**
 
-- ✅ 259 tests passing
+- ✅ 260 tests passing
 - ✅ 31.7M JARs (83 files, identical across platforms)
 - ✅ All native runners (no QEMU emulation)
 - ✅ Reproducible builds (pinned runner versions)
diff --git a/bindings/python/docs/development/ci-setup.md b/bindings/python/docs/development/ci-setup.md
@@ -98,11 +98,11 @@ After a successful release, you should see:
 
 ### Test Results (CI run #96)
 
-All 4 platforms passing 259 tests and example scripts:
+All 4 platforms passing 260 tests and example scripts:
 
 | Platforms | Wheel Size | JRE Size | Tests |
 |-----------|-----------|----------|-------|
-| linux/amd64, linux/arm64, darwin/arm64, windows/amd64 | ~68M | ~60M | 259 passed ✅ |
+| linux/amd64, linux/arm64, darwin/arm64, windows/amd64 | ~68M | ~60M | 260 passed ✅ |
 
 **All platforms include:**
 
diff --git a/bindings/python/docs/development/testing.md b/bindings/python/docs/development/testing.md
@@ -3,9 +3,9 @@
 Comprehensive testing documentation for ArcadeDB Python bindings.
 
 !!! success "Test Coverage"
-    **259 tests** across 20 test files, 100% passing
+    **260 tests** across 20 test files, 100% passing
 
-    - **Current package**: 259 passed, 6 skipped
+    - **Current package**: 260 passed, 6 skipped
     - All ArcadeDB features working (SQL, OpenCypher, Studio)
 
 ## Quick Navigation
diff --git a/bindings/python/docs/development/testing/overview.md b/bindings/python/docs/development/testing/overview.md
@@ -5,7 +5,7 @@ The ArcadeDB Python bindings have a comprehensive test suite covering all major
 ## Quick Statistics
 
 !!! success "Test Results"
-    - **Current package**: ✅ 259 passed, 6 skipped (259 collected)
+    - **Current package**: ✅ 260 passed, 6 skipped (260 collected)
     - All features available (SQL, OpenCypher, Studio UI, Vector search)
 
 ## What's Tested
@@ -129,7 +129,7 @@ pytest -m "not slow"
 When all tests pass, you should see:
 
 ```
-======================== 259 passed in 9.67s =========================
+======================== 260 passed in 9.67s =========================
 ```
 
 
diff --git a/bindings/python/docs/development/testing/test-concurrency.md b/bindings/python/docs/development/testing/test-concurrency.md
@@ -2,7 +2,7 @@
 
 [View source code]({{ config.repo_url }}/blob/{{ config.extra.version_tag }}/bindings/python/tests/test_concurrency.py){ .md-button }
 
-There are 4 tests demonstrating file locking, thread safety, sequential access, and concurrent limitation.
+There are 5 tests demonstrating file locking, thread safety, sequential access, concurrent limitation, and a mixed OLTP-style workload.
 
 ## Key Insight
 
@@ -28,6 +28,10 @@ Sequence: create+insert → close → reopen+query (1 record) → close → reop
 
 Attempts concurrent access from same process; documents that a second handle to locked DB raises `ArcadeDBError`.
 
+### 5) OLTP mixed workload (multi-thread)
+
+Runs a mixed read/write workload across multiple threads, measures basic latency/throughput, and includes retry/backoff for concurrent modifications.
+
 ## Architecture
 
 - **Single-process (embedded)**: Thread-safe; file lock prevents other processes
diff --git a/bindings/python/docs/examples/04_csv_import_documents.md b/bindings/python/docs/examples/04_csv_import_documents.md
@@ -48,6 +48,8 @@ python download_data.py movielens-large # movielens large dataset
 python download_data.py movielens-small # movielens small dataset
 ```
 
+**Note:** You must download the dataset before running this example if the auto-download is disabled or blocked by your environment.
+
 **Two dataset sizes available:**
 
 - **movielens-large**: ~86,000 movies, ~33M ratings (~265 MB) - Realistic performance testing
diff --git a/bindings/python/docs/examples/07_stackoverflow_multimodel.md b/bindings/python/docs/examples/07_stackoverflow_multimodel.md
@@ -17,6 +17,8 @@ system, showcasing ArcadeDB's true multi-model capabilities.
 
 The example supports multiple dataset sizes from the [Stack Exchange Data Dump](https://archive.org/details/stackexchange).
 
+**Note:** Download the dataset first with `python download_data.py stackoverflow-<size>` before running this example.
+
 | Dataset | Size (XML) | Records | Recommended Heap |
 | :--- | :--- | :--- | :--- |
 | **Tiny** (`cs.stackexchange.com`) | ~34 MB | ~100K | 2 GB |
diff --git a/bindings/python/docs/examples/08_server_mode_rest_api.md b/bindings/python/docs/examples/08_server_mode_rest_api.md
@@ -1,9 +1,9 @@
 # Example 08: Server Mode, Studio & Concurrent HTTP Clients
 
-This example demonstrates how to run ArcadeDB in **Server Mode**, which enables the HTTP REST API and the Studio Web UI while maintaining embedded Python access. It also showcases **concurrent load testing** and **polyglot querying** (SQL + OpenCypher).
-
 [View source code]({{ config.repo_url }}/blob/{{ config.extra.version_tag }}/bindings/python/examples/08_server_mode_rest_api.py){ .md-button }
 
+This example demonstrates how to run ArcadeDB in **Server Mode**, which enables the HTTP REST API and the Studio Web UI while maintaining embedded Python access. It also showcases **concurrent load testing** and **polyglot querying** (SQL + OpenCypher).
+
 ## Overview
 
 !!! note "Embedded vs Server"
diff --git a/bindings/python/docs/examples/basic.md b/bindings/python/docs/examples/basic.md
@@ -106,6 +106,7 @@ with arcadedb.create_database("./mydb") as db:
 Browse all examples:
 
 - **[Examples Overview](index.md)** - Complete list of examples
+- **[Dataset Downloader](download_data.md)** - Download and prepare example datasets
 - **[01 - Simple Document Store](01_simple_document_store.md)** - Document database basics
 - **[02 - Social Network Graph](02_social_network_graph.md)** - Graph database basics
 - **[03 - Vector Search](03_vector_search.md)** - AI-powered semantic search
diff --git a/bindings/python/docs/examples/download_data.md b/bindings/python/docs/examples/download_data.md
@@ -0,0 +1,68 @@
+# Dataset Downloader
+
+[View source code]({{ config.repo_url }}/blob/{{ config.extra.version_tag }}/bindings/python/examples/download_data.py){ .md-button }
+
+Use `download_data.py` to fetch and prepare datasets for the examples. Run it from the `examples/` directory so paths resolve correctly.
+
+## Quick Start
+
+```bash
+cd bindings/python/examples
+python download_data.py movielens-small
+```
+
+All datasets are stored under `bindings/python/examples/data/`.
+
+## Supported Datasets
+
+- **MovieLens**: `movielens-small`, `movielens-large`
+- **Stack Exchange**: `stackoverflow-small`, `stackoverflow-medium`, `stackoverflow-large`
+- **TPC-H**: `tpch-sf1`, `tpch-sf10`, `tpch-sf100`
+- **LDBC SNB Interactive v1**: `ldbc-snb-sf1`, `ldbc-snb-sf10`, `ldbc-snb-sf100`
+- **MSMARCO v2.1**: `msmarco-1m`, `msmarco-5m`, `msmarco-10m`
+
+## Usage
+
+```bash
+python download_data.py movielens-large
+python download_data.py stackoverflow-small
+python download_data.py tpch-sf1
+python download_data.py ldbc-snb-sf1
+python download_data.py msmarco-1m
+```
+
+## Notes
+
+- **MovieLens NULL injection** is enabled by default (use `--no-nulls` to skip).
+- **TPC-H** uses `dbgen` to generate `.tbl` files (pipe-delimited text, not SQL) via Docker (gcc image).
+    - Converted CSVs are written to `examples/data/tpch-sf<scale>/csv/`.
+    - A schema file is written to `examples/data/tpch-sf<scale>/schema.json`.
+- **LDBC SNB** is generated locally via Docker (ldbc/datagen).
+    - CSVs are stored under `examples/data/ldbc-snb-sf<scale>/`.
+    - A schema file is written to `examples/data/ldbc-snb-sf<scale>/schema.json` (inferred from CSV headers and samples).
+- **MSMARCO** downloads parquet shards and converts them to vector shards with a ground-truth file.
+
+## Dependencies
+
+Install only what you need for the datasets you plan to download:
+
+- Stack Exchange: `py7zr`
+- MSMARCO: `huggingface_hub`, `numpy`, `pyarrow`
+- TPC-H: Docker (gcc image for `dbgen`)
+- LDBC SNB: Docker (ldbc/datagen image)
+
+## Output Locations
+
+- MovieLens: `examples/data/movielens-<size>/`
+- Stack Exchange: `examples/data/stackoverflow-<size>/`
+- TPC-H: `examples/data/tpch-sf<scale>/`
+- LDBC SNB: `examples/data/ldbc-snb-sf<scale>/`
+- MSMARCO: `examples/data/MSMARCO-<size>/`
+
+## Formats & Schemas
+
+- **MovieLens**: CSV files, no schema file generated.
+- **Stack Exchange**: XML files, no schema file generated.
+- **TPC-H**: `.tbl` plus derived CSVs (pipe-delimited); schema in `schema.json`.
+- **LDBC SNB**: CSVs; schema in `schema.json` (inferred).
+- **MSMARCO**: binary vector shards (`.f32`) plus `.meta.json` and `.gt.jsonl`.
diff --git a/bindings/python/docs/examples/import.md b/bindings/python/docs/examples/import.md
@@ -2,6 +2,8 @@
 
 This page covers examples for importing data into ArcadeDB from various sources.
 
+Before running any import example, download the datasets using **[Dataset Downloader](download_data.md)**.
+
 ## CSV Import Examples
 
 ### Import Tabular Data (Documents)
diff --git a/bindings/python/docs/examples/index.md b/bindings/python/docs/examples/index.md
@@ -6,6 +6,9 @@ Hands-on examples demonstrating ArcadeDB Python bindings in real-world scenarios
 
 ### 🏁 Getting Started
 
+**[Dataset Downloader](download_data.md)**
+Download and prepare datasets used by the examples (MovieLens, Stack Exchange, TPC-H, LDBC SNB Interactive, MSMARCO).
+
 **[01 - Simple Document Store](01_simple_document_store.md)**
 Foundation example covering document types, CRUD operations, comprehensive data types (DATE, DATETIME, DECIMAL, FLOAT, INTEGER, STRING, BOOLEAN, LIST OF STRING), and NULL value handling (INSERT NULL, UPDATE to NULL, IS NULL queries).
 
diff --git a/bindings/python/docs/getting-started/distributions.md b/bindings/python/docs/getting-started/distributions.md
@@ -34,7 +34,7 @@ Pre-built **platform-specific** wheels are available for **4 platforms**. Sizes
 - ✅ All platforms use **platform-specific wheels** (not universal)
 - ✅ uv pip automatically selects the correct wheel for your system
 - ✅ Each platform has its own bundled JRE optimized for that architecture
-- ✅ All supported platforms tested and verified (259/259 tests passing)
+- ✅ All supported platforms tested and verified (260/260 tests passing)
 - ✅ Built on native runners (no emulation) for optimal performance
 
 ## What's Included
@@ -56,7 +56,7 @@ Pre-built **platform-specific** wheels are available for **4 platforms**. Sizes
 
 ## Test Results
 
-**259 out of 259 tests pass** on all platforms (100% success rate):
+**260 out of 260 tests pass** on all platforms (100% success rate):
 
 - ✅ All core database operations
 - ✅ SQL and OpenCypher queries
diff --git a/bindings/python/docs/index.md b/bindings/python/docs/index.md
@@ -9,7 +9,7 @@
     Native Python bindings for ArcadeDB with full test coverage
 
     - **Status**: ✅ Production Ready
-    - **Tests**: ✅ 259 + 7 Examples Passing
+    - **Tests**: ✅ 260 + 7 Examples Passing
 
 -   :fontawesome-brands-python:{ .lg .middle } **Pure Python API**
 
diff --git a/bindings/python/examples/README.md b/bindings/python/examples/README.md
@@ -14,6 +14,8 @@ cd bindings/python/examples
 python 01_simple_document_store.py
 ```
 
+See the dataset downloader guide: [docs/examples/download_data.md](../docs/examples/download_data.md).
+
 ## Available Examples
 
 ### 📄 [01_simple_document_store.py](./01_simple_document_store.py)
@@ -68,6 +70,8 @@ High-performance CSV import for document data:
 - Batch processing for optimal performance
 - Index creation strategies
 
+**Note:** Download the MovieLens dataset first with `python download_data.py movielens-<size>`.
+
 **Learn:** ETL patterns, bulk import, schema management, performance tuning
 
 **Status:** ✅ Fully functional - Imports 100K+ records efficiently
@@ -116,6 +120,8 @@ Comprehensive Stack Overflow dataset analysis:
 - Handling large datasets (XML parsing, batching)
 - Advanced schema design with inheritance and polymorphism
 
+**Note:** Download the Stack Exchange dataset first with `python download_data.py stackoverflow-<size>`.
+
 **Learn:** Multi-model architecture, complex ETL, advanced querying, full-stack analytics
 
 **Status:** ✅ Fully functional - Validated on Stack Exchange datasets
diff --git a/bindings/python/examples/download_data.py b/bindings/python/examples/download_data.py
diff --git a/bindings/python/mkdocs.yml b/bindings/python/mkdocs.yml
diff --git a/bindings/python/tests/README.md b/bindings/python/tests/README.md
diff --git a/bindings/python/tests/test_concurrency.py b/bindings/python/tests/test_concurrency.py
