Add changes for compatibility with WASM components and collocated UDF servers (#121)

Clean up Python SDK for WASM compatibility and add new Plugin API server for UDFs. This includes various changes such as:

  * Lazy load pandas, polars, pyarrow, numpy
  * Move other imports such as JWT inside functions that use them
  * Add experimental Plugin API server

Author: kesmit13

.github/workflows/fusion-docs.yml

@@ -1,7 +1,6 @@
 on:
-  push:
-    tags:
-      - 'v*.*.*'
+  release:
+    types: [published]
 
 name: Generate Fusion docs
 
@@ -36,6 +35,6 @@ jobs:
 
       - name: Upload release asset
         run: |
-          gh release upload ${{ github.ref_name }} fusion-docs.zip
+          gh release upload ${{ github.event.release.tag_name }} fusion-docs.zip
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/publish.yml

@@ -7,6 +7,12 @@
 name: Publish packages
 
 on:
+  push:
+    tags:
+      - 'v*-rc*'
+      - 'v*-test*'
+      - 'v*-alpha*'
+      - 'v*-beta*'
   release:
     types: [published]
   workflow_dispatch:
@@ -159,13 +165,15 @@ jobs:
     permissions:
       id-token: write   # Required for OIDC trusted publishing
       actions: read     # Required for actions/download-artifact
-      contents: read    # Required for repository access
+      contents: write   # Required for gh release create/upload
 
     environment:
       name: publish
       url: https://pypi.org/p/singlestoredb
 
     steps:
+      - uses: actions/checkout@v3
+
       - name: Download Linux wheels and sdist
         uses: actions/download-artifact@v4
         with:
@@ -184,6 +192,20 @@ jobs:
           name: artifacts-macOS
           path: dist
 
+      - name: Create GitHub Release
+        if: ${{ github.event_name == 'push' }}
+        run: |
+          gh release create ${{ github.ref_name }} dist/* --prerelease --title "${{ github.ref_name }}"
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Upload release assets
+        if: ${{ github.event_name == 'release' }}
+        run: |
+          gh release upload ${{ github.event.release.tag_name }} dist/* --clobber
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
       - name: Publish to PyPI
         if: ${{ github.event_name == 'release' || github.event.inputs.publish_pypi == 'true' }}
         uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -83,7 +83,7 @@ dev-docs
 **/.ipynb_checkpoints
 **/.benchmarks
 *.ipynb
-test*.py
+/test*.py
 certs
 **/*.prof
 **/*.pprof

ARCHITECTURE.md

@@ -99,7 +99,14 @@ singlestoredb/
 │       ├── mmap.py          # Memory-mapped execution
 │       ├── json.py          # JSON serialization
 │       ├── rowdat_1.py      # ROWDAT_1 format
-│       └── arrow.py         # Apache Arrow format
+│       ├── arrow.py         # Apache Arrow format
+│       └── plugin/          # Plugin UDF server (Unix socket)
+│           ├── __main__.py  # CLI entry point
+│           ├── server.py    # Socket server with thread/process pool
+│           ├── connection.py # Connection handling
+│           ├── control.py   # Control protocol
+│           ├── registry.py  # Function registry & discovery
+│           └── wasm.py      # WASM component interface
 │
 ├── ai/                      # AI/ML integration
 │   ├── chat.py              # Chat completion factory
@@ -603,9 +610,9 @@ Located in `singlestoredb/fusion/handlers/`:
 ## External Functions (UDFs)
 
 The functions module (`singlestoredb/functions/`) enables deploying Python functions
-as SingleStore external functions. UDF servers can be deployed as HTTP using
-an ASGI application or a collocated socket server that uses mmap files to
-transfer data.
+as SingleStore external functions. UDF servers can be deployed in three modes: as HTTP using an ASGI application
+(remote), a memory-mapped collocated server (lowest latency), or as a plugin
+server using Unix sockets with a thread/process pool (CLI-driven).
 
 ### Architecture
 
@@ -629,6 +636,7 @@ transfer data.
 ├─────────────────────────────────────────────────────────────────────┤
 │  asgi.py    │  HTTP server via ASGI (Uvicorn; JSON or ROWDAT_1)     │
 │  mmap.py    │  Memory-mapped shared memory (collocated; ROWDAT_1)   │
+│  plugin/    │  Plugin UDF server (Unix socket + thread/process pool) │
 │  json.py    │  JSON serialization over HTTP                         │
 │  rowdat_1.py│  ROWDAT_1 binary format                               │
 │  arrow.py   │  Apache Arrow columnar format                         │
@@ -659,6 +667,33 @@ python -m singlestoredb.functions.ext.asgi \
     my_functions
 ```
 
+### Plugin Server CLI
+
+The plugin server can be launched via the CLI for Unix socket-based UDF serving:
+
+```bash
+# Launch the plugin server
+python -m singlestoredb.functions.ext.plugin \
+    --plugin-name myfuncs \
+    --search-path /home/user/libs \
+    --socket /tmp/my-udf.sock
+
+# Or use the console_scripts entry point
+python-udf-server --plugin-name myfuncs
+```
+
+**CLI Arguments:**
+
+| Argument | Env Variable | Default | Description |
+|----------|-------------|---------|-------------|
+| `--plugin-name` | `PLUGIN_NAME` | (required) | Python module to import |
+| `--search-path` | `PLUGIN_SEARCH_PATH` | `""` | Colon-separated search dirs for the module |
+| `--socket` | `PLUGIN_SOCKET_PATH` | auto-generated | Unix socket path |
+| `--n-workers` | `PLUGIN_N_WORKERS` | `0` (CPU count) | Worker threads/processes |
+| `--max-connections` | `PLUGIN_MAX_CONNECTIONS` | `32` | Socket backlog |
+| `--log-level` | `PLUGIN_LOG_LEVEL` | `info` | Logging level (debug/info/warning/error) |
+| `--process-mode` | `PLUGIN_PROCESS_MODE` | `process` | Concurrency mode: `thread` or `process` |
+
 ### Type Mapping
 
 The `signature.py` module maps Python types to SQL types:
@@ -682,29 +717,30 @@ The `signature.py` module maps Python types to SQL types:
 ┌─────────────────────────────────────────────────────────────────────┐
 │                      SingleStore Database                           │
 └─────────────────────────────────────────────────────────────────────┘
-                   │                    │
-                   │ ASGI/HTTP          │ Memory-mapped
-                   │ (remote)           │ (collocated)
-                   ▼                    ▼
-             ┌─────────────┐      ┌─────────────┐
-             │   asgi.py   │      │   mmap.py   │
-             │   Uvicorn   │      │   Shared    │
-             │   HTTP/2    │      │   Memory    │
-             └─────────────┘      └─────────────┘
-                   │                    │
-                   └────────────────────┘
-                              │
-                              ▼
-                      ┌─────────────────┐
-                      │  Python UDF     │
-                      │  Functions      │
-                      └─────────────────┘
+                   │                    │                    │
+                   │ ASGI/HTTP          │ Memory-mapped      │ Plugin
+                   │ (remote)           │ (collocated)       │ (Unix socket)
+                   ▼                    ▼                    ▼
+             ┌─────────────┐      ┌─────────────┐      ┌─────────────┐
+             │   asgi.py   │      │   mmap.py   │      │   plugin/   │
+             │   Uvicorn   │      │   Shared    │      │   Unix sock │
+             │   HTTP/2    │      │   Memory    │      │   + pool    │
+             └─────────────┘      └─────────────┘      └─────────────┘
+                   │                    │                    │
+                   └────────────────────┴────────────────────┘
+                                        │
+                                        ▼
+                                ┌─────────────────┐
+                                │  Python UDF     │
+                                │  Functions      │
+                                └─────────────────┘
 ```
 
 | Mode | File | Use Case |
 |------|------|----------|
 | ASGI | `asgi.py` | Remote execution via HTTP, scalable |
 | Memory-mapped | `mmap.py` | Collocated execution, lowest latency |
+| Plugin | `plugin/` | Unix socket server, thread/process pool, CLI-driven |
 | JSON | `json.py` | Simple serialization, debugging |
 | ROWDAT_1 | `rowdat_1.py` | Binary format, efficient |
 | Arrow | `arrow.py` | Columnar format, analytics |
@@ -1043,6 +1079,13 @@ with free_tier.start() as server:
 | `SINGLESTOREDB_MANAGEMENT_TOKEN` | Management API token | None |
 | `SINGLESTORE_LICENSE` | License key for Docker | None |
 | `USE_DATA_API` | Use HTTP API for tests | 0 |
+| `PLUGIN_NAME` | Plugin server: Python module to import | None |
+| `PLUGIN_SEARCH_PATH` | Plugin server: colon-separated module search dirs | `""` |
+| `PLUGIN_SOCKET_PATH` | Plugin server: Unix socket path | auto-generated |
+| `PLUGIN_N_WORKERS` | Plugin server: worker count (0 = CPU count) | `0` |
+| `PLUGIN_MAX_CONNECTIONS` | Plugin server: socket backlog | `32` |
+| `PLUGIN_LOG_LEVEL` | Plugin server: logging level | `info` |
+| `PLUGIN_PROCESS_MODE` | Plugin server: `thread` or `process` | `process` |
 
 ### B. Cursor Type Matrix
 
@@ -1096,4 +1139,5 @@ Feature options:
 | Management API | `singlestoredb/management/workspace.py` |
 | Fusion handlers | `singlestoredb/fusion/handler.py` |
 | UDF decorator | `singlestoredb/functions/decorator.py` |
+| Plugin UDF server | `singlestoredb/functions/ext/plugin/server.py` |
 | Test fixtures | `singlestoredb/pytest.py` |

CONTRIBUTING.md

@@ -182,6 +182,10 @@ pytest -v --cov=singlestoredb.connection singlestoredb/tests/test_connection.py
 # Test UDF functionality
 pytest singlestoredb/tests/test_udf.py
 
+# Manual testing of the plugin UDF server
+python -m singlestoredb.functions.ext.plugin \
+    --plugin-name myfuncs --search-path /path/to/modules
+
 # Test against specific server (skips Docker)
 SINGLESTOREDB_URL=admin:pass@localhost:3306 pytest -v singlestoredb/tests
 

README.md

@@ -18,7 +18,7 @@ analytics and vector search.
 - **Vector Store**: Pinecone-compatible vector database API for similarity search
   applications with built-in connection pooling
 - **User-Defined Functions**: Deploy Python functions as SingleStore UDFs with
-  automatic type mapping
+  automatic type mapping (HTTP/ASGI or plugin-mode via CLI)
 - **SQLAlchemy Support**: Integrate with SQLAlchemy through the optional
   `sqlalchemy-singlestoredb` adapter
 - **Fusion SQL**: Extend SQL with custom client-side command handlers
@@ -53,6 +53,9 @@ pip install 'singlestoredb[rsa]'
 # Ed25519 key authentication
 pip install 'singlestoredb[ed22519]'
 
+# Pytest plugin with Docker test containers
+pip install 'singlestoredb[pytest,docker]'
+
 # Multiple extras can be combined
 pip install 'singlestoredb[vectorstore,sqlalchemy]'
 ```
@@ -66,6 +69,8 @@ pip install 'singlestoredb[vectorstore,sqlalchemy]'
 | `kerberos` / `gssapi` | Kerberos/GSSAPI authentication support |
 | `rsa` | RSA key exchange for encrypted connections |
 | `ed22519` | Ed25519 key authentication |
+| `docker` | Docker SDK for automated test container management |
+| `pytest` | Pytest plugin with SingleStoreDB Docker test fixtures |
 
 ## Documentation
 
@@ -250,6 +255,59 @@ conn.execute("""
 See [singlestoredb/fusion/README.md](singlestoredb/fusion/README.md)
 for details on writing custom Fusion SQL handlers.
 
+## Pytest Plugin
+
+The SDK includes a pytest plugin that automatically manages SingleStoreDB Docker
+containers for integration testing. Install with:
+
+```bash
+pip install 'singlestoredb[pytest,docker]'
+```
+
+The plugin provides these fixtures:
+
+- `singlestoredb_test_container` (session-scoped): Starts and stops a
+  SingleStoreDB Dev container, or reuses an existing server if
+  `SINGLESTOREDB_URL` is set
+- `singlestoredb_connection` (session-scoped): Database connection to the
+  test container
+- `singlestoredb_tempdb` (function-scoped): Cursor with a fresh temporary
+  database, dropped after the test
+
+The plugin supports `pytest-xdist` parallel execution with leader/follower
+coordination for container lifecycle.
+
+```python
+def test_query(singlestoredb_tempdb):
+    singlestoredb_tempdb.execute('CREATE TABLE t (id INT)')
+    singlestoredb_tempdb.execute('INSERT INTO t VALUES (1)')
+    singlestoredb_tempdb.execute('SELECT * FROM t')
+    assert singlestoredb_tempdb.fetchone() == (1,)
+```
+
+## Plugin UDF Server
+
+The SDK ships a high-performance plugin-mode UDF server that runs as a
+standalone process, communicating with SingleStoreDB over a Unix socket.
+
+```bash
+# Via the installed CLI entry point
+python-udf-server --plugin-name myfuncs --search-path /path/to/modules
+
+# Or as a Python module
+python -m singlestoredb.functions.ext.plugin --plugin-name myfuncs
+```
+
+| Option | Env Variable | Default | Description |
+|--------|-------------|---------|-------------|
+| `--plugin-name` | `PLUGIN_NAME` | (required) | Python module to import |
+| `--search-path` | `PLUGIN_SEARCH_PATH` | `""` | Colon-separated module search dirs |
+| `--socket` | `PLUGIN_SOCKET_PATH` | auto-generated | Unix socket path |
+| `--n-workers` | `PLUGIN_N_WORKERS` | `0` (CPU count) | Worker threads/processes |
+| `--max-connections` | `PLUGIN_MAX_CONNECTIONS` | `32` | Socket backlog |
+| `--log-level` | `PLUGIN_LOG_LEVEL` | `info` | Logging level |
+| `--process-mode` | `PLUGIN_PROCESS_MODE` | `process` | `thread` or `process` concurrency |
+
 ## Advanced Options
 
 ### SSL/TLS Configuration