Skip to content

Commit 19f6645

Browse files
sid-rlsid-rl
committed
smoke tests
1 parent 83f9b94 commit 19f6645

13 files changed

Lines changed: 3787 additions & 0 deletions

‎tests/smoketests/sdk/README.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# SDK End-to-End Smoke Tests
2+
3+
Comprehensive end-to-end tests for the object-oriented Python SDK (`runloop_api_client.sdk`). These tests run against the real Runloop API to validate critical workflows including devboxes, blueprints, snapshots, and storage objects.
4+
5+
## Overview
6+
7+
The Python SDK provides both synchronous and asynchronous interfaces:
8+
- **Synchronous SDK**: `RunloopSDK` with `Devbox`, `Blueprint`, `Snapshot`, `StorageObject`
9+
- **Asynchronous SDK**: `AsyncRunloopSDK` with `AsyncDevbox`, `AsyncBlueprint`, `AsyncSnapshot`, `AsyncStorageObject`
10+
11+
These tests ensure both interfaces work correctly in real-world scenarios.
12+
13+
## Test Files
14+
15+
### Infrastructure
16+
- `conftest.py` - Pytest fixtures for SDK client instances
17+
18+
### Devbox Tests
19+
- `test_devbox.py` - Synchronous devbox operations
20+
- `test_async_devbox.py` - Asynchronous devbox operations
21+
22+
**Test Coverage:**
23+
- Devbox lifecycle (create, get_info, shutdown)
24+
- Command execution (exec, exec_async) with streaming callbacks
25+
- File operations (read, write, upload, download)
26+
- State management (suspend, resume, await_running, await_suspended, keep_alive)
27+
- Networking (SSH keys, tunnels)
28+
- Creation from blueprints and snapshots
29+
- Snapshot creation
30+
- Context manager support
31+
32+
### Blueprint Tests
33+
- `test_blueprint.py` - Synchronous blueprint operations
34+
- `test_async_blueprint.py` - Asynchronous blueprint operations
35+
36+
**Test Coverage:**
37+
- Blueprint creation with dockerfiles and system setup commands
38+
- Blueprint listing and retrieval
39+
- Creating devboxes from blueprints
40+
- Blueprint deletion
41+
42+
### Snapshot Tests
43+
- `test_snapshot.py` - Synchronous snapshot operations
44+
- `test_async_snapshot.py` - Asynchronous snapshot operations
45+
46+
**Test Coverage:**
47+
- Snapshot creation from devboxes
48+
- Snapshot info and status tracking
49+
- Waiting for snapshot completion
50+
- Creating devboxes from snapshots
51+
- Snapshot listing and deletion
52+
53+
### Storage Object Tests
54+
- `test_storage_object.py` - Synchronous storage object operations
55+
- `test_async_storage_object.py` - Asynchronous storage object operations
56+
57+
**Test Coverage:**
58+
- Storage object lifecycle (create, upload, complete, delete)
59+
- Static upload methods (upload_from_text, upload_from_bytes, upload_from_file)
60+
- Download methods (download_as_text, download_as_bytes, get_download_url)
61+
- Storage object listing and retrieval
62+
- Mounting storage objects to devboxes
63+
64+
## Running the Tests
65+
66+
### Prerequisites
67+
68+
Set required environment variables:
69+
```bash
70+
export RUNLOOP_API_KEY=your_api_key_here
71+
# Optional: override the API base URL
72+
# export RUNLOOP_BASE_URL=https://api.runloop.ai
73+
```
74+
75+
### Run All SDK Smoke Tests
76+
```bash
77+
RUN_SMOKETESTS=1 uv run pytest -q -vv -m smoketest tests/smoketests/sdk/
78+
```
79+
80+
### Run Specific Test File
81+
```bash
82+
RUN_SMOKETESTS=1 uv run pytest -q -vv -m smoketest tests/smoketests/sdk/test_devbox.py
83+
```
84+
85+
### Run Specific Test
86+
```bash
87+
RUN_SMOKETESTS=1 uv run pytest -q -vv -m smoketest -k "test_devbox_lifecycle" tests/smoketests/sdk/
88+
```
89+
90+
### Run Only Sync or Async Tests
91+
```bash
92+
# Sync tests only (files without 'async' prefix)
93+
RUN_SMOKETESTS=1 uv run pytest -q -vv -m smoketest tests/smoketests/sdk/test_devbox.py tests/smoketests/sdk/test_blueprint.py tests/smoketests/sdk/test_snapshot.py tests/smoketests/sdk/test_storage_object.py
94+
95+
# Async tests only (files with 'async' prefix)
96+
RUN_SMOKETESTS=1 uv run pytest -q -vv -m smoketest tests/smoketests/sdk/test_async_*.py
97+
```
98+
99+
## Test Patterns
100+
101+
### Resource Management
102+
All tests include proper cleanup using try/finally blocks or pytest fixtures to ensure resources (devboxes, blueprints, etc.) are deleted after testing, even if tests fail.
103+
104+
### Timeouts
105+
Tests use appropriate timeouts based on operation types:
106+
- **30 seconds**: Quick operations (create, retrieve, delete)
107+
- **2+ minutes**: Long-running operations (devbox creation, blueprint builds)
108+
109+
### Sequential Tests
110+
Some tests within a file may be dependent on each other to save time and resources. Tests are designed to be run sequentially within each file.
111+
112+
### Naming Convention
113+
Tests use the `unique_name()` utility to generate unique resource names with timestamps, preventing conflicts between test runs.
114+
115+
## Differences from TypeScript SDK
116+
117+
The Python SDK includes both synchronous and asynchronous implementations, whereas the TypeScript SDK is async-only. This necessitates separate test files for each variant to ensure both work correctly.
118+
119+
Key differences:
120+
- Python uses `async`/`await` syntax for async operations
121+
- Python has separate classes: `Devbox` vs `AsyncDevbox`, etc.
122+
- Python uses context managers (`with` statement) for resource cleanup
123+
- Python async tests require `pytest.mark.asyncio` decorator
124+
125+
## CI Integration
126+
127+
These tests can be integrated into CI workflows similar to other smoketests. Set the `RUNLOOP_API_KEY` secret in your CI environment and run with the `RUN_SMOKETESTS=1` environment variable.
128+
129+
Example GitHub Actions workflow step:
130+
```yaml
131+
- name: Run SDK Smoke Tests
132+
env:
133+
RUNLOOP_API_KEY: ${{ secrets.RUNLOOP_SMOKETEST_API_KEY }}
134+
RUN_SMOKETESTS: 1
135+
run: |
136+
uv run pytest -q -vv -m smoketest tests/smoketests/sdk/
137+
```
138+

‎tests/smoketests/sdk/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# SDK End-to-End Smoke Tests
2+

‎tests/smoketests/sdk/conftest.py‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
"""Pytest fixtures for SDK end-to-end smoke tests."""
2+
3+
from __future__ import annotations
4+
5+
import os
6+
from typing import Iterator, AsyncIterator
7+
8+
import pytest
9+
10+
from runloop_api_client.sdk import RunloopSDK, AsyncRunloopSDK
11+
12+
13+
@pytest.fixture(scope="module")
14+
def sdk_client() -> Iterator[RunloopSDK]:
15+
"""Provide a synchronous RunloopSDK client for tests.
16+
17+
Reads configuration from environment variables:
18+
- RUNLOOP_API_KEY: Required API key
19+
- RUNLOOP_BASE_URL: Optional API base URL
20+
"""
21+
base_url = os.getenv("RUNLOOP_BASE_URL")
22+
bearer_token = os.getenv("RUNLOOP_API_KEY")
23+
24+
if not bearer_token:
25+
pytest.skip("RUNLOOP_API_KEY environment variable not set")
26+
27+
client = RunloopSDK(
28+
bearer_token=bearer_token,
29+
base_url=base_url,
30+
)
31+
32+
try:
33+
yield client
34+
finally:
35+
try:
36+
client.close()
37+
except Exception:
38+
pass
39+
40+
41+
@pytest.fixture(scope="module")
42+
async def async_sdk_client() -> AsyncIterator[AsyncRunloopSDK]:
43+
"""Provide an asynchronous AsyncRunloopSDK client for tests.
44+
45+
Reads configuration from environment variables:
46+
- RUNLOOP_API_KEY: Required API key
47+
- RUNLOOP_BASE_URL: Optional API base URL
48+
"""
49+
base_url = os.getenv("RUNLOOP_BASE_URL")
50+
bearer_token = os.getenv("RUNLOOP_API_KEY")
51+
52+
if not bearer_token:
53+
pytest.skip("RUNLOOP_API_KEY environment variable not set")
54+
55+
client = AsyncRunloopSDK(
56+
bearer_token=bearer_token,
57+
base_url=base_url,
58+
)
59+
60+
try:
61+
async with client:
62+
yield client
63+
except Exception:
64+
# If context manager fails, try manual cleanup
65+
try:
66+
await client.aclose()
67+
except Exception:
68+
pass

0 commit comments

Comments
 (0)