Skip to content

Commit a814c63

Browse files
vdusekclaude
vdusek
and
claude
authored
test: split integration tests into integration and e2e suites (#781)
## Summary - Moved Actor-based tests (which build and deploy Actors on the Apify platform) from `tests/integration/actor/` to `tests/e2e/` - Flattened `tests/integration/apify_api/` into `tests/integration/` (removed unnecessary nesting) - Created dedicated `tests/e2e/conftest.py` merging fixtures from `tests/integration/conftest.py` and the old `tests/integration/actor/conftest.py` - Added `e2e-tests` and `e2e-tests-cov` poe tasks in `pyproject.toml` - Added `e2e_tests` CI job in `.github/workflows/_tests.yaml` - Updated READMEs for both test directories - Added ruff lint exception for e2e tests (matching integration tests config) This gives a clearer separation: - **Integration tests** (`tests/integration/`): Make real API calls but run locally, no Actor builds needed - **E2E tests** (`tests/e2e/`): Build and deploy Actors on the Apify platform Test collection: 77 integration + 64 e2e = 141 total (unchanged). ## Test plan - [x] `uv run poe lint` — passes - [x] `uv run poe unit-tests` — 249 tests pass - [x] `uv run pytest --collect-only tests/integration` — 77 tests collected - [x] `uv run pytest --collect-only tests/e2e` — 64 tests collected - [x] CI integration tests pass - [x] CI e2e tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent cf36cc2 commit a814c63

31 files changed

+346
-176
lines changed

‎.github/workflows/_tests.yaml‎

Lines changed: 116 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,119 @@ jobs:
2020
tests_concurrency: "1"
2121

2222
integration_tests:
23-
name: Integration tests
24-
uses: apify/workflows/.github/workflows/python_integration_tests.yaml@main
25-
secrets: inherit
26-
with:
27-
python_versions: '["3.10", "3.14"]'
28-
operating_systems: '["ubuntu-latest"]'
29-
python_version_for_codecov: "3.14"
30-
operating_system_for_codecov: ubuntu-latest
31-
tests_concurrency: "16"
23+
name: Integration tests (${{ matrix.python-version }}, ${{ matrix.os }})
24+
25+
if: >-
26+
${{
27+
(github.event_name == 'pull_request' && github.event.pull_request.head.repo.owner.login == 'apify') ||
28+
(github.event_name == 'push' && github.ref == 'refs/heads/master')
29+
}}
30+
31+
strategy:
32+
matrix:
33+
os: ["ubuntu-latest"]
34+
python-version: ["3.10", "3.14"]
35+
36+
runs-on: ${{ matrix.os }}
37+
38+
env:
39+
TESTS_CONCURRENCY: "16"
40+
41+
steps:
42+
- name: Checkout repository
43+
uses: actions/checkout@v6
44+
45+
- name: Set up Python ${{ matrix.python-version }}
46+
uses: actions/setup-python@v6
47+
with:
48+
python-version: ${{ matrix.python-version }}
49+
50+
- name: Set up uv package manager
51+
uses: astral-sh/setup-uv@v7
52+
with:
53+
python-version: ${{ matrix.python-version }}
54+
55+
- name: Install Python dependencies
56+
run: uv run poe install-dev
57+
58+
- name: Run integration tests
59+
run: uv run poe integration-tests-cov
60+
env:
61+
APIFY_TEST_USER_API_TOKEN: ${{ secrets.APIFY_TEST_USER_PYTHON_SDK_API_TOKEN }}
62+
APIFY_TEST_USER_2_API_TOKEN: ${{ secrets.APIFY_TEST_USER_2_API_TOKEN }}
63+
64+
- name: Upload integration test coverage
65+
if: >-
66+
${{
67+
matrix.os == 'ubuntu-latest' &&
68+
matrix.python-version == '3.14' &&
69+
env.CODECOV_TOKEN != ''
70+
}}
71+
uses: codecov/codecov-action@v5
72+
env:
73+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
74+
with:
75+
token: ${{ env.CODECOV_TOKEN }}
76+
files: coverage-integration.xml
77+
flags: integration
78+
79+
e2e_tests:
80+
name: E2E tests (${{ matrix.python-version }}, ${{ matrix.os }})
81+
82+
if: >-
83+
${{
84+
(github.event_name == 'pull_request' && github.event.pull_request.head.repo.owner.login == 'apify') ||
85+
(github.event_name == 'push' && github.ref == 'refs/heads/master')
86+
}}
87+
88+
strategy:
89+
# E2E tests build and run Actors on the platform. Limit parallel workflows to 1 to avoid exceeding
90+
# the platform's memory limits. A single workflow with 16 pytest workers provides good test
91+
# parallelization while staying within platform constraints.
92+
max-parallel: 1
93+
matrix:
94+
os: ["ubuntu-latest"]
95+
python-version: ["3.10", "3.14"]
96+
97+
runs-on: ${{ matrix.os }}
98+
99+
env:
100+
TESTS_CONCURRENCY: "16"
101+
102+
steps:
103+
- name: Checkout repository
104+
uses: actions/checkout@v6
105+
106+
- name: Set up Python ${{ matrix.python-version }}
107+
uses: actions/setup-python@v6
108+
with:
109+
python-version: ${{ matrix.python-version }}
110+
111+
- name: Set up uv package manager
112+
uses: astral-sh/setup-uv@v7
113+
with:
114+
python-version: ${{ matrix.python-version }}
115+
116+
- name: Install Python dependencies
117+
run: uv run poe install-dev
118+
119+
- name: Run E2E tests
120+
run: uv run poe e2e-tests-cov
121+
env:
122+
APIFY_TEST_USER_API_TOKEN: ${{ secrets.APIFY_TEST_USER_PYTHON_SDK_API_TOKEN }}
123+
APIFY_TEST_USER_2_API_TOKEN: ${{ secrets.APIFY_TEST_USER_2_API_TOKEN }}
124+
125+
- name: Upload E2E test coverage
126+
if: >-
127+
${{
128+
matrix.os == 'ubuntu-latest' &&
129+
matrix.python-version == '3.14' &&
130+
env.CODECOV_TOKEN != ''
131+
}}
132+
uses: codecov/codecov-action@v5
133+
env:
134+
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
135+
with:
136+
token: ${{ env.CODECOV_TOKEN }}
137+
files: coverage-e2e.xml
138+
flags: e2e

‎pyproject.toml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -145,15 +145,13 @@ indent-style = "space"
145145
"**/{tests}/*" = [
146146
"D", # Everything from the pydocstyle
147147
"INP001", # File {filename} is part of an implicit namespace package, add an __init__.py
148+
"PLC0415", # `import` should be at the top-level of a file
148149
"PLR2004", # Magic value used in comparison, consider replacing {value} with a constant variable
149150
"S101", # Use of assert detected
150151
"SLF001", # Private member accessed: `{name}`
151152
"T20", # flake8-print
152-
"TRY301", # Abstract `raise` to an inner function
153153
"TID252", # Prefer absolute imports over relative imports from parent modules
154-
]
155-
"**/{tests}/{integration}/*" = [
156-
"PLC0415", # `import` should be at the top-level of a file
154+
"TRY301", # Abstract `raise` to an inner function
157155
]
158156
"**/{docs,website}/**" = [
159157
"D", # Everything from the pydocstyle
@@ -234,6 +232,8 @@ unit-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/unit
234232
unit-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-unit.xml tests/unit"
235233
integration-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/integration"
236234
integration-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-integration.xml tests/integration"
235+
e2e-tests = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} tests/e2e"
236+
e2e-tests-cov = "uv run pytest --numprocesses=${TESTS_CONCURRENCY:-auto} --cov=src/apify --cov-report=xml:coverage-e2e.xml tests/e2e"
237237
check-code = ["lint", "type-check", "unit-tests"]
238238

239239
[tool.poe.tasks.install-dev]

‎tests/e2e/README.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# E2E tests
2+
3+
These tests build and run Actors using the Python SDK on the Apify platform. They are slower than integration tests (see [`tests/integration/`](../integration/)) because they need to build and deploy Actors.
4+
5+
When writing new tests, prefer integration tests if possible. Only write E2E tests when you need to test something that requires building and running an Actor on the platform.
6+
7+
## Running
8+
9+
```bash
10+
export APIFY_TEST_USER_API_TOKEN=<your-token>
11+
uv run poe e2e-tests
12+
```
13+
14+
To run against a different environment, also set `APIFY_INTEGRATION_TESTS_API_URL`.
15+
16+
## Key fixtures
17+
18+
- **`apify_client_async`** — A session-scoped `ApifyClientAsync` instance configured with the test token and API URL.
19+
- **`prepare_test_env`** / **`_isolate_test_environment`** (autouse) — Resets global state and sets `APIFY_LOCAL_STORAGE_DIR` to a temporary directory before each test.
20+
- **`make_actor`** — Factory for creating temporary Actors on the Apify platform (built, then auto-deleted after the test).
21+
- **`run_actor`** — Starts an Actor run and waits for completion (10 min timeout).
22+
23+
## How to write tests
24+
25+
### Creating an Actor from a Python function
26+
27+
You can create Actors straight from a Python function. This is great because the test Actor source code gets checked by the linter.
28+
29+
```python
30+
async def test_something(
31+
make_actor: MakeActorFunction,
32+
run_actor: RunActorFunction,
33+
) -> None:
34+
async def main() -> None:
35+
async with Actor:
36+
print('Hello!')
37+
38+
actor = await make_actor(label='something', main_func=main)
39+
run_result = await run_actor(actor)
40+
41+
assert run_result.status == 'SUCCEEDED'
42+
```
43+
44+
The `src/main.py` file will be set to the function definition, prepended with `import asyncio` and `from apify import Actor`. You can add extra imports directly inside the function body.
45+
46+
### Creating an Actor from source files
47+
48+
Pass the `main_py` argument for a single-file Actor:
49+
50+
```python
51+
async def test_something(
52+
make_actor: MakeActorFunction,
53+
run_actor: RunActorFunction,
54+
) -> None:
55+
expected_output = f'ACTOR_OUTPUT_{crypto_random_object_id(5)}'
56+
main_py_source = f"""
57+
import asyncio
58+
from datetime import datetime
59+
from apify import Actor
60+
async def main():
61+
async with Actor:
62+
await Actor.set_value('OUTPUT', '{expected_output}')
63+
"""
64+
65+
actor = await make_actor(label='something', main_py=main_py_source)
66+
await run_actor(actor)
67+
68+
output_record = await actor.last_run().key_value_store().get_record('OUTPUT')
69+
assert output_record is not None
70+
assert output_record['value'] == expected_output
71+
```
72+
73+
Or pass `source_files` for multi-file Actors:
74+
75+
```python
76+
actor_source_files = {
77+
'src/utils.py': """
78+
from datetime import datetime, timezone
79+
def get_current_datetime():
80+
return datetime.now(timezone.utc)
81+
""",
82+
'src/main.py': """
83+
import asyncio
84+
from apify import Actor
85+
from .utils import get_current_datetime
86+
async def main():
87+
async with Actor:
88+
print('Hello! It is ' + str(get_current_datetime()))
89+
""",
90+
}
91+
actor = await make_actor(label='something', source_files=actor_source_files)
92+
```
93+
94+
### Assertions inside Actors
95+
96+
Since test Actors are not executed as standard pytest tests, we don't get introspection of assertion expressions. In case of failure, only a bare `AssertionError` is shown. Always include explicit assertion messages:
97+
98+
```python
99+
assert is_finished is False, f'is_finished={is_finished}'
100+
```

tests/integration/actor/__init__.py renamed to tests/e2e/__init__.py

File renamed without changes.

‎tests/e2e/_utils.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
from __future__ import annotations
2+
3+
from crawlee._utils.crypto import crypto_random_object_id
4+
5+
6+
def generate_unique_resource_name(label: str) -> str:
7+
"""Generates a unique resource name, which will contain the given label."""
8+
name_template = 'python-sdk-tests-{}-generated-{}'
9+
template_length = len(name_template.format('', ''))
10+
api_name_limit = 63
11+
generated_random_id_length = 8
12+
label_length_limit = api_name_limit - template_length - generated_random_id_length
13+
14+
label = label.replace('_', '-')
15+
assert len(label) <= label_length_limit, f'Max label length is {label_length_limit}, but got {len(label)}'
16+
17+
return name_template.format(label, crypto_random_object_id(generated_random_id_length))

tests/integration/actor/actor_source_base/Dockerfile renamed to tests/e2e/actor_source_base/Dockerfile

File renamed without changes.

tests/integration/actor/actor_source_base/requirements.txt renamed to tests/e2e/actor_source_base/requirements.txt

File renamed without changes.

tests/integration/actor/actor_source_base/server.py renamed to tests/e2e/actor_source_base/server.py

File renamed without changes.

tests/integration/actor/actor_source_base/src/__init__.py renamed to tests/e2e/actor_source_base/src/__init__.py

File renamed without changes.

tests/integration/actor/actor_source_base/src/__main__.py renamed to tests/e2e/actor_source_base/src/__main__.py

File renamed without changes.

0 commit comments

Comments
 (0)