Merge branch 'master' into apify-default-dataset-item-event

Author: Mantisus

.gitignore

@@ -15,9 +15,9 @@
 .serena
 .windsurf
 .zed-ai
-AGENTS.md
-CLAUDE.md
-GEMINI.md
+AGENTS.local.md
+CLAUDE.local.md
+GEMINI.local.md
 
 # Cache
 __pycache__

.rules.md

@@ -0,0 +1,116 @@
+# Coding guidelines
+
+This file provides guidance to programming agents when working with code in this repository.
+
+## Project Overview
+
+The Apify SDK for Python (`apify` package on PyPI) is the official library for creating [Apify Actors](https://docs.apify.com/platform/actors) in Python. It provides Actor lifecycle management, storage access (datasets, key-value stores, request queues), event handling, proxy configuration, and pay-per-event charging. It builds on top of the [Crawlee](https://crawlee.dev/python) web scraping framework and the [Apify API Client](https://docs.apify.com/api/client/python). Supports Python 3.10–3.14. Build system: hatchling.
+
+## Common Commands
+
+```bash
+# Install dependencies (including dev)
+uv sync --all-extras
+
+# Install dev dependencies + pre-commit hooks
+uv run poe install-dev
+
+# Format code (also auto-fixes lint issues via ruff check --fix)
+uv run poe format
+
+# Lint (format check + ruff check)
+uv run poe lint
+
+# Type check
+uv run poe type-check
+
+# Run all checks (lint + type-check + unit tests)
+uv run poe check-code
+
+# Unit tests (no API token needed)
+uv run poe unit-tests
+
+# Run a single test file
+uv run pytest tests/unit/actor/test_actor_lifecycle.py
+
+# Run a single test by name
+uv run pytest tests/unit/actor/test_actor_lifecycle.py -k "test_name"
+
+# Integration tests (needs APIFY_TEST_USER_API_TOKEN)
+uv run poe integration-tests
+
+# E2E tests (needs APIFY_TEST_USER_API_TOKEN, builds/deploys Actors on platform)
+uv run poe e2e-tests
+```
+
+## Code Style
+
+- **Formatter/Linter**: Ruff (line length 120, single quotes for inline, double quotes for docstrings)
+- **Type checker**: ty (targets Python 3.10)
+- **All ruff rules enabled** with specific ignores — see `pyproject.toml` `[tool.ruff.lint]` for the full ignore list
+- Tests are exempt from docstring rules (`D`), assert warnings (`S101`), and private member access (`SLF001`)
+- Unused imports are allowed in `__init__.py` files (re-exports)
+- **Pre-commit hooks**: lint check + type check run automatically on commit
+
+## Architecture
+
+### Core (`src/apify/`)
+
+- **`_actor.py`** — The `_ActorType` class is the central API. `Actor` is a lazy-object-proxy (`lazy-object-proxy.Proxy`) wrapping `_ActorType` — it acts as both a class (e.g. `Actor.is_at_home()`) and an instance-like context manager (`async with Actor:`). On `__aenter__`, the proxy's `__wrapped__` is replaced with the active `_ActorType` instance. It manages the full Actor lifecycle (`init`, `exit`, `fail`), provides access to storages (`open_dataset`, `open_key_value_store`, `open_request_queue`), handles events, proxy configuration, charging, and platform API operations (`start`, `call`, `metamorph`, `reboot`).
+
+- **`_configuration.py`** — `Configuration` extends Crawlee's `Configuration` with Apify-specific settings (API URL, token, Actor run metadata, proxy settings, charging config). Configuration is populated from environment variables (`APIFY_*`).
+
+- **`_charging.py`** — Pay-per-event billing system. `ChargingManager` / `ChargingManagerImplementation` handle charging events against pricing info fetched from the API.
+
+- **`_proxy_configuration.py`** — `ProxyConfiguration` manages Apify proxy setup (residential, datacenter, groups, country targeting).
+
+- **`_models.py`** — Pydantic models for API data structures (Actor runs, webhooks, pricing info, etc.).
+
+### Storage Clients (`src/apify/storage_clients/`)
+
+Four storage client implementations, all implementing Crawlee's abstract storage client interface:
+
+- **`_apify/`** — `ApifyStorageClient`: talks to the Apify API for dataset, key-value store, and request queue operations (separate sub-clients for single vs. shared request queues). Used when running on the Apify platform.
+- **`_file_system/`** — `FileSystemStorageClient` (alias `ApifyFileSystemStorageClient`): extends Crawlee's file system client with Apify-specific key-value store behavior.
+- **`_smart_apify/`** — `SmartApifyStorageClient`: hybrid client that writes to both API and local file system for resilience.
+- **`MemoryStorageClient`** — re-exported from Crawlee for in-memory storage.
+
+### Storages (`src/apify/storages/`)
+
+Re-exports Crawlee's `Dataset`, `KeyValueStore`, and `RequestQueue` classes.
+
+### Events (`src/apify/events/`)
+
+- **`_apify_event_manager.py`** — `ApifyEventManager` extends Crawlee's event system with platform-specific events received via WebSocket connection.
+
+### Request Loaders (`src/apify/request_loaders/`)
+
+- **`_apify_request_list.py`** — `ApifyRequestList` creates request lists from Actor input URLs (supports both direct URLs and "requests from URL" sources).
+
+### Scrapy Integration (`src/apify/scrapy/`)
+
+Optional integration (`apify[scrapy]` extra) providing Scrapy scheduler, middlewares, pipelines, and extensions for running Scrapy spiders as Apify Actors.
+
+### Key Dependencies
+
+- **`crawlee`** — Base framework providing storage abstractions, event system, configuration, service locator pattern
+- **`apify-client`** — HTTP client for the Apify API (`ApifyClientAsync`)
+- **`apify-shared`** — Shared constants and utilities (`ApifyEnvVars`, `ActorEnvVars`, etc.)
+
+## Testing
+
+Three test levels in `tests/`:
+
+- **`unit/`** — Fast tests with no external dependencies. Use mocked API clients (`ApifyClientAsyncPatcher` fixture). Run with `uv run poe unit-tests`.
+- **`integration/`** — Tests making real Apify API calls but not deploying Actors. Requires `APIFY_TEST_USER_API_TOKEN`. Run with `uv run poe integration-tests`.
+- **`e2e/`** — Full end-to-end tests that build and deploy Actors on the platform. Slowest. Requires `APIFY_TEST_USER_API_TOKEN`. Use `make_actor` and `run_actor` fixtures. Run with `uv run poe e2e-tests`.
+
+All test levels use `pytest-asyncio` with `asyncio_mode = "auto"` (no need for `@pytest.mark.asyncio`). Tests run in parallel via `pytest-xdist` (`--numprocesses`). Each test gets isolated state via the autouse `_isolate_test_environment` fixture which resets `Actor`, `service_locator`, and `AliasResolver` state. Conftest files live in each subdirectory (`tests/unit/conftest.py`, etc.) — there is no top-level `tests/conftest.py`.
+
+### Key Test Fixtures
+
+- **`apify_client_async_patcher`** (unit) — `ApifyClientAsyncPatcher` instance for mocking `ApifyClientAsync` methods. Patch by `method`/`submethod`, tracks call history in `.calls`.
+- **`make_httpserver`/`httpserver`** (unit) — session-scoped `HTTPServer` via `pytest-httpserver` for HTTP interception.
+- **`apify_client_async`** (integration/e2e) — real `ApifyClientAsync` using `APIFY_TEST_USER_API_TOKEN`.
+- **`make_actor`** (e2e) — creates a temporary Actor on the platform from a function, `main_py` string, or source files dict; cleans up after the session.
+- **`run_actor`** (e2e) — calls an Actor and waits up to 10 minutes for completion.

AGENTS.md

@@ -0,0 +1 @@
+.rules.md

CHANGELOG.md

@@ -3,15 +3,31 @@
 All notable changes to this project will be documented in this file.
 
 <!-- git-cliff-unreleased-start -->
-## 3.2.2 - **not yet released**
+## 3.3.1 - **not yet released**
+
+### 🐛 Bug Fixes
+
+- Fix f-string bugs in charging log message ([#817](https://github.com/apify/apify-sdk-python/pull/817)) ([bcb4050](https://github.com/apify/apify-sdk-python/commit/bcb4050b3f5ade0e577dd7499979dc65c0ba815e)) by [@vdusek](https://github.com/vdusek)
+- Fix BeforeValidator treating 0 as falsy in configuration fields ([#819](https://github.com/apify/apify-sdk-python/pull/819)) ([72efe88](https://github.com/apify/apify-sdk-python/commit/72efe883574ef0d05795934337c7f1c8c0d11877)) by [@vdusek](https://github.com/vdusek)
+- Clamp negative timedelta in _get_remaining_time() ([#818](https://github.com/apify/apify-sdk-python/pull/818)) ([69b8af9](https://github.com/apify/apify-sdk-python/commit/69b8af9b8d245cfed76875f983f374e05d93bba8)) by [@vdusek](https://github.com/vdusek)
+- **scrapy:** Close AsyncThread on scheduler open() failure ([#820](https://github.com/apify/apify-sdk-python/pull/820)) ([7dfaf1a](https://github.com/apify/apify-sdk-python/commit/7dfaf1a5c5af44743bd448a91140d9b074ac44bf)) by [@vdusek](https://github.com/vdusek)
+
+
+<!-- git-cliff-unreleased-end -->
+## [3.3.0](https://github.com/apify/apify-sdk-python/releases/tag/v3.3.0) (2026-02-25)
+
+### 🚀 Features
+
+- Support Actor schema storages with Alias mechanism ([#797](https://github.com/apify/apify-sdk-python/pull/797)) ([10986ac](https://github.com/apify/apify-sdk-python/commit/10986ac2f4a3d1112aa06eaf26f82884ab9c455a)) by [@Pijukatel](https://github.com/Pijukatel), closes [#762](https://github.com/apify/apify-sdk-python/issues/762)
+- Migrate to Scrapy&#x27;s native AsyncCrawlerRunner ([#793](https://github.com/apify/apify-sdk-python/pull/793)) ([01ad9da](https://github.com/apify/apify-sdk-python/commit/01ad9daf834894f798bbfa4362fc7d7f95bafe5c)) by [@vdusek](https://github.com/vdusek), closes [#638](https://github.com/apify/apify-sdk-python/issues/638)
 
 ### 🐛 Bug Fixes
 
 - Resolve LogRecord attribute conflict in event manager logging ([#802](https://github.com/apify/apify-sdk-python/pull/802)) ([e1bdbc9](https://github.com/apify/apify-sdk-python/commit/e1bdbc9e303c24571b9511f43ec0815e7e9f4b55)) by [@vdusek](https://github.com/vdusek)
 - Update models.py to align with the current API behavior ([#782](https://github.com/apify/apify-sdk-python/pull/782)) ([b06355d](https://github.com/apify/apify-sdk-python/commit/b06355dbc1c8276e9930ecbde72795b6570dde33)) by [@vdusek](https://github.com/vdusek), closes [#778](https://github.com/apify/apify-sdk-python/issues/778)
+- Handle `ServiceConflictError` when reusing `Actor` across sequential context ([#804](https://github.com/apify/apify-sdk-python/pull/804)) ([9e5078f](https://github.com/apify/apify-sdk-python/commit/9e5078fa7b1a19e44893bd3409b45108519aef63)) by [@Mantisus](https://github.com/Mantisus), closes [#678](https://github.com/apify/apify-sdk-python/issues/678)
 
 
-<!-- git-cliff-unreleased-end -->
 ## [3.2.1](https://github.com/apify/apify-sdk-python/releases/tag/v3.2.1) (2026-02-17)
 
 ### 🐛 Bug Fixes

CLAUDE.md

@@ -0,0 +1 @@
+.rules.md

GEMINI.md

@@ -0,0 +1 @@
+.rules.md

docs/02_concepts/11_pay_per_event.mdx

@@ -6,6 +6,7 @@ description: Monetize your Actors using the pay-per-event pricing model
 
 import ActorChargeSource from '!!raw-loader!roa-loader!./code/11_actor_charge.py';
 import ConditionalActorChargeSource from '!!raw-loader!roa-loader!./code/11_conditional_actor_charge.py';
+import ChargeLimitCheckSource from '!!raw-loader!roa-loader!./code/11_charge_limit_check.py';
 import ApiLink from '@site/src/components/ApiLink';
 import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
 
@@ -31,6 +32,22 @@ Then you just push your code to Apify and that's it! The SDK will even keep trac
 
 If you need finer control over charging, you can access call <ApiLink to="class/Actor#get_charging_manager">`Actor.get_charging_manager()`</ApiLink> to access the <ApiLink to="class/ChargingManager">`ChargingManager`</ApiLink>, which can provide more detailed information - for example how many events of each type can be charged before reaching the configured limit.
 
+### Handling the charge limit
+
+While the SDK automatically prevents overcharging by limiting how many events are charged and how many items are pushed, **it does not stop your Actor from running**. When the charge limit is reached, <ApiLink to="class/Actor#charge">`Actor.charge`</ApiLink> and `Actor.push_data` will silently stop charging and pushing data, but your Actor will keep running — potentially doing expensive work (scraping pages, calling APIs) for no purpose. This means your Actor may never terminate on its own if you don't check the charge limit yourself.
+
+To avoid this, you should check the `event_charge_limit_reached` field in the result returned by <ApiLink to="class/Actor#charge">`Actor.charge`</ApiLink> or `Actor.push_data` and stop your Actor when the limit is reached. You can also use the `chargeable_within_limit` field from the result to plan ahead — it tells you how many events of each type can still be charged within the remaining budget.
+
+<RunnableCodeBlock className="language-python" language="python">
+    {ChargeLimitCheckSource}
+</RunnableCodeBlock>
+
+Alternatively, you can periodically check the remaining budget via <ApiLink to="class/Actor#get_charging_manager">`Actor.get_charging_manager()`</ApiLink> instead of inspecting every `ChargeResult`. This can be useful when charging happens in multiple places across your code, or when using a crawler where you don't directly control the main loop.
+
+:::caution
+Always check the charge limit in your Actor, whether through `ChargeResult` return values or the `ChargingManager`. Without this check, your Actor will continue running and consuming platform resources after the budget is exhausted, producing no output.
+:::
+
 ## Transitioning from a different pricing model
 
 When you plan to start using the pay-per-event pricing model for an Actor that is already monetized with a different pricing model, your source code will need support both pricing models during the transition period enforced by the Apify platform. Arguably the most frequent case is the transition from the pay-per-result model which utilizes the `ACTOR_MAX_PAID_DATASET_ITEMS` environment variable to prevent returning unpaid dataset items. The following is an example how to handle such scenarios. The key part is the <ApiLink to="class/ChargingManager#get_pricing_info">`ChargingManager.get_pricing_info()`</ApiLink> method which returns information about the current pricing model.

docs/02_concepts/code/11_charge_limit_check.py

@@ -0,0 +1,29 @@
+import asyncio
+
+from apify import Actor
+
+
+async def main() -> None:
+    async with Actor:
+        urls = [
+            'https://example.com/1',
+            'https://example.com/2',
+            'https://example.com/3',
+        ]
+
+        for url in urls:
+            # Do some expensive work (e.g. scraping, API calls)
+            result = {'url': url, 'data': f'Scraped data from {url}'}
+
+            # highlight-start
+            # push_data returns a ChargeResult - check it to know if the budget ran out
+            charge_result = await Actor.push_data(result, 'result-item')
+
+            if charge_result.event_charge_limit_reached:
+                Actor.log.info('Charge limit reached, stopping the Actor')
+                break
+            # highlight-end
+
+
+if __name__ == '__main__':
+    asyncio.run(main())

docs/03_guides/06_scrapy.mdx

@@ -17,13 +17,13 @@ import SettingsExample from '!!raw-loader!./code/scrapy_project/src/settings.py'
 
 ## Integrating Scrapy with the Apify platform
 
-The Apify SDK provides an Apify-Scrapy integration. The main challenge of this is to combine two asynchronous frameworks that use different event loop implementations. Scrapy uses [Twisted](https://twisted.org/) for asynchronous execution, while the Apify SDK is based on [asyncio](https://docs.python.org/3/library/asyncio.html). The key thing is to install the Twisted's `asyncioreactor` to run Twisted's asyncio compatible event loop. This allows both Twisted and asyncio to run on a single event loop, enabling a Scrapy spider to run as an Apify Actor with minimal modifications.
+The Apify SDK provides an Apify-Scrapy integration. The main challenge of this is to combine two asynchronous frameworks that use different event loop implementations. Scrapy uses [Twisted](https://twisted.org/) for asynchronous execution, while the Apify SDK is based on [asyncio](https://docs.python.org/3/library/asyncio.html). The key thing is to install the Twisted's `asyncioreactor` to run Twisted's asyncio compatible event loop. The `apify.scrapy.run_scrapy_actor` function handles this reactor installation automatically. This allows both Twisted and asyncio to run on a single event loop, enabling a Scrapy spider to run as an Apify Actor with minimal modifications.
 
 <CodeBlock className="language-python" title="__main.py__: The Actor entry point ">
     {UnderscoreMainExample}
 </CodeBlock>
 
-In this setup, `apify.scrapy.initialize_logging` configures an Apify log formatter and reconfigures loggers to ensure consistent logging across Scrapy, the Apify SDK, and other libraries. The `apify.scrapy.run_scrapy_actor` bridges asyncio coroutines with Twisted's reactor, enabling the Actor's main coroutine, which contains the Scrapy spider, to be executed.
+In this setup, `apify.scrapy.initialize_logging` configures an Apify log formatter and reconfigures loggers to ensure consistent logging across Scrapy, the Apify SDK, and other libraries. The `apify.scrapy.run_scrapy_actor` installs Twisted's asyncio-compatible reactor and bridges asyncio coroutines with Twisted's reactor, enabling the Actor's main coroutine, which contains the Scrapy spider, to be executed.
 
 Make sure the `SCRAPY_SETTINGS_MODULE` environment variable is set to the path of the Scrapy settings module. This variable is also used by the `Actor` class to detect that the project is a Scrapy project, triggering additional actions.
 
@@ -47,7 +47,7 @@ Additional helper functions in the [`apify.scrapy`](https://github.com/apify/api
 - `apply_apify_settings` - Applies Apify-specific components to Scrapy settings.
 - `to_apify_request` and `to_scrapy_request` - Convert between Apify and Scrapy request objects.
 - `initialize_logging` - Configures logging for the Actor environment.
-- `run_scrapy_actor` - Bridges asyncio and Twisted event loops.
+- `run_scrapy_actor` - Installs Twisted's asyncio reactor and bridges asyncio and Twisted event loops.
 
 ## Create a new Apify-Scrapy project
 

docs/03_guides/code/scrapy_project/src/__main__.py

@@ -1,11 +1,5 @@
 from __future__ import annotations
 
-from scrapy.utils.reactor import install_reactor
-
-# Install Twisted's asyncio reactor before importing any other Twisted or
-# Scrapy components.
-install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
-
 import os
 
 from apify.scrapy import initialize_logging, run_scrapy_actor