Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion docs/02_concepts/01_actor_lifecycle.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,15 @@ When the Actor exits, either normally or due to an exception, the SDK performs a
</TabItem>
</Tabs>

You can also create an [`Actor`](https://docs.apify.com/sdk/python/reference/class/Actor) instance directly. This does not change its capabilities but allows you to specify optional parameters during initialization, such as disabling automatic `sys.exit()` calls or customizing timeouts. The choice between using a context manager or manual initialization depends on how much control you require over the Actor's startup and shutdown sequence.
You can also create an [`Actor`](https://docs.apify.com/sdk/python/reference/class/Actor) instance directly. This does not change its capabilities but allows you to specify optional parameters during initialization. The key parameters are:

- `configuration` — a custom [`Configuration`](https://docs.apify.com/sdk/python/reference/class/Configuration) instance to control storage paths, API URLs, and other settings.
- `configure_logging` — whether to set up default logging configuration (default `True`). Set to `False` if you configure logging yourself.
- `exit_process` — whether the Actor calls `sys.exit()` when the context manager exits. Defaults to `True`, except in IPython, Pytest, and Scrapy environments.
- `event_listeners_timeout` — maximum time to wait for Actor event listeners to complete before exiting.
- `cleanup_timeout` — maximum time to wait for cleanup tasks to finish (default 30 seconds).

The choice between using a context manager or manual initialization depends on how much control you require over the Actor's startup and shutdown sequence.

<Tabs groupId="request_queue">
<TabItem value="actor_instance_with_context_manager" label="Actor instance with context manager" default>
Expand Down
16 changes: 16 additions & 0 deletions docs/02_concepts/02_actor_input.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ description: Read and validate input data passed to your Actor at runtime.
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import InputExample from '!!raw-loader!roa-loader!./code/02_input.py';
import RequestListExample from '!!raw-loader!roa-loader!./code/02_request_list.py';
import ApiLink from '@site/src/components/ApiLink';

The Actor gets its [input](https://docs.apify.com/platform/actors/running/input) from the input record in its default [key-value store](https://docs.apify.com/platform/storage/key-value-store).

Expand All @@ -17,3 +19,17 @@ For example, if an Actor received a JSON input with two fields, `{ "firstNumber"
<RunnableCodeBlock className="language-python" language="python">
{InputExample}
</RunnableCodeBlock>

## Loading URLs from Actor input

Actors commonly receive a list of URLs to process via their input. The <ApiLink to="class/ApifyRequestList">`ApifyRequestList`</ApiLink> class (from `apify.request_loaders`) can parse the standard Apify input format for URL sources. It supports both direct URL objects (`{"url": "https://example.com"}`) and remote URL lists (`{"requestsFromUrl": "https://example.com/urls.txt"}`), where the remote file contains one URL per line.

<RunnableCodeBlock className="language-python" language="python">
{RequestListExample}
</RunnableCodeBlock>

## Secret input fields

The Apify platform supports [secret input fields](https://docs.apify.com/platform/actors/development/secret-input) that are encrypted before being stored. When you mark an input field as `"isSecret": true` in your Actor's [input schema](https://docs.apify.com/platform/actors/development/input-schema), the platform encrypts the value with the Actor's public key.

No special handling is needed in your code — when you call [`Actor.get_input`](../../reference/class/Actor#get_input), encrypted fields are automatically decrypted using the Actor's private key, which is provided by the platform via environment variables. You receive the plaintext values directly.
21 changes: 20 additions & 1 deletion docs/02_concepts/03_storages.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ description: Use datasets, key-value stores, and request queues to persist Actor
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import OpeningStoragesExample from '!!raw-loader!roa-loader!./code/03_opening_storages.py';
import OpeningStoragesAliasExample from '!!raw-loader!roa-loader!./code/03_opening_storages_alias.py';
import ApiLink from '@site/src/components/ApiLink';
import DeletingStoragesExample from '!!raw-loader!roa-loader!./code/03_deleting_storages.py';
import DatasetReadWriteExample from '!!raw-loader!roa-loader!./code/03_dataset_read_write.py';
import DatasetExportsExample from '!!raw-loader!roa-loader!./code/03_dataset_exports.py';
Expand Down Expand Up @@ -60,7 +62,7 @@ There are several methods for directly working with the default key-value store
- [`Actor.get_value('my-record')`](../../reference/class/Actor#get_value) reads a record from the default key-value store of the Actor.
- [`Actor.set_value('my-record', 'my-value')`](../../reference/class/Actor#set_value) saves a new value to the record in the default key-value store.
- [`Actor.get_input`](../../reference/class/Actor#get_input) reads the Actor input from the default key-value store of the Actor.
- [`Actor.push_data([{'result': 'Hello, world!'}, ...])`](../../reference/class/Actor#push_data) saves results to the default dataset of the Actor.
- [`Actor.push_data([{'result': 'Hello, world!'}, ...])`](../../reference/class/Actor#push_data) saves results to the default dataset of the Actor. When using the [pay-per-event pricing model](./pay-per-event), `push_data` returns a `ChargeResult` object that indicates whether the charge limit has been reached. You can also pass a `charged_event_name` parameter to charge for a custom event for each pushed item.

## Opening named and unnamed storages

Expand All @@ -70,6 +72,12 @@ The [`Actor.open_dataset`](../../reference/class/Actor#open_dataset), [`Actor.op
{OpeningStoragesExample}
</RunnableCodeBlock>

Besides `id` and `name`, the `open_*` methods also accept an `alias` parameter. An alias creates an unnamed storage scoped to the current Actor run — it does not persist across runs, but lets you reference the same storage within a single run using a human-readable label. The `alias` parameter is mutually exclusive with `id` and `name`.

<RunnableCodeBlock className="language-python" language="python">
{OpeningStoragesAliasExample}
</RunnableCodeBlock>

## Deleting storages

To delete a storage, you can use the [`Dataset.drop`](../../reference/class/Dataset#drop),
Expand Down Expand Up @@ -172,3 +180,14 @@ To check if all the requests in the queue are handled, you can use the [`Request
<RunnableCodeBlock className="language-python" language="python">
{RqExample}
</RunnableCodeBlock>

## Storage clients

Behind the scenes, the SDK uses storage clients to communicate with the storage backend. The SDK automatically selects the appropriate client based on the runtime environment:

- **`SmartApifyStorageClient`** (default on the Apify platform) — a hybrid client that writes to both the Apify API and the local filesystem for resilience.
- **`ApifyStorageClient`** — communicates directly with the Apify platform API for cloud storage.
- **`FileSystemStorageClient`** — stores data on the local filesystem (in the `storage/` directory). Used when running locally.
- **`MemoryStorageClient`** (from Crawlee) — stores data in memory. Useful for testing.

For most use cases, the default storage client selection is sufficient. All storage clients are available from the `apify.storage_clients` module. For details, see the <ApiLink to="class/ApifyStorageClient">`ApifyStorageClient`</ApiLink> API reference.
22 changes: 22 additions & 0 deletions docs/02_concepts/04_actor_events.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ description: Handle platform events like state persistence and graceful shutdown
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import ActorEventsExample from '!!raw-loader!roa-loader!./code/04_actor_events.py';
import UseStateExample from '!!raw-loader!roa-loader!./code/04_use_state.py';
import ApiLink from '@site/src/components/ApiLink';

During its runtime, the Actor receives Actor events sent by the Apify platform or generated by the Apify SDK itself.

Expand Down Expand Up @@ -69,6 +71,14 @@ During its runtime, the Actor receives Actor events sent by the Apify platform o
you can achieve the same effect by persisting the state regularly in an interval and listening for the migrating event.
</td>
</tr>
<tr>
<td><code>EXIT</code></td>
<td><code>None</code></td>
<td>
Emitted when the Actor is about to exit. You can use this event to perform final cleanup tasks,
such as closing external connections or sending notifications, before the Actor shuts down.
</td>
</tr>
</tbody>
</table>

Expand All @@ -80,3 +90,15 @@ and to remove them, you use the [`Actor.off`](../../reference/class/Actor#off) m
<RunnableCodeBlock className="language-python" language="python">
{ActorEventsExample}
</RunnableCodeBlock>

## Automatic state persistence with use_state

The example above shows how to manually persist state using the `PERSIST_STATE` event. For most use cases, you can use the <ApiLink to="class/Actor#use_state">`Actor.use_state`</ApiLink> method instead, which handles state persistence automatically.

`Actor.use_state` returns a dictionary that is automatically saved to the default key-value store at regular intervals and whenever a migration or shutdown occurs. You can modify the dictionary in place, and changes are persisted without any manual `set_value` calls.

You can optionally specify a `key` (the key-value store key under which the state is stored) and a `kvs_name` (the name of the key-value store to use). By default, the state is stored in the default key-value store under a default key.

<RunnableCodeBlock className="language-python" language="python">
{UseStateExample}
</RunnableCodeBlock>
13 changes: 13 additions & 0 deletions docs/02_concepts/05_proxy_management.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import ApifyProxyConfig from '!!raw-loader!roa-loader!./code/05_apify_proxy_conf
import CustomProxyFunctionExample from '!!raw-loader!roa-loader!./code/05_custom_proxy_function.py';
import ProxyActorInputExample from '!!raw-loader!roa-loader!./code/05_proxy_actor_input.py';
import ProxyHttpxExample from '!!raw-loader!roa-loader!./code/05_proxy_httpx.py';
import TieredProxyExample from '!!raw-loader!roa-loader!./code/05_tiered_proxy.py';
import ApiLink from '@site/src/components/ApiLink';

The Apify SDK provides built-in proxy management through the <ApiLink to="class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> class, supporting both [Apify Proxy](https://apify.com/proxy) and custom proxy servers. Proxies are essential for web scraping to avoid [IP address blocking](https://en.wikipedia.org/wiki/IP_address_blocking) and distribute requests across multiple addresses.
Expand Down Expand Up @@ -81,6 +82,18 @@ Or you can pass it a method (accepting one optional argument, the session ID), t
{CustomProxyFunctionExample}
</RunnableCodeBlock>

### Tiered proxy rotation

<ApiLink to="class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> supports tiered proxy URLs via the `tiered_proxy_urls` parameter. This accepts a list of lists of proxy URLs, where each inner list represents a tier. The proxy rotator starts with the first (cheapest) tier and automatically escalates to higher tiers when lower-tier proxies get blocked. This is useful for optimizing proxy costs — you use cheap datacenter proxies for most requests and only switch to expensive residential proxies when necessary.

:::info
The `tiered_proxy_urls` parameter is only available when constructing `ProxyConfiguration` directly. It is not supported by `Actor.create_proxy_configuration()`.
:::

<RunnableCodeBlock className="language-python" language="python">
{TieredProxyExample}
</RunnableCodeBlock>

### Configuring proxy based on Actor input

To make selecting the proxies that the Actor uses easier, you can use an input field with the editor [`proxy` in your input schema](https://docs.apify.com/platform/actors/development/input-schema#object). This input will then be filled with a dictionary containing the proxy settings you or the users of your Actor selected for the Actor run.
Expand Down
11 changes: 11 additions & 0 deletions docs/02_concepts/06_interacting_with_other_actors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import InteractingStartExample from '!!raw-loader!roa-loader!./code/06_interacti
import InteractingCallExample from '!!raw-loader!roa-loader!./code/06_interacting_call.py';
import InteractingCallTaskExample from '!!raw-loader!roa-loader!./code/06_interacting_call_task.py';
import InteractingMetamorphExample from '!!raw-loader!roa-loader!./code/06_interacting_metamorph.py';
import InteractingAbortExample from '!!raw-loader!roa-loader!./code/06_interacting_abort.py';
import ApiLink from '@site/src/components/ApiLink';

The Apify SDK lets you start, call, and transform (metamorph) other Actors directly from your Actor code. This is useful for composing complex workflows from smaller, reusable Actors.
Expand Down Expand Up @@ -52,4 +53,14 @@ For example, imagine you have an Actor that accepts a hotel URL on input, and th
{InteractingMetamorphExample}
</RunnableCodeBlock>

## Aborting an Actor run

The [`Actor.abort`](../../reference/class/Actor#abort) method aborts a running Actor on the Apify platform. You can use it to cancel a long-running Actor that is no longer needed.

When you set `gracefully=True`, the platform sends `ABORTING` and `PERSIST_STATE` events to the target Actor, giving it time to save its state, and then force-stops it after 30 seconds. Without the `gracefully` flag, the Actor is stopped immediately.

<RunnableCodeBlock className="language-python" language="python">
{InteractingAbortExample}
</RunnableCodeBlock>

For the full list of methods for interacting with other Actors, see the <ApiLink to="class/Actor">`Actor`</ApiLink> API reference.
18 changes: 18 additions & 0 deletions docs/02_concepts/10_configuration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ description: Customize Actor behavior through the Configuration class or environ
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

import ConfigExample from '!!raw-loader!roa-loader!./code/10_config.py';
import GetEnvExample from '!!raw-loader!roa-loader!./code/10_get_env.py';
import PlatformDetectionExample from '!!raw-loader!roa-loader!./code/10_platform_detection.py';
import ApiLink from '@site/src/components/ApiLink';

The <ApiLink to="class/Actor">`Actor`</ApiLink> class is configured through the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class, which reads its settings from environment variables. When running on the Apify platform or through the Apify CLI, configuration is automatic — manual setup is only needed for custom requirements.
Expand All @@ -33,4 +35,20 @@ This Actor run will not persist its local storages to the filesystem:
APIFY_PERSIST_STORAGE=0 apify run
```

## Reading the runtime environment

The <ApiLink to="class/Actor#get_env">`Actor.get_env`</ApiLink> method returns a dictionary with all `APIFY_*` environment variables parsed into their typed values. This is useful for inspecting the Actor's runtime context, such as the Actor ID, run ID, or default storage IDs. Variables that are not set or are invalid will have a value of `None`.

<RunnableCodeBlock className="language-python" language="python">
{GetEnvExample}
</RunnableCodeBlock>

## Platform detection

The <ApiLink to="class/Actor#is_at_home">`Actor.is_at_home`</ApiLink> method returns `True` when the Actor is running on the Apify platform, and `False` when running locally. This is useful for branching behavior based on the environment, such as using different storage backends or skipping proxy configuration during local development.

<RunnableCodeBlock className="language-python" language="python">
{PlatformDetectionExample}
</RunnableCodeBlock>

For the full list of configuration options, see the <ApiLink to="class/Configuration">`Configuration`</ApiLink> API reference.
18 changes: 18 additions & 0 deletions docs/02_concepts/11_pay_per_event.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ description: Monetize your Actors using the pay-per-event pricing model
import ActorChargeSource from '!!raw-loader!roa-loader!./code/11_actor_charge.py';
import ConditionalActorChargeSource from '!!raw-loader!roa-loader!./code/11_conditional_actor_charge.py';
import ChargeLimitCheckSource from '!!raw-loader!roa-loader!./code/11_charge_limit_check.py';
import AdvancedChargingExample from '!!raw-loader!roa-loader!./code/11_advanced_charging.py';
import ApiLink from '@site/src/components/ApiLink';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';

Expand Down Expand Up @@ -48,6 +49,23 @@ Alternatively, you can periodically check the remaining budget via <ApiLink to="
Always check the charge limit in your Actor, whether through `ChargeResult` return values or the `ChargingManager`. Without this check, your Actor will continue running and consuming platform resources after the budget is exhausted, producing no output.
:::

## Advanced charging management

The <ApiLink to="class/ChargingManager">`ChargingManager`</ApiLink> (accessed via <ApiLink to="class/Actor#get_charging_manager">`Actor.get_charging_manager()`</ApiLink>) provides methods for fine-grained budget control:

- `get_max_total_charge_usd()` — the configured budget limit for this run.
- `calculate_total_charged_amount()` — total USD charged so far.
- `calculate_max_event_charge_count_within_limit(event_name)` — how many more events of this type can be charged before reaching the limit.
- `get_charged_event_count(event_name)` — how many events of this type have been charged.
- `is_event_charge_limit_reached(event_name)` — whether the remaining budget is too low for even one more event of this type.
- `compute_chargeable()` — a dict of all event types and how many can still be charged.

These methods are useful for budget-aware crawling strategies, where you want to plan work based on the remaining budget rather than discovering the limit after the fact.

<RunnableCodeBlock className="language-python" language="python">
{AdvancedChargingExample}
</RunnableCodeBlock>

## Transitioning from a different pricing model

When you plan to start using the pay-per-event pricing model for an Actor that is already monetized with a different pricing model, your source code will need support both pricing models during the transition period enforced by the Apify platform. Arguably the most frequent case is the transition from the pay-per-result model which utilizes the `ACTOR_MAX_PAID_DATASET_ITEMS` environment variable to prevent returning unpaid dataset items. The following is an example how to handle such scenarios. The key part is the <ApiLink to="class/ChargingManager#get_pricing_info">`ChargingManager.get_pricing_info()`</ApiLink> method which returns information about the current pricing model.
Expand Down
29 changes: 29 additions & 0 deletions docs/02_concepts/code/02_request_list.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
import asyncio

from apify import Actor
from apify.request_loaders import ApifyRequestList


async def main() -> None:
async with Actor:
actor_input = await Actor.get_input() or {}

# The input may contain a list of URL sources in the standard Apify format
request_list_sources = actor_input.get('requestListSources', [])

# Create a request list from the input sources.
# Supports direct URLs and remote URL lists.
request_list = await ApifyRequestList.open(
request_list_sources_input=request_list_sources,
)

total = await request_list.get_total_count()
Actor.log.info(f'Loaded {total} requests from input')

# Process requests from the list
while request := await request_list.fetch_next_request():
Actor.log.info(f'Processing {request.url}')


if __name__ == '__main__':
asyncio.run(main())
20 changes: 20 additions & 0 deletions docs/02_concepts/code/03_opening_storages_alias.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import asyncio

from apify import Actor


async def main() -> None:
async with Actor:
# Open a dataset with an alias — this creates an unnamed dataset
# that can be referenced by this alias within the current run
dataset = await Actor.open_dataset(alias='intermediate-results')
await dataset.push_data({'step': 1, 'result': 'partial data'})

# Later, open the same dataset using the same alias
same_dataset = await Actor.open_dataset(alias='intermediate-results')
data = await same_dataset.get_data()
Actor.log.info(f'Items in dataset: {data.count}')


if __name__ == '__main__':
asyncio.run(main())
Loading
Loading