Skip to content

Commit dc12f42

Browse files
vdusekclaude
andcommitted
docs: Add missing content to concept pages
Add iterate_items() generator-based iteration section to the pagination page, closing notes with API references to async support and error handling pages, and a new Timeouts concept page covering the tiered timeout system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 64ee8ab commit dc12f42

File tree

8 files changed

+168
-0
lines changed

8 files changed

+168
-0
lines changed

docs/02_concepts/01_async_support.mdx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ description: Use the async client for non-blocking API calls with Python asyncio
77
import Tabs from '@theme/Tabs';
88
import TabItem from '@theme/TabItem';
99
import CodeBlock from '@theme/CodeBlock';
10+
import ApiLink from '@site/src/components/ApiLink';
1011

1112
import AsyncSupportExample from '!!raw-loader!./code/01_async_support.py';
1213

@@ -17,3 +18,5 @@ The following example demonstrates how to run an Actor asynchronously and stream
1718
<CodeBlock className="language-python">
1819
{AsyncSupportExample}
1920
</CodeBlock>
21+
22+
For the full async client API, see the <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> reference.

docs/02_concepts/04_error_handling.mdx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ description: Handle API errors with the ApifyApiError exception and automatic da
77
import Tabs from '@theme/Tabs';
88
import TabItem from '@theme/TabItem';
99
import CodeBlock from '@theme/CodeBlock';
10+
import ApiLink from '@site/src/components/ApiLink';
1011

1112
import ErrorAsyncExample from '!!raw-loader!./code/04_error_async.py';
1213
import ErrorSyncExample from '!!raw-loader!./code/04_error_sync.py';
@@ -25,3 +26,5 @@ When you use the Apify client, it automatically extracts all relevant data from
2526
</CodeBlock>
2627
</TabItem>
2728
</Tabs>
29+
30+
For a complete list of error classes, see the <ApiLink to="class/ApifyApiError">`ApifyApiError`</ApiLink> reference.

docs/02_concepts/08_pagination.mdx

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ import CodeBlock from '@theme/CodeBlock';
1111
import PaginationAsyncExample from '!!raw-loader!./code/08_pagination_async.py';
1212
import PaginationSyncExample from '!!raw-loader!./code/08_pagination_sync.py';
1313

14+
import IterateItemsAsyncExample from '!!raw-loader!./code/08_iterate_items_async.py';
15+
import IterateItemsSyncExample from '!!raw-loader!./code/08_iterate_items_sync.py';
16+
1417
Most methods named `list` or `list_something` in the Apify client return a [`ListPage`](/reference/class/ListPage) object. This object provides a consistent interface for working with paginated data and includes the following properties:
1518

1619
- `items` - The main results you're looking for.
@@ -37,3 +40,24 @@ The following example demonstrates how to fetch all items from a dataset using p
3740
</Tabs>
3841

3942
The [`ListPage`](/reference/class/ListPage) interface offers several key benefits. Its consistent structure ensures predictable results for most `list` methods, providing a uniform way to work with paginated data. It also offers flexibility, allowing you to customize the `limit` and `offset` parameters to control data fetching according to your needs. Additionally, it provides scalability, enabling you to efficiently handle large datasets through pagination. This approach ensures efficient data retrieval while keeping memory usage under control, making it ideal for managing and processing large collections.
43+
44+
## Generator-based iteration
45+
46+
For most use cases, `iterate_items()` is the recommended way to process all items in a dataset. It handles pagination automatically using a Python generator, fetching items in batches behind the scenes so you don't need to manage offsets or limits yourself.
47+
48+
<Tabs>
49+
<TabItem value="AsyncExample" label="Async client" default>
50+
<CodeBlock className="language-python">
51+
{IterateItemsAsyncExample}
52+
</CodeBlock>
53+
</TabItem>
54+
<TabItem value="SyncExample" label="Sync client">
55+
<CodeBlock className="language-python">
56+
{IterateItemsSyncExample}
57+
</CodeBlock>
58+
</TabItem>
59+
</Tabs>
60+
61+
`iterate_items()` accepts the same filtering parameters as `list_items()` (`clean`, `fields`, `omit`, `unwind`, `skip_empty`, `skip_hidden`), so you can combine automatic pagination with data filtering.
62+
63+
Similarly, `KeyValueStoreClient` provides an `iterate_keys()` method for iterating over all keys in a key-value store without manual pagination.

docs/02_concepts/11_timeouts.mdx

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
id: timeouts
3+
title: Timeouts
4+
description: Configure the tiered timeout system for controlling how long API requests can take.
5+
---
6+
7+
import Tabs from '@theme/Tabs';
8+
import TabItem from '@theme/TabItem';
9+
import CodeBlock from '@theme/CodeBlock';
10+
import ApiLink from '@site/src/components/ApiLink';
11+
12+
import TimeoutsAsyncExample from '!!raw-loader!./code/11_timeouts_async.py';
13+
import TimeoutsSyncExample from '!!raw-loader!./code/11_timeouts_sync.py';
14+
15+
The Apify client uses a tiered timeout system to set appropriate time limits for different types of API requests. Each tier has a default value suited to its use case:
16+
17+
| Tier | Default | Purpose |
18+
|---|---|---|
19+
| `short` | 5 seconds | Fast CRUD operations (get, update, delete) |
20+
| `medium` | 30 seconds | Batch, list, and data transfer operations |
21+
| `long` | 360 seconds | Long-polling, streaming, and heavy operations |
22+
| `no_timeout` || Disables the timeout entirely |
23+
24+
Every client method has a pre-assigned tier that matches the expected duration of the underlying API call. You generally don't need to change these unless you're working with unusually large payloads or slow network conditions.
25+
26+
## Configuring default timeouts
27+
28+
You can override the default values for each tier in the <ApiLink to="class/ApifyClient">`ApifyClient`</ApiLink> or <ApiLink to="class/ApifyClientAsync">`ApifyClientAsync`</ApiLink> constructor. The `timeout_max` parameter sets an upper cap on the timeout for any individual API request, limiting exponential growth during retries.
29+
30+
<Tabs>
31+
<TabItem value="AsyncExample" label="Async client" default>
32+
<CodeBlock className="language-python">
33+
{TimeoutsAsyncExample}
34+
</CodeBlock>
35+
</TabItem>
36+
<TabItem value="SyncExample" label="Sync client">
37+
<CodeBlock className="language-python">
38+
{TimeoutsSyncExample}
39+
</CodeBlock>
40+
</TabItem>
41+
</Tabs>
42+
43+
## Per-call overrides
44+
45+
Most client methods accept a `timeout` parameter that overrides the default tier for that specific call. You can pass either a `timedelta` for an exact duration or a tier literal (`'short'`, `'medium'`, `'long'`, `'no_timeout'`) to switch tiers.
46+
47+
```python
48+
from datetime import timedelta
49+
50+
# Use an exact timeout for this call.
51+
client.dataset('id').list_items(timeout=timedelta(seconds=120))
52+
53+
# Switch to a different tier.
54+
client.dataset('id').list_items(timeout='long')
55+
56+
# Disable the timeout entirely.
57+
client.dataset('id').list_items(timeout='no_timeout')
58+
```
59+
60+
## Interaction with retries
61+
62+
Timeouts work together with the [retry system](/api/client/python/docs/concepts/retries). When a request times out, it counts as a failed attempt and triggers a retry (up to `max_retries`). The timeout applies to each individual attempt, not the total time across all retries.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
from apify_client import ApifyClientAsync
2+
3+
TOKEN = 'MY-APIFY-TOKEN'
4+
5+
6+
async def main() -> None:
7+
apify_client = ApifyClientAsync(TOKEN)
8+
dataset_client = apify_client.dataset('dataset-id')
9+
10+
# Iterate through all items automatically.
11+
async for item in dataset_client.iterate_items():
12+
print(item)
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
from apify_client import ApifyClient
2+
3+
TOKEN = 'MY-APIFY-TOKEN'
4+
5+
6+
def main() -> None:
7+
apify_client = ApifyClient(TOKEN)
8+
dataset_client = apify_client.dataset('dataset-id')
9+
10+
# Iterate through all items automatically.
11+
for item in dataset_client.iterate_items():
12+
print(item)
13+
14+
15+
if __name__ == '__main__':
16+
main()
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
from datetime import timedelta
2+
3+
from apify_client import ApifyClientAsync
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
async def main() -> None:
9+
# Configure default timeout tiers globally.
10+
apify_client = ApifyClientAsync(
11+
token=TOKEN,
12+
timeout_short=timedelta(seconds=10),
13+
timeout_medium=timedelta(seconds=60),
14+
timeout_long=timedelta(seconds=600),
15+
timeout_max=timedelta(seconds=600),
16+
)
17+
18+
dataset_client = apify_client.dataset('dataset-id')
19+
20+
# Override the timeout for a single call using a timedelta.
21+
items = await dataset_client.list_items(timeout=timedelta(seconds=120))
22+
23+
# Or use a tier literal to select a predefined timeout.
24+
items = await dataset_client.list_items(timeout='long')
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
from datetime import timedelta
2+
3+
from apify_client import ApifyClient
4+
5+
TOKEN = 'MY-APIFY-TOKEN'
6+
7+
8+
def main() -> None:
9+
# Configure default timeout tiers globally.
10+
apify_client = ApifyClient(
11+
token=TOKEN,
12+
timeout_short=timedelta(seconds=10),
13+
timeout_medium=timedelta(seconds=60),
14+
timeout_long=timedelta(seconds=600),
15+
timeout_max=timedelta(seconds=600),
16+
)
17+
18+
dataset_client = apify_client.dataset('dataset-id')
19+
20+
# Override the timeout for a single call using a timedelta.
21+
items = dataset_client.list_items(timeout=timedelta(seconds=120))
22+
23+
# Or use a tier literal to select a predefined timeout.
24+
items = dataset_client.list_items(timeout='long')

0 commit comments

Comments
 (0)