Skip to content

Commit 175393f

Browse files
authored
docs: clarify and surface single vs. shared request queue (#1007)
@Pijukatel let me know if you would clarify/phrase something better, or add something, about single vs shared RQs.
1 parent e7240c5 commit 175393f

2 files changed

Lines changed: 5 additions & 3 deletions

File tree

docs/02_concepts/03_storages.mdx

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,8 @@ you can use the <ApiLink to="class/KeyValueStore#get_public_url">`KeyValueStore.
151151

152152
In this section we will show you how to work with [request queues](https://docs.apify.com/platform/storage/request-queue).
153153

154+
By default, a request queue is optimized for a single consumer. To let multiple Actor runs process the same queue at once, see [single vs. shared request queue](./storage-clients#single-vs-shared-request-queue).
155+
154156
### Adding requests to a queue
155157

156158
To add a request into the queue, you can use the <ApiLink to="class/RequestQueue#add_request">`RequestQueue.add_request`</ApiLink> method.
@@ -187,6 +189,6 @@ Behind the scenes, the SDK uses storage clients to communicate with the storage
187189

188190
## Conclusion
189191

190-
This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes.
192+
This page has covered the three storage types (datasets, key-value stores, and request queues): how they are emulated on the local filesystem, how to open named and unnamed storages, and how to read from and write to each through the `Actor` shortcuts and the storage classes. When multiple Actor runs need to process the same request queue, choose between a [single and shared request queue](./storage-clients#single-vs-shared-request-queue).
191193

192194
For comprehensive information about storage on the Apify platform, see the [storage documentation](https://docs.apify.com/platform/storage), including the pages on [datasets](https://docs.apify.com/platform/storage/dataset), [key-value stores](https://docs.apify.com/platform/storage/key-value-store), and [request queues](https://docs.apify.com/platform/storage/request-queue).

docs/02_concepts/12_storage_clients.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,8 +43,8 @@ For details, see the [Crawlee storage clients guide](https://crawlee.dev/python/
4343

4444
`ApifyStorageClient` supports two ways of accessing the Apify request queue, selected via its `request_queue_access` argument:
4545

46-
- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it is cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors.
47-
- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls.
46+
- **`'single'`** (default) - optimized for a single consumer. It makes fewer API calls, so it's cheaper and faster, but it doesn't support multiple clients consuming the same queue concurrently. This is the right choice for the majority of Actors.
47+
- **`'shared'`** - supports multiple consumers working on the same queue at the same time, at the cost of more API calls. Use it when several Actor runs share one named request queue. For example, you can split a large crawl across parallel runs, or feed the queue from a producer run while worker runs consume it.
4848

4949
To opt into the shared client, set it as the cloud client of the `SmartApifyStorageClient` in the [service locator](https://crawlee.dev/python/docs/guides/service-locator) before entering the Actor context:
5050

0 commit comments

Comments
 (0)