08_pagination.mdx

id	pagination
title	Pagination
description	Paginate through large result sets using page models or generator-based iteration.

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock';

import ApiLink from '@theme/ApiLink';

import PaginationAsyncExample from '!!raw-loader!./code/08_pagination_async.py'; import PaginationSyncExample from '!!raw-loader!./code/08_pagination_sync.py'; import IterateItemsAsyncExample from '!!raw-loader!./code/08_iterate_items_async.py'; import IterateItemsSyncExample from '!!raw-loader!./code/08_iterate_items_sync.py'; import IterateCollectionAsyncExample from '!!raw-loader!./code/08_iterate_collection_async.py'; import IterateCollectionSyncExample from '!!raw-loader!./code/08_iterate_collection_sync.py';

Most methods named list or list_something in the Apify client return a page model — a Pydantic model such as ListOfActors, ListOfDatasets, or ListOfRequests. Unstructured dataset items use the DatasetItemsPage dataclass instead. All page models share a consistent interface for working with paginated data and expose the following fields:

• items - The main results you're looking for.
• total - The total number of items available.
• offset - The starting point of the current page.
• count - The number of items in the current page.
• limit - The maximum number of items per page.

Some methods, such as list_keys or list_head, paginate differently. Regardless, the primary results are always stored under the items field, and the limit field can be used to control the number of results returned.

The following example shows how to fetch all items from a dataset using pagination:

{PaginationAsyncExample} {PaginationSyncExample}

The page model interface offers several key benefits. Its consistent structure ensures predictable results for most list methods, providing a uniform way to work with paginated data. It also offers flexibility, allowing you to customize the limit and offset parameters to control data fetching according to your needs. Additionally, it provides scalability, enabling you to efficiently handle large datasets through pagination. This approach ensures efficient data retrieval while keeping memory usage under control, making it ideal for managing and processing large collections.

Generator-based iteration

For collection clients, the iterate method returns an iterator that lazily fetches as many pages as needed to retrieve every item matching the filters. For dataset, key-value store and request queue clients, the matching helpers are iterate_items, iterate_keys and iterate_requests. They handle pagination automatically, so you don't need to manage offsets, limits or cursors yourself.

The example below iterates over every Actor owned by the current user using a collection client's iterate method:

{IterateCollectionAsyncExample} {IterateCollectionSyncExample}

The next example uses iterate_items on a dataset client to stream items past a given offset:

{IterateItemsAsyncExample} {IterateItemsSyncExample}