Skip to content

08_pagination.mdx

Latest commit

 

History

History
63 lines (49 loc) · 3.32 KB

08_pagination.mdx

File metadata and controls

63 lines (49 loc) · 3.32 KB
id pagination
title Pagination
description Paginate through large result sets using ListPage or generator-based iteration.

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import CodeBlock from '@theme/CodeBlock';

import PaginationAsyncExample from '!!raw-loader!./code/08_pagination_async.py'; import PaginationSyncExample from '!!raw-loader!./code/08_pagination_sync.py';

import IterateItemsAsyncExample from '!!raw-loader!./code/08_iterate_items_async.py'; import IterateItemsSyncExample from '!!raw-loader!./code/08_iterate_items_sync.py';

Most methods named list or list_something in the Apify client return a ListPage object. This object provides a consistent interface for working with paginated data and includes the following properties:

  • items - The main results you're looking for.
  • total - The total number of items available.
  • offset - The starting point of the current page.
  • count - The number of items in the current page.
  • limit - The maximum number of items per page.

Some methods, such as list_keys or list_head, paginate differently. Regardless, the primary results are always stored under the items property, and the limit property can be used to control the number of results returned.

The following example demonstrates how to fetch all items from a dataset using pagination:

{PaginationAsyncExample} {PaginationSyncExample}

The ListPage interface offers several key benefits. Its consistent structure ensures predictable results for most list methods, providing a uniform way to work with paginated data. It also offers flexibility, allowing you to customize the limit and offset parameters to control data fetching according to your needs. Additionally, it provides scalability, enabling you to efficiently handle large datasets through pagination. This approach ensures efficient data retrieval while keeping memory usage under control, making it ideal for managing and processing large collections.

Generator-based iteration

For most use cases, iterate_items() is the recommended way to process all items in a dataset. It handles pagination automatically using a Python generator, fetching items in batches behind the scenes so you don't need to manage offsets or limits yourself.

{IterateItemsAsyncExample} {IterateItemsSyncExample}

iterate_items() accepts the same filtering parameters as list_items() (clean, fields, omit, unwind, skip_empty, skip_hidden), so you can combine automatic pagination with data filtering.

Similarly, KeyValueStoreClient provides an iterate_keys() method for iterating over all keys in a key-value store without manual pagination.