Skip to content
This repository was archived by the owner on Mar 26, 2026. It is now read-only.
Merged
Changes from 6 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f6287f9
feat: adds augmented pagination to account for BQ family of APIs
chalmerlowe Mar 24, 2025
6979a63
updates with several options for an allowlist
chalmerlowe Mar 24, 2025
18523c4
updates comments, syntax formatting, linting
chalmerlowe Mar 25, 2025
2caa968
updates comments, removes unused variable
chalmerlowe Mar 25, 2025
34b6490
Updates with validate function
chalmerlowe Mar 26, 2025
00db7ce
updates paged_result_field with a validation function
chalmerlowe Mar 26, 2025
3bd99de
updates validation function with self arg
chalmerlowe Mar 26, 2025
5227acc
updates unittests with test__validate_paged_field_size_type
chalmerlowe Mar 26, 2025
1c4265a
updates parametrizations for test__validate_paged_field_size_type
chalmerlowe Mar 26, 2025
e334e8e
updates order and comments for test__validate_paged_field_size_type
chalmerlowe Mar 26, 2025
6ce132e
cleans up comments
chalmerlowe Mar 27, 2025
8895d64
adds test to improve code coverage
chalmerlowe Mar 27, 2025
a8a3ead
updates linting
chalmerlowe Mar 27, 2025
bdfe40f
renames _type, add pb_type names, removes now superfluous comments
chalmerlowe Mar 28, 2025
ddbe96a
updates return statement to resolve mypy error
chalmerlowe Mar 28, 2025
4e4f199
Merge branch 'main' into feat-360114649-augment-pagination-bq-poc
chalmerlowe Mar 28, 2025
921e62b
updates expected value in test parameters
chalmerlowe Mar 28, 2025
7e70e8d
adds proof-of-concept fragment test
chalmerlowe Mar 28, 2025
b30943e
Merge branch 'main' into feat-360114649-augment-pagination-bq-poc
chalmerlowe Mar 31, 2025
77366f8
as experiment support remove files
chalmerlowe Mar 31, 2025
a851880
add page_size focused test
chalmerlowe Mar 31, 2025
fbe3961
updates proto files for max_result and page_size
chalmerlowe Apr 1, 2025
01efee0
adds back the temporarily removed fragments tests
chalmerlowe Apr 1, 2025
20e145c
adds back the page_size proto
chalmerlowe Apr 1, 2025
a8c11b2
adds two experimental fragments
chalmerlowe Apr 1, 2025
654c7af
updates experimental fragments
chalmerlowe Apr 1, 2025
def1423
Update tests/fragments/test_pagination_max_results_and_wrapper.proto
chalmerlowe Apr 2, 2025
caea510
removes fragment proto focused solely on page_size
chalmerlowe Apr 9, 2025
42587ef
removes example fragments
chalmerlowe Apr 9, 2025
4459d9f
git pull is filling me
chalmerlowe Apr 9, 2025
eb59fcd
removes unneeded files
chalmerlowe Apr 9, 2025
bace0d6
Update gapic/schema/wrappers.py
chalmerlowe Apr 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 33 additions & 6 deletions gapic/schema/wrappers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1830,13 +1830,35 @@ def ident(self) -> metadata.Address:
"""Return the identifier data to be used in templates."""
return self.meta.address

def _validate_paged_field_size_type(page_field_size):
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _validate_paged_field_size_type(page_field_size):
def _validate_paged_field_size_type(self, page_field_size):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should help the unit tests! I have absolutely been burned by this before

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leahecole

Resolved.

"""Validates allowed paged_field_size type(s).

Confirms whether the paged_field_size.type is an allowed wrapper type:
The norm is for type to be int, but an additional check is included to
account for BigQuery legacy APIs which allowed UInt32Value and
Int32Value.
"""

_type = page_field_size.type

if _type == int or (
isinstance(_type, MessageType)
and _type.message_pb.name in {"UInt32Value", "Int32Value"}
):
return True
Comment thread
parthea marked this conversation as resolved.
Outdated

@utils.cached_property
def paged_result_field(self) -> Optional[Field]:
"""Return the response pagination field if the method is paginated."""
# If the request field lacks any of the expected pagination fields,
# then the method is not paginated.
"""Return the response pagination field if the method is paginated.

The request field must have a page_token field and a page_size field (or
for legacy APIs, a max_results field) and the response field
must have a next_token_field and a repeated field.

For the purposes of supporting legacy APIs, additional wrapper types are
allowed.
"""

# The request must have page_token and next_page_token as they keep track of pages
for source, source_type, name in (
(self.input, str, "page_token"),
(self.output, str, "next_page_token"),
Expand All @@ -1845,13 +1867,18 @@ def paged_result_field(self) -> Optional[Field]:
if not field or field.type != source_type:
return None

# The request must have max_results or page_size
# The request must have page_size (or max_results if legacy API)
page_fields = (
self.input.fields.get("max_results", None),
self.input.fields.get("page_size", None),
)
page_field_size = next((field for field in page_fields if field), None)
if not page_field_size or page_field_size.type != int:

if not page_field_size:
return None

# Confirm whether the paged_field_size is an allowed type.
if not self._validate_paged_field_size_type(page_field_size=page_field_size):
return None

# Return the first repeated field.
Expand Down
Loading