Skip to content

IFC-2536: Migrate InfrahubClient to graphql query for get and filters methods#9126

Open
solababs wants to merge 35 commits into
developfrom
optimize-prefect-queries-phase-3-ifc-2536
Open

IFC-2536: Migrate InfrahubClient to graphql query for get and filters methods#9126
solababs wants to merge 35 commits into
developfrom
optimize-prefect-queries-phase-3-ifc-2536

Conversation

@solababs
Copy link
Copy Markdown
Contributor

@solababs solababs commented May 4, 2026

Why

Prefect task files were making overfetching calls to client.all(), client.filters(), and client.get() that returned every attribute, relationship, and metadata field — but the tasks only consumed id (or a handful of fields). This caused unnecessary Neo4j reads, large GraphQL payloads, and slower task execution on every background worker cycle.

This PR replaces those calls with targeted execute_graphql() calls backed by purpose-built query model classes that request only the fields each task actually uses.

How to test

# Unit tests for every query model class
uv run pytest backend/tests/unit/display_labels/ backend/tests/unit/hfid/ \
    backend/tests/unit/git/test_git_repository_query.py \
    backend/tests/unit/generators/test_generator_instance_query.py \
    backend/tests/unit/computed_attribute/ -v

# Benchmark (200 nodes, 5 iterations — requires Neo4j testcontainer)
INFRAHUB_LOG_LEVEL=CRITICAL uv run pytest -v \
    backend/tests/benchmark/test_query_optimization.py --dist=no -s

# Full functional suite
uv run invoke backend.test-functional

Benchmark expected output (≥30% time reduction, ≥50% size reduction per task):

Task Full fetch Optimized Time ↓ Payload ↓
display_labels 485 ms / 65 KB 97 ms / 12 KB 80% 82%
hfid 120 ms / 65 KB 28 ms / 12 KB 76% 82%
git 263 ms / 164 KB 90 ms / 31 KB 66% 81%

Impact & rollout

  • Backward compatibility: no breaking changes — task outputs are identical, only the internal fetch mechanism changes
  • Performance: 65–80% execution time reduction and 80–82% payload size reduction for the six migrated call sites
  • Config/env changes: none
  • Deployment notes: safe to deploy; no migrations, no schema changes, no flag required

Checklist

  • Tests added/updated
  • Changelog entry added (uv run towncrier create ...)
  • External docs updated (if user-facing or ops-facing change)
  • Internal .md docs updated (internal knowledge and AI code tools knowledge)
  • I have reviewed AI generated content

@github-actions github-actions Bot added group/backend Issue related to the backend (API Server, Git Agent) type/spec A specification for an upcoming change to the project labels May 4, 2026
@solababs solababs changed the title Optimize prefect queries phase 3 ifc 2536 IFC-2536: Migrate InfrahubClient to graphql query for get and filters methods May 4, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 4, 2026

Merging this PR will not alter performance

✅ 12 untouched benchmarks


Comparing optimize-prefect-queries-phase-3-ifc-2536 (a38d4ed) with develop (1e09e82)

Open in CodSpeed

Comment thread backend/infrahub/generators/models.py Outdated
status: str


class GeneratorInstanceQuery(BaseModel):
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you using the Pydantic model generator from the SDK or have you generated this class using Claude directly ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is generated using claude

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we can do with the Pydantic model generator (https://docs.infrahub.app/infrahubctl/infrahubctl-graphql) is to have a reference to the GraphQL query in text and then by having it there we can run checks in the CI to validate that the query we've written is up to date based on the current core GraphQL schema shipped with Infrahub, i.e. the one stored here: https://github.com/opsmill/infrahub/blob/infrahub-v1.9.3/schema/schema.graphql. Basically together with a GraphQL query and the GraphQL schema we can generate Python bindings and validate that they are up to date in the CI. One of the thoughts around this JPD card was that we'd use this approach but it seems like this information was lost in the creation of that card. So the primary goal of the JPD card is to improve the performance by avoiding over fetching data, the secondary goal was to have queries defined as text and be able to both validate that they are up to date and that we have generated classes that match the queries. The intention is to start using this approach in multiple places within the backend but also get started in the frontend. This approach would not work for the changes in #9123 though since in that work we're directly querying for user defined nodes that doesn't exist in the core schema. But for the work in this PR it should be possible.

@solababs solababs marked this pull request as ready for review May 5, 2026 11:08
@solababs solababs requested a review from a team as a code owner May 5, 2026 11:08
def render_query(self) -> str:
filters: dict[str, str | list[str]] = (
{"ids": [self.transform_id]} if _is_uuid(self.transform_id) else {"name__value": self.transform_id}
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need a small change with regards to how this query would be executed. If you check this issue #8945. In short there's a caching feature of the backend that would cache the rendering and processing of the GraphQL queries, i.e. to convert them into a GraphQL document. But this only happens if the raw text of the query looks exactly the same. As such we don't want to have self.transform_id be hardcoded into the body of the query but instead we'd need to send it as variables in the SDK call to await client.execute_graphql()

if not transform_attribute.transform:
continue
transform_query = ComputedAttributeTransformQuery(transform_id=transform_attribute.transform)
transform_response = await client.execute_graphql(query=transform_query.render_query(), branch_name=branch_name)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a call like this should be fine with regards to pagination as we'd only expect to get a single result back, i.e. regarding the comments on pagination on the other PR.

@solababs solababs requested review from dgarros and ogenstad May 7, 2026 08:07
@github-actions github-actions Bot added the group/ci Issue related to the CI pipeline label May 7, 2026
Comment thread .github/workflows/ci.yml Outdated
uv run ruff format --check --diff . \
--exclude python_sdk \
--exclude backend/infrahub/computed_attribute/graphql_queries/computed_attribute_fetch_transform.py \
--exclude backend/infrahub/generators/graphql_queries/generator_instance_fetch.py
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These exclude lines are a bit problematic for anyone just running "ruff check ." which is a lot faster than using the invoke commands. I'd wonder what's failing in these generated files. Might it just be a matter of us having to run ruff check [filenames] --fix and ruff format [filenames]? I.e. when we generate them we should be able to format them correctly provided that there are no other violations in these files. For reference:

https://github.com/opsmill/infrahub/blob/infrahub-v1.9.3/tasks/backend.py#L260-L261

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we have a file called backend/infrahub/computed_attribute/graphql_queries.py in #9123 which would conflict with the module directory here backend/infrahub/computed_attribute/graphql_queries/. So could be that this PR should be rebased and reworked once the other one lives in develop

Actually this is probably the merge conflict for this PR.

…new query (#9123)

* IFC-2536: Migrate hfid and computed_attribute to use new query

* use pydantic basemodel

* add pagination

* rename and move files

* update fetch_all logic

* remove optimization tests

* update tasks
Base automatically changed from optimize-prefect-queries-phase-2-ifc-2536 to optimize-prefect-queries-infp-501 May 7, 2026 13:39
@github-actions github-actions Bot removed the group/ci Issue related to the CI pipeline label May 7, 2026
@solababs solababs requested a review from ogenstad May 7, 2026 15:04
@solababs solababs requested a review from a team May 12, 2026 08:52
Base automatically changed from optimize-prefect-queries-infp-501 to develop May 12, 2026 12:03
}
}
}
"""
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd want the same approach for this as you have a bit further down where we have generator_instance_fetch.gql as a separate file. It make it clearer that we'd expect to the bindings for this be generated from the actual file. As it is inline here it looks like it's just part of the code.

Also we can probably combine the two queries TRANSFORM_QUERY_BY_NAME and TRANSFORM_QUERY_BY_IDS into one, then we just have the queries accept both of the parameters and it would use whatever we send in with with the variables.

def render_query(self) -> str:
if _is_uuid(self.transform_id):
return TRANSFORM_QUERY_BY_IDS
return TRANSFORM_QUERY_BY_NAME
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Combining this into one query as per above would remove the if-statement here and we'd only have the get_variables() one.

)
if instances:
instance = instances[0]
response = await client.execute_graphql(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that we might want to have pagination for this query if there's a high number of generator instances.

Copy link
Copy Markdown
Contributor Author

@solababs solababs May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this request is for one instance, uses instance_nodes[0].id below

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

group/backend Issue related to the backend (API Server, Git Agent) type/spec A specification for an upcoming change to the project

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants