Skip to content

refactor: Give more instance caching flexibility to the storage clients#1425

Merged
Pijukatel merged 3 commits into
masterfrom
include-storag-client-in-additional-cache-key
Sep 25, 2025
Merged

refactor: Give more instance caching flexibility to the storage clients#1425
Pijukatel merged 3 commits into
masterfrom
include-storag-client-in-additional-cache-key

Conversation

@Pijukatel

@Pijukatel Pijukatel commented Sep 24, 2025

Copy link
Copy Markdown
Collaborator

Description

  • StorageClient type is not an individual storage key and is instead used in an additional cache key (can be overridden).
  • This allows StorageClients to have better control over caching.

Issues

Testing

  • Unit tests

- StorageClient is not individual storage key and is instead used in additional cache key
@github-actions github-actions Bot added this to the 124th sprint - Tooling team milestone Sep 24, 2025
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 24, 2025
@Pijukatel Pijukatel requested a review from janbuchar September 24, 2025 06:53
@Pijukatel Pijukatel marked this pull request as ready for review September 24, 2025 06:53

@janbuchar janbuchar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, found nits

Can be based on configuration or on the client itself. By default, returns a name of the client class.
"""
return ''
return self.__class__.__name__

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two points regarding this method:

  • self.__class__.__name__ is not unique enough - could you at least attach __module__ as well?
  • the name of the method will not be very fitting after this PR

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using module as well now.

But not sure about renaming, I am quite content with it. Do you have any better name in mind?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something like get_storage_client_cache_key?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

StorageClient.get_storage_client_cache_key feels kind of redundant, since the class is StorageClient and its children

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get it, but most people don't need to use it, so I wouldn't be too sad about that

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, renamed

Comment thread src/crawlee/storages/_storage_instance_manager.py Outdated
@Pijukatel Pijukatel requested a review from janbuchar September 24, 2025 08:00
@Pijukatel Pijukatel merged commit 5ada081 into master Sep 25, 2025
19 checks passed
@Pijukatel Pijukatel deleted the include-storag-client-in-additional-cache-key branch September 25, 2025 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants