`StorageInstanceManager` should support caching based on used configuration

[Pijukatel]

Context: https://github.com/apify/crawlee-python/pull/1368/files#r2300028783

[janbuchar]

https://github.com/apify/crawlee-python/pull/1368/files#r2298066749 is relevant as well

[Pijukatel]

I think we could go in the direction of StorageClient being explicitly tight with specific configuration and then we can cache only by the used StorageClient.

So instead of

storage_client=MemoryStorageClient()
kvs_client_1=storage_client.create_kvs_client(..., configuration=Configuration("ABC"))
kvs_client_2=storage_client.create_kvs_client(..., configuration=Configuration("XYZ"))

it would be

storage_client_1=MemoryStorageClient(configuration=Configuration("ABC"))
kvs_client_1=storage_client_1.create_kvs_client(...)

storage_client_2=MemoryStorageClient(configuration=Configuration("XYZ"))
kvs_client_2=storage_client_2.create_kvs_client(...)

And more top level:
instead of:

KeyValueStore.open(...,configuration=Configuration("ABC),  storage_clien=MemoryStorageClient())

KeyValueStore.open(...,  storage_clien=MemoryStorageClient(configuration=Configuration("ABC)))

[Pijukatel]

Configuration part of the StorageClient:

I investigated the previously mentioned proposal in https://github.com/apify/crawlee-python/tree/storage-clients-and-configurations
Unfortunately, moving the configuration into the storage client creation causes another issues downstream where the service_locator caches both the storage client and configuration and consumers of services can decide to pick just one. For example:

crawler = BasicCrawler(configuration=Configuration())

Such a crawler will take a custom configuration, but it will use storage_client from service_locator, such storage_client does not respect the custom configuration. This could lead to very unpredictable behavior.

Extended caching based on custom cache key defined by StorageClient::

So instead I tried this approach: #1386. (With SDK related changes here: apify/apify-sdk-python#576)

This should address all the mentioned issues, but it makes the caching more complex. It allows any storage client to define custom caching key to be used to make a clear distinction between storage instances created from it.

Example top-level behavior:

storage_client = FileSystemStorageClient()
d1= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path1"))
d2= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path2"))
d3= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path1"))

assert d2 is not d1
assert d3 is d1

storage_client_2 =MemoryStorageClient()
d4= await Dataset.open(storage_client=storage_client_2, configuration=Configuration(storage_dir="path1"))
d5= await Dataset.open(storage_client=storage_client_2, configuration=Configuration(storage_dir="path2"))
assert d4 is d5

The example above shows that specifically FileSystemStorageClient depends on Configuration.storage_dir which is included in the custom cache key for FileSystemStorageClient, but this is not true for MemoryStorageClient as the storage_dir is not relevant for it.
The example shows that it is the storage_client responsibility to define a specific caching key that can be used to distinguish two different storage instances.

The first crawler example is also handled well as the configuration instance is combined with storage_client instance to create storage instance. And thus it is not a problem to combine custom configuration with service_locator provided storage_client to create a new storage.

It also allows the following scenarios, which would cause a runtime error previously:

custom_configuration_1 = Configuration()
custom_event_manager_1 = LocalEventManager.from_config(custom_configuration_1)
custom_storage_client_1 = MemoryStorageClient()

custom_configuration_2 = Configuration()
custom_event_manager_2 = LocalEventManager.from_config(custom_configuration_2)
custom_storage_client_2 = MemoryStorageClient()

crawler_1 = BasicCrawler(
    configuration=custom_configuration_1,
    event_manager=custom_event_manager_1,
    storage_client=custom_storage_client_1,
)

crawler_2 = BasicCrawler(
    configuration=custom_configuration_2,
    event_manager=custom_event_manager_2,
    storage_client=custom_storage_client_2,
  )

# use crawlers without runtime crash...