[tests] refactor caching tests.#13235

sayakpaul wants to merge 8 commits intomainhuggingface/diffusers:mainfrom

refactor-caching-testshuggingface/diffusers:refactor-caching-testsCopy head branch name to clipboard