Merge pull request #10398 from circleci/update-caching-strategies

kevintse328 · web-flow · commit 02e19a941c8f · 2026-05-27T08:48:27.000-07:00
Updating Caching Strategies for Node and Python
diff --git a/docs/guides/modules/optimize/pages/caching-strategy.adoc b/docs/guides/modules/optimize/pages/caching-strategy.adoc
@@ -8,7 +8,7 @@ Caching is one of the most effective ways to make jobs faster on CircleCI. By re
 [#caching-and-self-hosted-runner]
 == Cache storage customization
 
-When using self-hosted runners, there is a network and storage usage limit included in your plan. Some actions related to artifacts will accrue network and storage usage. Once your usage exceeds your limit, charges will apply.
+When using self-hosted runners, there is a network and storage usage limit included in your plan. Some actions related to artifacts accrue network and storage usage. Once your usage exceeds your limit, charges apply.
 
 Retaining caches for a long period of time will have storage cost implications. It is best to determine why you are retaining caches, and how long caches need to be retained for your use case. To lower costs, consider a lower storage retention for caches, if that suits your needs.
 
@@ -27,14 +27,14 @@ Using cache keys that are too strict can mean that you will get a minimal number
 [#avoid-unnecessary-workflow-reruns]
 === Avoid unnecessary workflow reruns
 
-If your project has "flaky tests," workflows might be rerun unnecessarily. This will both use up your credits and increase your storage usage. To avoid this situation, address flaky tests. For help with identifying them, see xref:insights:insights-tests.adoc#flaky-tests[Test Insights].
+If your project has "flaky tests," workflows are rerun unnecessarily. This uses up your credits and increases your storage usage. To avoid this situation, address flaky tests. For help with identifying them, see xref:insights:insights-tests.adoc#flaky-tests[Test Insights].
 
 You can also consider configuring your projects to rerun failed jobs rather than entire workflows. To achieve this you can use the `when` step. For further information, see the xref:reference:ROOT:configuration-reference.adoc#the-when-attribute[Configuration Reference].
 
 [#split-cache-keys-by-directory]
 === Split cache keys by directory
 
-Having multiple directories under a single cache key increases the chance of there being a change to the cache. In the example below, there may be changes in the first two directories but no changes in the `a` or `b` directory. Saving all four directories under one cache key increases the potential storage usage. The cache restore step will also take longer than needed as all four sets of files will be restored.
+Having multiple directories under a single cache key increases the chance of there being a change to the cache. In the example below, there may be changes in the first two directories but no changes in the `a` or `b` directory. Saving all four directories under one cache key increases the potential storage usage. The cache restore step also takes longer than needed as all four sets of files are restored.
 
 [,yaml]
 ----
@@ -95,7 +95,7 @@ If you notice your cache usage is high and would like to reduce it:
 
 * Search for the `save_cache` and `restore_cache` commands in your `.circleci/config.yml` file to find all jobs utilizing caching and determine if their cache(s) need pruning.
 * Narrow the scope of a cache from a large directory to a smaller subset of specific files.
-* Ensure that your cache `key` is following xref:caching.adoc#further-notes-on-using-keys-and-templates[best practices]:
+* Ensure that your cache `key` is following xref:caching.adoc#further-notes-on-using-keys-and-templates[Best Practices]:
 +
 [,yaml]
 ----
@@ -106,12 +106,12 @@ If you notice your cache usage is high and would like to reduce it:
             - /usr/local/Homebrew
 ----
 +
-Notice in the above example that best practices are not followed. `brew-{{ epoch }}` will change every build causing an upload every time even if the value has not changed. This will cost you money, and never save you any time. Instead pick a cache `key` like the following:
+Notice in the above example that best practices are not followed. `brew-{{ epoch }}` changes every build, causing an upload every time even if the value has not changed. This costs you money, and never saves you any time. Instead pick a cache `key` like the following:
 +
 [,yaml]
 ----
       - save_cache:
-          key: brew-{{checksum “Brewfile”}}
+          key: brew-{{checksum "Brewfile"}}
           paths:
             - /Users/distiller/Library/Caches/Homebrew
             - /usr/local/Homebrew
@@ -136,7 +136,7 @@ Some dependency managers do not properly handle installing on top of partially r
             - gem-cache
 ----
 
-In the above example, if a dependency tree is partially restored by the second or third cache keys, some dependency managers will incorrectly install on top of the outdated dependency tree.
+In the above example, the second or third cache keys may produce a partial restore. Some dependency managers then incorrectly install on top of the outdated dependency tree.
 
 Instead of a cascading fallback, a more stable option is a single version-prefixed cache key:
 
@@ -237,7 +237,7 @@ Since Leiningen uses Maven under the hood, it behaves in a similar way.
 *Safe to Use Partial Cache Restoration?*
 Yes (with NPM5+).
 
-With NPM5+ and a lock file, you can safely use partial cache restoration.
+With NPM5+ and a lock file, you can use partial cache restoration. Cache the npm download cache at `~/.npm` rather than `node_modules` directly. The `~/.npm` path is the correct cache location regardless of npm version, and works with both `npm install` and `npm ci`.
 
 [,yaml]
 ----
@@ -248,19 +248,55 @@ With NPM5+ and a lock file, you can safely use partial cache restoration.
             - node-v1-{{ .Branch }}-{{ checksum "package-lock.json" }}
             - node-v1-{{ .Branch }}-
             - node-v1-
+      - run: npm install
       - save_cache:
           paths:
-            - ~/usr/local/lib/node_modules  # location depends on npm version
+            - ~/.npm
           key: node-v1-{{ .Branch }}-{{ checksum "package-lock.json" }}
 ----
 
+[NOTE]
+====
+If your job uses `npm ci` rather than `npm install`, be aware that `npm ci` deletes and rebuilds `node_modules` on every run by design. Caching `~/.npm` (the download cache) still provides a benefit because it avoids re-downloading packages from the registry, but the time saving is smaller than with `npm install`. If you find your `restore_cache` step is adding more time than it saves when using `npm ci`, consider switching to `npm install` with a lock file, or removing the cache step entirely.
+====
+
 [#pip-python]
 === `pip` (Python)
 
 *Safe to Use Partial Cache Restoration?*
-Yes (with Pipenv).
+Depends on what you cache.
+
+The safety of partial cache restoration for Python depends on whether you cache the *download cache* or the *installed environment*:
+
+* *Download cache* (`~/.cache/pip`): pip stores downloaded wheels here before installing them. Partially restoring this cache is safe because pip always resolves and installs the correct versions from your dependency file regardless of what wheels are already present. A stale partial restore causes some wheels to be re-downloaded.
+* *Installed environment* (a `virtualenv` or `site-packages` directory): partially restoring an installed environment is unsafe unless you use a lock file that pins every dependency to an exact version. Without pinned versions, pip may leave stale package versions in place rather than upgrading them, producing an environment that does not match a clean install.
+
+The examples below follow this distinction. Bare pip and `uv` cache the download cache and use fallback keys without risk. Pipenv and Poetry cache the installed environment and rely on their lock files to keep partial restores deterministic.
+
+[#pip-requirements-txt]
+==== Pip with requirements.txt
+
+Cache the pip download cache at `~/.cache/pip`. Partial restoration is safe here because pip resolves installs from `requirements.txt` independently of what is already cached.
+
+[,yaml]
+----
+    steps:
+      - restore_cache:
+          keys:
+            - pip-cache-v1-{{ .Branch }}-{{ checksum "requirements.txt" }}
+            - pip-cache-v1-{{ .Branch }}-
+            - pip-cache-v1-
+      - run: pip install -r requirements.txt
+      - save_cache:
+          paths:
+            - ~/.cache/pip
+          key: pip-cache-v1-{{ .Branch }}-{{ checksum "requirements.txt" }}
+----
 
-Pip can use files that are not explicitly specified in `requirements.txt`. Using link:https://docs.pipenv.org/[Pipenv] will include explicit versioning in a lock file.
+[#pip-pipenv]
+==== Pipenv
+
+Pipenv generates a `Pipfile.lock` that pins every dependency to an exact version. This makes partial restoration of the installed `virtualenv` safe, because the lock file ensures pip installs the correct versions even on top of a stale partial restore.
 
 [,yaml]
 ----
@@ -271,12 +307,58 @@ Pip can use files that are not explicitly specified in `requirements.txt`. Using
             - pip-packages-v1-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
             - pip-packages-v1-{{ .Branch }}-
             - pip-packages-v1-
+      - run: pipenv install
       - save_cache:
           paths:
             - ~/.local/share/virtualenvs/venv  # this path depends on where pipenv creates a virtualenv
           key: pip-packages-v1-{{ .Branch }}-{{ checksum "Pipfile.lock" }}
 ----
 
+[#pip-poetry]
+==== Poetry
+
+link:https://python-poetry.org/[Poetry] generates a `poetry.lock` file that pins all dependencies to specific versions, making it a reliable cache key. As with Pipenv, the lock file makes partial restoration of the installed `virtualenvs` directory safe.
+
+[,yaml]
+----
+    steps:
+      - restore_cache:
+          keys:
+            - poetry-v1-{{ .Branch }}-{{ checksum "poetry.lock" }}
+            - poetry-v1-{{ .Branch }}-
+            - poetry-v1-
+      - run: poetry install --no-interaction
+      - save_cache:
+          paths:
+            - ~/.cache/pypoetry/virtualenvs
+          key: poetry-v1-{{ .Branch }}-{{ checksum "poetry.lock" }}
+----
+
+[#pip-uv]
+==== `uv`
+
+link:https://docs.astral.sh/uv/[`uv`] is a fast Python package installer that maintains its own download cache at `~/.cache/uv`. Like bare pip, this is a download cache rather than an installed environment, so partial restoration is safe.
+
+[,yaml]
+----
+    steps:
+      - restore_cache:
+          keys:
+            - uv-cache-v1-{{ .Branch }}-{{ checksum "uv.lock" }}
+            - uv-cache-v1-{{ .Branch }}-
+            - uv-cache-v1-
+      - run: uv sync --frozen
+      - save_cache:
+          paths:
+            - ~/.cache/uv
+          key: uv-cache-v1-{{ .Branch }}-{{ checksum "uv.lock" }}
+----
+
+[NOTE]
+====
+`uv` installs packages faster than pip, so the time saving from caching may be smaller than with other tools. Whether caching is worthwhile depends on the size of your dependency tree and how frequently it changes.
+====
+
 [#yarn-node]
 === Yarn (Node)
 
@@ -294,6 +376,7 @@ Yarn has always used a lock file for the reasons explained above.
             - yarn-packages-v1-{{ .Branch }}-{{ checksum "yarn.lock" }}
             - yarn-packages-v1-{{ .Branch }}-
             - yarn-packages-v1-
+      - run: yarn --frozen-lockfile --cache-folder ~/.cache/yarn
       - save_cache:
           paths:
             - ~/.cache/yarn
@@ -302,7 +385,7 @@ Yarn has always used a lock file for the reasons explained above.
 
 We recommend using `yarn --frozen-lockfile --cache-folder ~/.cache/yarn` for two reasons:
 
-* `--frozen-lockfile` ensures a whole new lockfile is created and it also ensures your lockfile is not altered. This allows for the checksum to stay relevant and your dependencies should identically match what you use in development.
+* `--frozen-lockfile` ensures a whole new lockfile is created and it also ensures your lockfile is not altered. This allows for the checksum to stay relevant and your dependencies will identically match what you use in development.
 
 * The default cache location depends on OS. `--cache-folder ~/.cache/yarn` ensures you are explicitly matching your cache save location.
 
@@ -313,18 +396,18 @@ We recommend using `yarn --frozen-lockfile --cache-folder ~/.cache/yarn` for two
 
 In cases where the build tools for your language include elegant handling of dependencies, partial cache restores may be preferable to zero cache restores for performance reasons. If you get a zero cache restore, you have to reinstall all your dependencies, which can cause reduced performance. One alternative is to get a large percentage of your dependencies from an older cache, instead of starting from zero.
 
-However, for other language types, partial caches carry the risk of creating code dependencies that are not aligned with your declared dependencies and do not break until you run a build without a cache. If the dependencies change infrequently, consider listing the zero cache restore key first, and then track the costs over time.
+However, for other language types, partial caches carry the risk of creating code dependencies that are not aligned with your declared dependencies. These mismatches do not always surface until you run a build without a cache. If the dependencies change infrequently, consider listing the zero cache restore key first, and then track the costs over time.
 
 If the performance costs of zero cache restores (also referred to as a _cache miss_) prove significant over time, only then consider adding a partial cache restore key.
 
-Listing multiple keys for restoring a cache increases the chances of a partial cache hit. However, broadening your `restore_cache` scope to a wider history increases the risk of confusing failures. For example, if you have dependencies for Node v6 on an upgrade branch, but your other branches are still on Node v5, a `restore_cache` step that searches other branches might restore incompatible dependencies.
+Listing multiple keys for restoring a cache increases the chances of a partial cache hit. However, broadening your `restore_cache` scope to a wider history increases the risk of confusing failures. For example, consider a project with Node v6 on an upgrade branch and Node v5 on all other branches. A `restore_cache` step that searches other branches may restore incompatible dependencies.
 
 [#using-a-lock-file]
 == Using a lock file
 
 Language dependency manager lockfiles (for example, `Gemfile.lock` or `yarn.lock`) checksums may be a useful cache key.
 
-An alternative is to run the command `ls -laR your-deps-dir > deps_checksum` and reference it with `{{ checksum "deps_checksum" }}`. For example, in Python, to get a more specific cache than the checksum of your `requirements.txt` file, you could install the dependencies within a `virtualenv` in the project root `venv` and then run the command `ls -laR venv > python_deps_checksum`.
+An alternative is to run the command `ls -laR your-deps-dir > deps_checksum` and reference it with `{{ checksum "deps_checksum" }}`. For example, in Python, install the dependencies within a `virtualenv` in the project root `venv`. Then run `ls -laR venv > python_deps_checksum` to produce a more specific cache key than the checksum of your `requirements.txt` file.
 
 [#using-multiple-caches-for-different-languages]
 == Using multiple caches for different languages
@@ -333,14 +416,14 @@ It is also possible to lower the cost of a cache miss by splitting your job acro
 
 Consider splitting caches by language type (`npm`, `pip`, or `bundler`), if you know the following:
 
-* How each dependency manager stores its files
-* How it upgrades
-* How it checks dependencies
+* How each dependency manager stores its files.
+* How it upgrades.
+* How it checks dependencies.
 
 [#caching-expensive-steps]
 == Caching expensive steps
 
-Certain languages and frameworks include more expensive steps that can and should be cached. Scala and Elixir are two examples where caching the compilation steps will be especially effective. Rails developers could also notice a performance boost from caching frontend assets.
+Certain languages and frameworks include more expensive steps that are worth caching. Scala and Elixir are two examples where caching the compilation steps is effective. Rails developers also see a performance boost from caching frontend assets.
 
 Do not cache everything, but _do_ consider caching for costly steps like compilation.
 
