feat: SQL database-backed memoization cache

.features/pending/sql-memoization-cache.md

@@ -0,0 +1,13 @@
+Description: Add SQL database-backed memoization cache as an alternative to ConfigMaps.
+Authors: [droctothorpe](https://github.com/droctothorpe)
+Component: General
+Issues: 15952
+PRs: 15938
+
+Memoization can now store cache entries in a PostgreSQL or MySQL database instead of Kubernetes ConfigMaps.
+The SQL backend removes the 1 MB ConfigMap size limit and persists cache entries across cluster restarts.
+ConfigMaps remain the default; opt in by adding a `memoization` block to the `workflow-controller-configmap`.
+SQL-backed entries are stored in the configured table, which defaults to `cache_entries`.
+Each cache entry computes an `expires_at` timestamp at save time from the template's `maxAge` field (default: 30 days).
+The default max age can be overridden via the `DEFAULT_MAX_AGE` environment variable on the controller when SQL memoization is enabled.
+A periodic garbage collector deletes expired entries whose `expires_at` has elapsed.

.github/workflows/ci-build.yaml

@@ -251,6 +251,12 @@ jobs:
           - test: test-corefunctional
             profile: minimal
             use-api: false
+          - test: test-sqldbmemoize
+            profile: mysql
+            use-api: false
+          - test: test-sqldbmemoize
+            profile: postgres
+            use-api: false
           - test: test-functional
             profile: minimal
             use-api: false

.spelling

@@ -151,6 +151,7 @@ args
 async
 auth
 backend
+backends
 backoff
 backport
 backported
@@ -175,6 +176,7 @@ entrypoint
 enum
 env
 errored
+expires_at
 expr
 fibonacci
 filename

config/config.go

@@ -121,6 +121,10 @@ type Config struct {
 	// Synchronization via databases config
 	Synchronization *SyncConfig `json:"synchronization,omitempty"`
 
+	// Memoization configures memoization cache storage. When set, cache entries are stored in a
+	// database instead of ConfigMaps. ConfigMap-based caching remains the default when omitted.
+	Memoization *MemoizationConfig `json:"memoization,omitempty"`
+
 	// ArtifactDrivers lists artifact driver plugins we can use
 	ArtifactDrivers []ArtifactDriver `json:"artifactDrivers,omitempty"`
 
@@ -353,6 +357,17 @@ type SyncConfig struct {
 	SemaphoreLimitCacheSeconds *int64 `json:"semaphoreLimitCacheSeconds,omitempty"`
 }
 
+// MemoizationConfig contains memoization cache configuration for database-backed storage.
+// When configured, cache entries are stored in the specified database table instead of ConfigMaps.
+type MemoizationConfig struct {
+	DBConfig
+	// TableName is the name of the table to use for memoization cache entries.
+	// Defaults to "cache_entries" if not set.
+	TableName string `json:"tableName,omitempty"`
+	// SkipMigration skips automatic database migration on startup.
+	SkipMigration bool `json:"skipMigration,omitempty"`
+}
+
 // ConnectionPool contains database connection pool settings
 type ConnectionPool struct {
 	// MaxIdleConns sets the maximum number of idle connections in the pool

docs/environment-variables.md

@@ -27,6 +27,7 @@ This document outlines environment variables that can be used to customize behav
 | `CACHE_GC_PERIOD`                        | `time.Duration`     | `0s`                                                                                        | How often to perform memoization cache GC, which is disabled by default and can be enabled by providing a non-zero duration.                                                                                                                                             |
 | `CACHE_GC_AFTER_NOT_HIT_DURATION`        | `time.Duration`     | `30s`                                                                                       | When a memoization cache has not been hit after this duration, it will be deleted.                                                                                                                                                                                       |
 | `CRON_SYNC_PERIOD`                       | `time.Duration`     | `10s`                                                                                       | How often to sync cron workflows.                                                                                                                                                                                                                                        |
+| `DEFAULT_MAX_AGE`                        | `string`            | `""` (30 days)                                                                              | Default TTL for SQL-backed memoization cache entries when `memoize.maxAge` is not set on the template. Accepts a Go duration string (e.g. `720h`) or an integer number of seconds. If unset, entries expire after 30 days. This does not affect ConfigMap-backed memoization.                         |
 | `DEFAULT_REQUEUE_TIME`                   | `time.Duration`     | `10s`                                                                                       | The re-queue time for the rate limiter of the workflow queue.                                                                                                                                                                                                            |
 | `DISABLE_MAX_RECURSION`                  | `bool`              | `false`                                                                                     | Set to true to disable the recursion preventer, which will stop a workflow running which has called into a child template 100 times                                                                                                                                      |
 | `EXPRESSION_TEMPLATES`                   | `bool`              | `true`                                                                                      | Escape hatch to disable expression templates.                                                                                                                                                                                                                            |
@@ -40,6 +41,7 @@ This document outlines environment variables that can be used to customize behav
 | `LEADER_ELECTION_RENEW_DEADLINE`         | `time.Duration`     | `10s`                                                                                       | The duration that the acting master will retry refreshing leadership before giving up.                                                                                                                                                                                   |
 | `LEADER_ELECTION_RETRY_PERIOD`           | `time.Duration`     | `5s`                                                                                        | The duration that the leader election clients should wait between tries of actions.                                                                                                                                                                                      |
 | `MAX_OPERATION_TIME`                     | `time.Duration`     | `30s`                                                                                       | The maximum time a workflow operation is allowed to run for before re-queuing the workflow onto the work queue.                                                                                                                                                          |
+| `MEMO_CACHE_GC_PERIOD`                   | `time.Duration`     | `24h`                                                                                       | How often the SQL-backed memoization cache garbage collector runs to prune entries that have exceeded their TTL.                                                                                                                                                         |
 | `OFFLOAD_NODE_STATUS_TTL`                | `time.Duration`     | `5m`                                                                                        | The TTL to delete the offloaded node status. Currently only used for testing.                                                                                                                                                                                            |
 | `OPERATION_DURATION_METRIC_BUCKET_COUNT` | `int`               | `6`                                                                                         | The number of buckets to collect the metric for the operation duration.                                                                                                                                                                                                  |
 | `POD_NAMES`                              | `string`            | `v2`                                                                                        | Whether to have pod names contain the template name (v2) or be the node id (v1) - should be set the same for Argo Server.                                                                                                                                                |

docs/memoization.md

@@ -14,15 +14,72 @@ If you are using workflows prior to version 3.5 you should look at the [work avo
 
 In version 3.5 or later all steps can be memoized, whether or not they have outputs.
 
-## Cache Method
+## Cache Backends
 
-Currently, the cached data is stored in config-maps.
+Argo Workflows supports two backends for storing memoization cache entries:
+
+### ConfigMap (default)
+
+By default, cached data is stored in Kubernetes ConfigMaps.
 This allows you to easily manipulate cache entries manually through `kubectl` and the Kubernetes API without having to go through Argo.
-All cache config-maps must have the label `workflows.argoproj.io/configmap-type: Cache` to be used as a cache. This prevents accidental access to other important config-maps in the system
+All cache ConfigMaps must have the label `workflows.argoproj.io/configmap-type: Cache` to be used as a cache. This prevents accidental access to other important ConfigMaps in the system.
+
+### SQL Database
+
+> v4.0 and after
+
+Alternatively, cache entries can be stored in a PostgreSQL or MySQL database. This is recommended for production use — it has no size limits, supports long-term persistence, and includes automatic garbage collection.
+
+To enable SQL-backed memoization, add a `memoization` block to the `workflow-controller-configmap`:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: workflow-controller-configmap
+  namespace: argo
+data:
+  memoization: |
+    tableName: cache_entries
+    postgresql:
+      host: postgres
+      port: 5432
+      database: postgres
+      userNameSecret:
+        name: argo-postgres-config
+        key: username
+      passwordSecret:
+        name: argo-postgres-config
+        key: password
+```
+
+SQL-backed memoization stores entries in the configured table. Set `memoization.tableName` to override the default table name; if omitted, it defaults to `cache_entries`.
+The database connection settings remain under `postgresql` or `mysql`.
+
+Each cache entry stores its expiry time when it is written, derived from the template's `maxAge` field. If `maxAge` is not specified on the template, it defaults to 30 days (2592000 seconds). This default can be overridden by setting the `DEFAULT_MAX_AGE` environment variable on the workflow controller for SQL-backed memoization (accepts Go duration strings like `720h` or integer seconds like `2592000`).
+
+The garbage collector periodically deletes expired entries. The GC period defaults to 24 hours and can be configured via the `MEMO_CACHE_GC_PERIOD` environment variable.
+
+MySQL is also supported:
+
+```yaml
+  memoization: |
+    tableName: cache_entries
+    mysql:
+      host: mysql
+      port: 3306
+      database: argo
+      userNameSecret:
+        name: argo-mysql-config
+        key: username
+      passwordSecret:
+        name: argo-mysql-config
+        key: password
+```
 
 ## Using Memoization
 
-Memoization is set at the template level. You must specify a `key`, which can be static strings but more often depend on inputs.
+Memoization is configured at the template level via the `memoize` field. You must specify a `key`, which can be static strings but more often depend on inputs.
 You must also specify a name for the `config-map` cache.
 Optionally you can set a `maxAge` in seconds or hours (e.g. `180s`, `24h`) to define how long should it be considered valid. If an entry is older than the `maxAge`, it will be ignored.
 
@@ -43,18 +100,27 @@ spec:
                  name: print-message-cache
 ```
 
+### Fields
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `key` | Yes | The cache lookup key. |
+| `cache` | Yes | Specifies the cache storage. When using the ConfigMap backend, a ConfigMap is created. When using the SQL backend, `cache.configMap.name` acts as a logical group name in the database — no ConfigMap is created. |
+| `maxAge` | No | Maximum age of a cache entry (e.g. `"180s"`, `"24h"`). Entries older than this are treated as misses at lookup time. When omitted for SQL-backed memoization, it defaults to 30 days or the controller's `DEFAULT_MAX_AGE` setting. |
+
 [Find a simple example for memoization here](https://github.com/argoproj/argo-workflows/blob/main/examples/memoize-simple.yaml).
 
 !!! Note
-    In order to use memoization it is necessary to add the verbs `create` and `update` to the `configmaps` resource for the appropriate (cluster) roles. In the case of a cluster install the `argo-cluster-role` cluster role should be updated, whilst for a namespace install the `argo-role` role should be updated.
+    To use memoization with the ConfigMap backend, add the verbs `create` and `update` to the `configmaps` resource for the appropriate (cluster) roles. For a cluster install, update the `argo-cluster-role` cluster role; for a namespace install, update the `argo-role` role. This is not required when using the SQL database backend.
 
 ## FAQ
 
 1. If you see errors like `error creating cache entry: ConfigMap \"reuse-task\" is invalid: []: Too long: must have at most 1048576 characters`,
    this is due to [the 1MB limit placed on the size of `ConfigMap`](https://github.com/kubernetes/kubernetes/issues/19781).
    Here are a couple of ways that might help resolve this:
-    * Delete the existing `ConfigMap` cache or switch to use a different cache.
-    * Reduce the size of the output parameters for the nodes that are being memoized.
-    * Split your cache into different memoization keys and cache names so that each cache entry is small.
+    - Delete the existing `ConfigMap` cache or switch to use a different cache.
+    - Reduce the size of the output parameters for the nodes that are being memoized.
+    - Split your cache into different memoization keys and cache names so that each cache entry is small.
+    - Switch to the SQL database backend which has no size limit.
 1. My step isn't getting memoized, why not?
    If you are running workflows <3.5 ensure that you have specified at least one output on the step.