fix(enrichment tables): preserve memory enrichment table state on reload by esensar · Pull Request #25547 · vectordotdev/vector

esensar · 2026-06-01T12:51:24Z

Summary

While working on #25143, it was brought to my attention that reload was not handled for memory tables (#25143 (comment)) - that is, new components were generated, that were not attached to the tables that were queried. This PR resolves that by making these tables take over the state of previous components.

Vector configuration

enrichment_tables:
  memory_table:
    type: memory
    ttl: 60
    flush_interval: 5
    inputs: ["cache_generator"]


sources:
  demo_logs_test:
    type: "demo_logs"
    format: "json"

transforms:
  demo_logs_processor:
    type: "remap"
    inputs: ["demo_logs_test"]
    source: |
      . = parse_json!(.message)
      user_id = get!(., path: ["user-identifier"])

      existing, err = get_enrichment_table_record("memory_table", { "key": user_id })

      if err == null {
        . = existing.value
        .source = "cache"
      } else {
        .referer = parse_url!(.referer)
        .referer.host = encode_punycode!(.referer.host)
        .source = "transform"
      }      

  cache_generator:
    type: "remap"
    inputs: ["demo_logs_processor"]
    source: |
      existing, err = get_enrichment_table_record("memory_table", { "key": get!(., path: ["user-identifier"]) })
      if err != null {
        data = .
        . = set!(value: {}, path: [get!(data, path: ["user-identifier"])], data: data)
      } else {
        . = {}
      }      

sinks:
  console:
    inputs: ["demo_logs_processor"]
    target: "stdout"
    type: "console"
    encoding:
      codec: "json"

How did you test this PR?

Ran vector with the above configuration and the --watch-config flag. Changed TTL a couple of times and Vector properly reloaded and kept state, observed by seeing cached output data, instead of newly generated.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

Sponsored by Quad9

Related: vectordotdev#25143 Related: vectordotdev#25143 (comment)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6583fd70e4

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

pront · 2026-06-01T17:56:45Z

Thanks @esensar! Per our new policy I will come back to this once codex comments are resolved.

esensar · 2026-06-02T13:58:54Z

While resolving this I found that there were quite a few typos and missed cases for reload in enrichment tables that have source/sink. I think I covered them all now.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0044cc7f03

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…t types

pront

Thank you for this contribution. Can please add a couple of test cases, e.g. take_state_preserves_data and reload_preserves_state?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ed0732c66

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4cb301f9e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

esensar · 2026-06-08T13:25:01Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 961e72fe3b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

esensar · 2026-06-08T15:26:55Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

I have added the test - it works when run individually, but fails when run together with others - I assume because ENRICHMENT_TABLES is static and shared between tests. Not sure how to best resolve this...

pront · 2026-06-12T19:42:53Z

I am adding a topology reload test to confirm this, but I am having some issues with it. For some reason tables are missing on reload, that is TableRegistry::finish_load doesn't store tables properly during the test (it works fine in runtime)

I have added the test - it works when run individually, but fails when run together with others - I assume because ENRICHMENT_TABLES is static and shared between tests. Not sure how to best resolve this...

@esensar no worries, I will take a closer look and make suggestions.

pront · 2026-06-16T13:55:49Z

Updated this branch with a fix. Can you try the tests now?

esensar · 2026-06-16T16:01:57Z

Updated this branch with a fix. Can you try the tests now?

Thanks, that fixed it!

pront

Edit:

This makes memory table state preservation across reload unconditional when the table key remains the same. I can imagine users who treat reload as a cache invalidation point after changing the table’s producing logic or cached value schema.

This is currently breaking the existing reload behavior. Thoughts on adding an explicit reload behavior config option? Or preserving the current behavior and making this opt-in?

esensar · 2026-06-17T08:16:26Z

Edit:

This makes memory table state preservation across reload unconditional when the table key remains the same. I can imagine users who treat reload as a cache invalidation point after changing the table’s producing logic or cached value schema.

This is currently breaking the existing reload behavior. Thoughts on adding an explicit reload behavior config option? Or preserving the current behavior and making this opt-in?

Okay, that makes sense. Maybe a flag for state preservation on reload, or like you said reload behavior with just these 2 options for now so that we can expand it in the future if some other interesting behavior comes up.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 12939a04e6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…uts`

chatgpt-codex-connector

💡 Codex Review

vector/src/topology/running.rs

Lines 824 to 827 in e696b79

    
           .filter_map(|key| { 
        
               self.config 
        
                   .enrichment_table(key) 
        
                   .and_then(|t| t.as_source(key).map(|(key, _)| key))

Register added table sources from the new config

When a reload adds a memory enrichment table with source_config (or changes its source_key), diff.enrichment_tables.changed_and_added() contains the new source key, but this lookup uses the old self.config, where that table/source does not exist. As a result outputs_tap_metadata and component_type_names are never populated for the newly spawned table source, so topology/tap subscribers cannot see or tap it until Vector is restarted.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2564ca377f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

fix(enrichment): preserve memory enrichment table state on reload

6583fd7

Related: vectordotdev#25143 Related: vectordotdev#25143 (comment)

esensar requested a review from a team as a code owner June 1, 2026 12:51

github-actions Bot added the domain: topology Anything related to Vector's topology code label Jun 1, 2026

Add changelog entry

58b908e

esensar mentioned this pull request Jun 1, 2026

feat(enrichment tables): add cuckoo filter to memory table #25143

Open

9 tasks

chatgpt-codex-connector Bot reviewed Jun 1, 2026

View reviewed changes

Comment thread src/topology/running.rs Outdated

Comment thread src/topology/builder.rs Outdated

esensar changed the title ~~fix(enrichment): preserve memory enrichment table state on reload~~ fix(enrichment tables): preserve memory enrichment table state on reload Jun 2, 2026

esensar added 2 commits June 2, 2026 13:19

Restore old state when building the new table

04110a3

Fix reload handling for enrichment tables

0e0952c

Merge branch 'master' into fix/memory-table-reload-state

0044cc7

chatgpt-codex-connector Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread src/topology/builder.rs Outdated

esensar added 3 commits June 3, 2026 06:01

Add additional types for enrichment tables which act as more componen…

a696cb7

…t types

Change stateful flag check with sink check

664d2ae

Merge branch 'master' into fix/memory-table-reload-state

bf3173f

pront reviewed Jun 4, 2026

View reviewed changes

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

Comment thread lib/vector-vrl/enrichment/src/lib.rs Outdated

esensar added 3 commits June 5, 2026 11:28

Remove unused enrichment error

d36e743

Simplify state extraction from tables

ed83146

Remove another unused error from enrichment

1ed0732

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread lib/vector-vrl/enrichment/src/test_util.rs Outdated

esensar added 2 commits June 5, 2026 12:26

Add a test case for memory table state preservation

4ab2ce3

Remove obsolete trait methods from enrichment tests

f4cb301

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes

Comment thread src/topology/builder.rs

Prevent needless clone when extracting state from enrichment tables

e9f9738

Merge branch 'master' into fix/memory-table-reload-state

961e72f

chatgpt-codex-connector Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread src/enrichment_tables/memory/table.rs

Comment thread src/topology/running.rs

Add topology test for enrichment table state extraction

79eb2b1

esensar requested a review from pront June 8, 2026 15:27

Merge branch 'master' into fix/memory-table-reload-state

2fdd859

pront mentioned this pull request Jun 12, 2026

fix(topology): serialize enrichment table registry loads #25615

Merged

9 tasks

Merge branch 'master' into fix/memory-table-reload-state

b05daa4

pront approved these changes Jun 16, 2026

View reviewed changes

Add a configuration option for state preservation for memory table

12939a0

chatgpt-codex-connector Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread src/topology/running.rs

Comment thread src/topology/builder.rs

Comment thread src/enrichment_tables/memory/table.rs Outdated

esensar and others added 4 commits June 18, 2026 13:40

Reattach outputs from enrichment_table sinks in `reattach_severed_inp…

e7869bd

…uts`

Reload enrichment tables if they are added or changed too

c2bb5df

Finish enrichment tables load only on successful builds

0bded62

Merge branch 'master' into fix/memory-table-reload-state

a946993

pront enabled auto-merge June 22, 2026 17:13

This comment has been minimized.

Sign in to view

Generate component docs

e696b79

auto-merge was automatically disabled June 22, 2026 18:21
Head branch was pushed to by a user without write access

esensar requested a review from a team as a code owner June 22, 2026 18:21

github-actions Bot added the domain: external docs Anything related to Vector's external, public documentation label Jun 22, 2026

rtrieu approved these changes Jun 22, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 22, 2026

View reviewed changes

Comment thread src/topology/running.rs

Look up enrichment tables in correct config on reload

2564ca3

chatgpt-codex-connector Bot reviewed Jun 23, 2026

View reviewed changes

Comment thread src/topology/running.rs

Comment thread src/topology/test/mod.rs

	.filter_map(\|key\| {
	self.config
	.enrichment_table(key)
	.and_then(\|t\| t.as_source(key).map(\|(key, _)\| key))

Uh oh!

Conversation

esensar commented Jun 1, 2026

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

pront commented Jun 1, 2026

Uh oh!

esensar commented Jun 2, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

esensar commented Jun 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

esensar commented Jun 8, 2026

Uh oh!

pront commented Jun 12, 2026

Uh oh!

pront commented Jun 16, 2026

Uh oh!

esensar commented Jun 16, 2026

Uh oh!

pront left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

esensar commented Jun 17, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

pront left a comment •

edited

Loading