dead_letter_queue.flush_check_interval new config for flushing staled segment files.

[mashhurs]

Release notes

Introduces new dead_letter_queue.flush_check_interval config for flushing the staled segment files scheduler which can reduce frequent check overhead.

What does this PR do?

1. Introduces a new configurable dead_letter_queue.flush_check_interval param for the segment file stale check scheduler. See the problem description - Introduce a period for file flushing staled segment files #19037

• it cannot be less than 1sec
• it cannot be greater than dead_letter_queue.flush_interval

2. Validates dead_letter_queue.flush_interval for min 1s to keep consistency with the docs - https://www.elastic.co/docs/reference/logstash/dead-letter-queues: "Note that this value cannot be set to lower than 1000ms."

High level results

5-pipelines runs with 1-worker (to get comparable output), their configurations:

• pipeline-1: default, flush_interval: 5000 and flush_check_interval: 1000
• pipeline-2: default, flush_interval: 5000 and flush_check_interval: 2000
• pipeline-3: default, flush_interval: 5000 and flush_check_interval: 5000
• pipeline-4: default, flush_interval: 10000 and flush_check_interval: 5000
• pipeline-5: default, flush_interval: 10000 and flush_check_interval: 7000

Following is the result table which shows the efficiency:

Configuration	Flush Interval	Check Schedule Interval	CPU time	Total CPU/min
Pipeline 1 (baseline)	5s	1s	~50.67ms	~3,040ms (5.1%)
Pipeline 2	5s	2s	~50.67ms	~1,520ms (2.5%)
Pipeline 3	5s	5s	~50.67ms	~608ms (1.0%)
Pipeline 4	10s	5s	~50.67ms	~608ms (1.0%)
Pipeline 5	10s	7s	~50.67ms	~434ms (0.7%)

Why is it important/What is the impact to the user?

The users who are using intensive DLQ operations (write/read), the frequent flush check scheduler might give overhead to the pipeline, means uses much CPU. Introducing configurable scheduler cadence improves the pipeline efficiency by removing frequent operations.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

• pull this change
• create multiple pipelines with different flush_interval and flush_check_interval params, below is the example.
• run the Logstash

- pipeline.id: dlq-test-pipeline-1
  pipeline.workers: 1
  dead_letter_queue.enable: true
  dead_letter_queue.flush_interval: 5000
  dead_letter_queue.flush_check_interval: 1000
  dead_letter_queue.max_bytes: 14mb
  path.config: "config/tests/elasticsearch-output.conf"

- pipeline.id: dlq-test-pipeline-2
  pipeline.workers: 1
  dead_letter_queue.enable: true
  dead_letter_queue.flush_interval: 5000
  dead_letter_queue.flush_check_interval: 2000
  dead_letter_queue.max_bytes: 24mb
  path.config: "config/tests/elasticsearch-output.conf"

- pipeline.id: dlq-test-pipeline-3
  pipeline.workers: 1
  dead_letter_queue.enable: true
  dead_letter_queue.flush_interval: 5000
  dead_letter_queue.flush_check_interval: 5000
  dead_letter_queue.max_bytes: 24mb
  path.config: "config/tests/elasticsearch-output.conf"

- pipeline.id: dlq-test-pipeline-4
  pipeline.workers: 1
  dead_letter_queue.enable: true
  dead_letter_queue.flush_interval: 10000
  dead_letter_queue.flush_check_interval: 5000
  dead_letter_queue.max_bytes: 24mb
  path.config: "config/tests/elasticsearch-output.conf"

- pipeline.id: dlq-test-pipeline-5
  pipeline.workers: 1
  dead_letter_queue.enable: true
  dead_letter_queue.flush_interval: 10000
  dead_letter_queue.flush_check_interval: 7000
  dead_letter_queue.max_bytes: 24mb
  path.config: "config/tests/elasticsearch-output.conf"

ES configured in config/tests/elasticsearch-output.conf needs to reject events either 400 or 404 to be routed to the DLQ. Used the following config:

input {
  generator {
    id => "generator-id"
    ecs_compatibility => disabled
    count => 30000000
    threads => 2
    codec => json
    lines => [
	'{"fileset":{"module":"system","name":"auth"},"system":{"auth":{"timestamp":"May 17 05:17:00","ssh":{"source":{"ip":"123.123.123.123"}}}},"event":{"module":"cisco","data":{"User-Name":"mashhur"}},"client":{"ip":"123.123.123.123"},"DstIP":"123.123.123.123","SrcIP":"123.123.123.123","orginalClientSrcIP":"123.123.123.123","destination":{"ip":"123.123.123.123"},"source":{"ip":"123.123.123.123"},"ReferencedHost":"ip-192-168-1-2","DNSQuery":"example.com/my-path?query=value"}',
'{"fileset":{"module":"system","name":"syslog"},"system":{"auth":{"timestamp":"May 17  05:17:00","ssh":{"source":{"ip":"123.123.123.123"}}}},"event":{"module":"cisco","data":{"User-Name":"mashhur"}},"client":{"ip":"123.123.123.123"},"DstIP":"123.123.123.123","SrcIP":"123.123.123.123","orginalClientSrcIP":"123.123.123.123","destination":{"ip":"123.123.123.123"},"source":{"ip":"123.123.123.123"},"ReferencedHost":"ip-192-168-1-2","DNSQuery":"example.com/my-path?query=value"}',
'{"fileset":{"module":"system","name":"asa"},"system":{"auth":{"timestamp":"May 17  05:17:00","ssh":{"source":{"ip":"123.123.123.123"}}}},"event":{"category":"cisco-category", "type":"cisco-type", "data":{"User-Name":"mashhur"}},"client":{"ar_net":"123.123.123.123", "ongisac_ip":"123.123.123.123", "ip":"123.123.123.123"}, "destination": {"ar_net":"123.123.123.123", "ongisac_ip":"123.123.123.123"}, "source": {"ar_net":"123.123.123.123", "ongisac_ip":"123.123.123.123"}, "url":{"origin_domain": "ip-192-168-1-2"}, "DstIP":"123.123.123.123","SrcIP":"123.123.123.123","orginalClientSrcIP":"123.123.123.123","ReferencedHost":"ip-192-168-1-2", "dns":{"question": {"origin_domain":"example.com/my-path?query=value"}}}'
    ]
  }
}

output {
  elasticsearch {
    hosts => "http://127.0.0.1:9200"
    user => "elastic"
    password => "{pwd}"
    index => "test-dlq"
    action => "update"
    document_id => "nonexistent_id_12345"
    ecs_compatibility => disabled
  }
}

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify]

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[github-actions]

🔍 Preview links for changed docs

• docs/reference/dead-letter-queues.md

[github-actions]

✅ Vale Linting Results

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[alexcams]

Great implementation @mashhurs! I've compared results locally with before and after this feature and I see the CPU usage improvement on those pipelines using a value greater than the default (1s).
Just left one question about the value clamping.
Would be nice to hear @andsel thoughts too :)

[andsel]

Left some comment on how to make the changes more readable and testable.

[andsel]

There is a specific reason to do not use TimeValueSetting which should be preferred for time intervals?

[mashhurs]

Don't we need to keep the consistency with flush_interval ? I initially tried and in the docs it looked strange that flush_interval guides in milliseconds like 5000 and flush_check_interval in string format like 1s.

[andsel]

I understand your concern, but the setting is an interval and there is a specific setting type for this case, would be nice to use that because is less error prone. If we use a numeric value that are milliseconds, we have to specify in the doc the unit of time, while with TimeValueSetting is explicit. However, like you said, it created inconsistency with the existing, but this is a new setting.
Prefer consistency or use the appropriate setting? I don't have a strong motivation for one or the other.

[mashhurs]

Prefer consistency or use the appropriate setting?

I am also okay with either one. I was thinking from the user experience point of view that they are belong to the same domain, same type but in a different format.

[andsel]

@jsvd WDYT? about this thread? Prefer consistency or use the setting type that's more aligned with the use (it's an interval => TimeValue) ?

[jsvd]

++ on using the more aligned one. if we want we can change the type for 10.x

[andsel]

Left some suggestion to improve test code and a little rewording of the doc to avoid using "scheduler" without introducing the concept first.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 46f1568

Failed CI Steps

• :rspec: Ruby unit tests
• 🥼 Integration Tests - FIPS mode / part 4-of-6

History

• 💛 Build #4773 was flaky 86edfb5
• 💚 Build #4772 succeeded 32275e7
• 💚 Build #4771 succeeded 76741de
• 💚 Build #4749 succeeded ab4f453
• 💔 Build #4742 failed 355f18f

cc @mashhurs

[andsel]

LGTM!

Thanks @mashhurs great work 👍
I've asked @jsvd for an opinion about the type of setting, wait for his thought, but to me it's not blocking.

[mashhurs]

@Mergifyio backport 9.4

[mergify]

backport 9.4

✅ Backports have been created

Details

• #19090 [9.4] (backport #19036) dead_letter_queue.flush_check_interval new config for flushing staled segment files. has been created for branch 9.4