-
Notifications
You must be signed in to change notification settings - Fork 15
Test sampling rate increase cap #6412
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| # Unless explicitly stated otherwise all files in this repository are licensed under the the Apache License Version 2.0. | ||
| # This product includes software developed at Datadog (https://www.datadoghq.com/). | ||
| # Copyright 2021 Datadog, Inc. | ||
|
|
||
| import time | ||
|
|
||
| import requests | ||
|
|
||
| from utils import weblog, interfaces, scenarios, features, logger | ||
| from utils.proxy.mocked_response import ( | ||
| MOCKED_TRACER_RESPONSES_PATH, | ||
| SequentialJsonMockedTracerResponse, | ||
| _get_proxy_domain, | ||
| ) | ||
| from utils.proxy.ports import ProxyPorts | ||
|
|
||
|
|
||
| LOW_RATE = 0.1 | ||
| HIGH_RATE = 1.0 | ||
|
|
||
|
|
||
| def _send_mocked_tracer_responses(mocks: list) -> None: | ||
| """Send multiple mocked tracer responses in a single PUT request.""" | ||
| domain = _get_proxy_domain() | ||
| response = requests.put( | ||
| f"http://{domain}:{ProxyPorts.proxy_commands}{MOCKED_TRACER_RESPONSES_PATH}", | ||
| json=[m.to_json() for m in mocks], | ||
| timeout=30, | ||
| ) | ||
| response.raise_for_status() | ||
|
|
||
|
|
||
| @scenarios.sampling_rate_capping | ||
| @features.ensure_that_sampling_is_consistent_across_languages | ||
| class Test_SamplingRateCappedIncrease: | ||
| """When the agent returns a new higher sampling rate, the tracer should not jump directly | ||
| to the new rate. Instead, it should cap increases to 2x per flush interval, ramping up | ||
| gradually (e.g. 0.1 -> 0.2 -> 0.4 -> 0.8 -> 1.0). | ||
| """ | ||
|
|
||
| NUM_LOW_RATE_RESPONSES = 3 | ||
| NUM_HIGH_RATE_RESPONSES = 30 | ||
|
|
||
| def setup_sampling_rate_capped_increase(self): | ||
| low_rate_response = {"rate_by_service": {"service:,env:": LOW_RATE}} | ||
| high_rate_response = {"rate_by_service": {"service:,env:": HIGH_RATE}} | ||
|
|
||
| sequence = [low_rate_response] * self.NUM_LOW_RATE_RESPONSES + [ | ||
| high_rate_response | ||
| ] * self.NUM_HIGH_RATE_RESPONSES | ||
|
|
||
| # Send mocks for both trace endpoints in one call to avoid overwriting | ||
| mocks = [ | ||
| SequentialJsonMockedTracerResponse(path="/v0.4/traces", mocked_json_sequence=sequence), | ||
| SequentialJsonMockedTracerResponse(path="/v0.5/traces", mocked_json_sequence=sequence), | ||
| ] | ||
| _send_mocked_tracer_responses(mocks) | ||
|
|
||
| # Generate initial traffic until the tracer picks up the low rate | ||
| for i in range(40): | ||
| weblog.get(f"/sample_rate_route/{i}") | ||
|
|
||
| # Wait for a span with the low agent_psr to appear | ||
| def wait_for_low_rate(_data: dict) -> bool: | ||
| for _, span in interfaces.library.get_root_spans(): | ||
| agent_psr = span.get("metrics", {}).get("_dd.agent_psr") | ||
| if agent_psr is not None and abs(agent_psr - LOW_RATE) < 0.01: | ||
| return True | ||
| return False | ||
|
|
||
| interfaces.library.wait_for(wait_for_low_rate, timeout=30) | ||
|
|
||
| # Record how many spans exist before the ramp-up phase | ||
| self._spans_before_ramp = sum(1 for _ in interfaces.library.get_root_spans()) | ||
|
|
||
| # Generate traffic in bursts to trigger multiple flush cycles during ramp-up | ||
| # Each burst sends requests, then sleeps to allow the tracer to flush and receive | ||
| # the next mocked response, which should trigger a capped rate increase. | ||
| request_idx = 100 | ||
| for _ in range(10): | ||
| for _j in range(20): | ||
| weblog.get(f"/sample_rate_route/{request_idx}") | ||
| request_idx += 1 | ||
| time.sleep(2) | ||
|
|
||
| # Wait for a span with the high rate that appeared AFTER the low-rate phase | ||
| def wait_for_high_rate_after_ramp(_data: dict) -> bool: | ||
| for idx, (_, span) in enumerate(interfaces.library.get_root_spans()): | ||
| if idx < self._spans_before_ramp: | ||
| continue | ||
| agent_psr = span.get("metrics", {}).get("_dd.agent_psr") | ||
| if agent_psr is not None and abs(agent_psr - HIGH_RATE) < 0.01: | ||
| return True | ||
| return False | ||
|
|
||
| interfaces.library.wait_for(wait_for_high_rate_after_ramp, timeout=40) | ||
|
|
||
| def test_sampling_rate_capped_increase(self): | ||
| """Verify that the tracer ramps up sampling rate gradually instead of jumping directly.""" | ||
| # Only look at spans from AFTER the low-rate phase to avoid the default 1.0 at startup | ||
| agent_psr_values = set() | ||
|
|
||
| for idx, (_, span) in enumerate(interfaces.library.get_root_spans()): | ||
| if idx < self._spans_before_ramp: | ||
| continue | ||
| agent_psr = span.get("metrics", {}).get("_dd.agent_psr") | ||
| if agent_psr is not None: | ||
| agent_psr_values.add(round(agent_psr, 4)) | ||
|
|
||
| logger.info(f"Observed _dd.agent_psr values (ramp phase): {sorted(agent_psr_values)}") | ||
|
|
||
| assert any(abs(v - LOW_RATE) < 0.01 for v in agent_psr_values), ( | ||
| f"Expected to see the low rate ({LOW_RATE}) in _dd.agent_psr values: {sorted(agent_psr_values)}" | ||
| ) | ||
| assert any(abs(v - HIGH_RATE) < 0.01 for v in agent_psr_values), ( | ||
| f"Expected to see the high rate ({HIGH_RATE}) in _dd.agent_psr values: {sorted(agent_psr_values)}" | ||
| ) | ||
|
|
||
| # Key assertion: at least one intermediate value strictly between LOW_RATE and HIGH_RATE | ||
| intermediate_values = [v for v in agent_psr_values if LOW_RATE + 0.01 < v < HIGH_RATE - 0.01] | ||
| assert len(intermediate_values) > 0, ( | ||
| f"Expected at least one intermediate _dd.agent_psr value between {LOW_RATE} and {HIGH_RATE}, " | ||
| f"but only saw: {sorted(agent_psr_values)}. " | ||
| "The tracer should cap sampling rate increases to 2x per interval, not jump directly." | ||
| ) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This assertion can fail for a correctly implemented tracer because
_spans_before_rampis captured immediately after the first observed low-rate span, so the post-cutoff window may legitimately contain only ramped values (e.g., if no additional low-rate requests are emitted after the cutoff). In that case CI reports a regression even though capped increase behavior is correct, making the new test flaky across tracer flush timings.Useful? React with 👍 / 👎.