feat(observability): inject solution into outbound AWS SDK User-Agent

[scottschreckengaust]

Summary

Stamp an ABCA solution-attribution identifier into the outbound AWS SDK User-Agent on every AWS API call made by the platform, so AWS can attribute runtime API usage to this sample. This is the per-API-call complement to the deploy-time CloudFormation attribution token added in #292 (uksb-wt64nei4u6 in the stack description, cdk/src/main.ts:41).

Two cooperating fields, emitted on every request:

app/uksb-wt64nei4u6/{STACKNAME}             ← app-id form; STACKNAME sanitized + clipped to ≤34 chars (≤50 total)
md/uksb-wt64nei4u6#{IDENTIFIER}#{TRACE}     ← solution metadata; {TRACE} OPTIONAL, appended per-request

• app/uksb-wt64nei4u6/{STACKNAME} — the app-id form. uksb-wt64nei4u6/ is 16 chars, so {STACKNAME} is the first ≤ 34 chars of the sanitized stack name to keep the whole value ≤ 50. {STACKNAME} is ASCII-safe with / and # (and any other non-token char) replaced by - — see "Separators & sanitization". Set once per environment.
• md/uksb-wt64nei4u6#{IDENTIFIER}#{TRACE} — solution metadata, emitted verbatim (no length cap — see "Character budget"). {IDENTIFIER} names which ABCA component the calling code is (e.g. agent, orchestr, webhook, cli) — a stable per-component label baked at client construction. {TRACE} is an optional, per-component correlation token — whatever trace handle that component has (see "The {TRACE} field") — appended per-request without reissuing the connection (see "Connection sharing"). When no trace handle is in scope, the token is md/uksb-wt64nei4u6#{IDENTIFIER} with no trailing separator. (The stack name lives only in the app/ field, not here.)

Background

AWS samples/solutions are attributed two ways:

1. Deploy-time — a uksb-* token in the CloudFormation stack description. ✅ Done in feat(infra): add solution-tracking metrics ID to CloudFormation stack description #292.
2. Runtime — a solution identifier injected into the SDK User-Agent header so individual API calls are attributable. ❌ This issue.

Today no SDK client in the repo customizes its user agent. There are three independent SDK surfaces, each instantiating clients with empty config:

Surface	Lang / SDK	Examples	Injection mechanism
CDK Lambda handlers	TS / AWS SDK for JS v3	new DynamoDBClient({}), new BedrockAgentCoreClient({}), new ECSClient({}), new S3Client({}) (cdk/src/handlers/*.ts)	customUserAgent (verbatim metadata), or a UA middleware
Agent runtime	Python / boto3 + botocore	boto3.client(...) / boto3.Session(...) via agent/src/aws_session.py, agent/src/config.py	Config(user_agent_extra=...) (verbatim metadata), or a before-send event handler
CLI	TS / Cognito SDK v3	Cognito client in cli/src/auth.ts	same as CDK

Note: cdk/src/handlers/shared/gateway.ts:57,87 references user_agent but that is the inbound API Gateway caller UA from the request context — unrelated to outbound SDK calls.

Separators & sanitization (authoritative — see issue comments)

Research findings on this issue (with verbatim source snippets) are in the two comments below. The relevant constraints:

• Metadata (md/...) is emitted verbatim — botocore wraps user_agent_extra as a RawStringUserAgentComponent (no sanitization, no truncation); JS v3 customUserAgent is likewise passed through. So the # separators inside the md/uksb-wt64nei4u6#{IDENTIFIER}#{TRACE} string are valid and preserved. ✅
• The literal / between uksb-wt64nei4u6 and {STACKNAME} in the app/ segment is kept. If {STACKNAME} were set via the SDK app-id config field (AWS_SDK_UA_APP_ID / botocore user_agent_appid / JS userAgentAppId), that field is sanitized against an allowed charset that excludes / (→ -), which would mangle the uksb-wt64nei4u6/ separator. So the app/uksb-wt64nei4u6/{STACKNAME} string is contributed via the raw user-agent component path (same verbatim mechanism as md/), not the sanitizing app-id config field. The ≤ 50 length discipline is applied by us at construction.
• {STACKNAME}, {IDENTIFIER}, and {TRACE} are sanitized by us before emission. Because we emit via the raw path (no SDK sanitization), each value must be cleaned so it cannot break the segment structure or violate the UA token charset:
  • Replace / and # (the structural separators of this scheme) — and any other non-UA-token character — with -.
  • Enforce ASCII-safe: drop/replace any non-ASCII byte with -.
  • Target charset = UA token chars (A–Z a–z 0–9 ! $ % & ' * + - . ^ _ \ | ~). In practice CloudFormation stack names are already [A-Za-z0-9-]` and most trace handles (ULIDs, Lambda request IDs, PIDs) are too, so this is largely defensive — but it MUST be applied unconditionally to all three values.
  • For {STACKNAME}, sanitize first, then clip to ≤ 34 chars (so clipping can't re-expose a partial multi-byte sequence).

Character budget

Authoritative findings (verbatim snippets + URLs) are in the issue comments. Summary:

• md/ metadata has NO SDK length cap. The 50-char limit applies only to the app-id field; user_agent_extra / customUserAgent is emitted verbatim. The only other size limit in the UA code is 1024 bytes on the unrelated m/ business-metrics component. So md/uksb-wt64nei4u6#{IDENTIFIER}#{TRACE} is not budget-constrained — {IDENTIFIER} and the {TRACE} handle (ULID, Lambda request ID, etc.) fit with no truncation. (Keep it reasonable for HTTP-header practicality, but there is no hard SDK cap.)
• The app-id form keeps the ≤ 50 discipline because we mirror the documented app-id constraint even though we emit via the raw path: uksb-wt64nei4u6/ (16) + {STACKNAME} ≤ 50 → {STACKNAME} = first ≤ 34 chars of the sanitized stack name. The app/ marker itself is not counted toward the 50 (confirmed: the documented 50 is on the value, not the prefix).

Because {TRACE} rides in the uncapped md/ metadata, no {TRACE} elision is ever required — the full handle is always emitted.

The {IDENTIFIER} field — which component is making the call

{IDENTIFIER} lives in the uncapped md/ metadata and is baked once at construction. The stack name is no longer carried in md/ — it lives only in the app/ field (as the sanitized, ≤ 34-char {STACKNAME}).

{IDENTIFIER} names the component / "thing" the calling code is — i.e. which part of ABCA originated the AWS API call. It is a per-component constant, not a per-deployment value: each emitting surface uses a fixed label describing itself (the {STACKNAME} in the app/ field already carries the deployment identity).

Examples of the kind of value intended (illustrative — final names up to the implementer):

• agent — the Python agent runtime container
• orchestr — orchestrator / task-lifecycle Lambdas
• webhook — webhook-create-task path
• cli — the bgagent CLI

Requirements:

• Must be UA-token-safe (sanitized: /, #, and any non-UA-token / non-ASCII char → -).
• Recommended short — ≤ 10 characters — but NOT a hard requirement. There is no SDK length cap on md/ metadata (see comments), so longer values are allowed; ≤ 10 is a readability/conciseness guideline only.
• A given component uses a stable label (don't vary it per call); the only per-request variation in md/ is the optional #{TRACE} suffix.

The {TRACE} field — optional per-component correlation handle

{TRACE} is the optional trailing segment. It is not task-specific — it carries whatever correlation/trace handle the emitting component naturally has, so the same scheme works across every surface. Each component supplies its own kind of handle (or none):

Component ({IDENTIFIER})	Natural {TRACE} handle	Source
agent	task ULID	the task the agent is working
orchestr (Lambda)	AWS Lambda request ID	context.awsRequestId (JS) / context.aws_request_id
cli	process PID (or session id)	process.pid
any other	whatever trace handle it carries	component-specific

{TRACE} is always optional — when a component has no meaningful handle for a given call (or hasn't wired one up), it is omitted and the metadata is simply md/uksb-wt64nei4u6#{IDENTIFIER} (no trailing #). When present, it lets the md/ field answer "which ABCA component made this call, correlated to which unit of work" — e.g. md/uksb-wt64nei4u6#agent#01KTVY…, md/uksb-wt64nei4u6#orchestr#<aws-request-id>, md/uksb-wt64nei4u6#cli#12345.

{TRACE} must be UA-token-safe (same sanitization as above) and is appended per-request (never baked into client config — see "Connection sharing").

Connection sharing — {TRACE} must NOT pin a connection

agent/src/aws_session.py deliberately keeps a module-level singleton session (_session) with refreshable credentials so a single long-running task (up to the 8 h maxLifetime) reuses one session and its connection pool rather than re-assuming/reconnecting per call. There can also be durable/cached-client situations across task boundaries.

Baking a per-request {TRACE} handle into a client's Config(user_agent_extra=...) at construction time would pin that cached client (and its HTTP connection pool) to a single trace context, forcing a client/connection reissue just to update the UA. That is the failure mode to avoid.

Resolution — static baked, dynamic appended per-request:

• Static (app/uksb-wt64nei4u6/{STACKNAME} and md/uksb-wt64nei4u6#{IDENTIFIER}) is baked once at client/session construction. It never varies, so shared/cached clients and pooled connections are reused freely.
• Dynamic (#{TRACE}) is appended per request by mutating only the outgoing User-Agent header — never the client config — reading the current trace handle from ambient context:
  • Python / botocore: register a before-send (or before-sign) event handler on the session that reads a contextvar holding the current trace handle and appends #{TRACE} to request.headers['User-Agent']. The connection pool is untouched; only the header string changes per request.
  • JS SDK v3: add a build/finalizeRequest middleware to the client stack that reads the current trace handle from AsyncLocalStorage (or, in a Lambda, from context.awsRequestId) and appends #{TRACE}.
• {TRACE} is optional: when no trace handle is in scope, the header is simply md/uksb-wt64nei4u6#{IDENTIFIER} with no trailing separator.

Net effect on the wire: ... app/uksb-wt64nei4u6/{STACKNAME} md/uksb-wt64nei4u6#{IDENTIFIER}#{TRACE} ..., with the #{TRACE} segment present only when a handle is in scope — and reusing one connection across many trace contexts.

Acceptance criteria

• Static metadata baked once: all clients/sessions carry md/uksb-wt64nei4u6#{IDENTIFIER} baked at construction (boto3 Config(user_agent_extra=...), JS customUserAgent), and the app/uksb-wt64nei4u6/{STACKNAME} form contributed via the raw user-agent path (NOT the sanitizing app-id field — see the sanitization section).
• {STACKNAME} sanitized + clipped: the stack name is sanitized (replace /, #, and any non-UA-token / non-ASCII char with -) then clipped to the first ≤ 34 chars so the whole app-id value (uksb-wt64nei4u6/ + name) stays ≤ 50. A test asserts: (a) a stack name containing /, #, and a non-ASCII char emits with those replaced by -; (b) the value length for the longest realistic stack name is ≤ 50.
• {IDENTIFIER} is a stable per-component label identifying which ABCA component the calling code is (e.g. agent, orchestr, webhook, cli) — fixed per surface, not varied per call; UA-token-safe; recommended ≤ 10 chars (soft guideline, not enforced — no SDK cap on md/). A test asserts each surface emits its expected component label.
• {TRACE} appended per-request, never baked, and OPTIONAL: a botocore before-send handler (agent) and a JS SDK middleware (handlers/CLI as applicable) append #{TRACE} from ambient context (contextvar / AsyncLocalStorage / Lambda context.awsRequestId / PID) without reconstructing the client or its connection pool. Each component supplies its own handle kind (agent → task ULID, orchestrator Lambda → AWS request ID, CLI → PID, etc.) and may omit it. Tests assert: (a) two requests from the same cached client under different trace contexts emit different #{TRACE} segments (connection reused, not reissued); (b) with no handle, the metadata is exactly md/uksb-wt64nei4u6#{IDENTIFIER} (no trailing #); (c) {TRACE} is sanitized to UA-token-safe ASCII.
• Outgoing-header inspection tests (required for validation): tests must capture the actual outbound request's User-Agent header (not just the configured client value) and assert that both the app/uksb-wt64nei4u6/{STACKNAME} segment and the md/uksb-wt64nei4u6#{IDENTIFIER}[#{TRACE}] segment are present and intact in the emitted header — including verifying the literal / in the app/ segment survives (i.e. it was NOT routed through the sanitizing app-id field). Capture the header at the wire layer per surface — e.g. a botocore before-send-stage assertion or a stubbed transport that records request.headers['User-Agent'] (boto3), and a terminal middleware or request spy capturing the user-agent/x-amz-user-agent header (JS SDK v3). Verify both the trace-present (...#{TRACE}) and trace-absent (...#{IDENTIFIER} only) cases.
• Agent runtime (boto3): centralized in agent/src/aws_session.py / agent/src/config.py so no boto3.client(...) call site is missed; the singleton refreshable session keeps its pool across tasks.
• CDK Lambda handlers (JS SDK v3): all new XClient({}) instantiations carry the ABCA UA via a shared client factory or UA middleware.
• CLI (JS SDK v3): the Cognito client in cli/src/auth.ts carries the ABCA UA.
• {STACKNAME} sourced from the deployed stack name, threaded to each surface (Lambda env, agent container env, CLI config) — not hard-coded; sanitized to UA-token-safe ASCII.
• CDK compiles, synth succeeds, mise //cdk:test passes; agent tests pass.
• No functional behavior change beyond the added UA metadata.

Related

• feat(infra): add solution-tracking metrics ID to CloudFormation stack description #292 — solution-attribution metrics ID in the CloudFormation stack description (deploy-time counterpart; CLOSED/implemented). This issue is the runtime API-call counterpart.
• feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215 — Bedrock cost attribution (session tags, request metadata, FinOps). UA-based attribution is adjacent; coordinate so Bedrock-client UA work isn't duplicated.
• feat(observability): end-to-end task attribution and cross-plane trace correlation #245 — end-to-end task attribution & cross-plane trace correlation. The {TRACE} handle in the UA is another attribution vector; align on the identifier.

Out of scope

• Dashboards / querying of the attributed usage (belongs with feat: Bedrock cost attribution — session tags, request metadata, and operator FinOps guidance #215 / feat(observability): end-to-end task attribution and cross-plane trace correlation #245).
• Changing the deploy-time uksb token (already handled by feat(infra): add solution-tracking metrics ID to CloudFormation stack description #292).

[krokoko]

We also need to have it in the docs somewhere as instructions to the agent, that if we add additional SDK calls, we add the user agent header

[scottschreckengaust]

Research: AWS_SDK_UA_APP_ID value limit (authoritative)

TL;DR — the 50-char limit is on the value only (the app/ prefix is NOT counted); over-limit is a warning, not a rejection or truncation; and / is NOT an allowed character — it gets sanitized to -. # is allowed.

1. Max length = 50 chars, measured on the value (not including app/)

AWS SDKs & Tools Reference Guide — https://docs.aws.amazon.com/sdkref/latest/guide/feature-appid.html

Valid values: String with maximum length of 50. Letters, numbers and the following special characters are allowed: !,#,$,%,&,',*,+,-,.,^,_,`,|,~.

botocore botocore/args.py — https://github.com/boto/botocore/blob/develop/botocore/args.py

# Maximum allowed length of the ``user_agent_appid`` config field. Longer
# values result in a warning-level log message.
USERAGENT_APPID_MAXLEN = 50

The check runs on the raw config value before the app/ prefix is prepended at render time, so the prefix does not count toward the 50:

botocore botocore/useragent.py — https://github.com/boto/botocore/blob/develop/botocore/useragent.py

def _build_app_id(self):
    if self._client_config and self._client_config.user_agent_appid:
        appid = sanitize_user_agent_string_component(
            raw_str=self._client_config.user_agent_appid, allow_hash=True
        )
        return [RawStringUserAgentComponent(f'app/{appid}')]

AWS SDK for JS v3 — https://github.com/aws/aws-sdk-js-v3/blob/main/packages-internal/core/src/submodules/client/middleware-user-agent/configurations.ts

function isValidUserAgentAppId(appId: unknown): boolean {
  if (appId === undefined) return true;
  return typeof appId === "string" && appId.length <= 50;
}

(checked on appId itself, later formatted as app/${appId})

ABNF (smithy-rs user-agent spec) — https://github.com/smithy-lang/smithy-rs/blob/main/aws/rust-runtime/aws-runtime/src/user_agent.rs

app-id = "app/" name

The 50-char limit is on name; "app/" is a literal prefix.

2. Over-limit behavior: warning logged, value still used (NOT truncated, NOT rejected)

botocore botocore/args.py:

if user_agent_appid is not None and len(user_agent_appid) > USERAGENT_APPID_MAXLEN:
    logger.warning(
        'The configured value for user_agent_appid exceeds the '
        'maximum length of %d characters.', USERAGENT_APPID_MAXLEN,
    )
config_kwargs['user_agent_appid'] = user_agent_appid   # still used

JS v3 configurations.ts:

} else if (appId.length > 50) {
  logger?.warn("The provided userAgentAppId exceeds the maximum length of 50.");
}
return appId;   // still returned and used

3. Charset — / is NOT allowed (sanitized to -); # IS allowed for app-id

Allowed set per the docs above: ASCII letters, digits, !#$%&'*+-.^_`|~. Disallowed chars are replaced with -:

botocore botocore/useragent.py:

_USERAGENT_ALLOWED_CHARACTERS = ascii_letters + digits + "!$%&'*+-.^_`|~,"

def sanitize_user_agent_string_component(raw_str, allow_hash):
    """Replaces all not allowed characters in the string with a dash ("-")."""
    return ''.join(
        c if c in _USERAGENT_ALLOWED_CHARACTERS or (allow_hash and c == '#')
        else '-'
        for c in raw_str
    )

For app-id, _build_app_id passes allow_hash=True, so # survives but / does not (it is not in the allowed set) → replaced with -.

JS v3 — https://github.com/aws/aws-sdk-js-v3/blob/main/packages-internal/core/src/submodules/client/middleware-user-agent/constants.ts

UA_VALUE_ESCAPE_REGEX = /[^!$%&'*+\-.^_`|~\w#]/g   // disallowed -> "-"

Implication for this issue

Our proposed app/uksb-wt64nei4u6/{STACK} uses a / separator that is not in the allowed charset — botocore/JS would emit app/uksb-wt64nei4u6-{STACK} (slash → -). We must either accept the - form or pick an allowed separator (#, ., _, + are all valid). Budget remains: value uksb-wt64nei4u6<sep>{STACK} = 16 + STACK ≤ 50 → STACK ≤ 34, and the app/ prefix is not counted.

[scottschreckengaust]

Research: does the md/ metadata cap include md/? (authoritative)

TL;DR — there is NO length cap on md/ metadata entries at all. The 50-char limit applies ONLY to the app-id (app/) value; user_agent_extra / customUserAgent is emitted verbatim with no length check. So the question "does the 50 include md/?" is moot — there is no 50 on md/. (And the OTHER question — whether the app-id 50 includes app/ — is answered NO; see the app-id comment.)

This invalidates the issue's "≤ 50 chars on the content after md/" budget — there is no such cap. {IDENTIFIER} is not bounded to 5/7 by any SDK limit; choose its size on readability/collision grounds, not budget.

1. No length limit exists for md/ metadata

user_agent_extra is wrapped as a RawStringUserAgentComponent and emitted with no sanitization and no truncation. The only size constraint anywhere in botocore's user-agent code is 1024 bytes on the m/ business-metrics component — a different field, not md/:

botocore botocore/useragent.py — https://github.com/boto/botocore/blob/develop/botocore/useragent.py

size_config = UserAgentComponentSizeConfig(1024, ',')
return [
    UserAgentComponent('m', ','.join(features), size_config=size_config)
]

No size_config is ever applied to md/ metadata components. args.py's only length constant is USERAGENT_APPID_MAXLEN = 50 — for user_agent_appid only.

JS v3 (constants.ts) and Go v2 (user_agent.go) likewise define no length constant for metadata — only character-escape regexes.

2. The structured-grammar answer (for completeness): prefixes are literals, not counted in any value limit

Where length limits do exist (the app-id), the ABNF shows the prefix is a literal separate from the counted value:

smithy-rs user-agent ABNF — https://github.com/smithy-lang/smithy-rs/blob/main/aws/rust-runtime/aws-runtime/src/user_agent.rs

additional-metadata = "md/" ua-pair
ua-pair             = ua-name ["/" ua-value]
app-id              = "app/" name

"md/" and "app/" are literal prefixes; the variable part is ua-pair / name. So even if a limit applied, it would be on the value, not the md//app/ literal. (For app-id this is confirmed in code — see the app-id comment.) For md/ there is simply no limit to apply.

3. How md/ renders (botocore)

botocore UserAgentComponent.to_string() (botocore/useragent.py):

def to_string(self):
    """Create string like 'prefix/name#value' from a UserAgentComponent."""
    clean_prefix = sanitize_user_agent_string_component(self.prefix, allow_hash=True)
    clean_name = sanitize_user_agent_string_component(self.name, allow_hash=False)
    if self.value is None or self.value == '':
        clean_string = f'{clean_prefix}/{clean_name}'
    else:
        clean_value = sanitize_user_agent_string_component(self.value, allow_hash=True)
        clean_string = f'{clean_prefix}/{clean_name}#{clean_value}'

i.e. md/name#value (e.g. md/awscrt#0.16.0). Note user_agent_extra specifically is a RawStringUserAgentComponent — passed through verbatim, so our md/uksb-wt64nei4u6#{IDENTIFIER}#{TASK} string would appear exactly as composed (subject to whatever we sanitize ourselves).

Sources fetched

• botocore useragent.py, args.py, config.py, tests/functional/test_useragent.py — https://github.com/boto/botocore/tree/develop/botocore
• smithy-rs user_agent.rs, app_name.rs — https://github.com/smithy-lang/smithy-rs/tree/main/aws/rust-runtime
• AWS SDK JS v3 configurations.ts, constants.ts — https://github.com/aws/aws-sdk-js-v3/tree/main/packages-internal/core/src/submodules/client/middleware-user-agent
• AWS SDK Go v2 user_agent.go — https://github.com/aws/aws-sdk-go-v2/blob/main/aws/middleware/user_agent.go
• AWS SDKs & Tools Reference Guide — https://docs.aws.amazon.com/sdkref/latest/guide/feature-appid.html

Action items for this issue

• Drop the "≤ 50 on content after md/" budget and the derived {IDENTIFIER} hard-max-7 — neither is real. The md/ string can be as long as we want (HTTP header practicality aside).
• The only real SDK cap is the app-id 50 (value, excluding app/), and the / separator there is invalid (→ -); see the app-id comment.
• Keep {IDENTIFIER} size a design choice (readability + co-resident-deployment collision avoidance), not a budget-forced one.

[scottschreckengaust]

I've started working on this...branch and PR coming