Skip to content

LLM Obs: Patterns coverage + Playground record cap#38039

Open
thomaschin35 wants to merge 1 commit into
masterfrom
dd/thomas.chin/patterns-coverage-playground
Open

LLM Obs: Patterns coverage + Playground record cap#38039
thomaschin35 wants to merge 1 commit into
masterfrom
dd/thomas.chin/patterns-coverage-playground

Conversation

@thomaschin35

Copy link
Copy Markdown

What does this PR do? What is the motivation?

Three LLM Observability documentation pages are updated to reflect four merged backend PRs:

Changes

content/en/llm_observability/monitoring/patterns.md

  • Added Coverage datasets (optional) sub-item to the Scope configuration step, explaining how to associate datasets for offline coverage measurement.
  • Added Coverage column to the topic list table describing the coverage ratio and suggested datapoints count.
  • Extended the "Find evaluation coverage gaps" section to explain that Patterns now computes offline coverage automatically when a coverage dataset is configured, including global ratio and per-dataset breakdown.
  • Added a note to the "Export and act on interactions" section explaining that Patterns marks individual interactions as suggested to help close coverage gaps, and that they can be added to a dataset with Add to Dataset.

content/en/llm_observability/playground.md

  • Added an alert-info callout after the "Save the experiment" step noting that Playground experiments process up to 10,000 dataset records per run and that only the first 10,000 are processed if the dataset exceeds this limit.

Merge readiness

  • Ready for merge

AI assistance

Used Claude Code (Bits Code) to draft and apply all documentation changes based on PR descriptions provided.

Additional notes

Draft PR — pending review before merge.


PR by Bits - View session in Datadog

…tasets, suggested datapoints, and 10k record limit
@datadog-staging

Copy link
Copy Markdown

View session in Datadog

Bits Code status: ✅ Done

CI Auto-fix requires auto-push.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cd7f9191d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


The experiment runs across all records in the dataset—not only the 20-record preview sample. When complete, view results in [{{< ui >}}AI Observability{{< /ui >}} > {{< ui >}}Experiments{{< /ui >}}][2].

<div class="alert alert-info">Playground experiments process up to 10,000 dataset records per run. If your dataset exceeds this limit, only the first 10,000 records are processed.</div>

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconcile the full-dataset claim with the cap

For datasets with more than 10,000 records, this new note contradicts the paragraph immediately above that says the experiment runs across all records in the dataset. Readers can come away thinking a saved experiment evaluates every row even though this callout says the run is capped, so please update the full-dataset sentence or combine it with this note to make the capped behavior unambiguous.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thomaschin35 thanks for the PR! Since this feedback hasn't been addressed, I'll add the WORK IN PROGRESS label for now. Could you please address this feedback, or provide a rationale for ignoring it, and then remove the WORK IN PROGRESS label to re-open it for documentation team review?

Let me know if you have any questions!

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

@buraizu buraizu added the WORK IN PROGRESS No review needed, it's a wip ;) label Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bits AI WORK IN PROGRESS No review needed, it's a wip ;)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants