LLM Obs: Patterns coverage + Playground record cap#38039
Conversation
…tasets, suggested datapoints, and 10k record limit
|
Bits Code status: ✅ Done CI Auto-fix requires auto-push. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd7f9191d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| The experiment runs across all records in the dataset—not only the 20-record preview sample. When complete, view results in [{{< ui >}}AI Observability{{< /ui >}} > {{< ui >}}Experiments{{< /ui >}}][2]. | ||
|
|
||
| <div class="alert alert-info">Playground experiments process up to 10,000 dataset records per run. If your dataset exceeds this limit, only the first 10,000 records are processed.</div> |
There was a problem hiding this comment.
Reconcile the full-dataset claim with the cap
For datasets with more than 10,000 records, this new note contradicts the paragraph immediately above that says the experiment runs across all records in the dataset. Readers can come away thinking a saved experiment evaluates every row even though this callout says the run is capped, so please update the full-dataset sentence or combine it with this note to make the capped behavior unambiguous.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@thomaschin35 thanks for the PR! Since this feedback hasn't been addressed, I'll add the WORK IN PROGRESS label for now. Could you please address this feedback, or provide a rationale for ignoring it, and then remove the WORK IN PROGRESS label to re-open it for documentation team review?
Let me know if you have any questions!
Preview links (active after the
|
What does this PR do? What is the motivation?
Three LLM Observability documentation pages are updated to reflect four merged backend PRs:
coveragefield to Patterns configuration, allowing users to associate datasets with a Pattern for offline coverage computation.suggest_datapointsactivity that automatically marks individual production interactions as suggested for dataset addition.Changes
content/en/llm_observability/monitoring/patterns.mdcontent/en/llm_observability/playground.mdalert-infocallout after the "Save the experiment" step noting that Playground experiments process up to 10,000 dataset records per run and that only the first 10,000 are processed if the dataset exceeds this limit.Merge readiness
AI assistance
Used Claude Code (Bits Code) to draft and apply all documentation changes based on PR descriptions provided.
Additional notes
Draft PR — pending review before merge.
PR by Bits - View session in Datadog