Tutorial for LLM as a Judge, and documenting the apis in SRW pages. by epugh · Pull Request #12083 · opensearch-project/documentation-website

epugh · 2026-03-10T21:53:23Z

Description

Documentation in SRW about this new judgment creation capabilty, and then a dedicated tutorial inspired by tutorials/vector-search/neural-search-tutorial example.

Issues Resolved

n/a

Version

3.5

Frontend features

n/a

Checklist

[x ] By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2026-03-10T21:53:32Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference).

epugh · 2026-03-10T23:08:21Z

cc @wrigleyDan @heemin32

* Add RomanianAnalyzer to 3.x breaking changes Signed-off-by: Gabriel Weyer <159976942+gweyeratlassian@users.noreply.github.com> * Improve Romanian analyzer breaking change wording Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Gabriel Weyer <159976942+gweyeratlassian@users.noreply.github.com> --------- Signed-off-by: Gabriel Weyer <159976942+gweyeratlassian@users.noreply.github.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…-project#11978) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

@kolchfa-aws

…11981) * Add warning about truncation to ML documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Apply suggestion from @kolchfa-aws Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Signed-off-by: xiaoyuan0821 <yuan13215@163.com> Co-authored-by: x00815292 <niklaus.xiao@huawei.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…#12065) Signed-off-by: Mykola Shestopal <nick4eva@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…project#12020) * Add restoring snapshot from a remote-backed cluster docs Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Formatting Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Link fix and reword Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…roject#12071) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

* Add 2.19.5 to version history Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Updated release description Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…ct#11982) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…pensearch-project#12089) * Fix create sparse vector index error Signed-off-by: xiaoyuan0821 <yuan13215@163.com> * Create index with explicit mapping for sort by geo distance example Signed-off-by: xiaoyuan0821 <yuan13215@163.com> * Add copy buttons and intro sentence to mapping Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: xiaoyuan0821 <yuan13215@163.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: x00815292 <niklaus.xiao@huawei.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

@kolchfa-aws

) * Adding rerank documentation for agentic search Signed-off-by: Joshua Palis <jpalis@amazon.com> * Doc review Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Reformat requests and clarify steps Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestion from @kolchfa-aws Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…docs

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

pajuric · 2026-05-22T14:35:48Z

@kolchfa-aws - What's the status here? Is this ready to push or does it require more reviews?

kolchfa-aws · 2026-05-26T13:35:17Z

@pajuric This is documentation, not a blog. It is in my queue for review since today.

epugh · 2026-05-27T12:57:31Z

@kolchfa-aws I will make sure to prioritize any feedback... One thing I've been struggling with is should I weave in more screenshots of the SRW UI.... Steps 3,4,5 can all be done via the SRW UI, though instead I went with the API-first approach. I was thinking of saying "You can also do these following steps using the SRW UI"? I know we try to minimize the number of images....

epugh

Vale flagged a couple of false positive alerts on "LLM" in code chunk. Some correct style errors to be fixed!

epugh · 2026-05-27T12:58:35Z

+:--- | :--- | :---
+`name` | String | The name of the judgment list.
+`description` | String | Optional. A description of the judgment list.
+`type` | String | Set to `LLM_JUDGMENT`.


I wonder if we can tweak vale to not do this on code chunks? I think this is technically a false positive? Would you agree @kolchfa-aws ?

Vale already ignores code blocks and code font by default. Very rarely you'll see a false positive. You can try and tweak it but usually it's not worth the time.

epugh · 2026-05-27T13:05:08Z

+
+You can adapt the connector configuration for other providers:
+
+#### AWS Bedrock example:


Bedrock is a name, so maybe it needs adding to our vocabulary? AWS bedrock example seems werid!

It's because it should be "Amazon Bedrock", not "AWS Bedrock".

epugh · 2026-05-27T13:05:43Z

+Congratulations, you are now ready to [run an experiment to evaluate search quality]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/evaluate-search-quality/#creating-a-pointwise-experiment) using the LLM-generated judgments that you have just created.  You can reuse the search configuration and query set's that you have already created in this tutorial when you run your first evaluation.
+
+
+## Advanced features


@kolchfa-aws do you think this advacned features sections more properly belongs in the documentation? I'm getting that feeling now that I am re-reading....

Okay, I went ahead and moved it to the reference page since we don't talk to it in the tutorial.

Yes, that's better

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…ordered with the top level descirpiton, and rewrite a lot of passive language to make ti punchier Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

…e tutorial Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

epugh · 2026-05-27T14:58:52Z

Okay, went and reworked the judgments page to have a better structure now that we added llm-as-a-judge content. Plus revamped the text to be less passive!

kolchfa-aws · 2026-05-27T18:38:56Z

@epugh Normally, we try to minimize the number of screenshots because they go stale and are hard to maintain with each UI change. Instructions should be enough for users to do the tasks they want to do.

pajuric · 2026-05-27T19:25:18Z

@pajuric This is documentation, not a blog. It is in my queue for review since today.

I realize it's not a blog, but it requires internal review before I can publish the blog that relates to the topic. I appreciate your review and feedback.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws

Thank you, @epugh! I simplified the tutorial steps because this is a more advanced tutorial that assumes that users are already familiar with creating connectors, registering models, etc. Doc review complete and changes applied.

kolchfa-aws · 2026-05-28T17:57:26Z

@pajuric Review complete. Merging this PR.

pajuric · 2026-05-28T18:08:10Z

@kolchfa-aws thank you for the review and merge. It's appreciated.

@kolchfa-aws

…12083) (#12479) * Add RomanianAnalyzer to 3.x breaking changes (#12008) * Add RomanianAnalyzer to 3.x breaking changes * Improve Romanian analyzer breaking change wording --------- * first pass of tutorial * add reference docs. * Fix up vale violations * Standarize on the name wikipedia uses for this technique * Clarify the effect of bulk operations on ingest pipelines (#11978) * Add warning about truncation to ML documentation (#11981) * Add warning about truncation to ML documentation * Apply suggestions from code review * Apply suggestion from @kolchfa-aws --------- * Fix create sparse vector index error (#12066) * Fix formatting for Boolean AND examples in dql.md (#12065) * Add restoring snapshot from a remote-backed cluster docs (#12020) * Add restoring snapshot from a remote-backed cluster docs * Formatting * Link fix and reword --------- * Change copy curl buttons to copy only in workspace docs (#12071) * Add 2.19.5 to version history (#12078) * Add 2.19.5 to version history * Updated release description --------- * Add additional details to search backpressure stats (#11982) * Create index with explicit mapping for sort by geo distance example (#12089) * Fix create sparse vector index error * Create index with explicit mapping for sort by geo distance example * Add copy buttons and intro sentence to mapping --------- * Adding rerank documentation for agentic search (#12081) * Adding rerank documentation for agentic search * Doc review * Reformat requests and clarify steps * Apply suggestion from @kolchfa-aws --------- * fix up links * prune back extra verbiage and ideas that arent going to land * Backout local only change * Refinig text and links * Refining text * Try to be clearer in Step 6 * Responding to vale feedback * Ensure standard naming patterns followed * use ALL CAPS format for variables * Rework the order of the descriptions of the types of judgments to be ordered with the top level descirpiton, and rewrite a lot of passive language to make ti punchier * Move the advanced feature text to the ref page, they arent part of the tutorial * Doc review * Apply suggestion from @kolchfa-aws --------- Signed-off-by: Gabriel Weyer <159976942+gweyeratlassian@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: xiaoyuan0821 <yuan13215@163.com> Signed-off-by: Mykola Shestopal <nick4eva@users.noreply.github.com> Signed-off-by: Joshua Palis <jpalis@amazon.com> Co-authored-by: Eric Pugh <epugh@opensourceconnections.com> Co-authored-by: Gabriel Weyer <159976942+gweyeratlassian@users.noreply.github.com> Co-authored-by: Yuan <yuan13215@163.com> Co-authored-by: x00815292 <niklaus.xiao@huawei.com> Co-authored-by: Mykola Shestopal <nick4eva@users.noreply.github.com> Co-authored-by: Joshua Palis <jpalis@amazon.com>

epugh · 2026-05-28T22:35:38Z

Awesome @kolchfa-aws !

github-actions Bot assigned kolchfa-aws Mar 10, 2026

epugh mentioned this pull request Mar 10, 2026

[BLOG] Introducing LLM as a Judge: Scaling search relevance evaluation with AI opensearch-project/project-website#4093

Open

kolchfa-aws added In progress Issue/PR: The issue or PR is in progress. backport 3.5 labels Mar 11, 2026

gweyeratlassian and others added 17 commits April 6, 2026 09:14

first pass of tutorial

69ab1cd

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

add reference docs.

134a8ff

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Fix up vale violations

fe54981

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Standarize on the name wikipedia uses for this technique

d0fc81c

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Clarify the effect of bulk operations on ingest pipelines (opensearch…

ca4b090

…-project#11978) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Fix create sparse vector index error (opensearch-project#12066)

31e8ae3

Signed-off-by: xiaoyuan0821 <yuan13215@163.com> Co-authored-by: x00815292 <niklaus.xiao@huawei.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Fix formatting for Boolean AND examples in dql.md (opensearch-project…

a7627c0

…#12065) Signed-off-by: Mykola Shestopal <nick4eva@users.noreply.github.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Change copy curl buttons to copy only in workspace docs (opensearch-p…

caa0dda

…roject#12071) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Add additional details to search backpressure stats (opensearch-proje…

7953ca3

…ct#11982) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

fix up links

73d0191

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

prune back extra verbiage and ideas that arent going to land

223dfef

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

epugh force-pushed the add_llm_as_a_judge_docs branch from 735a0a0 to 223dfef Compare April 6, 2026 07:14

kolchfa-aws added backport 3.6 and removed backport 3.5 labels Apr 7, 2026

epugh added 2 commits May 21, 2026 21:08

Merge remote-tracking branch 'upstream/main' into add_llm_as_a_judge_…

63f0193

…docs

Backout local only change

09576a7

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

epugh marked this pull request as ready for review May 22, 2026 01:13

epugh requested a review from kolchfa-aws as a code owner May 22, 2026 01:13

epugh added 2 commits May 21, 2026 22:17

Refining text

08940be

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Try to be clearer in Step 6

10dcf88

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

kolchfa-aws added Doc review PR: Doc review in progress and removed Tech review PR: Tech review in progress labels May 26, 2026

epugh commented May 27, 2026

View reviewed changes

epugh added 5 commits May 27, 2026 09:06

Responding to vale feedback

ab83d43

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Ensure standard naming patterns followed

934d115

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

use ALL CAPS format for variables

ea7363c

Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Rework the order of the descriptions of the types of judgments to be …

3dd01b4

…ordered with the top level descirpiton, and rewrite a lot of passive language to make ti punchier Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

Move the advanced feature text to the ref page, they arent part of th…

6ab78ed

…e tutorial Signed-off-by: Eric Pugh <epugh@opensourceconnections.com>

epugh changed the title ~~first pass of dedicated tutorial for LLM as a Judge, and documenting the apis in SRW page.~~ Tutorial for LLM as a Judge, and documenting the apis in SRW pages. May 27, 2026

kolchfa-aws added 2 commits May 28, 2026 12:58

Merge branch 'main' into add_llm_as_a_judge_docs

ceca148

Doc review

c4e03a8

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws reviewed May 28, 2026

View reviewed changes

Comment thread _tutorials/llm-as-a-judge-tutorial.md Outdated

Apply suggestion from @kolchfa-aws

e9603f2

Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>

kolchfa-aws approved these changes May 28, 2026

View reviewed changes

kolchfa-aws merged commit 8e020f9 into opensearch-project:main May 28, 2026
6 checks passed

kolchfa-aws mentioned this pull request May 28, 2026

Backport PR #12083 to 3.6 #12479

Merged


		You can adapt the connector configuration for other providers:

		#### AWS Bedrock example:

		Congratulations, you are now ready to [run an experiment to evaluate search quality]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/evaluate-search-quality/#creating-a-pointwise-experiment) using the LLM-generated judgments that you have just created. You can reuse the search configuration and query set's that you have already created in this tutorial when you run your first evaluation.


		## Advanced features

Conversation

epugh commented Mar 10, 2026

Description

Issues Resolved

Version

Frontend features

Checklist

Uh oh!

github-actions Bot commented Mar 10, 2026

Uh oh!

epugh commented Mar 10, 2026

Uh oh!

pajuric commented May 22, 2026

Uh oh!

kolchfa-aws commented May 26, 2026

Uh oh!

epugh commented May 27, 2026

Uh oh!

epugh left a comment

Choose a reason for hiding this comment

Uh oh!

epugh May 27, 2026

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws May 27, 2026

Choose a reason for hiding this comment

Uh oh!

epugh May 27, 2026

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws May 27, 2026

Choose a reason for hiding this comment

Uh oh!

epugh May 27, 2026

Choose a reason for hiding this comment

Uh oh!

epugh May 27, 2026

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws May 27, 2026

Choose a reason for hiding this comment

Uh oh!

epugh commented May 27, 2026

Uh oh!

kolchfa-aws commented May 27, 2026

Uh oh!

pajuric commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kolchfa-aws left a comment

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented May 28, 2026

Uh oh!

Uh oh!

pajuric commented May 28, 2026

Uh oh!

epugh commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pajuric commented May 27, 2026 •

edited

Loading