feat(evaluator): add create-llm and rework upload replacement#188
feat(evaluator): add create-llm and rework upload replacement#188Ramon Nogueira (ramon-langchain) wants to merge 1 commit into
Conversation
Ports the LLM-as-judge evaluator support and evaluator-upload replacement fix that were made directly on the langchainplus vendored copy of this CLI but never synced back here. - `evaluator create-llm`: create structured LLM-as-judge run rules from a project or dataset, with an inline prompt+schema or a Prompt Hub ref (--hub-ref), a required serialized --model-config, and --variable-mapping. - `evaluator upload`: prepare the new rule before touching an existing one and PATCH in place on --replace (instead of delete-then-create), and add --trace-filter for parity with create-llm and the API.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| var result map[string]any | ||
| if err := c.RawPost(ctx, "/runs/rules", payload, &result); err != nil { | ||
| if existing != nil { | ||
| if err := c.RawPatch(ctx, fmt.Sprintf("/runs/rules/%s", existing.ID), payload, &result); err != nil { | ||
| ExitErrorf("replacing evaluator: %v", err) | ||
| } | ||
| } else if err := c.RawPost(ctx, "/runs/rules", payload, &result); err != nil { |
There was a problem hiding this comment.
🟡 Replacing an evaluator with a different type can leave stale scoring logic attached
When replacing an existing evaluator that is a different type than the new one (RawPatch at internal/cmd/evaluator.go:313 and internal/cmd/evaluator.go:410), the update only sends the new type's block and never clears the old one, so a rule can end up carrying both the old and new scoring logic.
Impact: A user who replaces a code evaluator with an LLM one (or vice versa) under the same name/target may unknowingly keep the old evaluator running alongside the new one.
PATCH merge vs. previous full-replacement behavior
The previous implementation deleted the existing rule and created a fresh one, guaranteeing the rule contained exactly the new evaluator type. The reworked flow issues a PATCH to /runs/rules/{id} with a payload that includes only code_evaluators (upload, internal/cmd/evaluator.go:299-301) or only evaluators (create-llm, internal/cmd/evaluator_llm.go:201). If the backend PATCH treats absent fields as "leave unchanged" (typical PATCH semantics for list fields), then replacing a code evaluator with an LLM evaluator (or vice versa) under the same name+target would leave the previous type's block in place, producing a rule with both code_evaluators and evaluators. findEvaluator (internal/cmd/evaluator.go:639-652) matches purely on name + dataset/project, so it can match an existing rule of the opposite type. Same-type replacement is fine because the array is overwritten wholesale. Whether this manifests depends on backend PATCH semantics, so it warrants verification.
Prompt for agents
The upload and create-llm commands now replace an existing evaluator via PATCH /runs/rules/{id} instead of the previous delete-then-create. The PATCH payload for upload includes only code_evaluators, and for create-llm includes only evaluators. findEvaluator matches an existing rule solely by display name plus dataset/project, so it can match a rule of the OPPOSITE evaluator type. If the backend PATCH leaves fields absent from the payload unchanged, then replacing a code evaluator with an LLM evaluator (or vice versa) would leave the old evaluator block attached, producing a rule that runs both. Verify the backend PATCH semantics for /runs/rules; if it does not clear omitted list fields, ensure the replacement payload explicitly clears the opposite evaluator array (e.g. send an empty code_evaluators when creating an LLM evaluator and an empty evaluators when uploading a code evaluator), or fall back to delete-then-create when the existing rule's type differs from the new one. Relevant code: internal/cmd/evaluator.go around lines 311-316 (upload) and internal/cmd/evaluator.go around lines 409-415 plus internal/cmd/evaluator_llm.go buildLLMEvaluatorPayload.
Was this helpful? React with 👍 or 👎 to provide feedback.
3 participants
Ports the two CLI changes that were made directly on the langchainplus vendored copy of this repo (
sdks/cli) but were never synced back here. Discovered while decouplinglangchainplusfrom the inline CLI vendor — these are the only functional pieces upstream was missing.Applied cleanly on top of current
main(upstream'sevaluator.gohasn't changed since the shared base, so no independent upstream work is touched).Changes
evaluator create-llm(langchainplus #28904): create structured LLM-as-judge run rules against a project or dataset. Supports an inline prompt+schema (--prompt/--schema) or a Prompt Hub ref (--hub-ref), a required serialized--model-config,--variable-mapping,--trace-filter, and--replace/--yes. Addsinternal/cmd/evaluator_llm.go.evaluator uploadreplacement rework (langchainplus #30192): build the new rule before touching an existing one, andPATCHin place on--replaceinstead of delete-then-create; adds--trace-filterfor parity.create-llmexamples.Test plan
go build ./...go vet ./internal/cmd/go test ./internal/cmd/(evaluator suite, incl. 5 new tests)gofmt -lcleanSupersedes the copybara full-mirror branch (#187), now closed — it would have reverted independent upstream work.