refactor: error from ontap_get retry as debug#103
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors integration-test logging around ontap_get retries so that expected/self-correcting validator errors are logged at debug level instead of being recorded as test errors, aligning test output with the agent’s retry behavior.
Changes:
- Refactors the tool-call error handling block into a
switchfor clearer branching. - Downgrades
ontap_get“invalid for field” errors (which the LLM corrects via retry) fromt.Errorftoslog.Debug. - Improves the default
t.Errorfto include tool arguments for easier debugging when failures are real.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
This is case where retried max value made as 3, |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound) | ||
| return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound) |
There was a problem hiding this comment.
fmt.Errorf(... %w, errFound) can return a nil error when errFound is nil (Go treats wrapping nil as producing nil). If maxIterations is reached without any tool error, this function would (1) call t.Errorf(...) with empty tool/args and (2) potentially return a nil error to callers, allowing tests to continue incorrectly.
Consider returning a non-wrapped error when errFound == nil (use %v or a separate errors.New/fmt.Errorf without %w), and only include tool/args context when you actually have a recorded tool failure.
| t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound) | |
| return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound) | |
| if errFound != nil { | |
| t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound) | |
| return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound) | |
| } | |
| t.Errorf("max iterations (%d) reached without completing response", maxIterations) | |
| return "", fmt.Errorf("max iterations (%d) reached without completing response", maxIterations) |
| failedTool = toolName | ||
| argsUsed = args | ||
| errFound = err | ||
| slog.Warn("LLM will retry", slog.String("tool", toolName), slog.Any("args", args), slog.Any("error", err)) | ||
|
|
There was a problem hiding this comment.
The PR description says retry-on-error for ontap_get should be logged as debug, but this change logs retries with slog.Warn(...). Warn will typically appear in default test output and can still create noisy logs.
If the intent is to reduce noise for self-correcting retries, switch this to slog.Debug (and optionally gate it to the specific ontap_get validation error case).
| if err != nil { | ||
| if expectedOntapErrorStr != "" && strings.Contains(err.Error(), expectedOntapErrorStr) { | ||
| slog.Debug("Expected tool error", slog.String("tool", toolName), slog.Any("error", err)) | ||
| } else { | ||
| t.Errorf("Tool %q returned error LLM will retry: %v", toolName, err) | ||
| return "", nil // test passed, expected error was observed | ||
| } | ||
| failedTool = toolName | ||
| argsUsed = args | ||
| errFound = err | ||
| slog.Warn("LLM will retry", slog.String("tool", toolName), slog.Any("args", args), slog.Any("error", err)) | ||
|
|
||
| result = "Error: " + err.Error() |
There was a problem hiding this comment.
Returning early when expectedOntapErrorStr matches (return "", nil) changes the semantics of ChatWithResponse: the tool error is never appended to messages, so the LLM never gets the error feedback to produce a final assistant message (and callers get an empty response).
If the goal is only to avoid failing the test on expected tool errors, consider keeping the loop behavior (append the tool error message and let the assistant respond) while suppressing t.Errorf/high-severity logging for that case, or return a non-empty response indicating the expected error was observed.
|
|
||
| var errFound error | ||
| failedTool := "" | ||
| argsUsed := make(map[string]any) |
There was a problem hiding this comment.
argsUsed := make(map[string]any) is immediately overwritten via argsUsed = args; the initial allocation is unused. You can declare it as var argsUsed map[string]any (or assign when the failure occurs) to avoid the extra allocation.
| argsUsed := make(map[string]any) | |
| var argsUsed map[string]any |
Validated with sample input prompt as
show all luns in marketing svm in umeng-aff300-05-06 cluster.Our LLM always do retry in case of error, but it's logged as error where it's already correcting itself in
ontap_getcase.Refactor this case in switch and made it debug.