Skip to content

refactor: error from ontap_get retry as debug#103

Merged
cgrinds merged 3 commits into
mainfrom
hl_llm_retry
Apr 10, 2026
Merged

refactor: error from ontap_get retry as debug#103
cgrinds merged 3 commits into
mainfrom
hl_llm_retry

Conversation

@Hardikl
Copy link
Copy Markdown
Contributor

@Hardikl Hardikl commented Apr 10, 2026

Validated with sample input prompt as show all luns in marketing svm in umeng-aff300-05-06 cluster.

Our LLM always do retry in case of error, but it's logged as error where it's already correcting itself in ontap_get case.
Refactor this case in switch and made it debug.

=== RUN   TestLuns/Show_luns
2026/04/10 14:34:02 INFO LLM would retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,size,uuid,state,volume.name,svm.name filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"size\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 14:34:05 INFO LLM would retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,uuid,state,volume.name,svm.name filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"state\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 14:34:09 INFO LLM would retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,uuid,volume.name,svm.name filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"volume.name\" is invalid for field \"fields\" (<field,...>) code: 262197"
--- PASS: TestLuns (19.13s)
--- PASS: TestLuns/Show_luns (19.13s)

Copilot AI review requested due to automatic review settings April 10, 2026 09:13
@cla-bot cla-bot Bot added the cla-signed label Apr 10, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors integration-test logging around ontap_get retries so that expected/self-correcting validator errors are logged at debug level instead of being recorded as test errors, aligning test output with the agent’s retry behavior.

Changes:

  • Refactors the tool-call error handling block into a switch for clearer branching.
  • Downgrades ontap_get “invalid for field” errors (which the LLM corrects via retry) from t.Errorf to slog.Debug.
  • Improves the default t.Errorf to include tool arguments for easier debugging when failures are real.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread integration/test/tools_test.go Outdated
Comment thread integration/test/tools_test.go Outdated
@Hardikl
Copy link
Copy Markdown
Contributor Author

Hardikl commented Apr 10, 2026

=== RUN   TestShow/show_luns
2026/04/10 19:52:04 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,svm.name,size,state,os_type filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"size\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 19:52:11 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,svm.name,state,os_type filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"state\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 19:52:20 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,svm.name,os_type filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"volume.name\" is invalid for field \"fields\" (<field,...>) code: 262197"
--- PASS: TestShow (36.84s)
--- PASS: TestShow/show_luns (36.84s)

@Hardikl
Copy link
Copy Markdown
Contributor Author

Hardikl commented Apr 10, 2026

This is case where retried max value made as 3,

=== RUN   TestShow/show_luns
2026/04/10 19:44:38 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,size,os_type,state filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"size\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 19:44:44 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,os_type,state filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"state\" is invalid for field \"fields\" (<field,...>) code: 262197"
2026/04/10 19:44:51 WARN LLM will retry tool=ontap_get args="map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,os_type filters:map[svm.name:marketing] path:/storage/luns]" error="error running the tool: ErrValidator: message: The value \"volume.name\" is invalid for field \"fields\" (<field,...>) code: 262197"
    tools_test.go:226: Tool "ontap_get" args map[cluster_name:umeng-aff300-05-06 fields:name,volume.name,os_type filters:map[svm.name:marketing] path:/storage/luns] returned error error running the tool: ErrValidator: message: The value "volume.name" is invalid for field "fields" (<field,...>) code: 262197
2026/04/10 19:44:51 ERROR Error processing input error="max iterations (3) reached without final assistant message"
--- FAIL: TestShow (23.35s)
--- FAIL: TestShow/show_luns (23.35s)

Comment thread integration/test/tools_test.go Outdated
Comment thread integration/test/tools_test.go Outdated
Copilot AI review requested due to automatic review settings April 10, 2026 15:12
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +226 to +227
t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound)
return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound)
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fmt.Errorf(... %w, errFound) can return a nil error when errFound is nil (Go treats wrapping nil as producing nil). If maxIterations is reached without any tool error, this function would (1) call t.Errorf(...) with empty tool/args and (2) potentially return a nil error to callers, allowing tests to continue incorrectly.

Consider returning a non-wrapped error when errFound == nil (use %v or a separate errors.New/fmt.Errorf without %w), and only include tool/args context when you actually have a recorded tool failure.

Suggested change
t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound)
return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound)
if errFound != nil {
t.Errorf("Tool %q args %v returned error %v", failedTool, argsUsed, errFound)
return "", fmt.Errorf("max iterations (%d) reached; last tool %q args %v error: %w", maxIterations, failedTool, argsUsed, errFound)
}
t.Errorf("max iterations (%d) reached without completing response", maxIterations)
return "", fmt.Errorf("max iterations (%d) reached without completing response", maxIterations)

Copilot uses AI. Check for mistakes.
Comment on lines +213 to +217
failedTool = toolName
argsUsed = args
errFound = err
slog.Warn("LLM will retry", slog.String("tool", toolName), slog.Any("args", args), slog.Any("error", err))

Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says retry-on-error for ontap_get should be logged as debug, but this change logs retries with slog.Warn(...). Warn will typically appear in default test output and can still create noisy logs.

If the intent is to reduce noise for self-correcting retries, switch this to slog.Debug (and optionally gate it to the specific ontap_get validation error case).

Copilot uses AI. Check for mistakes.
Comment on lines 209 to 218
if err != nil {
if expectedOntapErrorStr != "" && strings.Contains(err.Error(), expectedOntapErrorStr) {
slog.Debug("Expected tool error", slog.String("tool", toolName), slog.Any("error", err))
} else {
t.Errorf("Tool %q returned error LLM will retry: %v", toolName, err)
return "", nil // test passed, expected error was observed
}
failedTool = toolName
argsUsed = args
errFound = err
slog.Warn("LLM will retry", slog.String("tool", toolName), slog.Any("args", args), slog.Any("error", err))

result = "Error: " + err.Error()
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning early when expectedOntapErrorStr matches (return "", nil) changes the semantics of ChatWithResponse: the tool error is never appended to messages, so the LLM never gets the error feedback to produce a final assistant message (and callers get an empty response).

If the goal is only to avoid failing the test on expected tool errors, consider keeping the loop behavior (append the tool error message and let the assistant respond) while suppressing t.Errorf/high-severity logging for that case, or return a non-empty response indicating the expected error was observed.

Copilot uses AI. Check for mistakes.

var errFound error
failedTool := ""
argsUsed := make(map[string]any)
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

argsUsed := make(map[string]any) is immediately overwritten via argsUsed = args; the initial allocation is unused. You can declare it as var argsUsed map[string]any (or assign when the failure occurs) to avoid the extra allocation.

Suggested change
argsUsed := make(map[string]any)
var argsUsed map[string]any

Copilot uses AI. Check for mistakes.
@cgrinds cgrinds merged commit 37109a9 into main Apr 10, 2026
21 of 26 checks passed
@cgrinds cgrinds deleted the hl_llm_retry branch April 10, 2026 15:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants