fix: handle string tasks in AutoAgents configuration#802
fix: handle string tasks in AutoAgents configuration#802MervinPraison wants to merge 2 commits intomainfrom
Conversation
- Add _normalize_config method to convert string tasks to TaskConfig format - Normalize configuration before validation in _generate_config - Handle cases where LLMs (like Gemini) return tasks as strings - Add test script to verify the fix Fixes validation errors when using AutoAgents with custom LLMs Co-authored-by: Mervin Praison <MervinPraison@users.noreply.github.com>
|
Warning Rate limit exceeded@claude[bot] has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 5 minutes and 22 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (2)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
@claude review if this is the correct fix. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @MervinPraison, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the robustness of the AutoAgents system by addressing issues related to inconsistent task definitions returned by Large Language Models. It introduces a new mechanism to normalize task configurations, ensuring that whether tasks are provided as simple strings or incomplete dictionaries, they are transformed into a consistent, valid structure before being processed. This change improves compatibility with various LLM providers and is accompanied by new test coverage to verify the normalization logic.
Highlights
- Configuration Normalization: Introduced a new private method,
_normalize_config, withinAutoAgentsto standardize the structure of task configurations. This method converts tasks received as simple strings into a structured dictionary format (TaskConfig) and ensures that existing task dictionaries contain all necessary fields (name, description, expected_output, tools) by providing sensible defaults if missing. - Improved LLM Output Handling: Integrated the
_normalize_configmethod into the_generate_configprocess. This ensures that any configuration received from an LLM, whether parsed directly or extracted from a cleaned response, is normalized before Pydantic validation, preventing errors caused by inconsistent LLM output formats (e.g., tasks as strings instead of objects). - New Test Coverage: Added a new dedicated test file,
test_autoagents_fix.py, to specifically validate the functionality of the_normalize_configmethod. This test script verifies that string tasks are correctly converted to the expected dictionary format and that already-structured tasks remain valid after normalization.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
You are nearing your monthly Qodo Merge usage quota. For more information, please visit here. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Claude finished @MervinPraison's task —— View job Code ReviewI've reviewed this PR for handling string tasks in AutoAgents configuration. Todo List:
Overall Assessment✅ This fix correctly addresses the issue of LLMs returning tasks as strings instead of TaskConfig objects. The solution is well-implemented and should resolve the Pydantic validation errors when using custom LLMs like Gemini. Strengths
Issues and Recommendations1.
|
|
You are nearing your monthly Qodo Merge usage quota. For more information, please visit here. PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||
There was a problem hiding this comment.
Code Review
This pull request effectively addresses an issue with inconsistent LLM outputs by introducing a normalization step for task configurations. The approach is sound and the inclusion of a test script is commendable. I've provided a couple of suggestions to refine the normalization logic and strengthen the tests for better long-term maintainability and correctness.
| normalized2 = agents._normalize_config(config_dict2.copy()) | ||
| print("\n✅ Test passed: Dict tasks remain properly formatted") |
There was a problem hiding this comment.
This test case for dictionary-based tasks is missing assertions, which means it doesn't actually verify that the _normalize_config method works correctly for this scenario. A test without assertions can provide a false sense of security.
Add an assertion to ensure that a valid configuration is not modified by the normalization process.
Additionally, this test file appears to be a standalone script. For better maintainability and integration with CI/CD pipelines, structure it as a proper test using a framework like pytest. This would involve removing print statements in favor of framework-based reporting and removing the if __name__ == "__main__" block.
| normalized2 = agents._normalize_config(config_dict2.copy()) | |
| print("\n✅ Test passed: Dict tasks remain properly formatted") | |
| normalized2 = agents._normalize_config(config_dict2.copy()) | |
| assert normalized2 == config_dict2, "Normalization should not alter an already valid configuration." |
| if 'name' not in task: | ||
| task['name'] = task.get('description', 'Task')[:50] | ||
| if 'description' not in task: | ||
| task['description'] = task.get('name', 'Task description') |
There was a problem hiding this comment.
The current logic for setting a default description when it's missing results in it being a copy of the name. This can lead to redundant information and less descriptive task configurations, as the name is often a summary. Consider using a more informative default for the description to improve clarity.
| task['description'] = task.get('name', 'Task description') | |
| task['description'] = f"Complete the task: {task['name']}" |
|
@claude note that this is also another recommendation, so review again all and give me report
|
|
Claude finished @MervinPraison's task —— View job Comprehensive PR Review
Executive Summary✅ This fix is still necessary and valuable, even with the improved prompt recommendation. It provides essential defensive programming against unpredictable LLM outputs. Key Findings1. Value of the Fix Despite Improved PromptWhile the improved prompt explicitly instructs LLMs to return tasks as objects: "tasks": [
{
"name": "Task Name",
"description": "Detailed task description",
"expected_output": "What the task should produce",
"tools": ["tool1", "tool2"]
}
]The normalization logic is still valuable because:
2. Implementation IssuesAll reviewers identified several issues that should be addressed:
|
|
@claude review and find a proper fix |
|
Claude finished @MervinPraison's task —— View job Comprehensive PR Review
Executive Summary✅ This fix is still necessary and valuable, even with the improved prompt recommendation. It provides essential defensive programming against unpredictable LLM outputs. Key Findings1. Value of the Fix Despite Improved PromptWhile the improved prompt explicitly instructs LLMs to return tasks as objects: "tasks": [
{
"name": "Task Name",
"description": "Detailed task description",
"expected_output": "What the task should produce",
"tools": ["tool1", "tool2"]
}
]The normalization logic is still valuable because:
2. Implementation IssuesAll reviewers identified several issues that should be addressed:
|
|
@claude review above and fix, making sure its not too much, minimal cod change and backward compatibility with no feature removed |
|
Claude finished @MervinPraison's task —— View job Todo List:
SummaryI've successfully implemented all the requested fixes based on the code review feedback, with minimal changes to maintain backward compatibility: Changes Made:
What was NOT changed:
These minimal fixes address all the issues identified in the reviews while maintaining full backward compatibility. |
- Add word-aware truncation for task names to avoid breaking words - Add type safety by converting values to strings before operations - Extract duplicated JSON parsing logic into _parse_json_response method - Add missing assertion in test to verify dict tasks remain unchanged These minimal fixes address all review feedback while maintaining backward compatibility Co-authored-by: Mervin Praison <MervinPraison@users.noreply.github.com>
1 participant
User description
Fixes AutoAgents validation errors when using custom LLMs like Gemini.
Problem
LLMs sometimes return tasks as simple strings instead of TaskConfig objects, causing Pydantic validation errors.
Solution
_normalize_configmethod to convert string tasks to proper TaskConfig format_generate_configThis ensures AutoAgents works reliably with all LLM providers.
Related to PR #800
Generated with Claude Code
PR Type
Bug fix
Description
Add
_normalize_configmethod to handle string tasks from LLMsConvert string tasks to proper TaskConfig dictionary format
Ensure all required fields are present in task configurations
Add test script to verify the normalization functionality
Changes diagram
Changes walkthrough 📝
autoagents.py
Add task normalization for LLM responsessrc/praisonai-agents/praisonaiagents/agents/autoagents.py
_normalize_configmethod to convert string tasks to TaskConfigformat
_generate_configmethod before validationtest_autoagents_fix.py
Add test script for task normalizationsrc/praisonai-agents/test_autoagents_fix.py
_normalize_configfunctionality