fix: handle JSON parsing for Gemini self-reflection#800
Conversation
- Disable streaming for custom LLMs during self-reflection to avoid incomplete JSON chunks - Add manual JSON parsing for custom LLM reflection responses - Use clean_json_output method to handle markdown-wrapped JSON - Ensure reflection regeneration also uses non-streaming for custom LLMs This fixes the "Expecting property name enclosed in double quotes" error when using Gemini models with self-reflection enabled. Fixes #796 Co-authored-by: Mervin Praison <MervinPraison@users.noreply.github.com>
|
Caution Review failedThe pull request is closed. WalkthroughThe Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Agent
participant LLM (Custom/OpenAI)
User->>Agent: chat(prompt, reflection enabled)
Agent->>LLM (Custom): send prompt (stream=False for reflection)
LLM (Custom)-->>Agent: response (possibly malformed JSON)
Agent->>Agent: clean and parse JSON manually
Agent->>User: return structured reflection output
Assessment against linked issues
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
@claude review this solution again and is this the better fix with bacward compatibility and then no feature removed. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @MervinPraison, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request addresses a critical issue where custom LLMs, particularly Gemini, encountered JSON parsing errors during self-reflection due to incomplete streaming responses. My changes implement a robust solution by disabling streaming for these models during reflection, performing manual JSON parsing, and ensuring proper handling of markdown-wrapped JSON outputs. This significantly improves the reliability of self-reflection for non-OpenAI models.
Highlights
- Custom LLM Reflection Handling: I've modified the self-reflection logic to specifically handle custom LLMs (like Gemini) by disabling streaming (
stream=False) for reflection requests. This ensures complete JSON responses are received, preventing parsing errors that occur with partial streams. - Manual JSON Parsing and Cleaning: For custom LLM reflection responses, I've implemented manual JSON parsing. This includes calling
self.clean_json_outputto correctly extract JSON from potentially markdown-wrapped text and then usingjson.loads()to parse it. ACustomReflectionOutputclass was introduced to map the parsed data to the expected reflection object structure. - Reflection Regeneration Consistency: I've ensured that the reflection regeneration process also forces non-streaming for custom LLMs. This maintains consistency across all reflection attempts and prevents similar JSON parsing issues in subsequent iterations.
- New Test Case: A new dedicated test script (
test_gemini_fix.py) has been added to specifically verify the fix for Gemini models when using self-reflection, confirming the resolution of the JSON parsing problem.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
|
Claude finished @MervinPraison's task —— View job Code Review for Gemini Self-Reflection FixTodo List:
SummaryYes, this is a better fix that maintains backward compatibility and doesn't remove any features. The solution properly addresses the JSON parsing issues while preserving existing functionality for both OpenAI and custom LLM providers. Detailed Review✅ Strengths of the Solution:
✅ Key Benefits:
🎯 Conclusion: |
PR Code Suggestions ✨Explore these optional code suggestions:
|
||||||||||||
There was a problem hiding this comment.
Code Review
This pull request effectively addresses a JSON parsing error for Gemini models with self-reflection by disabling streaming and implementing manual parsing. The core logic changes in agent.py are sound, and the addition of a test script in test_gemini_fix.py is a great step towards ensuring the fix is robust. My review includes a few suggestions to enhance maintainability and adhere to Python best practices, such as using more specific exception types, reusing existing data structures for consistency, and properly structuring the new test script.
| reflection_response = self._chat_completion(messages, temperature=temperature, tools=None, stream=False, reasoning_steps=False) | ||
|
|
||
| if not reflection_response or not reflection_response.choices: | ||
| raise Exception("No response from reflection request") |
There was a problem hiding this comment.
Using the generic Exception class is generally discouraged as it can catch unexpected errors and make debugging more difficult. It's better to use a more specific exception type to allow for more granular error handling. In this case, ValueError would be more appropriate, as the error condition relates to an invalid or missing response from the reflection request.
| raise Exception("No response from reflection request") | |
| raise ValueError("No response from reflection request") |
| # Create a reflection output object manually | ||
| class CustomReflectionOutput: | ||
| def __init__(self, data): | ||
| self.reflection = data.get('reflection', '') | ||
| self.satisfactory = data.get('satisfactory', 'no').lower() | ||
|
|
||
| reflection_output = CustomReflectionOutput(reflection_data) |
There was a problem hiding this comment.
Defining a class inside a method is generally not recommended as it can be inefficient (the class is redefined on every call) and can harm readability.
For better consistency and maintainability, I suggest using the existing ReflectionOutput Pydantic model, which is already used for the OpenAI-specific path. This ensures a consistent data structure for reflection output across all LLM providers and leverages Pydantic's validation capabilities.
| # Create a reflection output object manually | |
| class CustomReflectionOutput: | |
| def __init__(self, data): | |
| self.reflection = data.get('reflection', '') | |
| self.satisfactory = data.get('satisfactory', 'no').lower() | |
| reflection_output = CustomReflectionOutput(reflection_data) | |
| # Create a reflection output object using the existing ReflectionOutput model | |
| # for consistency and validation. | |
| reflection_data['satisfactory'] = reflection_data.get('satisfactory', 'no').lower() | |
| if 'reflection' not in reflection_data: | |
| reflection_data['reflection'] = '' | |
| reflection_output = ReflectionOutput.model_validate(reflection_data) |
| # Test with a simple prompt | ||
| print("Testing Gemini with self-reflection...") | ||
| try: | ||
| response = agent.start("What is 2+2? Explain briefly.") | ||
| print(f"\nFinal response: {response}") | ||
| print("\nTest completed successfully!") | ||
| except Exception as e: | ||
| print(f"\nError occurred: {e}") | ||
| import traceback | ||
| traceback.print_exc() No newline at end of file |
There was a problem hiding this comment.
It's a Python best practice to wrap script execution logic in a if __name__ == '__main__': block. This prevents the code from running automatically if the script is imported into another module, making it more reusable and aligning with standard Python script structure.
| # Test with a simple prompt | |
| print("Testing Gemini with self-reflection...") | |
| try: | |
| response = agent.start("What is 2+2? Explain briefly.") | |
| print(f"\nFinal response: {response}") | |
| print("\nTest completed successfully!") | |
| except Exception as e: | |
| print(f"\nError occurred: {e}") | |
| import traceback | |
| traceback.print_exc() | |
| if __name__ == "__main__": | |
| # Test with a simple prompt | |
| print("Testing Gemini with self-reflection...") | |
| try: | |
| response = agent.start("What is 2+2? Explain briefly.") | |
| print(f"\nFinal response: {response}") | |
| print("\nTest completed successfully!") | |
| except Exception as e: | |
| print(f"\nError occurred: {e}") | |
| import traceback | |
| traceback.print_exc() |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🔭 Outside diff range comments (1)
src/praisonai-agents/praisonaiagents/agent/agent.py (1)
1386-1396: Avoid duplicating the JSON‐cleanup logicThe
clean_json_outputmethod is already implemented inpraisonaiagents.agents.agents. Please remove the local duplicate inagent.pyand import the existing one:• File: src/praisonai-agents/praisonaiagents/agent/agent.py
– Delete theclean_json_outputdefinition at lines 1386–1396
– At the top of the file, add:
python from praisonaiagents.agents.agents import clean_json_output
– Replace any calls toself.clean_json_output(...)withclean_json_output(...)Example diff:
--- a/src/praisonai-agents/praisonaiagents/agent/agent.py +++ b/src/praisonai-agents/praisonaiagents/agent/agent.py @@ -1,6 +1,7 @@ from praisonaiagents.knowledge import Knowledge +from praisonaiagents.agents.agents import clean_json_output class YourAgent: @@ -1383,14 +1383,6 @@ class YourAgent: def some_method(self, reflection_text: str): - def clean_json_output(self, output: str) -> str: - """Clean and extract JSON from response text.""" - cleaned = output.strip() - if cleaned.startswith("```json"): - cleaned = cleaned[len("```json"):].strip() - if cleaned.startswith("```"): - cleaned = cleaned[len("```"):].strip() - if cleaned.endswith("```"): - cleaned = cleaned[:-3].strip() - return cleaned # before: cleaned_json = self.clean_json_output(reflection_text) - cleaned_json = self.clean_json_output(reflection_text) + cleaned_json = clean_json_output(reflection_text)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/praisonai-agents/praisonaiagents/agent/agent.py(2 hunks)src/praisonai-agents/test_gemini_fix.py(1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-agents/CLAUDE.md:0-0
Timestamp: 2025-06-30T10:06:17.673Z
Learning: Use the `Agent` class from `praisonaiagents/agent/` for core agent implementations, supporting LLM integration, tool calling, and self-reflection.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.cursorrules:0-0
Timestamp: 2025-06-30T10:05:51.843Z
Learning: Applies to src/praisonai-ts/src/agent/agent.ts : The 'Agent' class in 'src/agent/agent.ts' should encapsulate a single agent's role, name, and methods for calling the LLM using 'aisdk'.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.cursorrules:0-0
Timestamp: 2025-06-30T10:05:51.843Z
Learning: Applies to src/praisonai-ts/src/llm/llm.ts : The 'LLM' class in 'llm.ts' should wrap 'aisdk.generateText' calls for generating text responses.
src/praisonai-agents/test_gemini_fix.py (5)
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-agents/CLAUDE.md:0-0
Timestamp: 2025-06-30T10:06:17.673Z
Learning: Applies to src/praisonai-agents/tests/**/*.py : Test files should be placed in the `tests/` directory and demonstrate specific usage patterns, serving as both test and documentation.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.windsurfrules:0-0
Timestamp: 2025-06-30T10:06:44.129Z
Learning: Applies to src/praisonai-ts/src/tools/test.ts : The 'src/tools/test.ts' file should serve as a script for running internal tests or examples for each tool.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-agents/CLAUDE.md:0-0
Timestamp: 2025-06-30T10:06:17.673Z
Learning: Applies to src/praisonai-agents/praisonaiagents/{memory,knowledge}/**/*.py : Place memory-related implementations in `praisonaiagents/memory/` and knowledge/document processing in `praisonaiagents/knowledge/`.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.cursorrules:0-0
Timestamp: 2025-06-30T10:05:51.843Z
Learning: Applies to src/praisonai-ts/src/tools/test.ts : The 'src/tools/test.ts' file should provide a script for running each tool's internal test or example.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-agents/CLAUDE.md:0-0
Timestamp: 2025-06-30T10:06:17.673Z
Learning: Run individual test files as scripts (e.g., `python tests/basic-agents.py`) rather than using a formal test runner.
src/praisonai-agents/praisonaiagents/agent/agent.py (3)
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-agents/CLAUDE.md:0-0
Timestamp: 2025-06-30T10:06:17.673Z
Learning: Use the `Agent` class from `praisonaiagents/agent/` for core agent implementations, supporting LLM integration, tool calling, and self-reflection.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.cursorrules:0-0
Timestamp: 2025-06-30T10:05:51.843Z
Learning: Applies to src/praisonai-ts/src/llm/llm.ts : The 'LLM' class in 'llm.ts' should wrap 'aisdk.generateText' calls for generating text responses.
Learnt from: CR
PR: MervinPraison/PraisonAI#0
File: src/praisonai-ts/.cursorrules:0-0
Timestamp: 2025-06-30T10:05:51.843Z
Learning: Applies to src/praisonai-ts/src/agent/agent.ts : The 'Agent' class in 'src/agent/agent.ts' should encapsulate a single agent's role, name, and methods for calling the LLM using 'aisdk'.
🧬 Code Graph Analysis (1)
src/praisonai-agents/praisonaiagents/agent/agent.py (4)
src/praisonai-agents/praisonaiagents/agents/agents.py (1)
clean_json_output(253-261)src/praisonai-agents/praisonaiagents/main.py (2)
json(390-393)ReflectionOutput(375-377)src/praisonai-agents/praisonaiagents/llm/openai_client.py (1)
sync_client(244-246)src/praisonai-agents/praisonaiagents/llm/llm.py (1)
response(1945-2029)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: Cursor BugBot
- GitHub Check: test-core (3.11)
- GitHub Check: quick-test
🔇 Additional comments (1)
src/praisonai-agents/praisonaiagents/agent/agent.py (1)
1354-1355: Good fix for ensuring complete reflection responses.Disabling streaming for custom LLMs during reflection regeneration is the correct approach to prevent incomplete JSON chunks. The conditional logic properly preserves the original streaming behavior for non-custom LLMs.
| # Check if we're using a custom LLM (like Gemini) | ||
| if self._using_custom_llm or self._openai_client is None: | ||
| # For custom LLMs, we need to handle reflection differently | ||
| # Use non-streaming to get complete JSON response | ||
| reflection_response = self._chat_completion(messages, temperature=temperature, tools=None, stream=False, reasoning_steps=False) | ||
|
|
||
| if not reflection_response or not reflection_response.choices: | ||
| raise Exception("No response from reflection request") | ||
|
|
||
| reflection_text = reflection_response.choices[0].message.content.strip() | ||
|
|
||
| # Clean the JSON output | ||
| cleaned_json = self.clean_json_output(reflection_text) | ||
|
|
||
| # Parse the JSON manually | ||
| reflection_data = json.loads(cleaned_json) | ||
|
|
||
| # Create a reflection output object manually | ||
| class CustomReflectionOutput: | ||
| def __init__(self, data): | ||
| self.reflection = data.get('reflection', '') | ||
| self.satisfactory = data.get('satisfactory', 'no').lower() | ||
|
|
||
| reflection_output = CustomReflectionOutput(reflection_data) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Refactor inline class definition and improve error handling.
The inline CustomReflectionOutput class should be defined at the module level for better maintainability and reusability. Additionally, the JSON parsing should validate required fields and provide more specific error messages.
Define the class at the module level near other imports:
class CustomReflectionOutput:
"""Custom reflection output for non-OpenAI LLMs."""
def __init__(self, data: dict):
if 'reflection' not in data:
raise ValueError("Missing required field 'reflection' in reflection response")
if 'satisfactory' not in data:
raise ValueError("Missing required field 'satisfactory' in reflection response")
self.reflection = data['reflection']
self.satisfactory = data.get('satisfactory', 'no').lower()Then update the reflection parsing:
-# Create a reflection output object manually
-class CustomReflectionOutput:
- def __init__(self, data):
- self.reflection = data.get('reflection', '')
- self.satisfactory = data.get('satisfactory', 'no').lower()
-
-reflection_output = CustomReflectionOutput(reflection_data)
+try:
+ reflection_output = CustomReflectionOutput(reflection_data)
+except ValueError as e:
+ raise Exception(f"Invalid reflection response format: {e}")🤖 Prompt for AI Agents
In src/praisonai-agents/praisonaiagents/agent/agent.py around lines 1277 to
1300, move the inline CustomReflectionOutput class definition to the module
level near other imports for better maintainability. Update the class
constructor to validate that the required fields 'reflection' and 'satisfactory'
exist in the input data and raise ValueError with clear messages if missing.
Then, in the reflection parsing code, instantiate this module-level class and
handle JSON parsing errors with specific error messages to improve error
handling.
| #!/usr/bin/env python3 | ||
| """Test script to verify the Gemini JSON parsing fix.""" | ||
|
|
||
| from praisonaiagents import Agent | ||
|
|
||
| # Test with minimal configuration to isolate the issue | ||
| llm_config = { | ||
| "model": "gemini/gemini-1.5-flash-latest", | ||
| "temperature": 0.7, | ||
| "max_tokens": 500, | ||
| } | ||
|
|
||
| # Create agent with self-reflection enabled | ||
| agent = Agent( | ||
| instructions="You are a helpful assistant. Be concise and clear.", | ||
| llm=llm_config, | ||
| verbose=True, | ||
| self_reflect=True, | ||
| max_reflect=2, | ||
| min_reflect=1 | ||
| ) | ||
|
|
||
| # Test with a simple prompt | ||
| print("Testing Gemini with self-reflection...") | ||
| try: | ||
| response = agent.start("What is 2+2? Explain briefly.") | ||
| print(f"\nFinal response: {response}") | ||
| print("\nTest completed successfully!") | ||
| except Exception as e: | ||
| print(f"\nError occurred: {e}") | ||
| import traceback | ||
| traceback.print_exc() No newline at end of file |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Move test to proper directory and add comprehensive test coverage.
According to the project's testing conventions from the learnings, test files should be placed in the tests/ directory. Additionally, the test lacks assertions to verify that self-reflection actually works correctly with the fix.
Consider moving this file to tests/test_gemini_reflection_fix.py and enhance it with proper assertions:
#!/usr/bin/env python3
"""Test script to verify the Gemini JSON parsing fix for self-reflection."""
import json
import logging
from unittest.mock import patch, MagicMock
from praisonaiagents import Agent
# Enable debug logging to capture reflection attempts
logging.basicConfig(level=logging.DEBUG)
def test_gemini_self_reflection_with_valid_json():
"""Test that Gemini handles self-reflection with properly formatted JSON."""
llm_config = {
"model": "gemini/gemini-1.5-flash-latest",
"temperature": 0.7,
"max_tokens": 500,
}
agent = Agent(
instructions="You are a helpful assistant. Be concise and clear.",
llm=llm_config,
verbose=True,
self_reflect=True,
max_reflect=2,
min_reflect=1
)
# Test with a simple prompt
response = agent.start("What is 2+2? Explain briefly.")
# Assertions
assert response is not None, "Agent should return a response"
assert len(response) > 0, "Response should not be empty"
assert "4" in response or "four" in response.lower(), "Response should contain the correct answer"
print(f"✓ Valid JSON test passed: {response}")
def test_gemini_self_reflection_with_malformed_json():
"""Test that Gemini handles malformed JSON in reflection responses gracefully."""
# This would require mocking the LLM response to return malformed JSON
# Implementation depends on the internal structure
pass
def test_gemini_without_self_reflection():
"""Test that Gemini works without self-reflection as a baseline."""
llm_config = {
"model": "gemini/gemini-1.5-flash-latest",
"temperature": 0.7,
"max_tokens": 500,
}
agent = Agent(
instructions="You are a helpful assistant. Be concise and clear.",
llm=llm_config,
verbose=True,
self_reflect=False # Disabled
)
response = agent.start("What is 2+2?")
assert response is not None, "Agent should return a response without self-reflection"
print(f"✓ No reflection test passed: {response}")
if __name__ == "__main__":
print("Testing Gemini with self-reflection JSON parsing fix...\n")
try:
test_gemini_self_reflection_with_valid_json()
test_gemini_without_self_reflection()
print("\n✅ All tests completed successfully!")
except AssertionError as e:
print(f"\n❌ Test failed: {e}")
import traceback
traceback.print_exc()
except Exception as e:
print(f"\n❌ Unexpected error: {e}")
import traceback
traceback.print_exc()🏁 Script executed:
#!/bin/bash
# Description: Check if tests directory exists and verify testing conventions
# Check for tests directory
echo "Checking for tests directory:"
fd -t d "tests" src/praisonai-agents/ --max-depth 2
# Check existing test files for patterns
echo -e "\nChecking existing test files:"
fd -e py "test" src/praisonai-agents/tests/ 2>/dev/null || echo "No test files found in tests/"
# Check if there are other test files in the root
echo -e "\nChecking for other test files in src/praisonai-agents/:"
fd -e py "^test_" src/praisonai-agents/ --max-depth 1Length of output: 1917
Move and strengthen Gemini JSON parsing fix tests
The test_gemini_fix.py file is currently mis-located and lacks coverage for key behaviors. To align with project conventions and ensure robustness:
• Relocate
- Move
src/praisonai-agents/test_gemini_fix.py
→
src/praisonai-agents/tests/test_gemini_reflection_fix.py
• Enhance assertions
- Verify that
agent.start(...)returns a non-empty response containing “4” (or “four”). - Assert that self-reflection steps were actually invoked (e.g., by inspecting logged reflection prompts or mocking the LLM).
• Add edge-case tests
- Mock the LLM to return malformed JSON during reflection and assert the agent handles it gracefully (no uncaught exceptions, fallback behavior).
- Include a baseline test with
self_reflect=Falseto confirm normal operation.
• Example structure (in tests/test_gemini_reflection_fix.py):
def test_reflection_valid_json():
# …assert response is correct and self-reflection occurred…
def test_reflection_malformed_json(monkeypatch):
# Monkeypatch LLM to return invalid JSON…
# Assert graceful handling
def test_without_self_reflection():
# self_reflect=False → assert normal responseBy moving the file into tests/ and expanding coverage as above, we’ll both satisfy the project’s testing conventions and validate the Gemini JSON parsing fix under varied scenarios.
🤖 Prompt for AI Agents
In src/praisonai-agents/test_gemini_fix.py lines 1 to 32, move the test file to
src/praisonai-agents/tests/test_gemini_reflection_fix.py to follow project
conventions. Enhance the tests by adding assertions that verify the
agent.start(...) returns a non-empty response containing "4" or "four" and
confirm that self-reflection steps are invoked, for example by inspecting logs
or mocking the LLM. Add edge-case tests that mock the LLM to return malformed
JSON during reflection and assert the agent handles it gracefully without
uncaught exceptions. Also include a baseline test with self_reflect=False to
confirm normal operation. Structure the new tests as separate functions for
valid reflection, malformed JSON handling, and no self-reflection scenarios.
|
@claude again i get this error. Implement fix and create a new pr branch code |
|
Claude finished @MervinPraison's task —— View job • Todo List
Solution ImplementedI've successfully fixed the AutoAgents validation error. The issue was that some LLMs (like Gemini) were returning tasks as simple strings instead of proper TaskConfig objects with required fields. Changes made:
The fix maintains backward compatibility while handling edge cases where LLMs return tasks in unexpected formats. Create PR |
There was a problem hiding this comment.
Bug: LLM Reflection Fails on Non-String Values
The reflection logic for custom LLMs calls .lower() on the 'satisfactory' field of the parsed JSON response. This causes an AttributeError if the LLM returns a non-string value (e.g., null, boolean, or number) for this field, as these types lack a .lower() method.
src/praisonai-agents/praisonaiagents/agent/agent.py#L1297-L1298
PraisonAI/src/praisonai-agents/praisonaiagents/agent/agent.py
Lines 1297 to 1298 in fc20e4c
Was this report helpful? Give feedback by reacting with 👍 or 👎
1 participant
User description
Fixes #796
Problem
When using Gemini models with self-reflection enabled, streaming responses were causing incomplete JSON chunks to be parsed, resulting in the error "Expecting property name enclosed in double quotes".
Solution
This fix ensures that Gemini and other custom LLM providers can use self-reflection without encountering JSON parsing errors.
Generated with Claude Code
PR Type
Bug fix
Description
Fix JSON parsing errors for Gemini self-reflection
Disable streaming for custom LLMs during reflection
Add manual JSON parsing for custom LLM responses
Ensure reflection regeneration uses non-streaming for custom LLMs
Changes diagram
Changes walkthrough 📝
agent.py
Implement custom LLM self-reflection with JSON parsingsrc/praisonai-agents/praisonaiagents/agent/agent.py
clean_json_outputmethod to handle markdown-wrapped JSONtest_gemini_fix.py
Add Gemini self-reflection test scriptsrc/praisonai-agents/test_gemini_fix.py
Summary by CodeRabbit
New Features
Bug Fixes
Chores