Merged
Conversation
…ove installation commands - Upgraded actions/checkout and actions/setup-python to version 4. - Changed UV installation method to use a script for better integration. - Modified dependency installation commands to include the --system flag for UV. - Enhanced unittest command to run with verbose output for clearer test results.
- Renamed workflow from 'Run specific unittest' to 'Quick Validation Tests'. - Introduced a new job 'quick-test' for streamlined testing. - Added installation of pytest and pytest-asyncio for enhanced testing capabilities. - Updated test command to run legacy example tests using pytest with improved output options. - Included environment variable for PYTHONPATH to facilitate module resolution.
- Introduced a new simple test runner (`simple_test_runner.py`) that operates without pytest dependency at import time. - Added a basic diagnostic test script (`test_basic.py`) to validate Python and module imports. - Updated `README.md` to include new test runner instructions and troubleshooting tips for pytest import issues. - Enhanced `test_runner.py` to conditionally import pytest and fallback to subprocess if not available, maintaining existing functionality. - Ensured backward compatibility with existing tests and workflows.
- Added `pytest-asyncio` as a dependency in `pyproject.toml` for improved async testing capabilities. - Updated GitHub Actions workflow to include `pytest-cov` for coverage reporting. - Renamed test functions in `advanced_example.py` and `basic_example.py` for better clarity and user-friendliness. - Removed unused event loop fixture from `conftest.py` to streamline test setup. - Enhanced assertions in various tests to ensure better error reporting and maintainability.
- Upgraded actions/checkout to version 4 and actions/setup-python to version 5 across multiple workflow files for improved functionality. - Updated actions/upload-artifact to version 4 in relevant workflows to enhance artifact management.
- Modified multiple workflow files to upgrade actions/setup-python from version 4 to version 5 for improved functionality and consistency. - Ensured minimal changes to existing code while enhancing the setup process for Python environments.
- Incremented PraisonAI version from 2.2.2 to 2.2.3 in `pyproject.toml`, `uv.lock`, and Dockerfiles for consistency. - Updated `.gitignore` to include exceptions for CrewAI test directories. - Added new test markers in `pytest.ini` for better categorisation of tests. - Enhanced GitHub Actions workflows to include new test patterns for AutoGen and CrewAI frameworks. - Refactored test runner to improve clarity and user experience, including warnings for real tests and full execution tests. - Ensured minimal changes to existing code while improving overall test structure and documentation.
Contributor
|
Caution Review failedThe pull request is closed. WalkthroughThis update introduces a comprehensive overhaul of the testing infrastructure for PraisonAI Agents. It adds extensive unit, integration, and end-to-end test suites, new test runner scripts, detailed documentation, and multiple GitHub Actions workflows for automated testing. Additionally, the Changes
Sequence Diagram(s)sequenceDiagram
participant Developer
participant GitHub Actions
participant Test Runner
participant Pytest
participant Mocked Services
participant Real APIs
Developer->>GitHub Actions: Push code / PR / Manual trigger
GitHub Actions->>Test Runner: Start test job (core, integration, e2e, etc.)
Test Runner->>Pytest: Run tests (unit, integration, etc.)
Pytest->>Mocked Services: Use fixtures/mocks for integration/unit tests
alt Real E2E Tests
Pytest->>Real APIs: Make real API calls (if keys and confirmation present)
end
Pytest-->>Test Runner: Return results and coverage
Test Runner-->>GitHub Actions: Upload reports/artifacts
GitHub Actions-->>Developer: Show test results and reports
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (50)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
✅ Deploy Preview for praisonai ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hello @MervinPraison, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
Hello team, gemini-code-assist here with a summary of this pull request titled "Develop". Based on the changes, the primary focus of this PR is a significant enhancement and restructuring of the project's testing suite, alongside a minor version bump for the praisonai package.
The testing infrastructure has been substantially improved by introducing dedicated directories for unit, integration (mock), and end-to-end (real) tests. New test runner scripts and pytest configuration have been added to provide more flexible and robust ways to execute tests, including specific patterns for different test types and warnings for tests that incur costs. Comprehensive documentation has also been added to explain the new testing structure and how to use it.
Additionally, the praisonai package version has been updated from 2.2.2 to 2.2.3 across the codebase, including Dockerfiles, documentation examples, and pyproject.toml. A small code change was made to adjust the verbose parameter in the agent generation logic.
Highlights
- Testing Suite Overhaul: The testing suite has been restructured into
unit,integration(mock), ande2e(real) directories with new tests added to cover different aspects of the framework integration (AutoGen, CrewAI) and core functionality. - Improved Test Runners: New and updated test runner scripts (
test_runner.py,simple_test_runner.py,test_basic.py) provide more options for running specific test subsets (e.g.,--fast,--unit,--pattern frameworks), better handling of dependencies via subprocess, and clear warnings for tests that make real API calls. - Enhanced Test Configuration & Docs: Pytest configuration (
pytest.ini) and fixtures (conftest.py) have been added/updated to support async tests, manage environment variables for real tests, and provide mock objects. Comprehensive documentation (tests/README.md,tests/TESTING_GUIDE.md,tests/integration/WORKFLOW_INTEGRATION.md) explains the new testing structure and usage. - Version Bump: The
praisonaipackage version has been updated from2.2.2to2.2.3inpyproject.toml, Dockerfiles, and documentation examples. - Minor Code Adjustment: The
verboseparameter in the agent generation logic (praisonai/agents_generator.py) was changed from2toTruefor both CrewAI and PraisonAI agent runs.
Changelog
Click here to see the changelog
- .gitignore
- Added exceptions to unignore
tests/integration/crewaiandtests/e2e/crewaidirectories, likely to ensure new test files are included.
- Added exceptions to unignore
- docker/Dockerfile
- Updated
praisonaipackage version from2.2.2to2.2.3in thepip installcommand.
- Updated
- docker/Dockerfile.chat
- Updated
praisonaipackage version from2.2.2to2.2.3in thepip installcommand.
- Updated
- docker/Dockerfile.dev
- Updated
praisonaipackage version from2.2.2to2.2.3in thepip installcommand.
- Updated
- docker/Dockerfile.ui
- Updated
praisonaipackage version from2.2.2to2.2.3in thepip installcommand.
- Updated
- docs/api/praisonai/deploy.html
- Updated the
praisonaipackage version from2.2.2to2.2.3in the embedded Dockerfile example.
- Updated the
- docs/developers/local-development.mdx
- Updated the
praisonaipackage version from2.2.2to2.2.3in the local development Dockerfile example.
- Updated the
- docs/ui/chat.mdx
- Updated the
praisonaipackage version from2.2.2to2.2.3in the chat UI local development Dockerfile example.
- Updated the
- docs/ui/code.mdx
- Updated the
praisonaipackage version from2.2.2to2.2.3in the code UI local development Dockerfile example.
- Updated the
- praisonai/agents_generator.py
- Changed the
verboseparameter from2toTruewhen initializingCrew(line 519). - Changed the
verboseparameter from2toTruewhen initializingPraisonAIAgents(line 633).
- Changed the
- praisonai/deploy.py
- Updated the
praisonaipackage version from2.2.2to2.2.3in the generated Dockerfile content (line 59).
- Updated the
- pyproject.toml
- Updated the project version from
2.2.2to2.2.3(line 3). - Updated the poetry tool version from
2.2.2to2.2.3(line 92). - Added
pytest-asyncio = ">=0.26.0"to the[tool.poetry.group.test.dependencies]section (line 160). - Added
pytest-asyncio = ">=0.26.0"to the[tool.poetry.group.dev.dependencies]section (line 168).
- Updated the project version from
- pytest.ini
- Added a new pytest configuration file.
- Configured
asyncio_mode = autoandasyncio_default_fixture_loop_scope = function. - Defined test paths, file/class/function patterns.
- Added default
addoptsfor verbose and short tracebacks. - Defined custom markers:
slow,integration,unit,asyncio,real.
- tests/README.md
- Added a comprehensive README detailing the test suite structure, categories (Unit, Integration), running tests, troubleshooting, features tested, configuration, adding new tests, coverage goals, interpreting results, dependencies, best practices, CI integration, and recent improvements to test runners.
- tests/TESTING_GUIDE.md
- Added a new guide explaining the difference between mock and real tests.
- Provided detailed instructions on running mock, real, and full execution tests using the test runner and pytest directly.
- Included sections on API key setup, safety features, test categories, and when to use each test type.
- tests/advanced_example.py
- Refactored the
advancedexample to separate thePraisonAIinstantiation from therun()call. - Introduced
advanced_agent_examplefunction to allow testing setup without full execution for fast tests (lines 4-21).
- Refactored the
- tests/basic_example.py
- Refactored the
mainexample to separate thePraisonAIinstantiation from therun()call. - Introduced
basic_agent_examplefunction to allow testing setup without full execution for fast tests (lines 4-17).
- Refactored the
- tests/conftest.py
- Added pytest fixtures for mocking LLM responses, sample agent/task configs, vector stores, and DuckDuckGo search.
- Added an
autousefixturesetup_test_environmentto manage test environment variables, specifically preserving real API keys for tests marked with@pytest.mark.real.
- tests/e2e/init.py
- Added package initialization file with a warning about real tests.
- tests/e2e/autogen/init.py
- Added package initialization file for AutoGen real tests.
- tests/e2e/autogen/test_autogen_real.py
- Added real end-to-end tests for AutoGen integration.
- Includes tests for simple conversation setup, environment checks, and a full execution test (marked as expensive).
- tests/e2e/crewai/init.py
- Added package initialization file for CrewAI real tests.
- tests/e2e/crewai/test_crewai_real.py
- Added real end-to-end tests for CrewAI integration.
- Includes tests for simple crew setup, environment checks, multi-agent setup, and a full execution test (marked as expensive).
- tests/integration/README.md
- Added a README detailing the integration test structure, framework tests (AutoGen, CrewAI), running tests, categories, dependencies, mock strategy, expected outcomes, adding new tests, best practices, and troubleshooting.
- tests/integration/WORKFLOW_INTEGRATION.md
- Added documentation explaining how AutoGen and CrewAI integration tests are integrated into GitHub workflows (Core Tests, Comprehensive Test Suite, new Framework Integration Tests workflow).
- tests/integration/autogen/init.py
- Added package initialization file for AutoGen integration tests.
- tests/integration/autogen/test_autogen_basic.py
- Added mock integration tests for basic AutoGen functionality.
- Includes tests for AutoGen import, basic agent creation, conversation flow, and configuration validation.
- tests/integration/crewai/init.py
- Added package initialization file for CrewAI integration tests.
- tests/integration/crewai/test_crewai_basic.py
- Added mock integration tests for basic CrewAI functionality.
- Includes tests for CrewAI import, basic agent creation, crew workflow, configuration validation, and agent collaboration.
- tests/integration/test_base_url_api_base_fix.py
- Updated tests to use
mock_completionfixture instead of patchinglitellmdirectly within each test method (e.g., line 30). - Adjusted assertions to check LLM instance attributes (
llm.base_url) and mock call assertions (mock_completion.assert_called()) (e.g., lines 51-59).
- Updated tests to use
- tests/integration/test_mcp_integration.py
- Added comprehensive mock tests for MCP (Model Context Protocol) integration.
- Includes tests for server connection, tool execution, tool wrapper, agent integration, server parameters, error handling, multiple servers, complex parameters, async tool integration, and tool registry.
- tests/integration/test_rag_integration.py
- Added comprehensive mock tests for RAG (Retrieval Augmented Generation) integration.
- Includes tests for RAG config creation, agent with knowledge config, vector store operations, knowledge indexing/retrieval simulation, different providers, Ollama integration, context injection, multi-document RAG, memory persistence, and knowledge updates.
- tests/simple_test_runner.py
- Added a new script for a simple test runner using subprocess.
- Includes options to run all tests, only fast tests (
--fast), or only unit tests (--unit). - Provides basic import and example execution checks for fast tests.
- tests/test_basic.py
- Added a new basic diagnostic script.
- Tests basic Python imports,
praisonaiagentsimport, and runs thebasic_agent_examplevia subprocess.
- tests/test_runner.py
- Updated the main test runner script.
- Added support for new
--patternchoices (e.g.,autogen,crewai,real,full-autogen). - Implemented warnings and confirmation prompts for running real and full execution tests.
- Set
PRAISONAI_RUN_FULL_TESTSenvironment variable for full execution tests. - Added
-x(stop on first failure) to default pytest options.
- uv.lock
- Updated the locked version of the
praisonaipackage to2.2.3.
- Updated the locked version of the
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Tests arrive in droves,
Unit, integration, e2e,
Mocked calls fly free.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
Contributor
There was a problem hiding this comment.
Code Review
This pull request primarily introduces a version bump for praisonai from 2.2.2 to 2.2.3 and adds a very comprehensive testing suite. The new testing infrastructure is well-organized, with clear distinctions between unit, integration (mocked), and end-to-end (real API calls) tests. The accompanying documentation (READMEs for test directories, testing guide) is excellent and will be highly beneficial for developers.
The changes to existing code are minor, mostly related to consistency (e.g., verbose=2 to verbose=True) or refactoring tests to be more idiomatic with pytest (e.g., using assert directly).
A more descriptive pull request title and a summary in the description would be helpful for a PR of this scope, but the code changes themselves are of good quality.
Overall, this is a valuable set of changes that significantly enhances the project's testability and maintainability.
Summary of Findings
- Verbose parameter consistency in
praisonai/agents_generator.py: Inpraisonai/agents_generator.py, theverboseparameter forCrewandPraisonAIAgentsinitialization was changed from2toTrue. This is a minor change, likely for explicitness or consistency, asverbose=2often equates to the highest level of verbosity, similar toTrue. This was not commented on due to review settings (severity: low). - Test refactoring in
tests/integration/test_base_url_api_base_fix.py: The tests intests/integration/test_base_url_api_base_fix.pywere refactored to mocklitellm.completionandlitellm.image_generationat a higher level. Assertions now focus more on the state of theLLMorAgentinstances and whether thelitellmfunctions are called, rather than inspecting the specificapi_baseargument passed to deeper mocks. This simplification is acceptable if the internalbase_urltoapi_basemapping within theLLMclass is robust and tested. This was not commented on due to review settings (severity: low).
Merge Readiness
The pull request appears to be in good shape and introduces significant improvements to the testing infrastructure. The code changes are sound. I am unable to approve the pull request, but based on this review, it seems ready for further review and merging after considering the minor points mentioned in the findings summary at the author's discretion. A more descriptive PR title and description would also be beneficial for future reference.
shaneholloman
pushed a commit
to shaneholloman/praisonai
that referenced
this pull request
Feb 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Bug Fixes
Documentation
Chores
praisonaiversion 2.2.3..gitignorerules for test directories.Style
Tests