Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/praisonai/tests/_pytest_plugins/test_gating.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ def pytest_collection_modifyitems(config, items):

# 2. Auto-detect and assign provider markers from file content
# Skip auto-detection entirely for excluded paths (plugin tests, etc.)
if item.fspath:
if item.fspath and test_type != 'unit':
filepath = Path(item.fspath)
filepath_str = str(filepath)

Expand Down
56 changes: 54 additions & 2 deletions src/praisonai/tests/integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,45 @@ tests/integration/
β”‚ └── test_crewai_basic.py # Basic CrewAI integration tests
β”œβ”€β”€ test_base_url_api_base_fix.py # API base URL integration tests
β”œβ”€β”€ test_mcp_integration.py # Model Context Protocol tests
β”œβ”€β”€ test_managed_real.py # Real agentic managed agent tests
└── test_rag_integration.py # RAG (Retrieval Augmented Generation) tests
```

## Real Agentic Tests

### Managed Agents Integration Tests
Located in `test_managed_real.py`

**IMPORTANT:** These tests make actual LLM API calls and are gated by environment variables.

**Test Coverage:**
- βœ… Anthropic managed agents with claude-haiku-4-5
- βœ… Local managed agents with gpt-4o-mini
- βœ… Multi-turn session persistence
- βœ… Tool execution in managed environments
- βœ… Streaming functionality
- βœ… Usage token tracking
- βœ… Session ID generation and persistence

**Environment Variables:**
- `RUN_REAL_AGENTIC=1` - Required to enable these tests
- `ANTHROPIC_API_KEY` - Required for Anthropic managed agent tests
- `OPENAI_API_KEY` - Required for local managed agent tests

**Example Usage:**
```bash
# Run all real agentic tests (requires API keys)
RUN_REAL_AGENTIC=1 pytest src/praisonai/tests/integration/test_managed_real.py -v

# Run only Anthropic tests
RUN_REAL_AGENTIC=1 PRAISONAI_TEST_PROVIDERS=anthropic pytest src/praisonai/tests/integration/test_managed_real.py::test_anthropic_managed_real -v

# Run only local/OpenAI tests
RUN_REAL_AGENTIC=1 PRAISONAI_TEST_PROVIDERS=openai pytest src/praisonai/tests/integration/test_managed_real.py::test_local_managed_real_openai -v
```
Comment on lines +49 to +54
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟑 Minor

PRAISONAI_TEST_PROVIDERS is documented but not used by the test module.

The example commands set PRAISONAI_TEST_PROVIDERS=anthropic/openai, but test_managed_real.py never reads this env var β€” per-provider selection actually happens via the explicit ::test_anthropic_managed_real / ::test_local_managed_real_openai node IDs (or via the provider_anthropic/provider_openai marks if -m is used). Either drop the PRAISONAI_TEST_PROVIDERS=... prefix in the examples, or implement the gating in the test module, to avoid misleading users.

πŸ€– Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/praisonai/tests/integration/README.md` around lines 49 - 54, The README
examples advertise PRAISONAI_TEST_PROVIDERS but test_managed_real.py does not
read that env var; update the README to remove the PRAISONAI_TEST_PROVIDERS=...
prefix in the example commands so they reflect the actual selection mechanism
(node IDs test_anthropic_managed_real and test_local_managed_real_openai, or
pytest marks provider_anthropic/provider_openai), or alternatively implement
env-var gating inside test_managed_real.py by reading PRAISONAI_TEST_PROVIDERS
and skipping/selecting tests accordingly; choose one approach and make the
README and the test module consistent (references: PRAISONAI_TEST_PROVIDERS,
test_managed_real.py, test_anthropic_managed_real,
test_local_managed_real_openai, provider_anthropic, provider_openai).


**Expected Output:** Each test prints the full LLM response for human verification, along with usage statistics and session IDs.

## Framework Integration Tests

### AutoGen Integration Tests
Expand Down Expand Up @@ -139,6 +175,7 @@ python -m pytest tests/integration/autogen/test_autogen_basic.py::TestAutoGenInt
### Feature Integration Tests
- **RAG**: Tests Retrieval Augmented Generation functionality
- **MCP**: Tests Model Context Protocol integration
- **Managed Agents**: Real agentic tests for managed agent backends (Anthropic & Local)
- **Base URL/API**: Tests API base configuration and URL handling

## Test Dependencies
Expand All @@ -153,27 +190,42 @@ pip install pyautogen
pip install crewai
```

### Required for Managed Agent Real Tests:
```bash
pip install anthropic>=0.94.0 # For Anthropic managed agents
pip install litellm>=1.81.0 # For local managed agents
```

### Required for all integration tests:
```bash
pip install pytest pytest-asyncio
```

## Mock Strategy

All integration tests use comprehensive mocking to avoid:
Most integration tests use comprehensive mocking to avoid:
- ❌ Real API calls (expensive and unreliable)
- ❌ Network dependencies
- ❌ Rate limiting issues
- ❌ Environment-specific failures

**Mocking Pattern:**
**Standard Mocking Pattern:**
```python
@patch('litellm.completion')
def test_framework_integration(self, mock_completion, mock_framework_completion):
mock_completion.return_value = mock_framework_completion
# Test logic here
```

**Exception: Real Agentic Tests**

The managed agent real tests (`test_managed_real.py`) are intentionally **NOT mocked** because they are designed to verify end-to-end LLM functionality. These tests:
- βœ… Make actual API calls to verify real behavior
- βœ… Are gated by `RUN_REAL_AGENTIC=1` environment variable
- βœ… Require actual API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY)
- βœ… Print full LLM responses for human verification
- βœ… Test real token usage, session management, and tool execution

## Expected Test Outcomes

### βœ… Success Scenarios
Expand Down
Loading
Loading