Skip to content

Develop #532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MervinPraison merged 11 commits into from
May 28, 2025
Merged

Develop #532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
d53f5b2
Add API key validation check to skip tests when invalid key is provided
MervinPraison May 28, 2025
5dd3153
Merge branch 'main' of https://github.com/MervinPraison/PraisonAI int…
MervinPraison May 28, 2025
aff990f
refactor: streamline API test skipping and result validation in test …
MervinPraison May 28, 2025
14ece72
chore: enhance test workflow and API key handling
MervinPraison May 28, 2025
4a65f77
chore: enhance GitHub Actions workflow and test cases
MervinPraison May 28, 2025
b1a8983
chore: add debugging steps to GitHub Actions workflow for CrewAI tests
MervinPraison May 28, 2025
b54568e
chore: update .gitignore to include additional test directories for C…
MervinPraison May 28, 2025
36ff14d
chore: enhance testing workflow and API test execution
MervinPraison May 28, 2025
954281e
chore: enhance GitHub Actions workflows with environment variable deb…
MervinPraison May 28, 2025
9191092
chore: add debugging steps for PraisonAI API key validation in GitHub…
MervinPraison May 28, 2025
b19a9ea
chore: update PraisonAI dependencies and enhance model testing in wor…
MervinPraison May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .github/workflows/python-package.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,36 @@ jobs:
with:
python-version: ${{ matrix.python-version }}

- name: Set environment variables
run: |
echo "OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY || 'sk-test-key-for-github-actions-testing-only-not-real' }}" >> $GITHUB_ENV
echo "OPENAI_API_BASE=${{ secrets.OPENAI_API_BASE || 'https://api.openai.com/v1' }}" >> $GITHUB_ENV
echo "OPENAI_MODEL_NAME=${{ secrets.OPENAI_MODEL_NAME || 'gpt-4o-mini' }}" >> $GITHUB_ENV
echo "LOGLEVEL=DEBUG" >> $GITHUB_ENV

- name: Install dependencies
run: |
cd src/praisonai
python -m pip install --upgrade pip
python -m pip install flake8 pytest
python -m pip install ."[ui,gradio,api,agentops,google,openai,anthropic,cohere,chat,code,realtime,call,crewai,autogen]"
python -m pip install duckduckgo_search
python -m pip install pytest-asyncio pytest-cov
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi

- name: Debug API Key Status
run: |
echo "🔍 Checking API key availability..."
if [ -n "${{ secrets.OPENAI_API_KEY }}" ]; then
echo "✅ GitHub secret OPENAI_API_KEY is available"
echo "🔑 API key starts with: $(echo "$OPENAI_API_KEY" | cut -c1-7)..."
else
echo "⚠️ GitHub secret OPENAI_API_KEY is NOT set - using fallback"
echo "🔑 Using fallback key: sk-test-key..."
fi
echo "🌐 API Base: $OPENAI_API_BASE"
echo "🤖 Model: $OPENAI_MODEL_NAME"

# - name: Lint with flake8
# run: |
# # stop the build if there are Python syntax errors or undefined names
Expand Down
185 changes: 178 additions & 7 deletions .github/workflows/test-core.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,134 @@ jobs:
echo "OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY || 'sk-test-key-for-github-actions-testing-only-not-real' }}" >> $GITHUB_ENV
echo "OPENAI_API_BASE=${{ secrets.OPENAI_API_BASE || 'https://api.openai.com/v1' }}" >> $GITHUB_ENV
echo "OPENAI_MODEL_NAME=${{ secrets.OPENAI_MODEL_NAME || 'gpt-4o-mini' }}" >> $GITHUB_ENV
echo "LOGLEVEL=DEBUG" >> $GITHUB_ENV
echo "PYTHONPATH=${{ github.workspace }}/src/praisonai-agents:$PYTHONPATH" >> $GITHUB_ENV
# Also export to current shell session for immediate availability
export OPENAI_API_KEY="${{ secrets.OPENAI_API_KEY || 'sk-test-key-for-github-actions-testing-only-not-real' }}"
export OPENAI_API_BASE="${{ secrets.OPENAI_API_BASE || 'https://api.openai.com/v1' }}"
export OPENAI_MODEL_NAME="${{ secrets.OPENAI_MODEL_NAME || 'gpt-4o-mini' }}"
export LOGLEVEL=DEBUG
# Verify immediate availability
echo "🔧 Immediate verification in same step:"
echo " OPENAI_API_KEY length in current session: ${#OPENAI_API_KEY}"
echo " OPENAI_API_KEY starts with sk-: $(echo "$OPENAI_API_KEY" | grep -q '^sk-' && echo 'YES' || echo 'NO')"

- name: Debug API Key Status
run: |
echo "🔍 Checking API key availability..."
if [ -n "${{ secrets.OPENAI_API_KEY }}" ]; then
echo "✅ GitHub secret OPENAI_API_KEY is available"
echo "🔑 API key starts with: $(echo "$OPENAI_API_KEY" | cut -c1-7)..."
else
echo "⚠️ GitHub secret OPENAI_API_KEY is NOT set - using fallback"
echo "🔑 Using fallback key: sk-test-key..."
fi
echo "🌐 API Base: $OPENAI_API_BASE"
echo "🤖 Model: $OPENAI_MODEL_NAME"
echo "🐛 Log Level: $LOGLEVEL"
echo "📊 Environment Check:"
echo " - OPENAI_API_KEY length: ${#OPENAI_API_KEY}"
echo " - OPENAI_API_BASE: $OPENAI_API_BASE"
echo " - OPENAI_MODEL_NAME: $OPENAI_MODEL_NAME"
echo " - LOGLEVEL: $LOGLEVEL"
echo "🔍 Is API key actually set?"
echo " - API key starts with sk-: $(echo "$OPENAI_API_KEY" | grep -q '^sk-' && echo 'YES' || echo 'NO')"
echo " - API key is not test key: $([ "$OPENAI_API_KEY" != 'sk-test-key-for-github-actions-testing-only-not-real' ] && echo 'YES' || echo 'NO')"

- name: Debug Environment Variables Raw
run: |
echo "🔧 Raw environment variable check:"
echo "OPENAI_API_KEY set: $(if [ -n "$OPENAI_API_KEY" ]; then echo 'YES'; else echo 'NO'; fi)"
echo "OPENAI_API_KEY length: ${#OPENAI_API_KEY}"
echo "OPENAI_API_KEY first 10 chars: ${OPENAI_API_KEY:0:10}"
echo "OPENAI_API_KEY last 5 chars: ${OPENAI_API_KEY: -5}"
printenv | grep OPENAI || echo "No OPENAI env vars found"

- name: Debug Python Environment Variables
run: |
python -c "
import os
print('🐍 Python Environment Variable Check:')
api_key = os.environ.get('OPENAI_API_KEY', 'NOT_SET')
if api_key != 'NOT_SET':
print(f' ✅ OPENAI_API_KEY: {api_key[:7]}... (length: {len(api_key)})')
else:
print(' ❌ OPENAI_API_KEY: NOT_SET')
print(f' 🌐 OPENAI_API_BASE: {os.environ.get(\"OPENAI_API_BASE\", \"NOT_SET\")}')
print(f' 🤖 OPENAI_MODEL_NAME: {os.environ.get(\"OPENAI_MODEL_NAME\", \"NOT_SET\")}')
print(f' 📋 All OPENAI env vars:')
for key, value in os.environ.items():
if key.startswith('OPENAI'):
print(f' {key}: {value[:10] if len(value) > 10 else value}...')
"

- name: Validate API Key
run: |
echo "🔑 Testing API key validity with minimal OpenAI call..."
python -c "
import os
try:
from openai import OpenAI
client = OpenAI(api_key=os.environ.get('OPENAI_API_KEY'))
response = client.models.list()
print('✅ API Key is VALID - OpenAI responded successfully')
print(f'📊 Available models: {len(list(response.data))} models found')
except Exception as e:
print(f'❌ API Key is INVALID - Error: {e}')
print('🔍 This explains why all API-dependent tests are failing')
print('💡 The GitHub secret OPENAI_API_KEY needs to be updated with a valid key')
"
continue-on-error: true

- name: Debug PraisonAI API Key Usage
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY || 'sk-test-key-for-github-actions-testing-only-not-real' }}
OPENAI_API_BASE: ${{ secrets.OPENAI_API_BASE || 'https://api.openai.com/v1' }}
OPENAI_MODEL_NAME: ${{ secrets.OPENAI_MODEL_NAME || 'gpt-4o-mini' }}
LOGLEVEL: DEBUG
run: |
echo "🔍 Testing PraisonAI API key usage directly..."
cd src/praisonai
python -c "
import os
import sys
sys.path.insert(0, '.')

print('🔧 Direct PraisonAI API Key Check:')
print(f'Environment OPENAI_API_KEY: {os.environ.get(\"OPENAI_API_KEY\", \"NOT_SET\")[:10]}...')

# Test PraisonAI initialization
from praisonai import PraisonAI
praisonai = PraisonAI()

print(f'PraisonAI config_list: {praisonai.config_list}')
api_key_from_config = praisonai.config_list[0].get('api_key', 'NOT_SET')
print(f'API key from PraisonAI config: {api_key_from_config[:10] if api_key_from_config != \"NOT_SET\" else \"NOT_SET\"}...')

# Test PraisonAIModel with explicit API key (the way CrewAI will use it)
from praisonai.inc.models import PraisonAIModel

print('\\n🧪 Testing PraisonAIModel with explicit API key (CrewAI method):')
model_with_explicit_key = PraisonAIModel(
model='openai/gpt-4o-mini',
base_url=praisonai.config_list[0].get('base_url'),
api_key=praisonai.config_list[0].get('api_key')
)
print(f' Model: {model_with_explicit_key.model}')
print(f' Model name: {model_with_explicit_key.model_name}')
print(f' API key var: {model_with_explicit_key.api_key_var}')
print(f' API key (explicit): {model_with_explicit_key.api_key[:10] if model_with_explicit_key.api_key != \"nokey\" else \"NOT_SET\"}...')
print(f' Base URL: {model_with_explicit_key.base_url}')

# Test if the model can be created without errors
try:
llm_instance = model_with_explicit_key.get_model()
print(f' ✅ LLM instance created successfully: {type(llm_instance).__name__}')
print(f' LLM instance API key: {getattr(llm_instance, \"openai_api_key\", \"NOT_FOUND\")[:10] if hasattr(llm_instance, \"openai_api_key\") else \"NO_API_KEY_ATTR\"}...')
except Exception as e:
print(f' ❌ Failed to create LLM instance: {e}')
"
continue-on-error: true

- name: Run Unit Tests
run: |
Expand All @@ -49,20 +177,63 @@ jobs:
run: |
cd src/praisonai && python -m pytest tests/integration/ -v --tb=short --disable-warnings

- name: Run AutoGen Framework Tests
- name: Debug Directory Structure
run: |
echo "🔍 Debugging directory structure for CrewAI tests..."
cd src/praisonai
echo "Current working directory: $(pwd)"
echo "📁 Contents of current directory:"
ls -la
echo ""
echo "📁 Contents of tests directory:"
ls -la tests/ || echo "❌ tests/ directory not found"
echo ""
echo "📁 Contents of tests/integration:"
ls -la tests/integration/ || echo "❌ tests/integration/ directory not found"
echo ""
echo "📁 Looking for crewai directory:"
find . -name "crewai" -type d 2>/dev/null || echo "❌ No crewai directories found"
echo ""
echo "📁 Full directory tree of tests:"
tree tests/ || find tests/ -type d 2>/dev/null || echo "❌ Cannot explore tests directory"

- name: Test AutoGen Framework
run: |
echo "🤖 Testing AutoGen Framework Integration..."
cd src/praisonai && python tests/test_runner.py --pattern autogen --verbose || echo "⚠️ AutoGen tests completed with issues"
continue-on-error: true
cd src/praisonai && python tests/test_runner.py --pattern autogen --verbose

- name: Run CrewAI Framework Tests
- name: Test CrewAI Framework
run: |
echo "⛵ Testing CrewAI Framework Integration..."
cd src/praisonai && python tests/test_runner.py --pattern crewai --verbose || echo "⚠️ CrewAI tests completed with issues"
continue-on-error: true
cd src/praisonai
echo "🔍 Trying test runner first..."
python tests/test_runner.py --pattern crewai --verbose || {
echo "❌ Test runner failed, trying direct pytest..."
echo "📁 Current directory: $(pwd)"
echo "📁 Looking for CrewAI tests..."
find . -name "*crewai*" -type f 2>/dev/null
echo "🧪 Trying direct pytest on integration/crewai..."
python -m pytest tests/integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ Direct path failed, trying relative path..."
python -m pytest integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ All CrewAI test attempts failed"
exit 1
}
}
}
Comment on lines +200 to +223
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improved test execution with intelligent fallback logic.

The enhanced test execution steps provide better error handling with fallback attempts on different paths. This approach is more robust than simply using continue-on-error: true.

However, consider adding a final status check to ensure at least one test path succeeded:

          python -m pytest integration/crewai/ -v --tb=short --disable-warnings || {
            echo "❌ All CrewAI test attempts failed"
+           echo "💡 Check if CrewAI tests exist at expected paths"
            exit 1
          }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Test AutoGen Framework
run: |
echo "🤖 Testing AutoGen Framework Integration..."
cd src/praisonai && python tests/test_runner.py --pattern autogen --verbose || echo "⚠️ AutoGen tests completed with issues"
continue-on-error: true
cd src/praisonai && python tests/test_runner.py --pattern autogen --verbose
- name: Run CrewAI Framework Tests
- name: Test CrewAI Framework
run: |
echo "⛵ Testing CrewAI Framework Integration..."
cd src/praisonai && python tests/test_runner.py --pattern crewai --verbose || echo "⚠️ CrewAI tests completed with issues"
continue-on-error: true
cd src/praisonai
echo "🔍 Trying test runner first..."
python tests/test_runner.py --pattern crewai --verbose || {
echo "❌ Test runner failed, trying direct pytest..."
echo "📁 Current directory: $(pwd)"
echo "📁 Looking for CrewAI tests..."
find . -name "*crewai*" -type f 2>/dev/null
echo "🧪 Trying direct pytest on integration/crewai..."
python -m pytest tests/integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ Direct path failed, trying relative path..."
python -m pytest integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ All CrewAI test attempts failed"
exit 1
}
}
}
- name: Test CrewAI Framework
run: |
echo "⛵ Testing CrewAI Framework Integration..."
cd src/praisonai
echo "🔍 Trying test runner first..."
python tests/test_runner.py --pattern crewai --verbose || {
echo "❌ Test runner failed, trying direct pytest..."
echo "📁 Current directory: $(pwd)"
echo "📁 Looking for CrewAI tests..."
find . -name "*crewai*" -type f 2>/dev/null
echo "🧪 Trying direct pytest on integration/crewai..."
python -m pytest tests/integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ Direct path failed, trying relative path..."
python -m pytest integration/crewai/ -v --tb=short --disable-warnings || {
echo "❌ All CrewAI test attempts failed"
echo "💡 Check if CrewAI tests exist at expected paths"
exit 1
}
}
}
🤖 Prompt for AI Agents 
In .github/workflows/test-core.yml around lines 138 to 161, the test steps for
CrewAI framework use fallback logic but lack a final status check to confirm
success. Add a final check after all fallback attempts to verify if any test
command succeeded, and if none did, explicitly fail the job by exiting with a
non-zero status. This ensures the workflow accurately reflects test failures
instead of silently passing.


- name: Run Legacy Tests
- name: Run Legacy Tests with API Key
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY || 'sk-test-key-for-github-actions-testing-only-not-real' }}
OPENAI_API_BASE: ${{ secrets.OPENAI_API_BASE || 'https://api.openai.com/v1' }}
OPENAI_MODEL_NAME: ${{ secrets.OPENAI_MODEL_NAME || 'gpt-4o-mini' }}
LOGLEVEL: DEBUG
run: |
echo "🧪 Running legacy tests with real API key..."
echo "🔧 Final environment check before pytest:"
echo " OPENAI_API_KEY set: $([ -n "$OPENAI_API_KEY" ] && echo 'YES' || echo 'NO')"
echo " OPENAI_API_KEY length: ${#OPENAI_API_KEY}"
echo " OPENAI_API_KEY starts with sk-: $(echo "$OPENAI_API_KEY" | grep -q '^sk-' && echo 'YES' || echo 'NO')"
cd src/praisonai && python -m pytest tests/test.py -v --tb=short --disable-warnings

- name: Upload Coverage Reports
Expand Down
12 changes: 12 additions & 0 deletions .github/workflows/unittest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,18 @@ jobs:
# Run the fastest, most essential tests
cd src/praisonai && python tests/test_runner.py --pattern fast

- name: Run Real API Tests
run: |
echo "🔑 Running real API tests with actual OpenAI API key..."
cd src/praisonai && python -m pytest tests/test_agents_playbook.py -v --tb=short --disable-warnings -m real
continue-on-error: true

- name: Run E2E Real Tests
run: |
echo "🧪 Running E2E real tests..."
cd src/praisonai && python -m pytest tests/e2e/ -v --tb=short --disable-warnings -m real
continue-on-error: true

- name: Run Legacy Example Tests
run: |
cd src/praisonai && python -m pytest tests/test.py -v --tb=short --disable-warnings
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ agentops.log
crewAI
!tests/integration/crewai
!tests/e2e/crewai
!src/praisonai/tests/integration/crewai
!src/praisonai/tests/e2e/crewai

# virtualenv
.venv
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install flask praisonai==2.2.16 gunicorn markdown
RUN pip install flask praisonai==2.2.17 gunicorn markdown
EXPOSE 8080
CMD ["gunicorn", "-b", "0.0.0.0:8080", "api:app"]
2 changes: 1 addition & 1 deletion docker/Dockerfile.chat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
RUN pip install --no-cache-dir \
praisonaiagents>=0.0.4 \
praisonai_tools \
"praisonai==2.2.16" \
"praisonai==2.2.17" \
"praisonai[chat]" \
"embedchain[github,youtube]"

Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.dev
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
RUN pip install --no-cache-dir \
praisonaiagents>=0.0.4 \
praisonai_tools \
"praisonai==2.2.16" \
"praisonai==2.2.17" \
"praisonai[ui]" \
"praisonai[chat]" \
"praisonai[realtime]" \
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile.ui
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
RUN pip install --no-cache-dir \
praisonaiagents>=0.0.4 \
praisonai_tools \
"praisonai==2.2.16" \
"praisonai==2.2.17" \
"praisonai[ui]" \
"praisonai[crewai]"

Expand Down
2 changes: 1 addition & 1 deletion docs/api/praisonai/deploy.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ <h2 id="raises">Raises</h2>
file.write(&#34;FROM python:3.11-slim\n&#34;)
file.write(&#34;WORKDIR /app\n&#34;)
file.write(&#34;COPY . .\n&#34;)
file.write(&#34;RUN pip install flask praisonai==2.2.16 gunicorn markdown\n&#34;)
file.write(&#34;RUN pip install flask praisonai==2.2.17 gunicorn markdown\n&#34;)
file.write(&#34;EXPOSE 8080\n&#34;)
file.write(&#39;CMD [&#34;gunicorn&#34;, &#34;-b&#34;, &#34;0.0.0.0:8080&#34;, &#34;api:app&#34;]\n&#39;)

Expand Down
2 changes: 1 addition & 1 deletion docs/developers/local-development.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ WORKDIR /app

COPY . .

RUN pip install flask praisonai==2.2.16 watchdog
RUN pip install flask praisonai==2.2.17 watchdog

EXPOSE 5555

Expand Down
2 changes: 1 addition & 1 deletion docs/ui/chat.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ To facilitate local development with live reload, you can use Docker. Follow the

COPY . .

RUN pip install flask praisonai==2.2.16 watchdog
RUN pip install flask praisonai==2.2.17 watchdog

EXPOSE 5555

Expand Down
2 changes: 1 addition & 1 deletion docs/ui/code.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ To facilitate local development with live reload, you can use Docker. Follow the

COPY . .

RUN pip install flask praisonai==2.2.16 watchdog
RUN pip install flask praisonai==2.2.17 watchdog

EXPOSE 5555

Expand Down
4 changes: 2 additions & 2 deletions src/praisonai-agents/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ build-backend = "setuptools.build_meta"

[project]
name = "praisonaiagents"
version = "0.0.89"
version = "0.0.90"
description = "Praison AI agents for completing complex tasks with Self Reflection Agents"
requires-python = ">=3.8"
requires-python = ">=3.10"
authors = [
{ name="Mervin Praison" }
]
Expand Down
Loading
Loading