[Phase 1] Integration tests for credential and environment modules

[frankbria]

Summary

Expand integration test coverage for core/credentials.py and core/environment.py modules.

Problem

These modules exist and work but lack comprehensive integration tests, which could lead to regressions.

Scope

Credential Manager Tests

• Keyring integration (store/retrieve/delete)
• Fallback to environment variables
• API key validation
• Error handling for missing credentials

Environment Validator Tests

• Tool detection (git, npm, uv, pytest, ruff)
• Version requirement validation
• Missing tool handling
• Environment report generation

Acceptance Criteria

• Credential manager tested with real keyring operations
• Environment validator tested with actual tool detection
• Edge cases covered (missing tools, invalid versions)
• Tests run as part of pytest tests/core/
• Target: Increase v2 test count from 70+ to 100+

Related

• Part of Phase 1 CLI completion
• Ensures robust foundation for all subsequent phases

[traycerai]

Plan

Observations

The codebase has comprehensive unit tests for file:codeframe/core/credentials.py and file:codeframe/core/environment.py modules with heavy mocking. Current tests in file:tests/core/test_credentials.py and file:tests/core/test_environment.py mock keyring operations and tool detection. The project uses pytest with markers (v2, integration) and has fixtures for temp directories and environment setup. The keyring library (v24.0.0+) is already a dependency, and the test infrastructure supports integration testing patterns.

Approach

Create dedicated integration test files that exercise real system operations without mocks. For credentials, test actual keyring storage/retrieval with cleanup fixtures. For environment validation, test real tool detection on the system. Use pytest markers to distinguish integration tests from unit tests, allowing selective execution. Structure tests to be idempotent and isolated, with proper setup/teardown to avoid polluting the system keyring or environment. Target 30+ new integration tests to reach the 100+ test count goal.

Implementation Steps

1. Create Integration Test File for Credentials

Create file:tests/core/test_credentials_integration.py with the following test classes:

Test Class: TestCredentialStoreIntegration

• Test real keyring storage and retrieval operations
• Use a dedicated test service name (e.g., "codeframe-test-credentials") to avoid conflicts
• Implement fixture integration_credential_store that creates a CredentialStore with test-specific storage directory and cleans up after tests
• Test scenarios:
  • Store credential in real keyring and verify retrieval
  • Store credential when keyring unavailable (force encrypted file fallback)
  • Verify encrypted file permissions are 0o600
  • Test credential deletion from both keyring and encrypted file
  • Test list_providers with mixed storage (some in keyring, some in file)
  • Test decryption failure handling (corrupt encrypted file)
  • Test salt file regeneration and key derivation consistency

Test Class: TestCredentialManagerIntegration

• Test high-level credential management with real storage
• Test scenarios:
  • Set and get credential through real keyring
  • Verify environment variable override works with real storage
  • Test credential rotation with real storage backend
  • Test list_credentials with mixed sources (env vars + stored)
  • Test expired credential handling with real timestamps
  • Test credential validation with actual API key formats
  • Test concurrent access to credential store (thread safety)

Test Class: TestEncryptionIntegration

• Test real encryption/decryption operations
• Test scenarios:
  • Verify machine-specific key derivation produces consistent keys
  • Test encryption round-trip with large credential values
  • Test salt file corruption detection and recovery
  • Verify encrypted data cannot be decrypted with different machine ID
  • Test file permissions enforcement on salt and encrypted files

Fixtures Required:

• integration_temp_dir: Temporary directory for test storage
• integration_credential_store: CredentialStore instance with test storage
• cleanup_keyring: Fixture to clean up test keyring entries after each test
• mock_machine_id: Fixture to provide consistent machine ID for testing

Markers:

• Mark all tests with @pytest.mark.integration
• Mark tests requiring keyring with @pytest.mark.skipif(not KEYRING_AVAILABLE, reason="Keyring not available")

2. Create Integration Test File for Environment Validation

Create file:tests/core/test_environment_integration.py with the following test classes:

Test Class: TestToolDetectionIntegration

• Test real tool detection on the system
• Test scenarios:
  • Detect git (should be available in CI/dev environments)
  • Detect python and verify version parsing from actual output
  • Detect pytest and verify version compatibility check
  • Test detection of non-existent tool returns NOT_FOUND status
  • Test version extraction from various real tool outputs (git, python, npm, etc.)
  • Test version compatibility with actual installed versions
  • Test timeout handling for slow tool version commands

Test Class: TestProjectTypeDetectionIntegration

• Test project type detection with real file structures
• Use tmp_path fixture to create test project structures
• Test scenarios:
  • Create pyproject.toml and verify Python project detection
  • Create package.json and verify JavaScript project detection
  • Create Cargo.toml and verify Rust project detection
  • Test priority when multiple markers exist (pyproject.toml + package.json)
  • Test unknown project type for empty directory
  • Test required tools list for each detected project type
  • Test optional tools list for each detected project type

Test Class: TestEnvironmentValidatorIntegration

• Test full environment validation workflow with real tools
• Test scenarios:
  • Validate environment for Python project with real tool detection
  • Test health score calculation with actual available tools
  • Test recommendation generation for missing tools
  • Test validation with custom required tools list
  • Test validation with version incompatibilities (if testable)
  • Test warning generation for critical missing tools
  • Test conflict detection (if applicable)
  • Test environment report generation with real data

Test Class: TestVersionParsingIntegration

• Test version parsing with real tool outputs
• Capture actual version strings from common tools
• Test scenarios:
  • Parse git version output: "git version 2.39.0"
  • Parse Python version output: "Python 3.11.4"
  • Parse pytest version output: "pytest 7.4.0"
  • Parse npm version output: "10.2.4"
  • Parse ruff version output with various formats
  • Test comparison with real version strings
  • Test edge cases from actual tool outputs

Fixtures Required:

• real_python_project: Creates temporary Python project structure
• real_js_project: Creates temporary JavaScript project structure
• system_tools: Fixture that detects available tools on the system
• skip_if_tool_missing: Parametrized fixture to skip tests if specific tools unavailable

Markers:

• Mark all tests with @pytest.mark.integration
• Mark tests requiring specific tools with @pytest.mark.skipif conditions

3. Update Test Configuration

Modify file:tests/core/conftest.py:

• Add fixture for integration test credential cleanup:

  @pytest.fixture
  def integration_credential_store(temp_dir):
      """Provide CredentialStore for integration testing with cleanup."""

• Add fixture for keyring cleanup:

  @pytest.fixture
  def cleanup_test_keyring():
      """Clean up test keyring entries after test."""

• Add fixture for system tool detection:

  @pytest.fixture(scope="session")
  def available_system_tools():
      """Detect available tools on the system once per session."""

Update file:pytest.ini or file:pyproject.toml:

• Ensure integration marker is registered
• Add option to run only integration tests: pytest -m integration
• Add option to exclude integration tests: pytest -m "not integration"

4. Add Edge Case Tests

In file:tests/core/test_credentials_integration.py:

• Test credential storage when keyring backend is "fail" keyring
• Test handling of corrupted encrypted file (invalid JSON)
• Test handling of corrupted salt file (wrong size)
• Test credential expiration with real datetime comparisons
• Test credential metadata persistence through storage round-trip
• Test multiple credentials stored simultaneously
• Test credential deletion when not found (should not raise)

In file:tests/core/test_environment_integration.py:

• Test tool detection with PATH modifications
• Test handling of tools that exist but fail version check
• Test handling of tools with non-standard version output
• Test environment validation with no tools available
• Test environment validation with all tools available
• Test custom tool detector registration
• Test install command generation for various tools

5. Documentation Updates

Create file:tests/core/README_INTEGRATION.md:

• Document purpose of integration tests vs unit tests
• Explain how to run integration tests: pytest -m integration tests/core/
• Document system requirements for integration tests (keyring, tools)
• Explain how to skip integration tests in CI if needed
• Document cleanup procedures and test isolation strategies

Update file:TESTING.md:

• Add section on integration test strategy
• Document test count progression (70+ → 100+)
• Explain integration test markers and execution
• Add troubleshooting for integration test failures

6. CI/CD Considerations

Ensure integration tests work in CI:

• Verify keyring backend availability in CI environment
• Configure CI to install required tools (git, python, pytest, ruff)
• Add CI job specifically for integration tests
• Consider using pytest-xdist for parallel execution
• Set up proper cleanup in CI to avoid state pollution

7. Test Execution and Validation

Verify test count increase:

• Run pytest tests/core/ --collect-only to count tests
• Ensure total count exceeds 100 tests
• Verify integration tests are properly marked
• Run integration tests separately: pytest -m integration tests/core/
• Run unit tests separately: pytest -m "not integration" tests/core/

Validate test quality:

• Ensure all integration tests are idempotent
• Verify proper cleanup in all fixtures
• Test isolation: run tests in random order
• Verify no test pollution between runs
• Check test execution time (integration tests may be slower)

Test Structure Diagram

graph TD
    A[tests/core/] --> B[test_credentials.py]
    A --> C[test_credentials_integration.py]
    A --> D[test_environment.py]
    A --> E[test_environment_integration.py]
    
    B --> B1[Unit Tests with Mocks]
    C --> C1[Real Keyring Tests]
    C --> C2[Real Encryption Tests]
    C --> C3[Edge Case Tests]
    
    D --> D1[Unit Tests with Mocks]
    E --> E1[Real Tool Detection]
    E --> E2[Real Project Detection]
    E --> E3[Real Validation Tests]
    
    F[conftest.py] --> G[Integration Fixtures]
    G --> G1[cleanup_keyring]
    G --> G2[integration_credential_store]
    G --> G3[available_system_tools]

Loading

Expected Test Count Breakdown

Test File	Unit Tests	Integration Tests	Total
test_credentials.py	~40	0	40
test_credentials_integration.py	0	~20	20
test_environment.py	~35	0	35
test_environment_integration.py	0	~15	15
Total	75	35	110

This achieves the target of 100+ tests with comprehensive integration coverage.

Import In IDE

🤖 Prompt for AI Agents

## Observations

The codebase has comprehensive unit tests for `file:codeframe/core/credentials.py` and `file:codeframe/core/environment.py` modules with heavy mocking. Current tests in `file:tests/core/test_credentials.py` and `file:tests/core/test_environment.py` mock keyring operations and tool detection. The project uses pytest with markers (v2, integration) and has fixtures for temp directories and environment setup. The keyring library (v24.0.0+) is already a dependency, and the test infrastructure supports integration testing patterns.

## Approach

Create dedicated integration test files that exercise real system operations without mocks. For credentials, test actual keyring storage/retrieval with cleanup fixtures. For environment validation, test real tool detection on the system. Use pytest markers to distinguish integration tests from unit tests, allowing selective execution. Structure tests to be idempotent and isolated, with proper setup/teardown to avoid polluting the system keyring or environment. Target 30+ new integration tests to reach the 100+ test count goal.

## Implementation Steps

### 1. Create Integration Test File for Credentials

Create `file:tests/core/test_credentials_integration.py` with the following test classes:

**Test Class: TestCredentialStoreIntegration**
- Test real keyring storage and retrieval operations
- Use a dedicated test service name (e.g., "codeframe-test-credentials") to avoid conflicts
- Implement fixture `integration_credential_store` that creates a CredentialStore with test-specific storage directory and cleans up after tests
- Test scenarios:
  - Store credential in real keyring and verify retrieval
  - Store credential when keyring unavailable (force encrypted file fallback)
  - Verify encrypted file permissions are 0o600
  - Test credential deletion from both keyring and encrypted file
  - Test list_providers with mixed storage (some in keyring, some in file)
  - Test decryption failure handling (corrupt encrypted file)
  - Test salt file regeneration and key derivation consistency

**Test Class: TestCredentialManagerIntegration**
- Test high-level credential management with real storage
- Test scenarios:
  - Set and get credential through real keyring
  - Verify environment variable override works with real storage
  - Test credential rotation with real storage backend
  - Test list_credentials with mixed sources (env vars + stored)
  - Test expired credential handling with real timestamps
  - Test credential validation with actual API key formats
  - Test concurrent access to credential store (thread safety)

**Test Class: TestEncryptionIntegration**
- Test real encryption/decryption operations
- Test scenarios:
  - Verify machine-specific key derivation produces consistent keys
  - Test encryption round-trip with large credential values
  - Test salt file corruption detection and recovery
  - Verify encrypted data cannot be decrypted with different machine ID
  - Test file permissions enforcement on salt and encrypted files

**Fixtures Required:**
- `integration_temp_dir`: Temporary directory for test storage
- `integration_credential_store`: CredentialStore instance with test storage
- `cleanup_keyring`: Fixture to clean up test keyring entries after each test
- `mock_machine_id`: Fixture to provide consistent machine ID for testing

**Markers:**
- Mark all tests with `@pytest.mark.integration`
- Mark tests requiring keyring with `@pytest.mark.skipif(not KEYRING_AVAILABLE, reason="Keyring not available")`

### 2. Create Integration Test File for Environment Validation

Create `file:tests/core/test_environment_integration.py` with the following test classes:

**Test Class: TestToolDetectionIntegration**
- Test real tool detection on the system
- Test scenarios:
  - Detect git (should be available in CI/dev environments)
  - Detect python and verify version parsing from actual output
  - Detect pytest and verify version compatibility check
  - Test detection of non-existent tool returns NOT_FOUND status
  - Test version extraction from various real tool outputs (git, python, npm, etc.)
  - Test version compatibility with actual installed versions
  - Test timeout handling for slow tool version commands

**Test Class: TestProjectTypeDetectionIntegration**
- Test project type detection with real file structures
- Use `tmp_path` fixture to create test project structures
- Test scenarios:
  - Create pyproject.toml and verify Python project detection
  - Create package.json and verify JavaScript project detection
  - Create Cargo.toml and verify Rust project detection
  - Test priority when multiple markers exist (pyproject.toml + package.json)
  - Test unknown project type for empty directory
  - Test required tools list for each detected project type
  - Test optional tools list for each detected project type

**Test Class: TestEnvironmentValidatorIntegration**
- Test full environment validation workflow with real tools
- Test scenarios:
  - Validate environment for Python project with real tool detection
  - Test health score calculation with actual available tools
  - Test recommendation generation for missing tools
  - Test validation with custom required tools list
  - Test validation with version incompatibilities (if testable)
  - Test warning generation for critical missing tools
  - Test conflict detection (if applicable)
  - Test environment report generation with real data

**Test Class: TestVersionParsingIntegration**
- Test version parsing with real tool outputs
- Capture actual version strings from common tools
- Test scenarios:
  - Parse git version output: "git version 2.39.0"
  - Parse Python version output: "Python 3.11.4"
  - Parse pytest version output: "pytest 7.4.0"
  - Parse npm version output: "10.2.4"
  - Parse ruff version output with various formats
  - Test comparison with real version strings
  - Test edge cases from actual tool outputs

**Fixtures Required:**
- `real_python_project`: Creates temporary Python project structure
- `real_js_project`: Creates temporary JavaScript project structure
- `system_tools`: Fixture that detects available tools on the system
- `skip_if_tool_missing`: Parametrized fixture to skip tests if specific tools unavailable

**Markers:**
- Mark all tests with `@pytest.mark.integration`
- Mark tests requiring specific tools with `@pytest.mark.skipif` conditions

### 3. Update Test Configuration

**Modify `file:tests/core/conftest.py`:**
- Add fixture for integration test credential cleanup:
  ```python
  @pytest.fixture
  def integration_credential_store(temp_dir):
      """Provide CredentialStore for integration testing with cleanup."""
  ```
- Add fixture for keyring cleanup:
  ```python
  @pytest.fixture
  def cleanup_test_keyring():
      """Clean up test keyring entries after test."""
  ```
- Add fixture for system tool detection:
  ```python
  @pytest.fixture(scope="session")
  def available_system_tools():
      """Detect available tools on the system once per session."""
  ```

**Update `file:pytest.ini` or `file:pyproject.toml`:**
- Ensure integration marker is registered
- Add option to run only integration tests: `pytest -m integration`
- Add option to exclude integration tests: `pytest -m "not integration"`

### 4. Add Edge Case Tests

**In `file:tests/core/test_credentials_integration.py`:**
- Test credential storage when keyring backend is "fail" keyring
- Test handling of corrupted encrypted file (invalid JSON)
- Test handling of corrupted salt file (wrong size)
- Test credential expiration with real datetime comparisons
- Test credential metadata persistence through storage round-trip
- Test multiple credentials stored simultaneously
- Test credential deletion when not found (should not raise)

**In `file:tests/core/test_environment_integration.py`:**
- Test tool detection with PATH modifications
- Test handling of tools that exist but fail version check
- Test handling of tools with non-standard version output
- Test environment validation with no tools available
- Test environment validation with all tools available
- Test custom tool detector registration
- Test install command generation for various tools

### 5. Documentation Updates

**Create `file:tests/core/README_INTEGRATION.md`:**
- Document purpose of integration tests vs unit tests
- Explain how to run integration tests: `pytest -m integration tests/core/`
- Document system requirements for integration tests (keyring, tools)
- Explain how to skip integration tests in CI if needed
- Document cleanup procedures and test isolation strategies

**Update `file:TESTING.md`:**
- Add section on integration test strategy
- Document test count progression (70+ → 100+)
- Explain integration test markers and execution
- Add troubleshooting for integration test failures

### 6. CI/CD Considerations

**Ensure integration tests work in CI:**
- Verify keyring backend availability in CI environment
- Configure CI to install required tools (git, python, pytest, ruff)
- Add CI job specifically for integration tests
- Consider using `pytest-xdist` for parallel execution
- Set up proper cleanup in CI to avoid state pollution

### 7. Test Execution and Validation

**Verify test count increase:**
- Run `pytest tests/core/ --collect-only` to count tests
- Ensure total count exceeds 100 tests
- Verify integration tests are properly marked
- Run integration tests separately: `pytest -m integration tests/core/`
- Run unit tests separately: `pytest -m "not integration" tests/core/`

**Validate test quality:**
- Ensure all integration tests are idempotent
- Verify proper cleanup in all fixtures
- Test isolation: run tests in random order
- Verify no test pollution between runs
- Check test execution time (integration tests may be slower)

## Test Structure Diagram

```mermaid
graph TD
    A[tests/core/] --> B[test_credentials.py]
    A --> C[test_credentials_integration.py]
    A --> D[test_environment.py]
    A --> E[test_environment_integration.py]
    
    B --> B1[Unit Tests with Mocks]
    C --> C1[Real Keyring Tests]
    C --> C2[Real Encryption Tests]
    C --> C3[Edge Case Tests]
    
    D --> D1[Unit Tests with Mocks]
    E --> E1[Real Tool Detection]
    E --> E2[Real Project Detection]
    E --> E3[Real Validation Tests]
    
    F[conftest.py] --> G[Integration Fixtures]
    G --> G1[cleanup_keyring]
    G --> G2[integration_credential_store]
    G --> G3[available_system_tools]
```

## Expected Test Count Breakdown

| Test File | Unit Tests | Integration Tests | Total |
|-----------|-----------|-------------------|-------|
| test_credentials.py | ~40 | 0 | 40 |
| test_credentials_integration.py | 0 | ~20 | 20 |
| test_environment.py | ~35 | 0 | 35 |
| test_environment_integration.py | 0 | ~15 | 15 |
| **Total** | **75** | **35** | **110** |

This achieves the target of 100+ tests with comprehensive integration coverage.

---

Execution Information

Branch: main
Commit: e768c1d

---

💡 Tips

Supported Commands (Inside Comments)

• Use @traycerai generate to iterate on the previous version of the implementation plan.

Supported Commands (Inside Description)

• Add @traycerai ignore anywhere in the ticket description to prevent this ticket from being processed.
• Add @traycerai branch:<branch-name> anywhere in the ticket description to specify the target branch for the implementation plan.

Community

• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#191 - feat(e2e): Enable conditional tests and fix discovery startup [merged]
#304 - test(cli): add comprehensive v2 CLI integration test suite [merged]
#305 - feat(cli): add comprehensive GitHub PR workflow commands [merged]
#306 - feat(env): add environment validation with tool detection [closed]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!