Update test workflow to use actual TrialOps testing instead of doc validation

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Author: Copilot

pkg/cli/workflows/test-projectops-pat-requirements.md

@@ -1,7 +1,20 @@
 ---
-description: Test ProjectOps PAT requirements documentation for user-owned and org-owned Projects v2
+description: Test ProjectOps PAT requirements with actual trial repository testing
 on:
   workflow_dispatch:
+    inputs:
+      test_user_projects:
+        description: "Test user-owned Projects (requires classic PAT in GH_AW_PROJECT_GITHUB_TOKEN)"
+        type: boolean
+        default: "true"
+      test_org_projects:
+        description: "Test org-owned Projects (requires PAT with org access)"
+        type: boolean
+        default: "true"
+      cleanup_trial_repos:
+        description: "Delete trial repositories after testing"
+        type: boolean
+        default: "true"
   schedule: weekly on monday
 permissions:
   contents: read
@@ -10,7 +23,7 @@ permissions:
   pull-requests: read
 name: Test ProjectOps PAT Requirements
 engine: copilot
-timeout-minutes: 30
+timeout-minutes: 45
 tools:
   bash:
     - "*"
@@ -21,96 +34,151 @@ safe-outputs:
   create-issue:
     max: 1
     expires: 7d
-    labels: [documentation, projectops, testing]
+    labels: [documentation, projectops, testing, trialops]
 ---
 
-# Test ProjectOps PAT Requirements Documentation
+# Test ProjectOps PAT Requirements with TrialOps
 
-This workflow validates the PAT setup documentation for GitHub Projects v2 by creating temporary trial repositories, testing different PAT configurations, and verifying the documented requirements are accurate.
+This workflow performs actual integration testing of ProjectOps PAT requirements by:
+1. Creating trial repositories
+2. Testing different PAT configurations with actual GitHub Projects v2 API calls
+3. Verifying documented requirements match real behavior
+4. Cleaning up trial repositories after testing
 
-## Test Objectives
+## Prerequisites
 
-1. **Verify User-owned Projects PAT Requirements**
-   - Classic PAT with `project` scope works
-   - Fine-grained PAT does NOT work (as documented)
+This test requires PATs to be configured:
+- `GH_AW_PROJECT_GITHUB_TOKEN`: PAT for testing (should have appropriate project scopes)
+- The workflow will test if the configured PAT works as documented
 
-2. **Verify Organization-owned Projects PAT Requirements**
-   - Classic PAT with `project` + `read:org` works
-   - Fine-grained PAT with explicit org access + Projects: Read+Write works
-   - Fine-grained PAT without explicit org access fails (as documented)
+## Test Execution
 
-3. **Validate Documentation Accuracy**
-   - Confirm all setup instructions are correct
-   - Verify scope requirements match actual API behavior
-   - Test both public and private repository scenarios
+Use `gh aw trial` to test ProjectOps workflows with different PAT configurations in isolated trial repositories:
 
-## Test Execution Plan
+### Test 1: User-owned Projects with Classic PAT
 
-**IMPORTANT**: This is a validation test. Do NOT create actual trial repositories or test real PATs. Instead:
+Test if a classic PAT with `project` scope can successfully manage user-owned Projects v2.
 
-1. **Review Documentation**: Read the current ProjectOps PAT documentation at:
-   - `docs/src/content/docs/reference/tokens.md` (GH_AW_PROJECT_GITHUB_TOKEN section)
-   - `docs/src/content/docs/examples/issue-pr-events/projectops.md` (Token Requirements section)
+**Steps**:
+1. Create a trial repository: `gh-aw-trial-projectops-user`
+2. Create a test workflow that uses `update-project` to:
+   - Add an issue to a user-owned Project
+   - Update project fields
+   - Verify the operations succeed
+3. Run the workflow with the configured PAT
+4. Check if operations succeed as documented
 
-2. **Verify Documented Scopes Match GitHub API Requirements**: 
-   - Check GitHub's official Projects v2 API documentation
-   - Confirm the documented scopes align with actual API permissions
-   - Verify the distinction between user and org Projects is accurate
+**Expected Result**: Operations should succeed if using classic PAT with `project` scope
 
-3. **Check for Consistency**: 
-   - Ensure all three documentation files have consistent information
-   - Verify links between documents are correct
-   - Check that examples match the documented requirements
+### Test 2: Organization-owned Projects with Classic PAT
 
-4. **Identify Gaps or Errors**:
-   - Are there any missing scenarios or edge cases?
-   - Are the instructions clear enough to follow?
-   - Are there any contradictions between documents?
+Test if a classic PAT with `project` + `read:org` can manage org-owned Projects v2.
 
-## Test Results
+**Steps**:
+1. Create a trial repository in an organization context
+2. Create a test workflow that uses `update-project` to:
+   - Add an issue to an org-owned Project
+   - Update project fields
+   - Verify the operations succeed
+3. Run with classic PAT having `project` + `read:org` scopes
 
-Create an issue with your findings:
+**Expected Result**: Operations should succeed with proper org permissions
 
-**Title**: "ProjectOps PAT Documentation Validation - [PASS/FAIL] - $(date +%Y-%m-%d)"
+### Test 3: Organization-owned Projects with Fine-grained PAT
 
-**Body** (use markdown format):
+Test if a fine-grained PAT with explicit org access + Projects: Read+Write can manage org-owned Projects.
 
-### Documentation Review
+**Steps**:
+1. Use the same trial repository
+2. Test with fine-grained PAT (if configured separately)
+3. Verify org-owned project operations work
 
-- **Consistency Check**: [PASS/FAIL]
-  - Note: [Explanation if any inconsistencies found]
+**Expected Result**: Should work if org access was explicitly granted
 
-- **Scope Accuracy**: [PASS/FAIL]  
-  - User-owned Projects requirements: [Verified/Issues found]
-  - Org-owned Projects requirements: [Verified/Issues found]
+### Test 4: Document Fine-grained PAT Limitation for User Projects
 
-- **Clarity and Completeness**: [PASS/FAIL]
-  - Missing information: [List if any]
-  - Unclear instructions: [List if any]
+Test and document that fine-grained PATs do NOT work with user-owned Projects.
 
-### Recommendations
+**Note**: This test documents the limitation but may not be executable if we don't have a fine-grained PAT configured for comparison.
 
-[List any improvements or clarifications needed]
+## Implementation
 
-### GitHub API Documentation Cross-Reference
+Execute the following tests using bash commands:
 
-[Link to relevant GitHub API docs that confirm the requirements]
+**Step 1**: Create a minimal test workflow that attempts ProjectOps operations
 
----
+**Step 2**: Use `gh aw trial` command to run the workflow in an isolated trial repository
+
+**Step 3**: Analyze the results to determine if the configured PAT works as documented
+
+**Step 4**: Report findings in an issue
+
+The AI agent will:
+1. Generate a test workflow dynamically
+2. Execute it using the gh-aw trial command with appropriate flags
+3. Parse the results to determine PAT permission compatibility
+4. Create a summary issue with findings
+
+This provides real integration testing of the documented PAT requirements using actual GitHub Projects v2 API operations.
+
+## Test Results Report
+
+Create an issue summarizing the test results:
+
+**Title**: "ProjectOps PAT TrialOps Test Results - $(date +%Y-%m-%d)"
+
+**Body**:
+
+### Test Summary
+
+**Test Date**: $(date -u +"%Y-%m-%d %H:%M:%S UTC")
+**Trial Repository**: [Link to trial repo if not deleted]
+**Workflow Run**: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
 
-**Test Metadata**:
-- Workflow run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
-- Repository: ${{ github.repository }}
-- Test date: $(date -u +"%Y-%m-%d %H:%M:%S UTC")
+### Test Configuration
 
-**Labels**: The issue will automatically have `documentation`, `projectops`, and `testing` labels applied.
+- User-owned Projects Test: Configured via workflow dispatch input
+- Org-owned Projects Test: Configured via workflow dispatch input
+- Cleanup Trial Repos: Configured via workflow dispatch input
+
+### Results
+
+#### User-owned Projects
+- **Classic PAT with project scope**: [PASS/FAIL/NOT_TESTED]
+  - Details: [Error messages or success confirmation]
+  
+- **Fine-grained PAT**: [EXPECTED_TO_FAIL/NOT_TESTED]
+  - Details: [Confirmation of documented limitation]
+
+#### Organization-owned Projects
+- **Classic PAT with project + read:org**: [PASS/FAIL/NOT_TESTED]
+  - Details: [Error messages or success confirmation]
+  
+- **Fine-grained PAT with org access**: [PASS/FAIL/NOT_TESTED]
+  - Details: [Error messages or success confirmation]
+
+### Documentation Validation
+
+- **Documentation Accuracy**: [CONFIRMED/ISSUES_FOUND]
+- **Issues Found**: [List any discrepancies between docs and actual behavior]
+
+### Recommendations
+
+[Any updates needed to documentation based on test results]
+
+### Trial Artifacts
+
+- Trial repository: [URL or "Deleted after test"]
+- Trial results JSON: [Path to local results file]
+- GitHub Actions logs: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+
+---
 
-## Notes
+**Note**: These tests use actual GitHub Projects v2 API calls with configured PATs. Results may vary based on:
+- PAT type and scopes configured
+- Organization settings and policies
+- Project ownership (user vs org)
+- GitHub API rate limits
 
-This test validates documentation accuracy without requiring actual PAT testing, which would require:
-- Creating GitHub Apps or PATs with various permission combinations
-- Setting up test organizations and user accounts
-- Creating and destroying trial repositories
-- Managing API rate limits and quota
+For manual testing, see the [TrialOps Guide](/gh-aw/guides/trialops/) and [ProjectOps Documentation](/gh-aw/examples/issue-pr-events/projectops/).
 
-For actual integration testing with real PATs, use the `gh aw trial` command with appropriate test repositories as described in the TrialOps guide.