Skip to content

GitHubWorkflow.md

Latest commit

 

History

History
278 lines (194 loc) · 18.5 KB

GitHubWorkflow.md

File metadata and controls

278 lines (194 loc) · 18.5 KB

The GitHub Workflow Server

The GitHub Workflow Server (neo.mjs-github-workflow) acts as the Agent OS's "Executive Function." It bridges the gap between the local development environment and the remote GitHub repository, enabling a "Local-First" project management philosophy.

1. Purpose: Context Engineering

Traditional autonomous agents rely heavily on GitHub APIs to read issues and Pull Requests (PRs). This is fragile, slow, and context-poor. An agent has to "ask" for information (via API) rather than "having" it.

This server transforms the workflow by implementing Context Engineering: giving the agent the precise, queryable, and persistent context it needs to be a true partner. By synchronizing issues and PRs as local Markdown files via a robust Bi-Directional Sync system, we enable:

  • Three-Dimensional Context:
    1. History: The git log provides the code's timeline.
    2. Requirements: The local tickets provide the formal plan and acceptance criteria.
    3. Intent: The agent's own Memory Core provides the reasoning behind past decisions.
  • Zero-Latency Access: Agents can read, search, and cross-reference tickets instantly without API round-trips.
  • Semantic Intelligence: The local markdown files are automatically indexed by the Knowledge Base Server (neo.mjs-knowledge-base). This enables semantic vector search, allowing agents to query tickets by meaning (e.g., "find past VDOM regressions") rather than just keywords.
  • Platform Independence: Your project management data lives in your repo, not just on GitHub's servers.

2. Real-World Use Case: The Autonomous Review Loop

The true power of this server is best understood through the "Autonomous Review Loop." This workflow removes the human maintainer as a bottleneck for routine code reviews.

The Workflow

  1. Discovery: The agent uses list_pull_requests to find open PRs.
  2. Context Gathering:
    • It reads the conversation history with get_conversation.
    • It analyzes the code changes with get_pull_request_diff.
    • If needed, it checks out the branch locally with checkout_pull_request to run tests (npm test).
  3. Analysis & Review: The agent synthesizes this information to form a review.
  4. Participation: The agent posts a structured review comment using create_comment.

Visualizing the Agent's Voice

When an agent posts a comment, the server automatically formats it with an "Agent Header" to ensure transparency and clear attribution:

Input from Gemini 2.5 Pro:

✦ Code Review Findings

I've analyzed the changes in this PR and found the following:

  1. Line 47: The null check should be moved before the dereference
  2. Tests: The edge case for empty arrays is not covered

This clear distinction between human and AI input is a core part of the server's design.

3. Architecture

The server is built on a robust service layer that orchestrates the flow of data between the local file system and GitHub.

3.1 Service Layer

The application logic is distributed across specialized services, all extending Neo.core.Base:

  • SyncService (The Conductor): The central singleton that coordinates the entire synchronization process. It manages the dependencies between fetching releases, archiving issues, pushing local changes, and pulling remote updates.
  • IssueSyncer: Handles the heavy lifting of converting between GitHub GraphQL nodes and local Markdown. It implements:
    • Content Hashing: To detect actual changes and prevent unnecessary writes.
    • Markdown Formatting: converting JSON data into frontmatter-enriched Markdown.
    • Archive Logic: Determining the correct path for closed issues based on release dates.
  • ReleaseSyncer: Fetches GitHub Releases and synchronizes their release notes to .github/RELEASE_NOTES/. This data is crucial for the "Self-Organizing Archive."
  • MetadataManager: Manages the persistent state (.github/.sync-metadata.json). It tracks content hashes, file paths, and timestamps to ensure efficient delta updates.
  • GraphqlService: A centralized wrapper around the GitHub GraphQL API. It handles authentication via the gh CLI, token caching, and executes complex queries defined in services/queries/.
  • HealthService: Acts as a gatekeeper, ensuring the gh CLI is installed, authenticated, and up-to-date. It implements intelligent caching (5-minute TTL for healthy results) to minimize overhead.

3.2 The Bi-Directional Sync Workflow

The core of the server is its ability to maintain state consistency between the local filesystem and GitHub. This isn't just downloading files; it's a true two-way synchronization engine:

  • Push (Local → Remote): The server scans for modified local Markdown files. It uses Content Hashing (SHA-256) to detect actual changes. If a file has changed, the server pushes updates (Title, Body) to GitHub via GraphQL mutations.
  • Pull (Remote → Local): The server fetches the latest data from GitHub. New issues are created locally, and updated issues overwrite local versions.

When sync_all is called, SyncService executes a precise 8-step orchestration logic to ensure data integrity:

  1. Load Metadata: Reads .sync-metadata.json to understand the previous state.
  2. Fetch Releases: ReleaseSyncer fetches the latest release data. This is done first because the archive logic depends on knowing available releases.
  3. Reconcile Archives: IssueSyncer scans the active .github/ISSUE directory for closed issues. It compares their closedAt date with release dates to move them to the correct archive folder (e.g., .github/ISSUE_ARCHIVE/v10.5.0/).
  4. Push Local Changes: IssueSyncer scans the active directory for modified Markdown files. It compares content hashes with the metadata. If changed, it pushes updates (title, body) to GitHub via the UPDATE_ISSUE GraphQL mutation.
  5. Pull Remote Changes: IssueSyncer executes the FETCH_ISSUES_FOR_SYNC GraphQL query. This powerful query fetches issues, comments, labels, assignees, and relationships in a single request. New issues are created locally; updated issues are overwritten.
  6. Sync Release Notes: ReleaseSyncer writes release notes to .github/RELEASE_NOTES/, ensuring the local documentation matches GitHub.
  7. Self-Heal: The system checks for and clears any resolved push failures from the metadata.
  8. Save Metadata: The new state, including updated content hashes and timestamps, is persisted to disk.

4. The Toolset

The server exposes a comprehensive suite of tools via the Model Context Protocol (MCP), mapped directly to service methods.

4.1 Synchronization & Health

  • sync_all: Triggers the full bi-directional sync workflow described above.
  • healthcheck: Verifies gh CLI status. Returns cached healthy results for speed, but performs immediate checks if the system was previously unhealthy.
  • get_viewer_permission: Returns the bot's permission level (ADMIN, WRITE, READ).

4.2 Issue Management

  • create_issue: Authoritative tool for creating tickets. Uses gh issue create.
  • list_issues: Lists issues with filtering. Uses client-side filtering for labels/assignees to reduce API complexity.
  • get_local_issue_by_id: Fast, offline retrieval of an issue's full Markdown content from the local file system.
  • add_labels / remove_labels: Manages labels via GraphQL mutations.
  • assign_issue / unassign_issue: Manages user assignments. Requires WRITE permission.
  • update_issue_relationship: Configures parent-child (Epic) or blocked-by dependencies using GraphQL mutations (ADD_SUB_ISSUE, ADD_BLOCKED_BY, etc.).

4.3 Pull Request Management

  • list_pull_requests: Lists PRs with status filtering.
  • checkout_pull_request: Checks out a PR branch locally using gh pr checkout.
  • get_pull_request_diff: Fetches the code difference via gh pr diff.
  • get_conversation: Retrieves the full conversation (title, body, comments) for a pull request or an issue — supply exactly one of pr_number / issue_number. The same comment selectors (comment_id / since_comment_id / last_n) narrow the result on both.
  • create_comment: Posts a new comment to a PR or Issue. Supports "Agent Headers" to identify which AI identity is speaking.
  • update_comment: Edits an existing comment.

4.4 Discovery

  • list_labels: Enumerates all valid labels in the repository. Always check this before applying labels.

4.5 Discussions

  • create_discussion: Creates a new GitHub Discussion (default category Ideas). Used for the Ideation Sandbox.
  • manage_discussion_comment: Creates or updates a comment on a Discussion (action: create | update).
  • manage_discussion: Updates the body of an existing Discussion (action: update_body). Enables post-publication corrections to Ideation Sandbox entries through the MCP surface instead of a raw gh api graphql workaround.

5. Configuration

The server is highly configurable via config.mjs or a custom configuration file passed via the -c or --config CLI flag. This allows you to override default settings—such as the target repository, file paths, or API limits—without modifying the source code.

Usage

node ai/mcp/server/github-workflow/mcp-server.mjs -c ./my-config.json

Configuration File Format

You can provide a JSON file or an ES Module (.mjs) that exports a configuration object. The custom configuration is deep-merged with the default settings.

Example my-config.json:

{
    "owner": "my-org",
    "repo": "my-project",
    "debug": true,
    "issueSync": {
        "syncStartDate": "2024-01-01T00:00:00Z",
        "droppedLabels": ["wontfix", "duplicate", "spam"]
    },
    "pullRequest": {
        "defaults": {
            "limit": 10
        }
    }
}

This flexibility is crucial for:

  • Multi-Repo Support: Quickly switching the server context between different projects.
  • Performance Tuning: Adjusting maxIssues or GraphQL limits based on your network or token constraints.
  • Environment Specifics: Using different directory structures for testing versus production.

6. Data Structures

The Directory Structure

  • .github/ISSUE/: The "Active Workspace." Contains all open tickets as Markdown files.
  • .github/ISSUE_ARCHIVE/: The "Historical Record." Contains closed tickets, organized by Release Version (e.g., v5.0.0/).
  • .github/RELEASE_NOTES/: Contains the content of GitHub Releases.

Archive Anomaly Detection & Sealed-Chunk Semantics

To ensure referential stability and institutionalize "Sealed-Chunk" archive integrity, the IssueSyncer enforces strict controls over issues once they are archived:

  • Sealed-Chunk Enforcements: Once an issue is written to an archive bucket (e.g., v10.5.0/chunk-1/), it is "sealed". If subsequent syncs pull the issue and detect a shift in its closedAt date or its milestone, the system will prevent the issue from "jumping" buckets. It forces retention at the oldAbsolutePath to prevent historical data regression.
  • Archive Anomaly Hooks: When an expected bucket shift occurs (e.g., the closedAt timestamp was modified), the syncer intercepts the discrepancy and emits an operator-actionable [ARCHIVE ANOMALY] log warning. This allows the Swarm to react to metadata drift while maintaining sync performance via delta updates.

The Markdown Format

Each issue is stored as a YAML-frontmatter Markdown file. The frontmatter contains rich metadata derived from the GraphQL response:

---
id: 7852
title: Documentation: Create Guide for "The GitHub Workflow Server"
state: OPEN
labels: [documentation, ai]
assignees: [tobiu]
author: tobiu
createdAt: 2025-10-26T10:00:00Z
updatedAt: 2025-10-26T12:00:00Z
parentIssue: 7850
subIssues: [7855, 7856]
blockedBy: []
---
# Description
Create a new guide...

## Comments

### @tobiu - 2025-10-26 12:05
This looks correct.

The Metadata Cache

Stored at .github/.sync-metadata.json, this file allows the server to perform "delta syncs." It tracks:

  • contentHash: SHA-256 hash of the Markdown content to detect local modifications.
  • updatedAt: Timestamp of the last update from GitHub to detect remote changes.
  • path: The current file path, allowing the system to track moves (e.g., archiving).

7. GitHub Projects v2 — First-Class Membership Primitive

The Neo swarm uses GitHub ProjectV2 as a first-class membership primitive wired directly into the create_issue and manage_issue_projects MCP tools. Per #11233 — the substrate-correct corrective for #10961's pilot scope, which proved labels-as-project-proxy is structurally wrong-shape.

Source-of-truth contract

  • Canonical membership: ProjectV2 board (addProjectV2ItemById / deleteProjectV2Item) — discoverable via gh project item-list <NUM> --owner <ORG> and the manage_issue_projects MCP tool
  • Canonical status: issue state + workflow labels (agent-task:in-progress, agent-task:review, etc.) — NOT release-tracking labels
  • Canonical owner: issue assignee
  • Project single-select fields (Status, Priority, etc.): authoritative for project-board view; not consumed by Dream Pipeline / Golden Path / Native Edge Graph (those layers remain on issue-state)

Migration history (#10961 → #11233): the pilot used release:v13 as a label-as-project-proxy, with a reconcileV13Project.mjs script to detect drift. Empirical V-B-A showed 31% structural drift (14 labeled-not-in-project + 196 in-project-not-labeled out of 250 items) — drift that cannot be patched away because labels (categorization primitive) and ProjectV2 items (membership primitive) are independent GitHub concepts. The script is deleted in Phase 3 of #11233; the release:v* label family is retired in Phase 2.

Agent ergonomics — atomic create-with-project

The create_issue MCP tool accepts an optional projects parameter:

{
  "title": "Some new ticket",
  "body": "Ticket body…",
  "labels": ["ai", "enhancement"],
  "assignees": ["@me"],
  "projects": [{"projectNumber": 12}]   // v13 Release project — direct attach
}

The issue is created via gh issue create, then attached to each ProjectV2 board via addProjectV2ItemById. Partial-attach is the graceful failure mode (issue exists; orphan-rollback would be worse for agent workflows).

Agent ergonomics — post-create membership management

The manage_issue_projects MCP tool mirrors the manage_issue_labels action shape:

  • {action: 'add', issue_number, projectNumbers: [12]}
  • {action: 'remove', issue_number, projectNumbers: [12]}
  • {action: 'update_field', issue_number, projectNumber: 12, fieldName: 'Status', value: 'In Progress'}

Field values for single-select fields (Status, Priority, etc.) are resolved by name; update_field returns OPTION_NOT_FOUND with the available option list if the value doesn't match. Multi-select / iteration / number fields are out of scope for this Phase 1 revision — extend the mutation surface as those use cases land.

The "If it's not on the Issue, it doesn't exist to the Swarm" rule (refined)

Per @neo-gemini-3-1-pro's original framing in Discussion #10959, the swarm's downstream substrate (Dream Pipeline, Golden Path Synthesizer, Native Edge Graph, list_issues, get_local_issue_by_id, Knowledge Base, Memory Core) consumes issue state — not Project field state. Project membership IS a first-class signal (it's how release-tracking and milestone-grouping work), but single-select field values (Status, Priority) remain Project-board observability and MUST NOT drive Dream synthesis or Golden Path decisions.

Concrete rule: project MEMBERSHIP is canonical (queryable via gh project item-list); project FIELD VALUES are observability-only for downstream agentic substrates.

Boundary with Sandman / Golden Path

ProjectV2 field state is NOT consumed by DreamService / GoldenPathSynthesizer / the Native Edge Graph. See learn/agentos/DreamPipeline.md §"Project state is observability-only" for the explicit non-input warning. (Membership signals — "this ticket is in v13 release" — are derivable via direct project query when needed for release-tracking workflows; they are not auto-ingested into the substrate.)

v2-only mandate

GitHub ProjectV2 (GraphQL projectsV2, ProjectV2, addProjectV2ItemById, deleteProjectV2Item, updateProjectV2ItemFieldValue). GitHub Projects classic / v1 was sunset on github.com in 2024 + classic REST API in 2025. NO classic columns/cards/REST endpoints anywhere in scripts or workflows.

Membership shape

ProjectV2 ownership is by Organization or User — Repository is NOT a ProjectV2Owner. A Project can be additionally linked to one or more repositories via linkProjectV2ToRepository, which makes it discoverable at the repo level. The v13 Project (#12) is org-owned (neomjs) and repo-linked to neomjs/neo.

8. Platform Independence & Vendor Lock-In

The GitHub Workflow Server's local-first architecture isn't just about performance—it's about sovereignty.

Git repositories, by default, do not contain issues, pull request conversations, or release notes. These critical project artifacts live inside GitHub's proprietary database. This creates significant vendor lock-in. If GitHub (owned by Microsoft) changes its terms, pricing, or strategic direction—such as moving infrastructure to Azure or restricting AI access—your project management data is trapped.

Our architecture solves this by treating GitHub as a sync target, not the source of truth.

Because every issue and release note is stored as a standard Markdown file inside your git repository:

  1. Migration is Feasible: Moving to GitLab, Gitea, or a custom solution becomes a matter of writing a new sync adapter. Your data is already in a portable format.
  2. Data is Yours: You have a complete, offline, version-controlled backup of your project's history.
  3. Workflow Continuity: If the remote platform goes down, your agents (and you) can continue to read, search, and modify tickets locally.

This aligns with the broader Agent OS philosophy: build tools that empower the user and ensure long-term resilience.