chore: scope package and align tooling with @reaatech/* standards (#12)

* chore: scope package and align tooling with @reaatech/* standards

Renames the npm package to @reaatech/agent-eval-harness and brings the repo
in line with the house standard set by ../a2a-reference-ts.

Naming
- npm package: agent-eval-harness -> @reaatech/agent-eval-harness
- Added publishConfig.access=public for scoped publishing
- Bin command, OTel service names, MCP server name, Docker tag, repo URL
  all kept unscoped (identity strings, not package references)
- Updated install/import examples in README, CLAUDE.md, AGENTS.md, all
  skills/*/skill.md, and runtime getLibraryInfo()
- Added top-level LICENSE (was missing despite "license": "MIT")

Tooling
- Replaced ESLint + Prettier with Biome 1.9.4 (single tool, faster)
- Removed eslint.config.mjs, .prettierrc, all eslint/prettier devDeps
- Tightened tsconfig: strictFunctionTypes, strictBindCallApply,
  strictPropertyInitialization, alwaysStrict, isolatedModules,
  verbatimModuleSyntax (full reference parity)
- Removed invalid ignoreDeprecations="6.0" that was breaking builds on TS 5.x
- Fixed bogus version specs (TS ^6.0.3, vitest ^4.1.5, eslint ^10.x didn't
  exist on registry); aligned to reference's ^5.8.3 / ^3.2.4

Package manager
- Switched from npm to pnpm@10.22.0
- Added .npmrc with strict-peer-dependencies=true
- Generated pnpm-lock.yaml, removed package-lock.json
- Surfaced and fixed real peer-dep conflict (otel api 1.9 vs sdk-* needing
  <1.9): pinned api to ~1.8.0
- Added pnpm.overrides to force uuid >=14.0.0 (resolves moderate audit finding)
- Dropped --legacy-peer-deps workaround from Dockerfile

CI/Release
- Rewrote ci.yml to mirror reference shape: install -> {audit, format, lint,
  typecheck} -> build (uploads artifact) -> {test (matrix node 20+22),
  coverage, docker-build, docker-compose, eval} -> all-checks final gate
- Updated eval.yml to use pnpm
- release.yml: kept tag-trigger pattern (no changesets, single-package),
  switched to pnpm, added npm provenance, GitHub Packages mirror

Code cleanup
- Eliminated all 46 noNonNullAssertion sites for full reference parity
  (rule now at error level, matching reference)
- Production: refactored Levenshtein to use ?? fallbacks; added explicit
  null guards in pairwise loop iteration; replaced filter().map(t => t.opt!)
  with explicit for-loop in eval.command
- Tests: replaced expect(x.opt!).toBeLessThan(y.opt!) with `as number`
  casts; replaced find()!.prop with optional chaining + toBeDefined()

Documentation
- Added SCOPED_REMEDIATION.md: phased checklist for applying these same
  standards to other @reaatech/* repos, including pre-flight inventory,
  decision points, common pitfalls (git stash hazards, biome --unsafe
  breakage, audit overrides), and a final verification matrix

Local pipeline now all green:
- pnpm typecheck: clean
- pnpm lint: 0 errors, 0 warnings
- pnpm test: 735 passing
- pnpm build: clean
- pnpm audit --audit-level moderate: no vulnerabilities

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): call CLI via node, grant pull-requests permission

Two issues surfaced by PR #12 first CI run:

1. `pnpm exec agent-eval-harness` failed with "Command not found".
   Unlike npm, pnpm does not link a package's own bin into
   node_modules/.bin/, so the CLI was unreachable. Switched to
   `node dist/cli.js` which works under both managers and avoids
   the magic-bin-linking quirk entirely.

2. The PR-comment step hit a 403 "Resource not accessible by
   integration" because the workflow lacked pull-requests:write.
   This was a pre-existing gap that only surfaced when issue (1)
   caused the comment step to actually run with an error body.
   Added explicit permissions block.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(ci): grant pull-requests permission at eval reusable-workflow call site

A reusable workflow's permissions block can only be a subset of what the
caller grants. When eval.yml started declaring pull-requests:write, ci.yml's
eval: job invocation needed to grant it explicitly — without that, the
workflow fails at startup before any job runs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* docs: capture two CI pitfalls discovered on PR #12 first run

Adds two new entries to SCOPED_REMEDIATION.md based on real failures:

1. pnpm does not link the package's own bin into node_modules/.bin/
   the way npm does — `pnpm exec <own-bin>` fails. Use `node dist/cli.js`
   in CI scripts and Dockerfiles.

2. Reusable workflow `permissions:` blocks must be mirrored at every
   `uses:` call site in the parent workflow. Without that, the run
   fails with `startup_failure` before any job executes — and there are
   no job logs to inspect, only a workflow-level error.

Both were caught by PR #12's CI run; both are documented now so a
future remediation against another @reaatech/* repo doesn't need to
rediscover them.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

Author: reaatech

.github/workflows/ci.yml

@@ -9,21 +9,27 @@ on:
 jobs:
   evaluate:
     runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      issues: write
     steps:
       - uses: actions/checkout@v4
 
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+
       - name: Setup Node.js
         uses: actions/setup-node@v4
         with:
           node-version: '22'
-          cache: 'npm'
-          cache-dependency-path: package-lock.json
+          cache: 'pnpm'
 
       - name: Install dependencies
-        run: npm ci --legacy-peer-deps
+        run: pnpm install --frozen-lockfile
 
       - name: Build
-        run: npm run build
+        run: pnpm build
 
       - name: Download baseline results
         if: github.event_name == 'pull_request'
@@ -40,23 +46,23 @@ jobs:
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
         run: |
           mkdir -p results
-          npx agent-eval-harness eval \
+          node dist/cli.js eval \
             trajectories/examples/*.jsonl \
             --config trajectories/examples/config.yaml \
             --output results/
 
       - name: Run regression gates
         if: github.event_name == 'pull_request' && hashFiles('baseline/') != ''
         run: |
-          npx agent-eval-harness compare \
+          node dist/cli.js compare \
             baseline/results.json \
             results/results.json \
             --format markdown \
             --output results/comparison.md
 
       - name: Check gates
         run: |
-          npx agent-eval-harness gate \
+          node dist/cli.js gate \
             results/results.json \
             --preset standard \
             --exit-code

.github/workflows/eval.yml

@@ -4,75 +4,102 @@ on:
   push:
     tags:
       - 'v*'
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: false
+
+env:
+  NODE_VERSION: 22
 
 jobs:
   release:
+    name: Release
     runs-on: ubuntu-latest
     permissions:
       contents: write
       packages: write
+      id-token: write
     steps:
-      - uses: actions/checkout@v4
-      
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+
       - name: Setup Node.js
         uses: actions/setup-node@v4
         with:
-          node-version: '22'
-          cache: 'npm'
-          cache-dependency-path: package-lock.json
+          node-version: ${{ env.NODE_VERSION }}
+          cache: 'pnpm'
           registry-url: 'https://registry.npmjs.org'
-      
+
       - name: Install dependencies
-        run: npm ci
-      
+        run: pnpm install --frozen-lockfile
+
       - name: Run tests
-        run: npm test
-      
+        run: pnpm test
+
       - name: Build
-        run: npm run build
-      
+        run: pnpm build
+
       - name: Publish to npm
-        run: npm publish --access public
+        run: pnpm publish --access public --no-git-checks
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
-      
+          NPM_CONFIG_PROVENANCE: 'true'
+
+      - name: Mirror to GitHub Packages
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          cat > .npmrc <<EOF
+          @reaatech:registry=https://npm.pkg.github.com
+          //npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}
+          EOF
+          pnpm publish --registry=https://npm.pkg.github.com --no-git-checks
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
-      
+
       - name: Login to Docker Hub
         uses: docker/login-action@v3
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
-      
+
       - name: Build and push Docker image
-        uses: docker/build-push-action@v5
+        uses: docker/build-push-action@v6
         with:
           context: .
           push: true
           tags: |
             ${{ github.repository }}:${{ github.ref_name }}
             ${{ github.repository }}:latest
-          cache-from: type=registry,ref=${{ github.repository }}:buildcache
-          cache-to: type=inline
-      
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
       - name: Create GitHub Release
-        uses: softprops/action-gh-release@v1
+        uses: softprops/action-gh-release@v2
         with:
           generate_release_notes: true
           files: |
             dist/*.js
           body: |
             ## Changes
             See the [CHANGELOG](https://github.com/${{ github.repository }}/blob/main/CHANGELOG.md) for details.
-            
+
             ## Installation
-            
+
             ### npm
             ```bash
-            npm install agent-eval-harness
+            npm install @reaatech/agent-eval-harness
             ```
-            
+
             ### Docker
             ```bash
             docker pull ${{ github.repository }}:${{ github.ref_name }}
+            ```

.github/workflows/release.yml

@@ -1,13 +1,4 @@
 {
-  "src/**/*.{ts,js}": [
-    "eslint --fix",
-    "prettier --write"
-  ],
-  "tests/**/*.{ts,js}": [
-    "eslint --fix",
-    "prettier --write"
-  ],
-  "*.{json,md,yaml,yml}": [
-    "prettier --write"
-  ]
+  "*.{ts,js,json,jsonc}": ["biome check --write --no-errors-on-unmatched"],
+  "*.{md,yaml,yml}": ["biome format --write --no-errors-on-unmatched"]
 }

.lintstagedrc.json

@@ -0,0 +1,2 @@
+shamefully-hoist=false
+strict-peer-dependencies=true

.npmrc

@@ -196,7 +196,7 @@ Golden trajectories serve as reference implementations for regression testing.
 ### Comparing Against Golden
 
 ```typescript
-import { compareAgainstGolden } from 'agent-eval-harness';
+import { compareAgainstGolden } from '@reaatech/agent-eval-harness';
 
 const result = compareAgainstGolden(trajectory, goldenTrajectory, {
   similarityThreshold: 0.85,
@@ -251,7 +251,7 @@ judge:
 4. **Apply calibration** to future judge scores
 
 ```typescript
-import { calibrate, applyCalibration } from 'agent-eval-harness';
+import { calibrate, applyCalibration } from '@reaatech/agent-eval-harness';
 
 await calibrate({
   humanLabelsPath: 'calibration/human-labels.jsonl',
@@ -363,7 +363,7 @@ latency:
 ### Latency Monitoring
 
 ```typescript
-import { monitorLatency } from 'agent-eval-harness';
+import { monitorLatency } from '@reaatech/agent-eval-harness';
 
 const budget = {
   per_turn_p99: 5000,
@@ -405,7 +405,7 @@ tool_validation:
 ### Validation Example
 
 ```typescript
-import { validateTrajectory, validateSchema } from 'agent-eval-harness';
+import { validateTrajectory, validateSchema } from '@reaatech/agent-eval-harness';
 
 const toolSchemas = {
   send_reset_email: {

.prettierrc

@@ -187,7 +187,7 @@ describe('MyEvaluator', () => {
 
 ```typescript
 import { describe, it, expect } from 'vitest';
-import { loadFromFile, evaluate } from 'agent-eval-harness';
+import { loadFromFile, evaluate } from '@reaatech/agent-eval-harness';
 
 describe('Integration: Load and Evaluate', () => {
   it('should load and evaluate trajectory', () => {

AGENTS.md

@@ -1,28 +1,27 @@
 # Stage 1: Build
 FROM node:22-alpine AS builder
 
-WORKDIR /app
+RUN npm install -g pnpm@10
 
-# Copy package files
-COPY package.json package-lock.json ./
+WORKDIR /app
 
-# Install dependencies (full install for build)
-RUN npm ci --legacy-peer-deps && npm cache clean --force
+COPY package.json pnpm-lock.yaml ./
+RUN pnpm install --frozen-lockfile
 
-# Copy source
 COPY tsconfig.json ./
 COPY src ./src
 
-# Build
-RUN npm run build
+RUN pnpm build
 
 # Stage 2: Install production deps only
 FROM node:22-alpine AS prod-deps
 
+RUN npm install -g pnpm@10
+
 WORKDIR /app
 
-COPY package.json package-lock.json ./
-RUN npm ci --legacy-peer-deps --only=production --ignore-scripts && npm cache clean --force
+COPY package.json pnpm-lock.yaml ./
+RUN pnpm install --prod --frozen-lockfile --ignore-scripts
 
 # Stage 3: Runtime
 FROM node:22-alpine AS runtime

CLAUDE.md

@@ -22,7 +22,7 @@ End-to-end agent evaluation harness for full agent runs. Supports trajectory eva
 
 ```bash
 # npm
-npm install agent-eval-harness
+npm install @reaatech/agent-eval-harness
 
 # Or use without installing
 npx agent-eval-harness eval trajectories/*.jsonl