Important

This repository has moved to https://github.com/PDP-Connect/data-connectors. It is now maintained as part of the PDP-Connect lab under LF Decentralized Trust. This copy is archived and read-only; please open issues and pull requests at the new home.

Data Connectors

Playwright-based data connectors for local desktop collection. Each connector exports your data from a web platform using browser automation. Credentials never leave your device.

This repo bundles its own standalone runner (playwright-runner/) -- no other repo is required to run a connector. See Running a connector below.

Connector status

Each connector has a status indicating its maturity level:

• Stable -- Production-ready. Well-tested across multiple releases, reliable login and data extraction, complete schemas.
• Beta -- Functional but less tested. Works in most cases but may have edge cases or require more login handling.
• Experimental -- New or untested. May not work reliably; contributed recently with limited real-world usage.

Connectors

This table is an overview, not the public contract. scope-catalog.json is the authoritative machine-readable contract for sources and scopes. SCOPES.md is its generated human-readable view.

The catalog describes the local desktop collection path: registered Playwright connectors run on your device and use your browser session.

Source	Published scopes	Desktop connector(s) / maturity
amazon	amazon.orders, amazon.profile	amazon-playwright (beta)
chatgpt	chatgpt.conversations, chatgpt.memories	chatgpt-playwright (stable)
claude	claude.conversations, claude.projects	claude-export-playwright (experimental)
doordash	doordash.orders	doordash-playwright (beta)
github	github.contributions, github.events, github.history, github.profile, github.repositories, github.starred	github-playwright (stable)
heb	heb.nutrition, heb.orders, heb.profile	heb-playwright (experimental)
icloud_notes	icloud_notes.folders, icloud_notes.notes	icloud-notes-playwright (experimental)
instagram	instagram.ads, instagram.following, instagram.posts, instagram.profile	instagram-playwright (stable); instagram-ads-playwright (beta)
linkedin	linkedin.connections, linkedin.education, linkedin.experience, linkedin.languages, linkedin.profile, linkedin.skills	linkedin-playwright (stable)
oura	oura.activity, oura.readiness, oura.sleep	oura-playwright (stable)
shop	shop.orders	shop-playwright (beta)
spotify	spotify.playlists, spotify.profile, spotify.savedTracks	spotify-playwright (stable)
steam	steam.friends, steam.games, steam.profile	steam-playwright (experimental)
uber	uber.receipts, uber.trips	uber-playwright (beta)
wholefoods	wholefoods.nutrition, wholefoods.orders, wholefoods.profile	wholefoods-playwright (experimental)
youtube	youtube.history, youtube.likes, youtube.playlistItems, youtube.playlists, youtube.profile, youtube.subscriptions, youtube.watchLater	youtube-playwright (beta)

Context Gateway

Open Data Labs exposes source data through the Context Gateway API. The table below lists its permissions by source.

These read:* values are Context Gateway permissions, not canonical scope IDs. Use scope-catalog.json to find canonical source and scope IDs, payload schemas, collection paths, limits, and maturity.

Source	Status	Scopes
GitHub	Available	read:profile, read:repositories, read:starred
iCloud Notes	Available	read:notes, read:folders
Instagram	Available	read:profile, read:following, read:ads
Oura Ring	Available	read:readiness, read:sleep, read:activity
Spotify	Coming soon	read:profile, read:savedTracks, read:playlists

See the Context Gateway docs for full schemas and example responses.

Running a connector

# One-time setup: install the bundled runner's dependencies + Chromium
bash skills/pdp-connect/scripts/setup.sh

node run-connector.cjs ./connectors/github/github-playwright.js              # JSON output (for agents)
node run-connector.cjs ./connectors/github/github-playwright.js --pretty      # colored output (for humans)
node run-connector.cjs ./connectors/github/github-playwright.js --inputs '{"username":"x","password":"y"}'

run-connector.cjs resolves the bundled playwright-runner/ in this repo automatically -- no other repo needs to be cloned. See skills/pdp-connect/SETUP.md for details, and skills/pdp-connect/ for the agent skill: setup, running, creating new connectors, and data recipes.

Repository structure

Folder	What's inside	Audience
connectors/	All platform connectors (<company>/<name>-playwright.js + .json + connector-local schemas/ + local assets like icons/)	Everyone
scripts/	Developer tooling: scaffold, test, validate, session capture	Human developers
skills/	AI agent skill for creating/running connectors (pdp-connect/)	AI agents (Claude, etc.)
schemas/	Shared meta-schemas such as manifest.schema.json	Validation
types/	TypeScript type definitions (connector.d.ts)	TypeScript consumers
fixture-index.json	Public index of schema-valid connector fixtures with main-branch raw GitHub URLs and checksums	Builders and AI agents

connectors/                        # All platform connectors
├── <company>/
│   ├── <name>-playwright.js       #   Connector script (plain JS)
│   ├── <name>-playwright.json     #   Metadata (login URL, selectors, scopes)
│   ├── schemas/
│   │   └── <platform>.<scope>.json
│   ├── fixtures/
│   │   └── <platform>.<scope>.<scenario>.json
│   └── icons/
│
scripts/                           # Developer tooling (human-driven)
├── create-connector.sh            #   End-to-end connector scaffold + test
├── capture-session.cjs            #   Browser session capture (manual login)
├── test-connector.cjs             #   Run connector against a real browser
├── validate-connector.cjs         #   Structure + output validator
└── reference/                     #   Templates and API docs
│
skills/pdp-connect/                # Agent skill (AI-agent-driven)
├── SKILL.md                       #   Skill entry point (setup, connect, recipes)
├── CREATE.md                      #   Full walkthrough for building connectors
└── scripts/                       #   Agent-facing scripts (runner, validator, etc.)
│
schemas/                           # Shared meta-schemas
├── manifest.schema.json
│
registry.json                      # Central registry (checksums, versions, OTA)
fixture-index.json                 # Public fixture catalog for builders and agents
scope-catalog.json                 # Generated public source/scope contract
run-connector.cjs                  # Symlink → skills/pdp-connect/scripts/run-connector.cjs
test-connector.cjs                 # Standalone test runner
create-connector.sh                # Quick autonomous scaffold script

Distribution contract

data-connectors is an open collection of data connectors that any consumer can pin and resolve as a versioned dependency. Vana's Context Gateway and Unity Surfaces are two example consumers; nothing here is specific to them.

• connector-index.json is the authoritative release index.
• scope-catalog.json is the generated, schema-validated public contract for source IDs, scope IDs, payload schema paths, desktop fulfillment, material limits, and maturity. SCOPES.md is generated from the same model.
• Pin scope-catalog.json from an immutable GitHub release named connectors-<commit12>, for example https://github.com/PDP-Connect/data-connectors/releases/download/connectors-<commit12>/scope-catalog.json. Release catalogs embed the full source commit and an immutable schema.url for every payload schema. Checked-in catalogs retain schema.path for local repository use. A raw main URL is not a version pin.
• scope-catalog.json.sigstore.json is the catalog's detached Sigstore bundle.
• The same immutable release includes scope-catalog.schema.json and its detached Sigstore bundle. The catalog's catalogSchema object names both the checked-in schema path and the flat release-asset filename.
• fixture-index.json is the public fixture catalog for schema-valid synthetic sample data. Builders and agents should download fixture raw URLs instead of pasting large JSON payloads into prompts or terminals. Raw URLs point at latest main; use the included SHA-256 checksum when exact bytes matter.
• connector-index.json.sigstore.json is the detached Sigstore bundle for the authoritative signed index published at the stable connectors-latest release URL.
• registry.json is the sole publishability gate for connectors and their scopes. Only registry-selected manifests enter generated public catalogs and release indexes.
• artifacts/<connector>/<connector>-<version>.tgz is the immutable bundle format consumed by installers.
• artifacts/**/*.tgz.sigstore.json is the detached Sigstore bundle for each immutable artifact.
• @opendatalabs/data-connectors-tools/installer-core exposes the supported install/update contract used by consumer repos.
• connector-installer is the supported CLI wrapper for lock generation, installation, verification, and update checks.

Connectors

Each connector lives in connectors/<company>/. A connector bundle keeps its runtime assets together inside that directory. A connector usually consists of:

• <name>-playwright.js -- the connector script (plain JS, runs inside the Playwright runner sidecar)
• <name>-playwright.json -- metadata (display name, login URL, selectors, scopes)
• schemas/<platform>.<scope>.json -- connector-owned JSON Schemas for every declared scope
• fixtures/<platform>.<scope>.<scenario>.json -- optional synthetic sample payloads for local builder tests
• icons/... -- canonical local icon assets referenced by the manifest via a path relative to that connector directory

Some connectors also include a README with platform-specific setup instructions (e.g., API keys).

Builder fixtures

Fixtures are public, synthetic scope payloads for app builders and AI agents. Use them when a realistic local sample is needed but a live account export would be too large to paste into an agent prompt.

Fixture files live next to the connector:

connectors/<company>/fixtures/<scope>.<scenario>.json

Start each shipped scope with these scenarios when useful:

• empty -- valid no-data state
• small -- a compact sample for UI smoke tests
• large -- a larger sample for pagination, performance, and summarization tests

Each fixture must be valid JSON, map to a scope declared by a connector manifest in the same directory, and conform to connectors/<company>/schemas/<scope>.json. The generated fixture-index.json records the fixture id, connector id, source id, scope, scenario, record count, repo path, latest-main raw URL, schema path, byte size, and SHA-256 checksum.

npm run fixtures:validate
npm run fixture-index:generate
npm run fixture-index:check

Scripts vs. skills

The repo has two interfaces for building and running connectors. They serve different audiences but share the same connector format and output:

	scripts/	skills/pdp-connect/
Audience	Human developers at a terminal	AI agents (Claude, etc.)
Entry point	scripts/create-connector.sh	skills/pdp-connect/SKILL.md
Login	Manual browser login via capture-session.cjs	Runner-driven (run-connector.cjs)
Testing	scripts/test-connector.cjs	run-connector.cjs
When to use	Local development, debugging, manual QA	Automated connector creation and data export

Both produce the same connector files (connectors/<company>/<name>-playwright.js + .json) and use the same schemas, registry, and validation scripts.

---

How connectors work

Connectors run in a sandboxed Playwright browser managed by the bundled runner (playwright-runner/) or, in the desktop app, an equivalent driver. The runner provides a page API object (not raw Playwright). The browser starts headless; connectors call page.showBrowser() when login is needed and page.goHeadless() after.

Two-phase architecture

Phase 1 -- Login (visible browser)

1. Navigate to the platform's login page (headless)
2. Check if the user is already logged in via persistent session
3. If not, show the browser so the user can log in manually
4. Extract auth tokens/cookies once logged in

Phase 2 -- Data collection (headless)

1. Switch to headless mode (browser disappears)
2. Fetch data via API calls, network capture, or DOM scraping
3. Report structured progress to the UI
4. Return the collected data with an export summary

Canonical result shape

Connectors return a flat result object. Top-level keys are either reserved metadata or scope keys (source.category format, e.g., linkedin.profile, chatgpt.conversations). The frontend auto-detects scoped keys (any key containing a . that isn't a reserved key) and POSTs each scope separately to the Personal Server at POST /v1/data/{scope}.

const result = {
  // ── Reserved metadata (always required) ──
  requestedScopes: ['platform.scope1', 'platform.scope2'],
  timestamp: new Date().toISOString(),
  version: '2.0.0-playwright',
  platform: 'platform-name',
  exportSummary: { count: 42, label: '42 items exported', details: { /* ... */ } },
  errors: [],

  // ── Scope keys (one per collected scope) ──
  'platform.scope1': { /* scope data */ },
  'platform.scope2': { /* scope data */ },
};

Reserved metadata keys

These top-level keys are reserved and must not be used as scope names:

Key	Type	Required	Description
requestedScopes	string[]	yes	The resolved execution target for this run — explicit, deduplicated, non-empty.
timestamp	string	yes	ISO 8601 timestamp of when the run completed.
version	string	yes	Connector version string.
platform	string	yes	Platform identifier.
exportSummary	object	yes	Summary with count (number), label (string), and optional details.
errors	ConnectorError[]	yes	Unresolved output-affecting errors. Empty array if none.

All other top-level keys are treated as candidate scope keys and must be members of requestedScopes.

Scope keys

Scope keys use the platform.category format (e.g., github.repositories). Rules:

• Every produced scope key must appear in requestedScopes.
• A scope not listed in requestedScopes must never be produced (protocol violation).
• Empty-but-present scope payloads ({}) are valid data, not failures.
• A requested scope may be: produced cleanly, produced in degraded form, or omitted.

requestedScopes

requestedScopes represents the resolved execution target for this run — not raw caller input.

• If the caller omits scopes, the runtime expands to the connector manifest's full canonical scope set before execution.
• requestedScopes must always be present, explicit, deduplicated, and non-empty.
• Missing, empty, or non-array requestedScopes is a protocol violation.

errors[] — honest outcome reporting

errors[] carries unresolved, output-affecting problems only. It must always be present (empty array is valid).

Each error entry:

{
  errorClass: 'auth_failed',        // from the shared taxonomy (see below)
  reason: 'Session cookie expired',  // human-readable explanation
  disposition: 'omitted',           // 'omitted' | 'degraded' | 'fatal'
  scope: 'platform.scope1',         // required for omitted/degraded; optional for fatal
  phase: 'auth',                    // optional diagnostic context
}

Dispositions:

Disposition	Meaning	scope required?	Scope key present?
omitted	The requested scope did not produce a trustworthy payload.	yes	no
degraded	The scope produced a payload, but completeness or fidelity is materially affected.	yes	yes
fatal	A run-level problem makes the entire run untrustworthy.	no	n/a

What belongs in errors[]:

• Auth failures preventing collection
• Selector drift causing omission
• Pagination failures making a produced scope materially incomplete
• Scope emitted outside requestedScopes

What does NOT belong (if fully recovered):

• A navigation attempt that failed then succeeded with no output lost
• A fetch that failed once, retry succeeded, no output lost

Error taxonomy

Connectors and runners share one error taxonomy:

auth_failed · rate_limited · upstream_error · navigation_error · network_error · selector_error · timeout · protocol_violation · runtime_error · personal_server_unavailable · unknown

Protocol violations

The following are protocol violations and must be treated as hard failures:

• Missing required metadata fields (requestedScopes, timestamp, version, platform, exportSummary, errors)
• requestedScopes missing, empty, or non-array
• errors missing or not an array
• Top-level scope key produced outside requestedScopes
• Invalid error entries (missing errorClass, reason, or disposition)
• omitted/degraded errors missing scope

Collection outcomes

The runner classifies each run into one of:

Outcome	Meaning
success	All requested scopes produced, no unresolved output-affecting errors.
partial	At least one scope produced, but at least one scope was omitted or degraded.
failure	No scopes successfully produced, or a fatal/protocol-level issue makes the run untrustworthy.
cancelled	Run was cancelled.

Data extraction patterns

Pattern	When to use	Example connector
API fetch via page.evaluate()	Platform has REST/JSON APIs	connectors/openai/chatgpt-playwright.js
Network capture via page.captureNetwork()	Platform uses GraphQL/XHR that fires on navigation	connectors/meta/instagram-playwright.js
DOM scraping via page.evaluate()	No API available, data only in rendered HTML	connectors/linkedin/linkedin-playwright.js

---

Building a new connector

See skills/pdp-connect/CREATE.md for the full walkthrough. Summary:

1. Scaffold: node skills/pdp-connect/scripts/scaffold.cjs <platform> [company] -- generates script, metadata, and connector-local stub schema
2. Implement: Write login + data collection logic (see CREATE.md for auth patterns, extraction strategies, and reference connectors)
3. Validate structure: node scripts/validate-connector.cjs connectors/<company>/<name>-playwright.js
4. Test: node run-connector.cjs connectors/<company>/<name>-playwright.js --inputs '{"username":"x","password":"y"}'
5. Validate output: node scripts/validate-connector.cjs connectors/<company>/<name>-playwright.js --check-result ~/.pdp-connect/desktop/last-result.json
6. Register: node skills/pdp-connect/scripts/register.cjs connectors/<company>/<name>-playwright.js -- adds entry + checksums to registry.json

---

Page API reference

The page object is available as a global in connector scripts. The runner implementation is bundled in this repo at playwright-runner/.

Method	Description
page.evaluate(jsString)	Run JS in browser context, return result
page.screenshot()	Take a JPEG screenshot, returns base64 string
page.requestInput({message, schema?})	Request data from the driver (credentials, 2FA codes, etc.)
page.goto(url, options?)	Navigate to URL
page.sleep(ms)	Wait for milliseconds
page.setData(key, value)	Send data to host ('status', 'error', 'result')
page.setProgress({phase, message, count})	Structured progress for the UI
page.showBrowser(url?)	Escalate to headed mode; returns { headed: true/false }
page.goHeadless()	Switch to headless mode (no-op if already headless)
page.promptUser(msg, checkFn, interval)	Poll checkFn until truthy
page.captureNetwork({urlPattern, bodyPattern, key})	Register a network capture
page.getCapturedResponse(key)	Get captured response or null
page.hasCapturedResponse(key)	Check if a response was captured
page.clearNetworkCaptures()	Clear all captures
page.closeBrowser()	Close browser, keep process for HTTP work
page.httpFetch(url, options?)	Node.js fetch with auto-injected cookies from the browser session

showBrowser — headed escalation

showBrowser switches the browser to headed mode for cases that require live human interaction (e.g., interactive CAPTCHAs). It returns { headed: true } on success or { headed: false } if the driver doesn't support headed mode. Connectors should check the return value and handle the fallback:

const { headed } = await page.showBrowser(url);
if (!headed) {
  // Headed not available — retry, skip, or report error
}

For normal login flows, use requestInput to ask the driver for credentials without showing a browser:

const { email, password } = await page.requestInput({
  message: 'Log in to ChatGPT',
  schema: {
    type: 'object',
    properties: {
      email: { type: 'string', format: 'email' },
      password: { type: 'string', format: 'password' }
    },
    required: ['email', 'password']
  }
});

The runner relays the request to the driver (Tauri app, agent, CLI) and resolves with the response. The schema field uses JSON Schema — the same format used by OpenAI, Anthropic, and Google for LLM tool definitions.

Progress reporting

await page.setProgress({
  phase: { step: 1, total: 3, label: 'Fetching memories' },
  message: 'Downloaded 50 of 200 items...',
  count: 50,
});

• phase.step / phase.total -- drives the step indicator ("Step 1 of 3")
• phase.label -- short label for the current phase
• message -- human-readable progress text
• count -- numeric count for progress tracking

---

Testing locally

Standalone test runner

Test connectors without any other app or repo. run-connector.cjs spawns the bundled playwright-runner/ (see Running a connector for one-time setup) as a child process and outputs JSON protocol messages.

# Run a connector (headed by default, browser visible)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js

# Colored, human-readable output
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --pretty

# Pre-supply credentials
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --inputs '{"username":"x","password":"y"}'

# Run headless (no visible browser)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --headless

# Override the initial URL
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --url https://linkedin.com/feed

# Save result to a custom path (default: ./connector-result.json)
node run-connector.cjs ./connectors/linkedin/linkedin-playwright.js --output ./my-result.json

The runner reads the connector's sibling .json metadata to resolve the connectURL. In headed mode, goHeadless() becomes a no-op so the browser stays visible throughout.

---

Contributing

Adding a new connector

1. Fork this repo
2. Create a branch: git checkout -b feat/<platform>-connector
3. Add your files in connectors/<company>/:
   • connectors/<company>/<name>-playwright.js -- connector script
   • connectors/<company>/<name>-playwright.json -- metadata
   • connectors/<company>/schemas/<platform>.<scope>.json -- data schema (required for shipped connectors)
4. Test locally using the instructions above
5. Update registry.json with your connector entry and checksums
6. Open a pull request

Modifying an existing connector

1. Fork and branch
2. Make your changes to the connector script and/or metadata
3. Test locally
4. Update the version in the metadata JSON
5. Regenerate checksums and update registry.json
6. Open a pull request

Guidelines

• Credentials stay on-device. Never send tokens or passwords to external servers.
• Use page.setProgress() to report progress during long exports.
• Include exportSummary in the result. The UI uses it to display what was collected.
• Handle errors. Use page.setData('error', message) with clear error messages.
• Prefer API fetch over DOM scraping. APIs are more stable than DOM structure.
• Avoid obfuscated CSS class names. Use structural selectors, heading text, and content heuristics.
• Rate-limit API calls. Add page.sleep() between requests.
• Test pagination edge cases -- empty results, single page, large datasets.

Registry checksums

The registry uses SHA-256 checksums to verify file integrity during OTA updates. Always regenerate checksums when modifying connector files:

shasum -a 256 connectors/<company>/<name>-playwright.js | awk '{print "sha256:" $1}'
shasum -a 256 connectors/<company>/<name>-playwright.json | awk '{print "sha256:" $1}'

---

How consumers get connectors

Any project can consume connectors from this repo as pinned dependencies. A consumer declares version constraints in a connector-dependencies.json file and resolves them with a connectors:resolve script (naming and invocation are up to the consumer's own tooling), for example:

# Example: a consumer's package.json script
npm run connectors:resolve

This fetches matching versions from the signed connectors-latest connector-index.json, verifies the detached Sigstore bundles plus the artifact/manifest/script checksums, and writes the scripts + manifests to a local snapshot directory. It's analogous to npm install.

Vana's Context Gateway and Unity Surfaces are two examples of consumers that resolve connectors this way.