Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
184 lines (128 loc) · 10.2 KB

ARCHITECTURE.md

File metadata and controls

184 lines (128 loc) · 10.2 KB
Raw

Architecture: Fetching Data from Loop & SharePoint

Overview

The tool exports Microsoft Loop workspaces to local Markdown files. It talks to two separate services using two different authentication tokens:

Service Base URL Token env var Purpose
Loop API substrate.office.com LOOP_BEARER_TOKEN Workspace & page metadata
SharePoint API *.sharepoint.com SHAREPOINT_BEARER_TOKEN Fluid snapshots (hierarchy) and page HTML content

Both tokens are validated at startup by getConfig() (in config.mts), which reads from .env via dotenv and memoizes the result.

Data flow

Loop API (v1.1)           SharePoint API (v2.1)          SharePoint API (v2.0)
───────────────           ─────────────────────          ─────────────────────
  │                              │                              │
  │  1a. /workspaces             │                              │
  │  ───────────────►            │                              │
  │  canonical workspace list    │                              │
  │  (incl. Personal workspace)  │                              │
  │                              │                              │
  │  1b. /recent                 │                              │
  │  ───────────────►            │                              │
  │  recently-active workspaces  │                              │
  │  + pages                     │                              │
  │                              │                              │
  │  1c. /deltasync              │                              │
  │  ───────────────►            │                              │
  │  full component graph        │                              │
  │  + pages                     │                              │
  │                              │                              │
  │  ── merge & deduplicate ──   │                              │
  │                              │                              │
  │                     2. opStream/snapshots                   │
  │                     ────────────────────►                   │
  │                     Fluid snapshot (JSON)                   │
  │                     → page hierarchy tree                   │
  │                                                             │
  │                                          3. items/{id}/content
  │                                          ─────────────────────►
  │                                          Page HTML (per page)
  │                                          → converted to Markdown

Step 1 — Loop workspace & page discovery

No single Loop API endpoint returns all workspaces and pages. The tool queries three endpoints and merges the results, deduplicating by id:

1a. /workspaces — canonical workspace list

Endpoint: GET https://substrate.office.com/recommended/api/v1.1/loop/workspaces?rs=en-us

Returns the user's workspace list. This is the only endpoint that reliably includes the Personal workspace (titled "My workspace" by the API). It returns workspace metadata only — no pages.

1b. /recent — recently-active workspaces + pages

Endpoint: GET https://substrate.office.com/recommended/api/v1.1/loop/recent?top=30&settings=true&rs=en-us

Returns workspaces and pages ordered by recent activity. Newly created workspaces typically appear here first. Capped at 30 items per request.

1c. /deltasync — full component graph

Endpoint: GET https://substrate.office.com/recommended/api/v1.1/loop/deltasync?loopComponents=true&rs=en-us

Returns the full workspace + page + component graph. May include workspaces not touched recently. This is where the bulk of the page metadata comes from.

Merge logic

All three responses share the same LoopData shape:

  • workspaces[] — each with an id, title, and mfs_info.pod_id (base64-encoded pointer to the SharePoint backing store)
  • pages[] — each with an id, title, type, workspace_id, is_deleted, onedrive_info.drive_id, and sharepoint_info.site_url
  • activities[] — not used by this tool

Auth: Standard Authorization: Bearer {token} header.

Pagination: Each response may include next_page_link (a query string). The tool follows these links, merging and deduplicating results by id, until no more links are returned.

Key detail: This step gives us page metadata but not the page content or hierarchy (folder structure). Those come from SharePoint. Workspaces without mfs_info.pod_id are filtered out with a warning (they are typically empty or not yet synced to SharePoint).

Step 2 — Fluid snapshot (hierarchy)

Each Loop workspace is backed by a Fluid Framework container stored in SharePoint. The workspace's mfs_info.pod_id is a base64-encoded string of the format …|{host}|{driveId}|{itemId} that points to this container.

Endpoint: GET https://{host}/_api/v2.1/drives/{driveId}/items/{itemId}/opStream/snapshots/trees/latest?ump=1

Auth: SharePoint's multipart POST convention (see below).

What it returns: A JSON snapshot of the Fluid container, containing an array of blobs with base64-encoded content.

Extracting the hierarchy from the snapshot

  1. Find the SharedTree blob — scan the blobs for one whose decoded JSON has both editHistory and internedStrings properties.

  2. Find Build 0 — walk editHistory.editChunks[*].chunk[*].changes[*] looking for a change with type: 5 (a Build change) whose source array has more than one node. This is the initial state of the workspace tree.

  3. Extract page metadata — Build 0's source array contains nodes. Each node with a LoopPage label has traits:

    • displayText → page title
    • icon → emoji (e.g. { type: "emoji", data: "🚀" })
    • odspMetadata → { itemId } (the SharePoint item ID, used to match against Loop page IDs)

  4. Extract the tree — The first node in the source array is a LoopWorkspace node. Its values trait contains the ordered list of top-level pages. Each page entry may recursively contain child pages in its own values trait, forming the full hierarchy.

  5. Flatten — The tree is flattened into a list of { pageId, title, emoji, spoItemId, path } entries where path is a slugified filesystem path like design/components/button-styles.

Step 3 — Page HTML export

Pages are exported from two sources, processed in order:

Primary — Loop API pages: every non-deleted, non-shared page returned by Step 1 that belongs to the selected workspace.

Secondary — orphan hierarchy pages: pages present in the Fluid hierarchy (Step 2) but absent from the Loop API (e.g. due to API pagination limits or sync lag). These are fetched using host + driveId inferred from a known page in the same workspace, falling back to the workspace pod_id coordinates if no known pages exist.

Endpoint: GET https://{spHost}/_api/v2.0/drives/{driveId}/items/{itemId}/content?format=html&ump=1

  • spHost is derived from the page's sharepoint_info.site_url (primary), or from the workspace pod_id host (orphans)
  • driveId comes from onedrive_info.drive_id (primary), or from the workspace pod_id (orphans)
  • itemId is extracted from the Loop page ID (the segment after the last _) for primary pages, or directly from the hierarchy's odspMetadata.itemId for orphans

Auth: Same SharePoint multipart POST convention.

What it returns: The full page content as HTML.

The HTML is then converted to Markdown using Turndown and written to disk at the path determined by the hierarchy from Step 2.

SharePoint authentication

SharePoint endpoints don't accept a normal Authorization header from browser-origin requests. Instead, the tool uses a multipart form POST that smuggles the credentials:

POST {url}
Content-Type: multipart/form-data;boundary={uuid}
Origin: https://loop.cloud.microsoft
Referer: https://loop.cloud.microsoft/

--{uuid}
Authorization: Bearer {token}
X-HTTP-Method-Override: GET
_post: 1

--{uuid}--

This is the same mechanism the Loop web client uses. The X-HTTP-Method-Override: GET header inside the multipart body tells SharePoint to treat this POST as a GET. Both the snapshot (v2.1) and content (v2.0) endpoints use this pattern.

Module structure

src/
├── main.mts              CLI entry point, arg parsing, orchestration
└── lib/
    ├── types.mts          Shared interfaces (Workspace, LoopPage, LoopData, …)
    ├── config.mts         Token validation & memoization (getConfig)
    ├── sharepoint.mts     SP multipart auth helper (spGet) with retry + backoff
    ├── loop-api.mts       Step 1: /workspaces + /recent + /deltasync with merge
    ├── hierarchy.mts      Step 2: Fluid snapshot → page tree → flat entries
    └── export.mts         Step 3: HTML fetch → Turndown → .md files

Output directory

All exports write to workspaces/, with each workspace in its own slugified subdirectory:

workspaces/{slug}/     ← single workspace
workspaces/*/          ← --all or picker "0"

A workspace directory is cleared before re-export only if it already exists (re-export) or if --all / "Export all" was used. A first-time export to a new directory skips the clear.

CLI flags

Flag Description
-w, --workspace NAME Select workspace by name or ID (also accepts "Personal workspace")
-a, --all Export all workspaces, each into workspaces/{slug}/
-p, --pick-workspace Interactive numbered picker (option 0 exports all workspaces)
--page TITLE Export a single page by substring match
--dump-html Save raw HTML alongside markdown
-d, --delay MS Delay between page requests (default: 50)
-n, --dry-run Show what would be exported without fetching
-h, --help Show help