diff --git a/AGENTS.md b/AGENTS.md index dd7aca42..90c6d62f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,8 +75,10 @@ out on the system. ## Misc rules -- Git committing and pushing is for humans, not agents. +- Git commits and pushes are for humans, not agents. - No blank lines in functions. +- API endpoint functions should start with their REST verbs, + e.g., `post_something` or `get_something`. - Search inputs should always be clearable. - Changes to the UI state, e.g., a selected tab or a modal open, should typically be part of query params so a link will show a similar state. diff --git a/LATEX_EDITOR_PLAN.md b/LATEX_EDITOR_PLAN.md new file mode 100644 index 00000000..cf13fabe --- /dev/null +++ b/LATEX_EDITOR_PLAN.md @@ -0,0 +1,879 @@ +# LaTeX Editor ("Overleaf replacement") — Implementation Plan + +## Goal + +Let a user open a publication on the publications page, click **Edit**, and get a +full-screen, closable LaTeX editor — an "app within Calkit" — where: + +- The `.tex` source is edited in a code editor (Overleaf-like split view). +- Compilation to PDF happens **client-side in WebAssembly** (no compile server). +- Changes flow back into the project's git repo. We start with **auto-git-commit** + (matching the existing `PUT contents` behavior) and later move to an + **editing-session = branch** model that squashes into a single project commit. +- The feel is collaborative, but the styling is our own (Chakra UI), not an Overleaf clone. +- **Onboarding is as easy as Overleaf**: sign up with Google / email / university SSO, join + a project via a shareable link, and start editing **with no GitHub account** — edits still + land as real commits in the project repo. This requires decoupling identity from GitHub + (see §2) and is as important to the goal as the editor itself. + +We want to reuse compilation code from [TeXlyre](https://github.com/TeXlyre/texlyre) +where it makes sense, but with eyes open about licensing (see below). Calkit is open source +(MIT), which shapes both the licensing and self-hosting choices below. + +--- + +## Decisions log + +Settled during planning (see referenced sections for rationale): + +| Topic | Decision | Ref | +|---|---|---| +| **License** | Path 1 — our own loader around the WASM binaries; copy no TeXlyre source | §0 | +| **TeX engine** | Upstream **busytex/busytex** (MIT, TeX Live 2023 + SyncTeX). The TeXlyre TeX Live 2026 build is AGPL — rejected for Path 1. | §0, Phase 0 | +| **Compile role** | Preview-only; never a pipeline artifact; pipeline stays source of truth | §3.1 | +| **Preview latency** | Phase 1–2: BusyTeX ~0.4 s full recompile (warm + debounce + keep last PDF). Instant-preview later via **WASM linear-memory snapshotting** (client-side analog of TeXpresso's `fork()`); server-side TeXpresso only as opt-in fallback. TeXpresso itself is native-only (no WASM). | §3.2 | +| **Preview download** | None — preview is view-only in the editor | §3.1 | +| **Git hosting** | GitHub-backed; git-backend abstraction added up front; self-host deferred | §2.4, I4 | +| **Push credential** | Done — existing Calkit GitHub App installation; only authorship routing remains | §2.2 | +| **Onboarding (near-term)** | Google sign-in + email/password signup via invite links (pick password on first sign-in) | §2.3 | +| **GitHub-less users** | Can be **collaborators**, but **cannot own/create projects** until git hosting is decoupled (I4). Owners must have a linked GitHub account. | §2.2, I1 | +| **University SSO** | Deferred | I3 | +| **Sequencing** | I1 + I2 land before/with editor Phase 1; Phase 0 spike runs in parallel | §6 | +| **Phase 3 order** | 3a (real-time collaboration) first, then 3b (sessions-as-branches) | Phase 3 | +| **TeX Live packages** | Upstream server for Phase 0/1; self-hosted cached proxy in Phase 2 | Phase 2 | + +**Open verification tasks (not decisions):** confirm BusyTeX + SwiftLaTeX engine `.wasm` +artifact licenses fit Path 1 redistribution before Phase 1 ships. + +The near-term critical path: **Phase 0 compile spike (BusyTeX)** in parallel with **I1** +(Google/email signup + GitHub-less authorship) and **I2** (native membership + invite +links), then **editor Phase 1** (single-file edit → preview → auto-commit). The detailed +task breakdown for that work is in §8. + +--- + +## 0. The licensing decision (must resolve before writing code) + +This is the single most important gate on the plan. + +| Project | License | Implication | +|---|---|---| +| **calkit-cloud** | **MIT** (`LICENSE`) | Permissive; what we ship today | +| **TeXlyre** | **AGPL-3.0** | Network-copyleft. Linking/integrating its code into our hosted web app would arguably obligate us to release Calkit's source under AGPL. | +| **SwiftLaTeX** (engine TeXlyre uses) | main repo **AGPL-3.0**; engine wrapper files dual **EPL-2.0 / GPL-2.0 w/ Classpath exception** | The on-disk `.wasm` engines derive from TeX Live (mostly permissive/LPPL), but SwiftLaTeX's own loader/wrapper code is copyleft. | + +**Why this matters:** AGPL-3.0 is incompatible with keeping calkit-cloud MIT if we copy +their source into our bundle — and since **Calkit is itself open source (MIT)**, pulling +AGPL code into the tree would force the whole project (or at least the editor) to relicense. +That reinforces Path 1 below (treat the engine as an arms-length binary dependency, write +our own loader). We have three realistic paths: + +1. **Clean-room reuse of the WASM engine binaries only.** Treat the compiled SwiftLaTeX + (`pdftex`/`xetex`) `.wasm` artifacts as a black-box dependency loaded in a Web Worker, + and write *our own* thin TypeScript loader/bridge (do **not** copy TeXlyre's React/TS + source). This is the cleanest separation but still needs the engine's own license + (EPL-2.0/GPL-2.0-classpath) verified as acceptable for redistribution. **Recommended + starting assumption**, pending a real license review. +2. **Use a permissively-licensed engine instead.** Evaluate alternatives whose licensing is + friendlier (e.g. engines distributed under MIT/Apache, or texlive.net-style remote + compile as a fallback). Trade-off: less mature browser story than SwiftLaTeX/BusyTeX. +3. **Accept AGPL for an isolated, separately-licensed sub-package.** Ship the editor as a + distinct AGPL module/micro-frontend with its own LICENSE, loaded at runtime. Legally + fragile for a hosted SaaS; only with counsel sign-off. + +> **DECIDED: Path 1.** We write our own loader/bridge around the WASM binaries and copy +> **no** TeXlyre React/TS source. +> +> **Engine license verification — DONE, and it has teeth (2026-06-17):** +> - `texlyre-busytex` (npm, TeX Live **2026**) is **AGPL-3.0-or-later** — does **not** fit +> Path 1 for an MIT project. Its AGPL covers the TS wrapper + build tooling. +> - Upstream `busytex/busytex` is **MIT** (code/scripts); its published `.wasm`/`.data` +> binaries carry TeX Live/LPPL (permissive) licenses — Path-1-clean — but bundle TeX Live +> **2023**, not 2026. (No npm package; GitHub-releases only.) +> - **Implication:** the license-clean engine is **busytex 2023**, not the TeXlyre 2026 +> build. See the revised engine decision below / in the Decisions log. + +A second gotcha that rides along with the engine choice: **SwiftLaTeX fetches TeX Live +packages on demand from a remote package server at compile time.** We must either +(a) point at SwiftLaTeX's public package server, (b) host our own package repository, or +(c) bundle a fixed TeX Live subset. For reproducibility and uptime we'll likely want our +own cached package endpoint eventually (see Phase 2, "TeX Live package proxy"). + +--- + +## 1. How this fits the existing codebase + +Grounded in the current architecture (researched, not assumed): + +### Frontend (`frontend/`) +- React 18 + TypeScript + Vite, **Chakra UI** components, **TanStack Router** (file-based) + + **TanStack Query**, auto-generated OpenAPI client in `src/client/`. +- Publications live at + `src/routes/_layout/$accountName/$projectName/_layout/publications.tsx` with components in + `src/components/Publications/` (`PublicationView.tsx`, `NewPublication.tsx`, + `ImportOverleaf.tsx`, `PdfAnnotator.tsx`). +- Full-screen modal pattern already exists (Chakra `Modal` with `size="full"`); see + `ArtifactCompareModal.tsx` / `FileViewModal.tsx` for large-modal precedent. +- **No editor/CRDT deps yet** — `codemirror`, `yjs`, `swiftlatex` are all net-new. +- File I/O today goes through the OpenAPI client: `getProjectContents()` (base64 or signed + URL per file), `putProjectContents()` (multipart upload → backend commits & pushes), + `postProjectFsBatchOp()` (batch file ops). History via `getProjectHistory()` / + `getProjectFileHistory()`; refs via `searchProjectRefs()`. + +### Backend (`backend/app/`) +- FastAPI + SQLModel + Postgres; **GitPython** for repo ops (`app/git.py`), DVC integration + in `app/dvc.py`, project/file logic in `app/projects.py` and + `app/api/routes/projects/core.py`. +- Repos are cloned per-user under `/tmp/{github_username}/{owner}/{project}/repo/`, guarded + by `FileLock`. `PUT contents` already does: write file → `git add` → `git commit` → + `git push origin ` (max 1 MB/file). +- **Publications** are entries in `calkit.yaml` (`Publication` model in + `app/models/core.py`): `path`, `title`, `type`, optional DVC `stage`, `storage` + (`git`/`dvc`/`dvc-zip`), and optional `overleaf` sync config. The PDF is usually a DVC + output of a pipeline stage; the `.tex`/`.bib` sources are typically git-tracked. +- Branch support is **read-only today**: refs can be listed and read without checkout, the + working tree always reflects the default branch, and there are **no branch + create/switch/merge endpoints** yet. This is the main backend gap for the + session-as-branch phase. +- Permissions: `get_project()` resolves Read < Write < Admin < Owner. Editing requires + **Write**. + +### What we can reuse vs. build +- **Reuse:** Chakra full-screen modal pattern, OpenAPI file endpoints for the auto-commit + MVP, existing Overleaf sync as a sibling feature, DVC URL resolution for figure assets. +- **Build:** CodeMirror-based editor, WASM compile worker + our loader, virtual filesystem + bridge (repo files ↔ engine FS), and (later) Yjs collaboration + branch/session backend + endpoints. + +--- + +## 2. Identity, onboarding & git hosting + +An Overleaf-grade editor is only as good as its onboarding. The requirement is: a new user +should be able to **sign up with Google / email / their university SSO, click a share link, +and start editing — with no GitHub account** — and their edits must still land as real +commits in the project's repo. This collides head-on with how Calkit works today, so it's a +first-class part of this plan, not an afterthought. + +### 2.1 The starting reality (researched) + +Calkit is currently **deeply GitHub-coupled**: + +- **Login is GitHub-only** in practice. Email/password infra exists (`UserRegister`/ + `UserCreate`, bcrypt, JWT reset tokens) but `POST /users/signup` is intentionally + **disabled (501)**. A `google-auth.tsx` callback route already exists on the frontend. +- **Every `Account` requires a non-null `github_name`** (`app/models/core.py`), used to + derive `github_username`, default repo URLs, and API calls. +- **Every `Project` requires a `git_repo_url` on github.com** (`Project.git_repo_url` is + non-nullable and validated to github.com; the repo is created via the GitHub API at + project-creation time). +- **Write access is derived from GitHub**: `UserProjectAccess` is a *cache* of + `GET /repos/{owner}/{repo}/collaborators/{user}/permission`. There is **no native + collaborators table, and no invite/share-link mechanism** today. +- **Pushes use the requesting user's own GitHub token** via the credential helper in + `app/git.py`; a user with no GitHub token gets a 401 and cannot push. + +Useful foundations already in place to build on: the multi-provider +`UserExternalCredential` table (GitHub/Zenodo/Overleaf/**Google**), bcrypt password support, +refresh tokens, an `Account` abstraction already separate from `User`, and a native +`UserOrgMembership` table (proof we can do native membership without GitHub). + +### 2.2 The key decoupling: two identities in every commit + +The conceptual unlock is separating the two identities bundled into a git push today: + +1. **Authorship** — the `Author:` on the commit (name + email). Costs nothing, needs **no + GitHub account**. A browser editor can author commits as any signed-in Calkit user. +2. **Push credential** — write access to the *remote*. This is the only part that needs a + real token. + +Today both come from one person's GitHub token. If we split them, a GitHub-less contributor +can author commits that are **pushed under a project-level credential**. + +**DECIDED — and the push side is already built.** Calkit already has a **GitHub App +installation** that supplies push access for users with write permission, so the +project-level push credential exists; we don't need to build it. A short-lived, repo-scoped +installation token pushes the commit while the commit is *authored* by the real contributor. + +What remains (in I1/I2) is **not** the credential — it's wiring a **GitHub-less Calkit +identity's authorship** (name + verified email) through that existing App push path, so a +contributor who joined via a share link and has no GitHub token still produces a properly +attributed commit that the App pushes. + +### 2.3 Recommended direction: decouple identity, keep GitHub as the git backend (for now) + +This delivers the full onboarding requirement **without** taking on the risk of self-hosting +git. Changes, roughly in dependency order: + +1. **Turn on real onboarding (the near-term priority).** + - Make creating a Calkit Cloud account dead simple: **"Sign in with Google"** (callback + already stubbed) and **plain email/password signup** (infra exists, just re-enable). + - **Invite-link-driven signup is the primary path:** a project share link lands a new + user on a signup screen where they either continue with Google or **pick a password on + first sign-in**, and are dropped straight into the project. This ties §2.3.3's invite + links to the account-creation flow so onboarding is one continuous motion. + - **University SSO (SAML/OIDC) is deferred** (see I3) — not needed for the initial goal. + When we do it, lean toward buying a broker over hand-rolling SAML (§2.5 caveat). +2. **Make `github_name` optional** on `Account`; mint a Calkit account `name` independent of + GitHub. GitHub becomes *one linkable identity/credential*, not the root of identity. +3. **Native membership + invites.** Add a `ProjectMembership` table (role per user) and a + `ProjectInvitation` / **shareable join-link** table (token, role, expiry, max-uses). + Project access resolves from native membership **first**, with GitHub-collaborator sync + kept as one contributing source. This is what makes "click a link → start editing" + possible for non-GitHub users. +4. **Push via the existing GitHub App installation** (§2.2 — already built). The editor + commits with the contributor's authorship and pushes via the App's installation token; + the only new work is routing a GitHub-less contributor's authorship through that path. +5. **Decouple `git_repo_url` from github.com** behind a small **git-backend abstraction** + (see §2.4) so we're not hard-wired even while GitHub remains the only backend we ship + first. + +Net effect: a student signs in with Google, opens a share link, edits the `.tex` in the WASM +editor, and their commits push to the project's GitHub repo authored as them — no GitHub +account, no friction. + +### 2.4 The bigger bet: self-host git, optionally mirror to GitHub + +You raised hosting repos ourselves. It's attractive (truly breaks the GitHub dependence; no +contributor ever needs GitHub) but it's a large, separable bet: + +- **How:** `dulwich` is *already a dependency* and can serve git smart-HTTP; alternatively + run Gitea/Forgejo. Store repos on our infra (object storage + metadata), optional **push + mirror to GitHub** so GitHub-native users keep their workflow. +- **Costs/risks you flagged are real:** scaling git hosting (packfiles, storage, the + DVC/large-file interplay), migrating existing github.com-backed projects, ops/on-call + burden, and spooking users who *want* their work on GitHub. +- **DECIDED:** don't self-host now. Introduce the **git-backend abstraction (§2.3.5) up + front** so `Project` isn't welded to github.com, ship the **GitHub-backed** implementation + first, and keep **self-hosted git as a pluggable backend** (I4) we can enable later or + per-deployment without re-architecting. + +### 2.5 Open-source implications + +- Calkit being **MIT/open source** reinforces the §0 licensing stance (no AGPL in-tree). +- Cloud-only onboarding features (paid SSO broker, GitHub App credentials, hosted SAML) + must **degrade gracefully in self-hosted OSS builds** — gate them behind config so a + community deployment still works with plain email/password (and, ironically, self-hosted + git is the *most* OSS-friendly backend since it needs no github.com at all). + +### 2.6 New identity workstream (phases) + +These are largely independent of the editor's compile work but **gate its collaborative +value** — non-GitHub contributors can't meaningfully use the editor until I1–I2 land. + +- **I1 — Onboarding & push decoupling:** Google + email signup; `github_name` optional; + route GitHub-less contributor authorship through the **existing** GitHub App push path. + *(Enables: non-GitHub user edits → commit authored as them → pushed by the App.)* +- **I2 — Native membership & share links:** `ProjectMembership` + `ProjectInvitation` + (join links); access checks resolve natively first. *(Enables: "click link → start + editing.")* +- **I3 — University SSO (deferred):** SAML/OIDC, lean toward a broker. Not required for the + initial launch; revisit after I1/I2 prove out the Google + email + invite-link flow. +- **I4 — (optional, bigger) Self-hosted git:** git-backend abstraction + dulwich/Gitea + backend + optional GitHub mirror. + +> **Sequencing — DECIDED:** **I1 + I2 land alongside or before editor Phase 1.** The first +> editor release must be usable by GitHub-less contributors (Google/email signup + invite +> links), since that's the audience the whole feature targets. Editor Phase 0 (the compile +> spike) can proceed in parallel with I1/I2, but Phase 1 does not ship without them. + +--- + +## 3. Target architecture + +``` +┌───────────────────────────────────────────────────────────────────────┐ +│ Publications page ──[Edit]──▶ (Chakra size=full) │ +│ │ +│ ┌─────────────┬───────────────────────┬──────────────────────────┐ │ +│ │ File tree │ CodeMirror editor │ PDF preview (pdf.js) │ │ +│ │ (.tex/.bib │ (LaTeX mode, errors) │ + log / SyncTeX jumps │ │ +│ │ figures) │ │ │ │ +│ └─────────────┴───────────────────────┴──────────────────────────┘ │ +│ │ │ ▲ │ +│ │ ▼ │ │ +│ │ ┌──────────────────────┐ compiled PDF + log │ +│ │ │ Compile Web Worker │───────────┘ │ +│ │ │ (SwiftLaTeX WASM + │ │ +│ │ │ our TS loader) │ │ +│ │ └──────────┬───────────┘ │ +│ │ │ reads/writes │ +│ ▼ ▼ │ +│ ┌──────────────────────────────────────┐ │ +│ │ Virtual FS (in-memory / IndexedDB) │ ◀── seeded from repo │ +│ └──────────────────────────────────────┘ via getProjectContents │ +│ │ │ +│ ▼ save (debounced / on close) │ +│ OpenAPI client ── putProjectContents / fsBatchOp ──▶ backend │ +│ backend: git add/commit/push (auto-commit MVP) │ +└───────────────────────────────────────────────────────────────────────┘ +``` + +Three layers, each independently testable: + +1. **Editor UI** — modal, file tree, CodeMirror, PDF preview. Pure frontend. +2. **Compile core** — Web Worker hosting the WASM engine + a virtual filesystem. No + network at compile time except the TeX Live package fetch. +3. **Persistence** — virtual FS ⇄ project repo. MVP = per-file auto-commit through existing + endpoints; later = session branch + squash. + +### 3.1 Compilation is preview-only — provenance lives in the pipeline + +A load-bearing principle for the whole feature: **the WASM compile exists to support editing +the writing, not to produce artifacts.** A PDF compiled in the browser is a disposable +preview, **never a valid pipeline output**, and carries no provenance. + +- **Source of truth stays the pipeline.** The official, citable PDF is produced by the + project's DVC/pipeline stage and cached as it is today. The editor never writes a PDF into + the repo, DVC, or object storage, and never updates a publication's canonical artifact. +- **Preview PDFs are ephemeral.** They live in the browser (worker memory / IndexedDB cache) + for the editing session and are thrown away. Nothing server-side persists them. +- **No download (DECIDED).** The draft preview is **view-only in the editor** — no download + button. This is the strongest guard against a dirty working copy masquerading as "the + paper." Users who need a file run the pipeline, which produces the official, provenance- + tracked artifact. (Revisit only if there's real demand; if ever added, it would be a + clearly-labeled, commit-named "dirty" file.) +- **UI framing.** The editor surfaces the preview as a "draft preview," visually distinct + from the published artifact shown on the publications page. Regenerating the *official* + PDF remains a pipeline run, never an editor action. + +**Future step (out of scope here): provenance-perfect builds with user compute.** Eventually +we may let a user attach their own compute to a Calkit project and run the real build there +via `calkit`, so an "official" compile is reproducible and fully tracked — the pipeline +produces and caches it, exactly as the canonical path does now. That would be the *only* +sanctioned way to promote a compiled PDF to a real artifact; the in-browser WASM path stays +preview-only regardless. + +### 3.2 Preview latency & TeXpresso (evaluated) + +BusyTeX gives a **full** recompile in ~0.4 s (measured, §8.1) — fine for a manual/debounced +"compile" button, but not the keystroke-live feel of [TeXpresso](https://github.com/let-def/texpresso). +Pete flagged TeXpresso as the bar for fast previews; here's the assessment. + +**What TeXpresso is.** MIT-licensed, the best live-LaTeX UX available: edit → the render +updates almost instantly, with immediate inline errors. Stack = a **custom XeTeX** (Tectonic +fork) + native **MuPDF** rendering + **libSDL** viewer + a C driver. + +**Why it's fast — and why it doesn't port.** The "live magic" is *"the cute hack of forking +the process near the part of the input being edited"*: it `fork()`s the TeX engine at +checkpoints, so on an edit it rolls back to the nearest checkpoint and replays only the +changed tail. That's **Unix-`fork()`-only — native Linux/macOS, no Windows, no browser, no +WASM.** The author's own roadmap is WSL → "WebAssembly with a capable VM" → *implement +snapshotting inside the XeTeX engine* — and he states he lacks capacity for that engine work. +So there is **nothing to adopt off-the-shelf for a browser/WASM editor**, and rendering is +native (MuPDF/SDL), not canvas/pdf.js. Adopting TeXpresso as-is would mean a **server-side** +preview service — reintroducing the per-session compute we chose WASM to avoid. + +**The portable idea: WASM linear-memory snapshotting.** TeXpresso's `fork()` is really +"snapshot the engine's mutable state, restore it later." In a single-threaded WASM module, +that state *is* its **linear memory** (+ globals/table). So the browser-native analog is: +snapshot BusyTeX's linear memory at checkpoints (e.g. per page/paragraph boundary), and on +edit `restore nearest checkpoint → replay the tail`. Memory copies of tens of MB are +millisecond-cheap. This is exactly the author's "long-term" engine-snapshotting goal, done at +the **WASM-memory level** instead of in XeTeX source — and it keeps everything **client-side, +zero-infra**. It's real engineering (checkpoint placement, determinism, output diffing) but +architecturally aligned with our decisions. + +**Decision / sequencing.** +- **Phase 1–2 (now):** ship BusyTeX full recompile (~0.4 s) made to *feel* fast — keep the + engine warm (already preloaded), debounce, and keep the last good PDF on screen while + recompiling. No new infra. +- **Instant-preview upgrade (post-Phase 2):** prototype **WASM linear-memory snapshotting** + on BusyTeX as the client-side path to TeXpresso-class latency. Fallback if that proves too + hard: an **opt-in server-side TeXpresso** fast-preview tier (still preview-only — §3.1 + provenance unaffected; default stays WASM). +- Re-evaluate after Phase 2 with real editing-latency feedback before investing. + +--- + +## 4. Phased delivery + +### Phase 0 — Spike & decisions (no production code) +- Resolve the **licensing path** (§0) — blocker for everything else. +- Stand up a throwaway spike: load the **upstream MIT busytex** WASM (TeX Live 2023, SyncTeX + support) in a Web Worker, compile a hello-world `.tex` to PDF entirely in the browser, + render with pdf.js. Confirm bundle size, cold-start time, and where TeX Live packages come + from. (Engine license already verified — §0: MIT busytex is Path-1-clean; the TeXlyre 2026 + build is AGPL and rejected.) +- Decide editor lib (**CodeMirror 6** recommended — it's what TeXlyre uses, lighter than + Monaco, good LaTeX support, and the same core we'd need for Yjs later via `y-codemirror`). +- **Exit criteria:** a documented yes/no on engine + license, and a measured compile of a + real publication `.tex` from one of our projects. + +### Phase 1 — Single-user editor MVP (auto-commit) +Scope: one publication, its primary `.tex`, edit + compile + preview + save. + +**Frontend** +- `Edit` button on the publications page (gated on Write permission) opens + `LatexEditorModal` (Chakra `Modal size="full"`, closable, with unsaved-changes guard). +- `LatexEditor` component: CodeMirror (LaTeX syntax + error squiggles) | PDF preview pane. +- On open: fetch the publication's source file(s) via `getProjectContents()` into the + virtual FS. Initially handle the single `.tex` and any sibling `.bib`. +- "Compile" (manual + debounced auto) posts the FS to the compile worker, renders the + returned PDF, surfaces the log/errors in a collapsible panel. +- "Save": write changed files back via `putProjectContents()` (per-file). Auto-commit on + the backend gives us versioning for free. Debounce + save-on-close. + +**Backend** +- Likely **no new endpoints** for the MVP — reuse `PUT contents`. Possible small additions: + raise/relax the 1 MB limit awareness for `.tex`, and confirm `getProjectContents` returns + raw text suitably for editing. + +**Out of scope for Phase 1:** multi-file projects with `\input`, DVC figures, collaboration, +branches. + +**Exit criteria:** edit a real paper's `.tex`, compile to PDF in-browser, save, and see the +auto-commit land on the default branch with a push to GitHub. + +### Phase 2 — Real projects: multi-file, figures, bib, SyncTeX +- **Virtual FS seeding for a whole publication directory**: resolve all dependencies + (`\input`/`\include`, `\bibliography`, `\includegraphics`). Pull git-tracked sources as + text and **DVC/large binary figures via their signed URLs** (`getProjectContents` already + returns `url` for DVC-stored files; figures may be pipeline outputs). +- **File tree** panel (read + edit text files; figures shown read-only). +- **bibtex/biber + multi-pass** compile orchestration in the worker. +- **SyncTeX** forward/inverse search (click PDF ↔ jump to source) — BusyTeX build supports + SyncTeX; factor into engine choice. +- **TeX Live package proxy** (optional but recommended): a backend route that caches the + packages the engine requests, so compiles are reproducible and don't depend on a + third-party package server. +- **Batch save** via `postProjectFsBatchOp()` to commit multiple changed files in one + commit instead of N. + +**Exit criteria:** a multi-file paper with figures and a `.bib` compiles to the same PDF +the pipeline produces (or close enough), with figures resolved. + +### Phase 3 — Collaboration and/or editing sessions + +Two related but separable upgrades. **Decided order: 3a (real-time collaboration) first**, +then 3b (sessions-as-branches) — collaboration delivers the headline Overleaf feel soonest +and reuses the Phase 1 auto-commit model unchanged. + +**3a. Real-time collaboration (Yjs)** +- Add `yjs` + `y-codemirror.next`. Shared doc, live cursors/selections. +- Needs a sync transport: a **WebSocket relay** (`y-websocket`, server-authoritative — fits + our hosted model better than TeXlyre's P2P WebRTC) or WebRTC w/ a signaling server. + Server-authoritative is the recommended fit for Calkit since we already have a backend. +- Presence/awareness UI in Chakra. Persistence of the live doc (Redis/Postgres or a doc + service) is the main new infra. + +**3b. Editing sessions = branches (squash on finish)** +This is the bigger backend lift because branch *writes* don't exist yet. +- New backend capability in `app/git.py` / projects routes: + - Start session → create branch `editor-session/` from default, check it out in the + per-user repo clone. + - Commit edits to the session branch (auto-commit, frequent, cheap). + - Finish session → **squash-merge** the branch into the default branch as a single, + well-described commit; delete the session branch. Handle conflicts/abort. + - Discard session → delete branch, no merge. +- Data model: a `EditingSession` (or reuse `FileLock`-style table) tracking branch name, + owner, publication path, status, base commit. +- Frontend: session lifecycle UI (start/resume/finish/discard), "draft" vs "published" + state, and a diff/review of the squash before it merges (we already have + `ArtifactCompareModal` + `react-diff-viewer` to lean on). +- Interaction with the current "working tree = default branch" assumption and the per-user + `/tmp` clone model needs care — concurrent sessions and the existing `PUT contents` + auto-commit path must not stomp each other. + +**Exit criteria (3b):** start a session, make several edits/compiles, finish → exactly one +clean commit on the default branch; discard → no trace. + +--- + +## 5. Key technical risks & open questions + +1. **License (§0)** — gating. Resolve first. +2. **TeX Live package delivery** — on-demand fetch vs. self-hosted proxy vs. bundled subset. + Affects reproducibility, offline, and cold-start latency. +3. **Bundle size / cold start** — WASM engines are large (tens of MB). Lazy-load the worker + only when the editor opens; cache aggressively (IndexedDB / service worker). +4. **Figure & big-file handling** — DVC outputs are large and may be pipeline-generated. + Pull read-only via signed URLs; don't try to round-trip them through the editor. +5. **Fidelity vs. the pipeline build** — in-browser compile may differ from the project's + canonical (Docker/DVC stage) build. This is acceptable *because* the WASM compile is + preview-only and the pipeline stays source of truth (see §3.1) — but the UI must make the + draft-vs-published distinction obvious so the difference never causes confusion. +6. **Branch-write model (Phase 3b)** — net-new backend surface; concurrency with the + existing per-user clone + auto-commit path is the trickiest part. +7. **Concurrent edits before Yjs** — Phase 1/2 are single-writer; reuse `FileLock` to avoid + two users (or the editor + Overleaf sync) clobbering each other. +8. **Relationship to existing Overleaf import/sync** — is this editor a *replacement* for + that flow, or complementary? Affects whether we keep `ImportOverleaf` prominent. +9. **Identity decoupling (§2)** — making `github_name` optional and resolving access from a + native membership table touches core auth; risk of regressing existing GitHub-derived + permissions. Needs careful migration + keeping GitHub-collaborator sync working. +10. **Push attribution & abuse** — pushing GitHub-less contributors' commits under a GitHub + App token means commit *authorship* is only as trustworthy as our auth; verify emails, + and rate-limit/scope join-link roles to avoid a share link becoming a write-access leak. +11. **SSO build-vs-buy & OSS degradation (§2.5)** — a paid SSO broker is the pragmatic path + for university IdPs but must not become a hard dependency for self-hosted OSS builds. + +--- + +## 6. Rough sequencing / sizing + +Two interleaved tracks — **Editor** (compile/UX) and **Identity** (onboarding/git): + +| Phase | Track | Outcome | Relative size | +|---|---|---|---| +| 0 | Editor | License decision + WASM compile spike | Small (but blocking) | +| I1 | Identity | Google + email signup; `github_name` optional; GitHub App push credential | Medium | +| I2 | Identity | Native `ProjectMembership` + shareable join links; native access checks | Medium | +| 1 | Editor | Single-file editor modal, compile, auto-commit save | Medium | +| 2 | Editor | Multi-file, figures, bib, SyncTeX, batch save, package proxy | Large | +| 3a | Editor | Real-time collaboration (Yjs + WS relay) | Large | +| 3b | Editor | Editing sessions as branches w/ squash-merge | Large (backend-heavy) | +| I3 | Identity | (Deferred) University SSO (SAML/OIDC via broker) | Medium | +| I4 | Identity | (Optional) Self-hosted git backend + GitHub mirror | Large (infra-heavy) | + +**Decided ordering:** Editor Phase 0 (compile spike) runs in parallel with **I1 + I2, which +land before/with editor Phase 1** — the first editor release must serve GitHub-less +contributors. Editor phases 1–2 then deliver "edit & preview in the browser, versioned in +git." Editor Phase 3 is where it becomes truly collaborative; 3a and 3b can ship in either +order — **3a (collaboration) first**. I3 (university SSO) and I4 (self-hosted git) are both +deferred. + +--- + +## 7. Proposed first concrete steps + +1. Phase 0 spike in a scratch branch: **BusyTeX** WASM in a Web Worker compiling a real + project `.tex` → PDF in browser; measure & document. (See §8.1 for the full task list.) +2. Add `codemirror`, pdf.js (already partly present via `pdfjs-dist`), and the engine + artifact to `frontend/`; scaffold `src/components/Publications/LatexEditor/`. +3. Wire the `Edit` button + full-screen modal shell with file-load and save stubs. +4. Land the manual-compile + auto-commit MVP behind a feature flag. + +The fully decomposed task breakdown for the near-term critical path (Phase 0 + I1 + I2 + +editor Phase 1) is in **§8**. + +--- + +### Open questions for review +- ~~Licensing path~~ — **DECIDED: Path 1** (§0). Only follow-up: verify the engine `.wasm` + artifact licenses before Phase 1. +- ~~Engine choice~~ — **DECIDED: upstream MIT `busytex/busytex`** (TeX Live 2023 + SyncTeX). + License verified Path-1-clean; the TeXlyre TeX Live 2026 build is AGPL and was rejected. +- ~~Git hosting strategy~~ — **DECIDED: GitHub-backed, with the git-backend abstraction + (§2.4) introduced up front.** Self-hosted git stays a pluggable backend deferred to I4. +- ~~Push credential~~ — **DECIDED/DONE: existing Calkit GitHub App installation** supplies + push for write-access users (§2.2). Remaining work is authorship routing, not credentials. +- ~~University SSO~~ — **DECIDED: deferred (I3).** Near-term onboarding = Google sign-in + + email/password signup via invite links (pick password on first sign-in). Broker-vs-build + revisited when we actually start I3. +- ~~I1+I2 sequencing~~ — **DECIDED: I1 + I2 land before/with editor Phase 1**; Phase 0 + spike runs in parallel. +- ~~Phase 3 priority~~ — **DECIDED: 3a (real-time collaboration) first**, then 3b. +- ~~Preview download~~ — **DECIDED: no download; preview is view-only** (§3.1). +- ~~TeX Live package server~~ — **DECIDED: defer.** Use the upstream/public package server + for Phase 0/1; self-host a cached package proxy in Phase 2 (§ Phase 2) for reproducibility + and uptime. + +--- + +## 8. Task breakdown — near-term critical path + +Four workstreams. **§8.1 (Phase 0 spike)** can start immediately and run in parallel with +**§8.2 (I1)** and **§8.3 (I2)**; **§8.4 (editor Phase 1)** depends on all three. File paths +are the current locations found during research — verify before editing. + +### 8.1 Phase 0 — BusyTeX compile spike (throwaway, scratch branch) + +Goal: prove an in-browser compile of a real Calkit paper before committing to UI work. + +**STATUS: DONE — verdict GO.** Spike lives in `spikes/latex-wasm-busytex/` (throwaway; +binaries gitignored, re-fetch via `download-assets.sh`). Verified headless in Chrome. + +- [x] ~~Obtain artifact + confirm license fits Path 1.~~ **Done (§0):** upstream MIT + `busytex/busytex` (TeX Live 2023); TeXlyre's 2026 build is AGPL and rejected. +- [x] ~~Page loads engine in a Web Worker, compiles a hello-world `.tex` to PDF, renders it.~~ + Our own loader (`main.js`) around the MIT busytex worker; PDF shown via blob-URL iframe. +- [x] ~~Compile a real-ish `.tex` with packages.~~ article + `amsmath`/`graphicx`/`hyperref` + → 121.6 KB PDF, `exit_code 0`, all packages resolved from the `texlive-basic` bundle. + *(Follow-up: try a heavier real paper from an actual project in §8.4.)* +- [x] ~~Measure & document.~~ **Cold-start ~1.5–1.8 s, compile ~0.4 s, total ~1.9–2.2 s.** + Asset size dominates: `busytex.wasm` ≈ 29 MB + `texlive-basic.data` ≈ 100 MB one-time. +- [ ] FS-seed contract decision (how files reach the worker) — carried into §8.4 (the + `{path, contents}[]` shape busytex expects maps cleanly onto `getProjectContents`). +- **Exit met:** GO with numbers + a working compile. **Key productionization takeaway:** the + ~130 MB one-time asset download — not compile speed — is the cost to manage (lazy-load + when the editor opens; cache in IndexedDB / service worker). + +### 8.2 I1 — Onboarding & GitHub-less authorship + +Goal: a user can create a Calkit Cloud account without GitHub, and their edits can be +authored as them and pushed via the existing GitHub App. + +**Constraint (from Pete):** GitHub-less users may **collaborate** but **cannot own/create +projects** until git hosting is decoupled (I4). Owners must have a linked GitHub account. + +**Backend** +- [x] ~~Re-enable signup~~ — `POST /users/signup` now creates an email/password user + (bcrypt) with no GitHub account. (`app/api/routes/users.py`) +- [x] ~~Make `Account.github_name` nullable + migration.~~ Column nullable + (`app/models/core.py`); migration `f3a9c1d2b4e6_make_account_github_name_nullable` + (applies cleanly to head). `create_user` no longer forces a `github_name`; None-safe + types on `User.github_username` / `UserPublic` / `AccountPublic` and the derived + comment/file-lock props; **invariant guards** keep `Org.github_name` / + `Project.owner_github_name` typed `str` (owners/orgs always have one). +- [x] ~~Owner guard.~~ `post_project` returns **403** "A linked GitHub account is required to + create or own projects" for GitHub-less users. Tests: GitHub-less signup + owner-guard + added; **full backend suite green (89 passed, 4 skipped)**. +- [ ] Finalize **Google sign-in** on the backend (token exchange + user/account creation), + pairing with the existing `google-auth.tsx` callback; store identity via + `UserExternalCredential` (provider=google). +- [x] ~~**Authorship routing.**~~ `get_repo` now uses a **GitHub App installation token** + for GitHub-less users (`github.get_app_installation_token(owner, repo)` mints a + repo-scoped token via the App JWT) instead of a personal token; GitHub users are + unchanged. `_configure_committer` already authors as the Calkit user (name = full_name + / email, email = `user.email`) and is None-safe; `get_repo`'s temp-path now falls back + to `account.name` when `github_username` is None. Unit-tested the token exchange (mocked + GitHub API) + 502 handling. **Caveat:** the live App-token network path and the + GitHub-less clone/push round-trip can't be exercised locally (needs the App private key + + real repo) — verify on staging. Email **verification** for authored commits is still + a TODO (currently trusts the signup email). + +**Frontend** +- [ ] Login/signup UI (`src/routes/login/`): add "Continue with Google" + email/password + sign-up alongside the existing GitHub button. +- [ ] Wire the Google callback (`src/routes/google-auth.tsx`) end-to-end through `lib/auth.ts` + token storage. + +- **Exit:** create an account via Google and via email/password (no GitHub); make an edit + through an existing write path and see a commit authored as the Calkit user, pushed by the + App. + +### 8.3 I2 — Native membership & shareable invite links + +Goal: project access resolves from a native table first, and a share link lets a new user +join and start editing. + +**Backend — DONE (full suite green, 92 passed / 4 skipped).** +- [x] ~~`ProjectMembership` table.~~ `(user_id, project_id, role_id)` mirroring + `UserOrgMembership`, with `role_name` computed (`app/models/core.py`). +- [x] ~~`ProjectInvitation` table + endpoints.~~ Token is high-entropy + (`generate_refresh_token`), only its **SHA-256 hash** is stored; `role_id`, `expires`, + `max_uses`, `use_count`, `revoked`, `is_valid`. Endpoints: **create** / **list** / + **revoke** (admin-only) + **redeem** (`POST /project-invitations/{token}` → creates + membership; 410 if revoked/expired/used-up; owners aren't downgraded). Migration + `b7e2f4a1c9d8_add_project_membership_and_invitations` (FK cascades; applies to head). +- [x] ~~Access resolution.~~ `get_project` now checks native `ProjectMembership` **first** + in the collaborator branch, falling back to the GitHub-derived `UserProjectAccess` + cache (extracted into `_resolve_github_collaborator_access`). GitHub access unchanged. +- [x] ~~Role-escalation guard.~~ Invites cap at **admin** (never owner); create/list/revoke + require admin access. Tests cover create+redeem→access, admin-only, revoked→410. +- [x] ~~**Repo-write for native members.**~~ Done via the §8.2 authorship-routing work — a + GitHub-less `write` member's git operations run through the App installation token, + authored as them. (Live push verification deferred to staging — see §8.2 caveat.) + +**Frontend** +- [x] ~~"Invite / share" UI.~~ Built on the **Collaborators page** (per Pete) — + `components/Projects/InviteLinks.tsx`: create-link modal (role + optional expiry/ + max-uses), one-time link reveal with copy, list with status/uses/expiry + revoke. + Client regenerated (`make client`); tsc + biome clean. *(Admin-facing — works for + existing GitHub users; not blocked by the `_layout` gate below.)* +- [x] ~~Invite landing route (`/join/{token}`).~~ `routes/join/$token.tsx`: bounces + unauthenticated visitors to `/signup` (storing the redirect), then redeems via + `postProjectInvitationRedemption` and navigates into the project. +- [x] ~~Email/password signup UI.~~ `routes/signup.tsx` (register → **auto-login** → honor + redirect) + email/password sign-in and a "Create an account" link on the login page. + *(Google sign-in still TODO — needs a backend `login/google` endpoint; the existing + `google-auth.tsx` is account-linking only.)* +- [x] ~~**Gate relaxed.**~~ `routes/_layout.tsx` now only enforces the GitHub-App install + for users **with** a `github_username`; GitHub-less users skip the query, the + install-redirect, and the loading-wait. +- [x] **Verified end-to-end** (headless Chrome): email/password signup → GitHub-less user + created (`github_username: null`) → auto-login → **lands on the dashboard, not the + GitHub-install gate**. (Required `alembic upgrade head` on the dev DB — see note below.) +- [ ] Google sign-in (backend `login/google` + login-page button). + +- **Exit (backend met):** a GitHub-less user signs up, redeems an invite, and gains native + access to a private project (verified by test). Remaining for full exit: the frontend + flow + repo-write authorship routing. + +### 8.4 Editor Phase 1 — single-file editor MVP (depends on 8.1–8.3) + +Goal: open a publication, edit its `.tex`, compile-preview in-browser, save via auto-commit. + +**Frontend — first slice landed (tsc + biome clean; engine verified in-app).** +- [x] ~~Deps + engine packaging.~~ Added `codemirror` v6 + `@codemirror/legacy-modes` (stex + LaTeX mode). Engine served from `frontend/public/tex/` — MIT glue + (`busytex_worker.js`, `busytex_pipeline.js`) committed; the ~130 MB binaries are + gitignored + fetched via `scripts/download-tex-engine.sh`. Base URL is configurable via + `VITE_TEX_ENGINE_URL` (default `/tex`) for a CDN in prod. **Confirmed: no COOP/COEP + needed** (busytex runs without cross-origin isolation — verified). +- [x] ~~Compiler wrapper.~~ `src/lib/latexCompiler.ts` — our own `LatexCompiler` class around + the MIT worker (lazy `init()`, `compile()`, `terminate()`). +- [x] ~~Editor modal.~~ `components/Publications/LatexEditor.tsx`: full-screen Chakra modal, + CodeMirror pane | **pdf.js** preview (`PdfDocumentViewer` — no download, §3.1) | + collapsible log panel, "Draft preview — not the published PDF" label, + unsaved-changes guard. +- [x] ~~**Edit** button.~~ On the publications `PubInfo` panel, gated on `userHasWriteAccess`. + Source `.tex` derived from the publication output path (`paper.pdf`→`paper.tex`) — + Phase-1 heuristic; later resolve from the stage deps. +- [x] ~~Load `.tex`~~ via `getProjectContents` (base64→UTF-8) and ~~save~~ via + `putProjectContents` (auto-commit) with an unsaved guard. +- [x] **Engine verified in the app origin** (headless): assets serve at `/tex/`, worker + compiles a sample → 31 KB PDF, exit 0, ~2.8 s cold. +- [ ] Manual/debounced **auto-compile** (currently a manual "Compile preview" button). +- [ ] Feature flag (currently always-on for write users — gate before shipping). +- [ ] Full UI E2E (click Edit → type → compile → see PDF) — needs an authed project + + publication fixture; engine path already proven in-app. +- [ ] Smarter `.tex` resolution (multi-file `\input`/figures/bib is Phase 2). + +**Backend** +- [x] No new endpoints — reuses `PUT contents` (auto-commit). `.tex` round-trips as raw text. + *(Still verify the 1 MB limit is comfortable for large sources.)* + +- **Exit:** edit a real paper's `.tex`, compile to PDF in-browser, save, and see the + auto-commit land + push to GitHub — as a non-GitHub user who joined via an invite link. + +### 8.5 Editor Phase 2 — multi-file, figures, bib (in progress) + +First slice landed; engine capabilities verified headless in the app origin. + +- [x] ~~**Multi-file project loading.**~~ `src/lib/latexProject.ts` lists the publication's + directory (recursive, depth/count-capped) and fetches all text sources (`.tex/.bib/ + .cls/.sty/.bst/…`) + images (`.png/.jpg/.pdf/.eps/…`) — text via base64, DVC figures via + signed URL. Seeds them all into the compile. +- [x] ~~**File tree + multi-buffer editing.**~~ `LatexEditor` rewritten: left file-tree + column, per-file CodeMirror buffers, dirty tracking, **batch save** of all changed + files, unsaved-changes guard. +- [x] ~~**`\input`/`\include` + figures verified.**~~ Engine compiles a real multi-file doc + (`\input` + `\includegraphics` of a valid PNG) → PDF, exit 0 (headless, app origin). +- [x] **Production build green** (`npm run build`) — editor + CodeMirror + worker bundle. +- [x] ~~**Render-timing bug fixed**~~ (reported: "source tex doesn't load"). `EditorPane` + was mounting before `buffersRef` was populated and never remounting (the initial + `activePath` already equalled the main file, so its `key` never changed) → the main + `.tex` showed blank. Added a `ready` gate (mount only after the project loads) + robust + main-file detection (prefer the `\documentclass` file, not just the basename heuristic). + Verified: the main file content shows immediately on open. +- [x] ~~**Broader package coverage**~~ (reported: `sectsty.sty not found`). Now serve all + available bundles (`texlive-basic` + `ubuntu-texlive-latex-base/recommended/extra/ + science/fonts-recommended`); the engine resolves `\usepackage` names against them and + loads the needed `.data` **on demand** (preload only the base). Verified: a `booktabs` + doc (in the recommended bundle, not basic) loads that bundle on demand and compiles. + Missing packages now surface a clear "Missing from the in-browser TeX bundle: X" message. +- [ ] ⚠️ **Package/class coverage is bundle-limited (no on-demand fetch) — the headline + Phase 2 risk.** busytex only has the *bundled* TeX Live subset; anything absent from all + bundles cannot be fetched (busytex has no package server, unlike SwiftLaTeX). Confirmed + real-world gaps: + - **`sectsty`** (used by `example-basic`) and **`siunitx`** — not in any bundle. + - **Journal document classes**: `petebachant/boom-paper` uses **`aastex631`** (AAS + astronomy class). The repo vendors `aastex631.cls`, it loads, but the document + produces **"No pages of output"** even with **all** bundles loaded (~340 MB) — it + warns *"Please update your system to include revtex4-1.cls"* and the needed + revtex/support simply isn't in TeX Live 2023's subset here. Verified it's **not** a + loading bug: the full 79 KB `main.tex` reaches the engine intact. Many real users + (astronomy: aastex; physics: revtex; Elsevier: elsarticle; etc.) will hit this. + Real fix = a **package/file proxy** (fetch missing `.cls`/`.sty` + their deps on demand + into the FS), a self-built **fuller busytex bundle** (full TeX Live), or revisiting the + engine. This is now clearly the **critical** next investment for "real papers compile." +- [x] ~~**Package-proxy spike — DONE**~~ (`spikes/latex-package-proxy/`). Validated the + concept end-to-end: a self-hosted **texmf proxy** (`texlive/texlive` + `kpsewhich`, the + Texlive-Ondemand model — the public `texlive.swiftlatex.com` is dead) serves any TeX + file by name, and on-demand fetching resolved a **deep real tree** — **~94 files** + (`revtex4-1` + 8 society `.rtx` + the whole `tikz`/`pgf` core) — driving boom-paper from + "no output" deep into a real compile. **Conclusion:** coverage is *solvable* and the + file source is easy; the iterative "scan log → fetch → recompile" delivery is the + **wrong mechanism** (O(N) recompiles = minutes; TeX's many missing-file error formats + are whack-a-mole). **Recommended path:** an **in-engine kpathsea hook** — the engine + requests the *exact* missing file from our self-hosted proxy and continues in **one** + compile. See `spikes/latex-package-proxy/README.md`. +- [x] ~~**Engine-hook decision (assessed)**~~ — **patch busytex with our own MIT kpathsea + hook.** SwiftLaTeX's engine already has the hook but is **AGPL-3.0** (the hook *is* + their AGPL code) → rejected for our MIT editor (would force AGPL on Calkit). busytex is + MIT + TeX Live 2023 and already builds via Emscripten + source patches, so a clean-room + kpathsea patch keeps licensing clean. Pure-JS FS hooks won't do — kpathsea decides a + file is absent *before* `fopen`, so the search logic needs patching + a wasm rebuild. + **Cost:** a one-time engine build (Emscripten/cmake/Docker) + a small focused patch + + sync-fetch glue (sync XHR in the worker, or Asyncify). **Fallback for the heaviest + papers:** an opt-in **server-side compile** with full TeX Live (simplest coverage, + at the cost of per-session server compute; ties to §3.1 user-compute builds). +- [x] ~~**Patched engine BUILT**~~ (`spikes/busytex-remote-fetch/`). `build.sh` ran green under + `emscripten/emsdk:3.1.43` → **`busytex.js` (~297 KB) + `busytex.wasm` (~30 MB)** with our + hook embedded (verified: `calkitTexmfEndpoint`/`__calkitCache` in the JS; `_malloc`/ + `stringToUTF8`/`lengthBytesUTF8`/`UTF8ToString` exported). Staged at + `frontend/public/tex/busytex.patched.{js,wasm}` (gitignored). **Design that worked:** + kpathsea stays **pure C** — `apply_patch.py` patches `kpathsea_find_file` to call a real, + always-defined `kpse_remote_fetch` that delegates through a NULL-default function pointer; + the **EM_JS** browser fetch lives only in `busytex.c` (from `remote_fetch.c`) and a + constructor installs it into the pointer at engine startup. This was the crux: `EM_JS` + symbols are JS imports, and busytex's ~6 **standalone applet** links (kpsewhich, bibtex8, …) + reject a JS-import symbol pulled in via `libkpathsea`; the indirection makes every binary + link (applets get a no-op) while only the engine carries the fetch. +- [x] ~~**Engine wired + on-demand fetch verified end-to-end**~~. `latexCompiler.ts` reads + `VITE_TEXMF_PROXY` and passes it through `busytex_worker.js` → `busytex_pipeline.js` + (`Module.calkitTexmfEndpoint`); patched engine swapped into `frontend/public/tex/`. In a + real headless browser, a doc using a package **absent from all bundles** compiles in a + **single pass**, pulling the whole tree on demand (e.g. tikz-cd → ~100 pgf/tikz files; + revtex4-1 tree). Confirmed on the actual **boom-paper**: it fetched `revtex4-1.cls`, + `aps*.rtx`, `revsymb4-1.sty`, the full tikz/pgf stack, etc. +- [x] ~~**Fonts: `mktexpk` fork() fatal — fixed via proxy generation**~~. PK/TFM lookups use + `kpathsea_path_search` (not `find_file`), and on a miss kpathsea forks `mktexpk`/`mktextfm` + — impossible in WASM (boom died on `Font tctt1000 at 600 not found`). `apply_patch.py` now + also patches **`tex-make.c`**: `kpathsea_make_tex()` tries `kpse_remote_fetch` for the file + the script would generate (`.pk`, `.tfm`) before the doomed fork; the + proxy (`proxy-server.py`, `kpsewhich -mktex=pk -mktex=tfm`) generates it where fork works. + Verified: the exact `tctt1000` case that killed boom now fetches `tctt1000.600pk` and + compiles to a PDF. Remaining: user re-runs boom-paper in the editor for the full-document + confirmation; productionize the proxy (hosted, CORS-restricted, TL version pinned to 2023). +- [ ] ⚠️ **Bibliography is an engine limitation.** busytex reuses one WASM module across the + bibtex multi-pass, so pdftex asserts on the 2nd run (`pdfinitmapfile`); XeTeX driver + also fails. **Worked around** by forcing a single pdflatex pass (`bibtex: false`) so a + preview always renders — but **citations stay unresolved** (`[?]`). Real fix needs a + patched/newer build or per-pass module recreation (or running bibtex across separate + `compile()` calls while persisting the FS). Tracked for a follow-up. +- [x] ~~**Verified end-to-end through the UI**~~ (headless, mocked contents API): open + editor → loader builds the file tree → main file loads → real-engine compile of a + multi-file doc (`\input` + on-demand `booktabs` + figure) → PDF renders. Loader contract + also checked against the **real** API (dir listing + base64 file content) on a live + project. Still TODO: a true end-to-end against a real git repo (clone + commit on save). +- [x] ~~**Dependency-driven loading (cross-dir figures + DVC)**~~ (reported: a figure at + `\includegraphics{../figures/voltage-time-series.png}` not found). The loader now drives + the file set from the publication's **pipeline-stage `deps`** (`stage_info.deps`), + expanding directory deps (e.g. `figures`) and skipping `.calkit/…`, and fetches + DVC-tracked figures (via base64 `content` or signed `url`). Files are seeded at their + **full repo path**, so relative refs like `../figures/x.png` resolve against the real + layout. Falls back to scanning the `.tex`'s directory when there's no stage. Verified + end-to-end (mocked): a paper in `paper/` with a figure in a sibling `figures/` dir loads + the figure and **compiles ✓**. +- [ ] SyncTeX (forward/inverse search) and the **TeX Live package proxy** (self-hosted cached + packages) remain for later in Phase 2. +- [ ] A size/count budget surfaced to the user when a dep dir is large. + +**All open questions resolved.** Remaining follow-ups are verification tasks, not decisions: +confirm the BusyTeX + SwiftLaTeX engine artifact licenses fit Path 1 before Phase 1 ships. + +--- + +## 9. Future ideas & open product questions (from `TODO.md`) + +Captured from Pete's `TODO.md` — product direction beyond the core editor. + +- **Editor open-state in the URL.** Make the editor's open/closed state (and active file) a + search/route param so a session is shareable and restorable by URL — instead of local + component state. Fits the TanStack Router search-param pattern already used on the + publications page. +- **External files: `map-paths` stage vs. show-in-place — LEANING show-in-place.** When a + paper references results/figures from the broader project (outside the paper dir), do we + (a) encourage a `map-paths`/copy stage that brings them into the paper directory, or + (b) surface them in the editor in place (e.g. `../figures/whatever.png`)? The Phase-2 + dependency-driven loader (§8.5) already does **(b)** — it loads pipeline-stage deps at + their real repo paths, so `../figures/x.png` just works. A `map-paths` stage could still be + offered for users who want a self-contained paper dir, but it's no longer required to + compile. +- **Mapping results into the text (no copy/paste).** Make it easy to reference computed + numerical results and tables from the pipeline inside the LaTeX (so authors don't paste + stale numbers). E.g. a stage that emits `\newcommand`/`.tex` snippets or a `siunitx`-style + values file the paper `\input`s, surfaced in the editor. Product + pipeline design needed. +- **Figure provenance on hover.** Hovering a figure (in the editor or PDF preview) shows a + link to the pipeline stage that produced it — leveraging the same `stage_info`/deps data + the loader now uses. +- **User-attached compute for real builds (follow-on PR).** Let users connect a local + workspace or another machine to run pipeline stages and serve back produced files (their + workspace commits + pushes, we pull). Complex; keep it for later. **Design implication + now:** keep the in-browser LaTeX compilation **modular** so this compute path can slot in + later — and it dovetails with §3.1's "provenance-perfect builds with user compute" (the + only sanctioned way to promote a compile to a real artifact; the WASM path stays + preview-only). diff --git a/backend/app/alembic/versions/b7e2f4a1c9d8_add_project_membership_and_invitations.py b/backend/app/alembic/versions/b7e2f4a1c9d8_add_project_membership_and_invitations.py new file mode 100644 index 00000000..575e6a08 --- /dev/null +++ b/backend/app/alembic/versions/b7e2f4a1c9d8_add_project_membership_and_invitations.py @@ -0,0 +1,87 @@ +"""Add project membership and invitations + +Native (non-GitHub) project membership plus shareable invite links, so users +without GitHub accounts can be granted collaborator access. + +Revision ID: b7e2f4a1c9d8 +Revises: f3a9c1d2b4e6 +Create Date: 2026-06-17 00:00:00.000000 + +""" + +from alembic import op +import sqlalchemy as sa +import sqlmodel.sql.sqltypes + + +# revision identifiers, used by Alembic. +revision = "b7e2f4a1c9d8" +down_revision = "f3a9c1d2b4e6" +branch_labels = None +depends_on = None + + +def upgrade(): + op.create_table( + "projectmembership", + sa.Column("user_id", sa.Uuid(), nullable=False), + sa.Column("project_id", sa.Uuid(), nullable=False), + sa.Column("role_id", sa.Integer(), nullable=False), + sa.Column("created", sa.DateTime(), nullable=False), + sa.Column( + "updated", + sa.DateTime(), + server_default=sa.func.now(), + nullable=False, + ), + sa.Column("invited_by_user_id", sa.Uuid(), nullable=True), + sa.ForeignKeyConstraint( + ["user_id"], ["user.id"], ondelete="CASCADE" + ), + sa.ForeignKeyConstraint( + ["project_id"], ["project.id"], ondelete="CASCADE" + ), + sa.ForeignKeyConstraint( + ["invited_by_user_id"], ["user.id"], ondelete="SET NULL" + ), + sa.PrimaryKeyConstraint("user_id", "project_id"), + ) + op.create_table( + "projectinvitation", + sa.Column("id", sa.Uuid(), nullable=False), + sa.Column("project_id", sa.Uuid(), nullable=False), + sa.Column( + "token_hash", + sqlmodel.sql.sqltypes.AutoString(), + nullable=False, + ), + sa.Column("role_id", sa.Integer(), nullable=False), + sa.Column("created_by_user_id", sa.Uuid(), nullable=True), + sa.Column("created", sa.DateTime(), nullable=False), + sa.Column("expires", sa.DateTime(), nullable=True), + sa.Column("max_uses", sa.Integer(), nullable=True), + sa.Column("use_count", sa.Integer(), nullable=False), + sa.Column("revoked", sa.Boolean(), nullable=False), + sa.ForeignKeyConstraint( + ["project_id"], ["project.id"], ondelete="CASCADE" + ), + sa.ForeignKeyConstraint( + ["created_by_user_id"], ["user.id"], ondelete="SET NULL" + ), + sa.PrimaryKeyConstraint("id"), + ) + op.create_index( + op.f("ix_projectinvitation_token_hash"), + "projectinvitation", + ["token_hash"], + unique=True, + ) + + +def downgrade(): + op.drop_index( + op.f("ix_projectinvitation_token_hash"), + table_name="projectinvitation", + ) + op.drop_table("projectinvitation") + op.drop_table("projectmembership") diff --git a/backend/app/alembic/versions/f3a9c1d2b4e6_make_account_github_name_nullable.py b/backend/app/alembic/versions/f3a9c1d2b4e6_make_account_github_name_nullable.py new file mode 100644 index 00000000..777c1657 --- /dev/null +++ b/backend/app/alembic/versions/f3a9c1d2b4e6_make_account_github_name_nullable.py @@ -0,0 +1,35 @@ +"""Make account github_name nullable + +Allows accounts created without GitHub (email/Google signup). Project owners +must still have a github_name (enforced in the app layer) until git hosting is +decoupled from GitHub; collaborators need not. + +Revision ID: f3a9c1d2b4e6 +Revises: f3a9c1b7e240 +Create Date: 2026-06-17 00:00:00.000000 + +""" + +from alembic import op +import sqlalchemy as sa + + +# revision identifiers, used by Alembic. +revision = "f3a9c1d2b4e6" +down_revision = "f3a9c1b7e240" +branch_labels = None +depends_on = None + + +def upgrade(): + op.alter_column( + "account", "github_name", existing_type=sa.VARCHAR(), nullable=True + ) + + +def downgrade(): + # Note: rows with NULL github_name (GitHub-less accounts) must be handled + # before downgrading, or this will fail. + op.alter_column( + "account", "github_name", existing_type=sa.VARCHAR(), nullable=False + ) diff --git a/backend/app/api/routes/accounts.py b/backend/app/api/routes/accounts.py index c83a2d3b..94bdc800 100644 --- a/backend/app/api/routes/accounts.py +++ b/backend/app/api/routes/accounts.py @@ -17,7 +17,7 @@ class AccountPublic(SQLModel): name: str - github_name: str + github_name: str | None display_name: str kind: Literal["user", "org"] role: Literal["self", "read", "write", "admin", "owner"] | None = None diff --git a/backend/app/api/routes/projects/core.py b/backend/app/api/routes/projects/core.py index 8218427c..14675415 100644 --- a/backend/app/api/routes/projects/core.py +++ b/backend/app/api/routes/projects/core.py @@ -9,7 +9,7 @@ import uuid import zipfile from copy import deepcopy -from datetime import datetime +from datetime import datetime, timedelta from fnmatch import fnmatch from io import StringIO from pathlib import Path @@ -48,6 +48,7 @@ ) from app.api.routes.orgs import OrgPost, post_org from app.config import settings +from app.security import generate_refresh_token, hash_refresh_token from app.core import ( CATEGORIES_PLURAL_TO_SINGULAR, CATEGORIES_SINGULAR_TO_PLURAL, @@ -95,6 +96,12 @@ ProjectComment, ProjectCommentPatch, ProjectCommentPost, + ProjectInvitation, + ProjectInvitationCreated, + ProjectInvitationPost, + ProjectInvitationPublic, + ProjectInvitationRedeemed, + ProjectMembership, ProjectPost, ProjectPublic, ProjectsPublic, @@ -105,6 +112,7 @@ UserOrgMembership, UserProjectAccess, ) +from app.models.core import ROLE_IDS from app.models.projects import ( Showcase, ShowcaseFigure, @@ -276,6 +284,13 @@ def post_project( project_in: ProjectPost, ) -> ProjectPublic: """Create new project.""" + # Project owners must have a linked GitHub account until git hosting is + # decoupled from GitHub. GitHub-less users can still collaborate. + if current_user.account.github_name is None: + raise HTTPException( + 403, + "A linked GitHub account is required to create or own projects.", + ) project_in.name = project_in.name.lower() if project_in.git_repo_exists and project_in.git_repo_url is None: raise HTTPException( @@ -1196,6 +1211,7 @@ def put_project_contents( file: Annotated[UploadFile, File()], session: SessionDep, current_user: CurrentUser, + message: Annotated[str | None, Form()] = None, ) -> ContentsItem: project = app.projects.get_project( owner_name=owner_name, @@ -1216,7 +1232,8 @@ def put_project_contents( f.write(file.file.read()) repo.git.add(path) if repo.git.diff(["--staged", path]): - repo.git.commit(["-m", f"Upload {path} from web"]) + commit_message = message or f"Upload {path} from web" + repo.git.commit(["-m", commit_message]) repo.git.push(["origin", repo.active_branch.name]) else: raise HTTPException( @@ -3930,6 +3947,159 @@ def delete_project_collaborator( return Message(message="Success") +@router.post("/projects/{owner_name}/{project_name}/invitations") +def post_project_invitation( + owner_name: str, + project_name: str, + req: ProjectInvitationPost, + current_user: CurrentUser, + session: SessionDep, +) -> ProjectInvitationCreated: + """Create a shareable invite link granting native project membership. + + The raw token is returned only here; the DB stores its hash. Invites can + grant up to admin, never ownership. + """ + project = app.projects.get_project( + owner_name=owner_name, + project_name=project_name, + session=session, + current_user=current_user, + min_access_level="admin", + ) + token = generate_refresh_token() + expires = ( + utcnow() + timedelta(days=req.expires_days) + if req.expires_days is not None + else None + ) + invitation = ProjectInvitation( + project_id=project.id, + token_hash=hash_refresh_token(token), + role_id=ROLE_IDS[req.role], + created_by_user_id=current_user.id, + expires=expires, + max_uses=req.max_uses, + ) + session.add(invitation) + session.commit() + session.refresh(invitation) + url = f"{settings.frontend_host.rstrip('/')}/join/{token}" + return ProjectInvitationCreated( + id=invitation.id, + role_name=invitation.role_name, + created=invitation.created, + expires=invitation.expires, + max_uses=invitation.max_uses, + use_count=invitation.use_count, + revoked=invitation.revoked, + token=token, + url=url, + ) + + +@router.get("/projects/{owner_name}/{project_name}/invitations") +def get_project_invitations( + owner_name: str, + project_name: str, + current_user: CurrentUser, + session: SessionDep, +) -> list[ProjectInvitationPublic]: + project = app.projects.get_project( + owner_name=owner_name, + project_name=project_name, + session=session, + current_user=current_user, + min_access_level="admin", + ) + invitations = session.exec( + select(ProjectInvitation) + .where(ProjectInvitation.project_id == project.id) + .order_by(sqlalchemy.desc(ProjectInvitation.created)) # type: ignore + ).all() + return list(invitations) # type: ignore[return-value] + + +@router.delete( + "/projects/{owner_name}/{project_name}/invitations/{invitation_id}" +) +def delete_project_invitation( + owner_name: str, + project_name: str, + invitation_id: uuid.UUID, + current_user: CurrentUser, + session: SessionDep, +) -> Message: + project = app.projects.get_project( + owner_name=owner_name, + project_name=project_name, + session=session, + current_user=current_user, + min_access_level="admin", + ) + invitation = session.get(ProjectInvitation, invitation_id) + if invitation is None or invitation.project_id != project.id: + raise HTTPException(404, "Invitation not found") + invitation.revoked = True + session.add(invitation) + session.commit() + return Message(message="Invitation revoked") + + +@router.post("/project-invitations/{token}") +def post_project_invitation_redemption( + token: str, + current_user: CurrentUser, + session: SessionDep, +) -> ProjectInvitationRedeemed: + """Redeem an invite link, granting the current user native membership.""" + invitation = session.exec( + select(ProjectInvitation).where( + ProjectInvitation.token_hash == hash_refresh_token(token) + ) + ).first() + if invitation is None: + raise HTTPException(404, "Invitation not found") + if not invitation.is_valid: + raise HTTPException(410, "Invitation is no longer valid") + project = session.get(Project, invitation.project_id) + if project is None: + raise HTTPException(404, "Project not found") + # Project owners already have full access; don't create a lesser membership. + if project.owner_account.user_id == current_user.id: + return ProjectInvitationRedeemed( + owner_name=project.owner_account.name, + project_name=project.name, + role_name="owner", + ) + existing = session.exec( + select(ProjectMembership) + .where(ProjectMembership.project_id == project.id) + .where(ProjectMembership.user_id == current_user.id) + ).first() + if existing is None: + session.add( + ProjectMembership( + user_id=current_user.id, + project_id=project.id, + role_id=invitation.role_id, + invited_by_user_id=invitation.created_by_user_id, + ) + ) + elif invitation.role_id > existing.role_id: + # Upgrade if the invite grants more than they already have. + existing.role_id = invitation.role_id + session.add(existing) + invitation.use_count += 1 + session.add(invitation) + session.commit() + return ProjectInvitationRedeemed( + owner_name=project.owner_account.name, + project_name=project.name, + role_name=invitation.role_name, + ) + + class Issue(BaseModel): id: int number: int diff --git a/backend/app/api/routes/users.py b/backend/app/api/routes/users.py index e03e014d..432a85ef 100644 --- a/backend/app/api/routes/users.py +++ b/backend/app/api/routes/users.py @@ -161,8 +161,11 @@ def delete_current_user( @router.post("/users/signup") def register_user(session: SessionDep, user_in: UserRegister) -> UserPublic: - """Create new user without the need to be logged in.""" - raise HTTPException(501) + """Create a new user with email + password, without a GitHub account. + + Such users can collaborate on projects (e.g. via invite links) but cannot + own projects until git hosting is decoupled from GitHub. + """ user = users.get_user_by_email(session=session, email=user_in.email) if user: raise HTTPException( diff --git a/backend/app/git.py b/backend/app/git.py index fd1fa369..226fd434 100644 --- a/backend/app/git.py +++ b/backend/app/git.py @@ -21,7 +21,7 @@ from ruamel.yaml import YAMLError from sqlmodel import Session -from app import users +from app import github, users from app.core import logger, ryaml from app.models import GitRef, Project, User @@ -147,7 +147,10 @@ def get_repo( # Add the file to the repo(s) -- we may need to clone it. # Ref-based reads should not mutate this working tree checkout. if user is not None: - base_dir = f"/tmp/{user.github_username}/{owner_name}/{project_name}" + # github_username is None for GitHub-less users; fall back to the + # (always-present, unique) account name for a stable temp path. + user_dir = user.github_username or user.account.name + base_dir = f"/tmp/{user_dir}/{owner_name}/{project_name}" else: base_dir = f"/tmp/anonymous/{owner_name}/{project_name}" repo_dir = os.path.join(base_dir, "repo") @@ -161,9 +164,24 @@ def get_repo( # Clone the repo if it doesn't exist -- it will be in a "repo" dir access_token: str | None = None if user is not None: - logger.info(f"Getting {user.email}'s access token for Git operations") - with _timed("get-github-token", user=user.github_username): - access_token = users.get_github_token(session=session, user=user) + if user.account.github_name is not None: + # GitHub user: operate with their personal token. + logger.info(f"Getting {user.email}'s token for Git operations") + with _timed("get-github-token", user=user.github_username): + access_token = users.get_github_token( + session=session, user=user + ) + else: + # GitHub-less member: access was authorized natively upstream + # (e.g. via an invite). Operate via the GitHub App installation + # token for the repo; commits are still authored as this user. + logger.info( + f"Getting GitHub App installation token for {user.email}" + ) + with _timed("get-app-installation-token", user=user.email): + access_token = github.get_app_installation_token( + owner_name, project_name + ) # Plain URL with no embedded token -- credentials handled in helper git_plain_url = project.git_repo_url if not git_plain_url.endswith(".git"): diff --git a/backend/app/github.py b/backend/app/github.py index b2188a78..6f46ac93 100644 --- a/backend/app/github.py +++ b/backend/app/github.py @@ -4,6 +4,8 @@ import time import jwt +import requests +from fastapi import HTTPException def create_app_token() -> str: @@ -25,6 +27,43 @@ def create_app_token() -> str: return encoded_jwt +def get_app_installation_token(owner_name: str, repo_name: str) -> str: + """Mint a GitHub App installation access token scoped to one repo. + + Used to perform git operations on behalf of users who have native Calkit + access to a project but no personal GitHub token (e.g. email/Google + signups). The caller must have authorized the user's access first. + """ + app_jwt = create_app_token() + headers = { + "Authorization": f"Bearer {app_jwt}", + "Accept": "application/vnd.github+json", + } + resp = requests.get( + f"https://api.github.com/repos/{owner_name}/{repo_name}/installation", + headers=headers, + timeout=15, + ) + if resp.status_code != 200: + raise HTTPException( + 502, + "Could not find the Calkit GitHub App installation for this repo", + ) + installation_id = resp.json()["id"] + resp = requests.post( + f"https://api.github.com/app/installations/{installation_id}" + "/access_tokens", + headers=headers, + json={"repositories": [repo_name]}, + timeout=15, + ) + if resp.status_code not in (200, 201): + raise HTTPException( + 502, "Could not mint a GitHub App installation token" + ) + return resp.json()["token"] + + def token_resp_text_to_dict(resp_text: str) -> dict: items = resp_text.split("&") out = {} diff --git a/backend/app/models/core.py b/backend/app/models/core.py index 0c9833cc..e1d06b63 100644 --- a/backend/app/models/core.py +++ b/backend/app/models/core.py @@ -39,7 +39,10 @@ class Account(SQLModel, table=True): org_id: uuid.UUID | None = Field( default=None, foreign_key="org.id", nullable=True ) - github_name: str + # Null for accounts created without GitHub (email/Google signup). Project + # owners must still have a github_name until git hosting is decoupled from + # GitHub (see LATEX_EDITOR_PLAN.md I4); collaborators need not. + github_name: str | None = Field(default=None) # Relationships owned_projects: list["Project"] = Relationship( back_populates="owner_account", @@ -71,7 +74,7 @@ class UserBase(SQLModel): class UserCreate(UserBase): password: str = Field(min_length=8, max_length=40) account_name: str | None = Field(default=None, max_length=64) - github_username: str = Field(default=None, max_length=64) + github_username: str | None = Field(default=None, max_length=64) class UserRegister(SQLModel): @@ -223,7 +226,7 @@ class User(UserBase, table=True): @computed_field @property - def github_username(self) -> str: + def github_username(self) -> str | None: return self.account.github_name @property @@ -242,7 +245,7 @@ def get_external_credential( # Properties to return via API, id is always required class UserPublic(UserBase): id: uuid.UUID - github_username: str + github_username: str | None subscription: Union["UserSubscription", None] @@ -268,6 +271,9 @@ def display_name(self) -> str: @computed_field @property def github_name(self) -> str: + # Orgs are always created with a GitHub name. + if self.account.github_name is None: + raise ValueError("Org account has no github_name") return self.account.github_name @property @@ -538,6 +544,13 @@ class Project(ProjectBase, table=True): user_access_records: list["UserProjectAccess"] = Relationship( back_populates="project", cascade_delete=True ) + # Native (non-GitHub) project membership and shareable invite links. + memberships: list["ProjectMembership"] = Relationship( + back_populates="project", cascade_delete=True + ) + invitations: list["ProjectInvitation"] = Relationship( + back_populates="project", cascade_delete=True + ) # TODO: Figure out how to do self-referential relationships with parent # and children projects questions: list["Question"] = Relationship( @@ -577,6 +590,10 @@ def owner_account_type(self) -> str: @computed_field @property def owner_github_name(self) -> str: + # Project owners must have a GitHub account (collaborators need not) + # until git hosting is decoupled from GitHub. + if self.owner_account.github_name is None: + raise ValueError("Project owner account has no github_name") return self.owner_account.github_name @property @@ -652,6 +669,101 @@ class UserProjectAccess(SQLModel, table=True): project: Project = Relationship(back_populates="user_access_records") +class ProjectMembership(SQLModel, table=True): + """Native (non-GitHub) membership of a user in a project. + + Resolved before GitHub-derived access in ``get_project``, and the only + access path for GitHub-less collaborators. + """ + + user_id: uuid.UUID = Field(foreign_key="user.id", primary_key=True) + project_id: uuid.UUID = Field(foreign_key="project.id", primary_key=True) + # Membership cannot grant ownership; capped at admin by the API. + role_id: int = Field(ge=min(ROLE_IDS.values()), le=max(ROLE_IDS.values())) + created: datetime = Field(default_factory=utcnow) + updated: datetime = Field( + default_factory=utcnow, + sa_column_kwargs=dict( + server_onupdate=sqlalchemy.func.now(), + server_default=sqlalchemy.func.now(), + ), + ) + invited_by_user_id: uuid.UUID | None = Field( + default=None, foreign_key="user.id" + ) + # Relationships (no User relationship: two user FKs would be ambiguous) + project: Project = Relationship(back_populates="memberships") + + @computed_field + @property + def role_name(self) -> str: + return ROLE_NAMES[self.role_id] + + +class ProjectInvitation(SQLModel, table=True): + """A shareable invite link granting project membership when redeemed.""" + + id: uuid.UUID = Field(default_factory=uuid.uuid4, primary_key=True) + project_id: uuid.UUID = Field(foreign_key="project.id") + # Only the SHA-256 hash of the token is stored; the raw token lives in the + # invite URL and is shown to the creator once. + token_hash: str = Field(unique=True, index=True) + role_id: int = Field(ge=min(ROLE_IDS.values()), le=max(ROLE_IDS.values())) + created_by_user_id: uuid.UUID | None = Field( + default=None, foreign_key="user.id" + ) + created: datetime = Field(default_factory=utcnow) + expires: datetime | None = Field(default=None) + max_uses: int | None = Field(default=None) + use_count: int = Field(default=0) + revoked: bool = Field(default=False) + # Relationships + project: Project = Relationship(back_populates="invitations") + + @computed_field + @property + def role_name(self) -> str: + return ROLE_NAMES[self.role_id] + + @property + def is_valid(self) -> bool: + if self.revoked: + return False + if self.expires is not None and self.expires < utcnow(): + return False + if self.max_uses is not None and self.use_count >= self.max_uses: + return False + return True + + +class ProjectInvitationPost(SQLModel): + role: Literal["read", "write", "admin"] = "write" + expires_days: int | None = Field(default=None, ge=1, le=365) + max_uses: int | None = Field(default=None, ge=1) + + +class ProjectInvitationPublic(SQLModel): + id: uuid.UUID + role_name: str + created: datetime + expires: datetime | None + max_uses: int | None + use_count: int + revoked: bool + + +class ProjectInvitationCreated(ProjectInvitationPublic): + # Raw token + ready-to-share URL, returned only at creation time. + token: str + url: str + + +class ProjectInvitationRedeemed(SQLModel): + owner_name: str + project_name: str + role_name: str + + class DvcPipelineStage(SQLModel): cmd: str deps: list[str] | None = None @@ -773,7 +885,7 @@ class ProjectComment(SQLModel, table=True): @computed_field @property - def user_github_username(self) -> str: + def user_github_username(self) -> str | None: return self.user.github_username @computed_field @@ -915,7 +1027,7 @@ class FileLock(SQLModel, table=True): @computed_field @property - def user_github_username(self) -> str: + def user_github_username(self) -> str | None: return self.user.github_username @computed_field diff --git a/backend/app/projects.py b/backend/app/projects.py index fae1e4b2..8a3d931f 100644 --- a/backend/app/projects.py +++ b/backend/app/projects.py @@ -44,6 +44,7 @@ def _yaml_load(data: bytes | str): Notebook, Org, Project, + ProjectMembership, Publication, User, UserProjectAccess, @@ -81,6 +82,64 @@ def _yaml_load(data: bytes | str): _ck_dvc_cache_lock = threading.Lock() +def _resolve_github_collaborator_access( + session: Session, project: Project, current_user: User +) -> None: + """Resolve a non-member user's access from the cached GitHub permission, + querying GitHub and caching the result on a miss. Sets + ``project.current_user_access`` (left None if it can't be determined). + """ + # TODO: There may be a race here with concurrent requests, though it does + # not appear to cause a real problem despite the failed writes. + access_query = ( + select(UserProjectAccess) + .where(UserProjectAccess.project_id == project.id) + .where(UserProjectAccess.user_id == current_user.id) + .with_for_update() + ) + access = session.exec(access_query).first() + if access is not None: + project.current_user_access = access.access + return + # Query GitHub for permissions + try: + github_token = app.users.get_github_token(session, current_user) + except HTTPException: + github_token = None + logger.info(f"User {current_user.email} has no GitHub token") + if github_token is None: + return + logger.info("Fetching permissions from GitHub") + url = ( + f"https://api.github.com/repos/{project.github_repo}" + f"/collaborators/{current_user.github_username}/permission" + ) + resp = requests.get( + url, + headers={"Authorization": f"Bearer {github_token}"}, + timeout=15, + ) + if resp.status_code == 200: + logger.info("Fetched permissions from GitHub") + permissions = resp.json()["permission"] + if permissions == "none": + permissions = None + else: + permissions = None + logger.info( + f"Failed to fetch permissions from GitHub ({resp.status_code})" + ) + project.current_user_access = permissions + session.add( + UserProjectAccess( + project_id=project.id, + user_id=current_user.id, + access=permissions, + ) + ) + session.commit() + + def get_project( session: Session, owner_name: str, @@ -131,64 +190,20 @@ def get_project( if project.current_user_access is None and project.is_public: project.current_user_access = "read" else: - # Query for permissions in our database, and if they aren't set, - # query GitHub and save - # TODO: We seem to have a race condition here with multiple - # requests causing this to run concurrently, though it doesn't - # seem to actually cause a problem despite the failure to write - # to the database in all but one - access_query = ( - select(UserProjectAccess) - .where(UserProjectAccess.project_id == project.id) - .where(UserProjectAccess.user_id == current_user.id) - .with_for_update() - ) - access = session.exec(access_query).first() - if access is not None: - project.current_user_access = access.access + # Non-owner: native Calkit membership takes precedence over + # GitHub-derived access, and is the only access path for + # GitHub-less collaborators. + membership = session.exec( + select(ProjectMembership) + .where(ProjectMembership.project_id == project.id) + .where(ProjectMembership.user_id == current_user.id) + ).first() + if membership is not None: + project.current_user_access = membership.role_name else: - # Query GitHub for permissions - try: - github_token = app.users.get_github_token( - session, current_user - ) - except HTTPException: - github_token = None - logger.info( - f"User {current_user.email} has no GitHub token" - ) - if github_token is not None: - logger.info("Fetching permissions from GitHub") - url = ( - f"https://api.github.com/repos/{project.github_repo}" - f"/collaborators/{current_user.github_username}/" - "permission" - ) - resp = requests.get( - url, - headers={"Authorization": f"Bearer {github_token}"}, - timeout=15, - ) - if resp.status_code == 200: - logger.info("Fetched permissions from GitHub") - permissions = resp.json()["permission"] - if permissions == "none": - permissions = None - else: - permissions = None - logger.info( - "Failed to fetch permissions from GitHub " - f"({resp.status_code})" - ) - project.current_user_access = permissions - session.add( - UserProjectAccess( - project_id=project.id, - user_id=current_user.id, - access=permissions, - ) - ) - session.commit() + _resolve_github_collaborator_access( + session, project, current_user + ) if project.is_public and project.current_user_access is None: project.current_user_access = "read" if project.current_user_access is None: diff --git a/backend/app/tests/api/routes/projects/test_core.py b/backend/app/tests/api/routes/projects/test_core.py index 44686ac5..12bb7c41 100644 --- a/backend/app/tests/api/routes/projects/test_core.py +++ b/backend/app/tests/api/routes/projects/test_core.py @@ -1,12 +1,17 @@ """Tests for app.api.routes.projects.core endpoints.""" +import uuid from types import SimpleNamespace from unittest.mock import ANY, patch +from app import users from app.api.routes.projects.core import get_project_comments from app.config import settings -from app.models.core import ContentsItem +from app.models import Project, UserCreate +from app.models.core import ContentsItem, ProjectMembership +from app.tests import authentication_token_from_email, create_random_user from fastapi.testclient import TestClient +from sqlmodel import Session, select def test_get_project_contents_forwards_ref(client: TestClient) -> None: @@ -801,3 +806,136 @@ def test_get_project_presentations_reads_declared_at_ref( _ref_aware_endpoint_reads_declared_at_ref( client, "presentations", "presentations" ) + + +# --- Project membership & invitation links (I2) --------------------------- + + +def _make_owner_with_project( + db: Session, client: TestClient +) -> tuple[Project, dict[str, str]]: + """Create a project owner (with GitHub) + a private project, return the + project and the owner's auth headers. + """ + suffix = uuid.uuid4().hex[:8] + owner = users.create_user( + session=db, + user_create=UserCreate( + email=f"owner-{suffix}@example.com", + password="ownerpassword123", + account_name=f"owner{suffix}", + github_username=f"owner{suffix}", + ), + ) + project = Project( + name=f"proj-{suffix}", + title="Invite Test Project", + git_repo_url=f"https://github.com/owner{suffix}/proj-{suffix}", + owner_account_id=owner.account.id, + owner_account=owner.account, + ) + db.add(project) + db.commit() + db.refresh(project) + headers = authentication_token_from_email( + client=client, email=owner.email, db=db + ) + return project, headers + + +def test_invitation_create_and_redeem_grants_access( + client: TestClient, db: Session +) -> None: + project, owner_headers = _make_owner_with_project(db, client) + owner_name = project.owner_account.name + base = f"{settings.API_V1_STR}/projects/{owner_name}/{project.name}" + + # A GitHub-less user has no access to the private project yet. + ghless = create_random_user(db) + assert ghless.account.github_name is None + ghless_headers = authentication_token_from_email( + client=client, email=ghless.email, db=db + ) + r = client.get(base, headers=ghless_headers) + assert r.status_code == 403 + + # Owner creates an invite link. + r = client.post( + f"{base}/invitations", + headers=owner_headers, + json={"role": "write", "max_uses": 5}, + ) + assert r.status_code == 200, r.text + invite = r.json() + assert invite["role_name"] == "write" + assert invite["token"] + assert f"/join/{invite['token']}" in invite["url"] + token = invite["token"] + + # The GitHub-less user redeems it and gains write membership. + r = client.post( + f"{settings.API_V1_STR}/project-invitations/{token}", + headers=ghless_headers, + ) + assert r.status_code == 200, r.text + redeemed = r.json() + assert redeemed["owner_name"] == owner_name + assert redeemed["project_name"] == project.name + assert redeemed["role_name"] == "write" + + # Membership row exists and the user can now read the project. + membership = db.exec( + select(ProjectMembership) + .where(ProjectMembership.project_id == project.id) + .where(ProjectMembership.user_id == ghless.id) + ).first() + assert membership is not None and membership.role_name == "write" + r = client.get(base, headers=ghless_headers) + assert r.status_code == 200 + + +def test_invitation_create_requires_admin( + client: TestClient, db: Session +) -> None: + project, _ = _make_owner_with_project(db, client) + owner_name = project.owner_account.name + # A random non-member cannot create invitations. + other = create_random_user(db) + other_headers = authentication_token_from_email( + client=client, email=other.email, db=db + ) + r = client.post( + f"{settings.API_V1_STR}/projects/{owner_name}/{project.name}" + "/invitations", + headers=other_headers, + json={"role": "write"}, + ) + assert r.status_code == 403 + + +def test_redeem_revoked_invitation_fails( + client: TestClient, db: Session +) -> None: + project, owner_headers = _make_owner_with_project(db, client) + owner_name = project.owner_account.name + base = f"{settings.API_V1_STR}/projects/{owner_name}/{project.name}" + r = client.post( + f"{base}/invitations", headers=owner_headers, json={"role": "read"} + ) + assert r.status_code == 200 + invite = r.json() + # Revoke it. + r = client.delete( + f"{base}/invitations/{invite['id']}", headers=owner_headers + ) + assert r.status_code == 200 + # Redeeming a revoked invite is rejected. + redeemer = create_random_user(db) + redeemer_headers = authentication_token_from_email( + client=client, email=redeemer.email, db=db + ) + r = client.post( + f"{settings.API_V1_STR}/project-invitations/{invite['token']}", + headers=redeemer_headers, + ) + assert r.status_code == 410 diff --git a/backend/app/tests/api/routes/test_users.py b/backend/app/tests/api/routes/test_users.py index e3f66e9a..e3a77931 100644 --- a/backend/app/tests/api/routes/test_users.py +++ b/backend/app/tests/api/routes/test_users.py @@ -286,15 +286,24 @@ def test_update_password_me_same_password_error( def test_register_user(client: TestClient, db: Session) -> None: - username = random_email() + """A user can self-register with email + password and no GitHub account.""" + email = random_email() password = random_lower_string() full_name = random_lower_string() - data = {"email": username, "password": password, "full_name": full_name} + data = {"email": email, "password": password, "full_name": full_name} r = client.post( f"{settings.API_V1_STR}/users/signup", json=data, ) - assert r.status_code == 501 + assert 200 <= r.status_code < 300 + created = r.json() + assert created["email"] == email + # GitHub-less signup: no GitHub username on the public payload + assert created["github_username"] is None + user = users.get_user_by_email(session=db, email=email) + assert user is not None + assert user.account.github_name is None + assert verify_password(password, user.hashed_password) def test_register_user_already_exists_error(client: TestClient) -> None: @@ -309,7 +318,31 @@ def test_register_user_already_exists_error(client: TestClient) -> None: f"{settings.API_V1_STR}/users/signup", json=data, ) - assert r.status_code == 501 + assert r.status_code == 400 + + +def test_github_less_user_cannot_create_project(client: TestClient) -> None: + """GitHub-less users can sign up but cannot own projects (yet).""" + email = random_email() + password = random_lower_string() + r = client.post( + f"{settings.API_V1_STR}/users/signup", + json={"email": email, "password": password}, + ) + assert 200 <= r.status_code < 300 + login = client.post( + f"{settings.API_V1_STR}/login/access-token", + data={"username": email, "password": password}, + ) + assert login.status_code == 200 + headers = {"Authorization": f"Bearer {login.json()['access_token']}"} + r = client.post( + f"{settings.API_V1_STR}/projects", + headers=headers, + json={"name": "ghless-project", "title": "GitHub-less project"}, + ) + assert r.status_code == 403 + assert "GitHub" in r.json()["detail"] def test_update_user( diff --git a/backend/app/tests/test_git.py b/backend/app/tests/test_git.py index c18b272b..9dcb0390 100644 --- a/backend/app/tests/test_git.py +++ b/backend/app/tests/test_git.py @@ -3,11 +3,23 @@ from pathlib import Path import git +import pytest +from fastapi import HTTPException import app.git +import app.github import app.projects +class _FakeResp: + def __init__(self, status_code: int, payload: dict) -> None: + self.status_code = status_code + self._payload = payload + + def json(self) -> dict: + return self._payload + + def _init_repo(repo_dir: Path) -> tuple[git.Repo, str]: repo = git.Repo.init(repo_dir) repo.git.config(["user.name", "CI Test"]) @@ -125,6 +137,45 @@ def test_get_file_history_dvc_lock(tmp_path, monkeypatch): assert history[0]["committed_date"] >= history[-1]["committed_date"] +def test_get_app_installation_token(monkeypatch) -> None: + """The App JWT is exchanged for a repo-scoped installation token.""" + calls: dict = {} + monkeypatch.setattr(app.github, "create_app_token", lambda: "fake-jwt") + + def fake_get(url, headers=None, timeout=None): + calls["get_url"] = url + calls["get_auth"] = headers["Authorization"] + return _FakeResp(200, {"id": 12345}) + + def fake_post(url, headers=None, json=None, timeout=None): + calls["post_url"] = url + calls["post_json"] = json + return _FakeResp(201, {"token": "ghs_installationtoken"}) + + monkeypatch.setattr(app.github.requests, "get", fake_get) + monkeypatch.setattr(app.github.requests, "post", fake_post) + + token = app.github.get_app_installation_token("owner-acct", "my-repo") + assert token == "ghs_installationtoken" + assert calls["get_url"].endswith("/repos/owner-acct/my-repo/installation") + assert calls["get_auth"] == "Bearer fake-jwt" + assert "/app/installations/12345/access_tokens" in calls["post_url"] + assert calls["post_json"] == {"repositories": ["my-repo"]} + + +def test_get_app_installation_token_no_installation(monkeypatch) -> None: + """A missing installation surfaces as a 502, not a crash.""" + monkeypatch.setattr(app.github, "create_app_token", lambda: "fake-jwt") + monkeypatch.setattr( + app.github.requests, + "get", + lambda *a, **k: _FakeResp(404, {}), + ) + with pytest.raises(HTTPException) as exc: + app.github.get_app_installation_token("owner", "repo") + assert exc.value.status_code == 502 + + def test_get_ck_info_from_repo_valid(tmp_path): """A well-formed calkit.yaml is loaded into a dict.""" repo, _ = _init_repo(tmp_path / "repo") diff --git a/backend/app/users.py b/backend/app/users.py index f0bfd017..d56fc407 100644 --- a/backend/app/users.py +++ b/backend/app/users.py @@ -120,7 +120,9 @@ def create_user(*, session: Session, user_create: UserCreate) -> User: account_name = user_create.account_name or user_create.github_username if not account_name: account_name = user_create.email.split("@")[0] - github_name = user_create.github_username or account_name + # Only set a GitHub name when the user actually has a GitHub account; + # GitHub-less (email/Google) signups leave it null. + github_name = user_create.github_username if account_name.lower() in INVALID_ACCOUNT_NAMES: raise HTTPException(422, "Invalid account name") existing = session.exec( diff --git a/frontend/package-lock.json b/frontend/package-lock.json index cdb619ff..77690db9 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -10,6 +10,10 @@ "dependencies": { "@chakra-ui/icons": "2.1.1", "@chakra-ui/react": "2.8.2", + "@codemirror/language": "^6.12.4", + "@codemirror/legacy-modes": "^6.5.3", + "@codemirror/state": "^6.7.0", + "@codemirror/view": "^6.43.3", "@emotion/react": "11.13.0", "@emotion/styled": "11.13.0", "@stripe/react-stripe-js": "^2.8.0", @@ -19,6 +23,7 @@ "@tanstack/react-router": "^1.48.1", "@types/react-syntax-highlighter": "^15.5.13", "axios": "^1.15.0", + "codemirror": "^6.0.2", "d3-selection": "^3.0.0", "d3-zoom": "^3.0.0", "form-data": "4.0.4", @@ -1910,6 +1915,96 @@ "resolved": "https://registry.npmjs.org/commander/-/commander-2.20.3.tgz", "integrity": "sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==" }, + "node_modules/@codemirror/autocomplete": { + "version": "6.20.3", + "resolved": "https://registry.npmjs.org/@codemirror/autocomplete/-/autocomplete-6.20.3.tgz", + "integrity": "sha512-tlosUqb+3BbxCxZdu4tKeRghPFC+QM7q4X5YhKV2eCmPG+1r2F3f4AaSz5sCrFqUtX4Jh20VFTKecl16MgiV9g==", + "license": "MIT", + "dependencies": { + "@codemirror/language": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.17.0", + "@lezer/common": "^1.0.0" + } + }, + "node_modules/@codemirror/commands": { + "version": "6.10.4", + "resolved": "https://registry.npmjs.org/@codemirror/commands/-/commands-6.10.4.tgz", + "integrity": "sha512-Ryk9y9T0FFVF0cUGhAknveAyUOl/A1qReTFi+qPKtOh2Z9F4AUBz3XOrYD4ZEgZirdugVzHvd/2/Wcwy5OliTg==", + "license": "MIT", + "dependencies": { + "@codemirror/language": "^6.0.0", + "@codemirror/state": "^6.7.0", + "@codemirror/view": "^6.27.0", + "@lezer/common": "^1.1.0" + } + }, + "node_modules/@codemirror/language": { + "version": "6.12.4", + "resolved": "https://registry.npmjs.org/@codemirror/language/-/language-6.12.4.tgz", + "integrity": "sha512-1q4PaT+o6PbgpkJt4Q8Fv5XJxTy4FUZ4MWETtyiDw3J0Pyr9E2vqcKL+k9wcvjNTIsauxvE7OfmWj3FRPHQ76A==", + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.23.0", + "@lezer/common": "^1.5.0", + "@lezer/highlight": "^1.0.0", + "@lezer/lr": "^1.0.0", + "style-mod": "^4.0.0" + } + }, + "node_modules/@codemirror/legacy-modes": { + "version": "6.5.3", + "resolved": "https://registry.npmjs.org/@codemirror/legacy-modes/-/legacy-modes-6.5.3.tgz", + "integrity": "sha512-xCsmIzH78MyWkib9jlPaaun57XNkfbMIhagfaZVd0iLTqlpw3jXaIcbZm72MTmmn64eTZpBVNjbyYh+QXnxRsg==", + "license": "MIT", + "dependencies": { + "@codemirror/language": "^6.0.0" + } + }, + "node_modules/@codemirror/lint": { + "version": "6.9.7", + "resolved": "https://registry.npmjs.org/@codemirror/lint/-/lint-6.9.7.tgz", + "integrity": "sha512-28/+iWLYxKxsvGYhSYL7zaCZqLz5+FFFDq9tVsvGv9kv8RY4fFAchJ5WX9M3YrrRlTIsECjsXPqeNgnSmNP2dg==", + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.42.0", + "crelt": "^1.0.5" + } + }, + "node_modules/@codemirror/search": { + "version": "6.7.1", + "resolved": "https://registry.npmjs.org/@codemirror/search/-/search-6.7.1.tgz", + "integrity": "sha512-uMe5UO6PamJtSHrXhhHOzSX3ReWtiJrva6GnPMwSOrZtiExb5X5eExhr2OUZQVvdxPsKpY3Ro2mFbQadpPWmHA==", + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.37.0", + "crelt": "^1.0.5" + } + }, + "node_modules/@codemirror/state": { + "version": "6.7.0", + "resolved": "https://registry.npmjs.org/@codemirror/state/-/state-6.7.0.tgz", + "integrity": "sha512-Zbl9NyscLMZkfXPQnNAIIAFftidrA1UbcJEIMp24C0Bukc2I5T8wJS0wsXYsnDOqCFJUeJ1BITGNs5CqPDSmSg==", + "license": "MIT", + "dependencies": { + "@marijn/find-cluster-break": "^1.0.0" + } + }, + "node_modules/@codemirror/view": { + "version": "6.43.3", + "resolved": "https://registry.npmjs.org/@codemirror/view/-/view-6.43.3.tgz", + "integrity": "sha512-MwEwCAr/o0agJefhC2+reBv5kfOQpMcDRUNQrRYZgWlhH8IwQcerMZrpqWyUFSyO0ebgN2cnh/w87F7G4BGSng==", + "license": "MIT", + "dependencies": { + "@codemirror/state": "^6.7.0", + "crelt": "^1.0.6", + "style-mod": "^4.1.0", + "w3c-keyname": "^2.2.4" + } + }, "node_modules/@csstools/color-helpers": { "version": "5.1.0", "resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-5.1.0.tgz", @@ -2687,6 +2782,30 @@ "dev": true, "license": "MIT" }, + "node_modules/@lezer/common": { + "version": "1.5.2", + "resolved": "https://registry.npmjs.org/@lezer/common/-/common-1.5.2.tgz", + "integrity": "sha512-sxQE460fPZyU3sdc8lafxiPwJHBzZRy/udNFynGQky1SePYBdhkBl1kOagA9uT3pxR8K09bOrmTUqA9wb/PjSQ==", + "license": "MIT" + }, + "node_modules/@lezer/highlight": { + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/@lezer/highlight/-/highlight-1.2.3.tgz", + "integrity": "sha512-qXdH7UqTvGfdVBINrgKhDsVTJTxactNNxLk7+UMwZhU13lMHaOBlJe9Vqp907ya56Y3+ed2tlqzys7jDkTmW0g==", + "license": "MIT", + "dependencies": { + "@lezer/common": "^1.3.0" + } + }, + "node_modules/@lezer/lr": { + "version": "1.4.10", + "resolved": "https://registry.npmjs.org/@lezer/lr/-/lr-1.4.10.tgz", + "integrity": "sha512-rnCpTIBafOx4mRp43xOxDJbFipJm/c0cia/V5TiGlhmMa+wsSdoGmUN3w5Bqrks/09Q/D4tNAmWaT8p6NRi77A==", + "license": "MIT", + "dependencies": { + "@lezer/common": "^1.0.0" + } + }, "node_modules/@mapbox/geojson-rewind": { "version": "0.5.2", "resolved": "https://registry.npmjs.org/@mapbox/geojson-rewind/-/geojson-rewind-0.5.2.tgz", @@ -2841,6 +2960,12 @@ "resolved": "https://registry.npmjs.org/tinyqueue/-/tinyqueue-3.0.0.tgz", "integrity": "sha512-gRa9gwYU3ECmQYv3lslts5hxuIa90veaEcxDYuu3QGOIAEM2mOZkVHp48ANJuu1CURtRdHKUBY5Lm1tHV+sD4g==" }, + "node_modules/@marijn/find-cluster-break": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@marijn/find-cluster-break/-/find-cluster-break-1.0.2.tgz", + "integrity": "sha512-l0h88YhZFyKdXIFNfSWpyjStDjGHwZ/U7iobcK1cQQD8sejsONdQtTVU+1wVN1PBw40PiiHB1vA5S7VTfQiP9g==", + "license": "MIT" + }, "node_modules/@playwright/test": { "version": "1.56.1", "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.56.1.tgz", @@ -5190,6 +5315,21 @@ "node": ">=6" } }, + "node_modules/codemirror": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/codemirror/-/codemirror-6.0.2.tgz", + "integrity": "sha512-VhydHotNW5w1UGK0Qj96BwSk/Zqbp9WbnyK2W/eVMv4QyF41INRGpjUhFJY7/uDNuudSc33a/PKr4iDqRduvHw==", + "license": "MIT", + "dependencies": { + "@codemirror/autocomplete": "^6.0.0", + "@codemirror/commands": "^6.0.0", + "@codemirror/language": "^6.0.0", + "@codemirror/lint": "^6.0.0", + "@codemirror/search": "^6.0.0", + "@codemirror/state": "^6.0.0", + "@codemirror/view": "^6.0.0" + } + }, "node_modules/color-alpha": { "version": "1.0.4", "resolved": "https://registry.npmjs.org/color-alpha/-/color-alpha-1.0.4.tgz", @@ -5405,6 +5545,12 @@ "resolved": "https://registry.npmjs.org/country-regex/-/country-regex-1.1.0.tgz", "integrity": "sha512-iSPlClZP8vX7MC3/u6s3lrDuoQyhQukh5LyABJ3hvfzbQ3Yyayd4fp04zjLnfi267B/B2FkumcWWgrbban7sSA==" }, + "node_modules/crelt": { + "version": "1.0.6", + "resolved": "https://registry.npmjs.org/crelt/-/crelt-1.0.6.tgz", + "integrity": "sha512-VQ2MBenTq1fWZUH9DJNGti7kKv6EeAuYr3cLwxUWhIu1baTaXh4Ib5W2CqHVqib4/MqbYGJqiL3Zb8GJZr3l4g==", + "license": "MIT" + }, "node_modules/css-box-model": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/css-box-model/-/css-box-model-1.2.1.tgz", @@ -18050,6 +18196,12 @@ "webpack": "^5.27.0" } }, + "node_modules/style-mod": { + "version": "4.1.3", + "resolved": "https://registry.npmjs.org/style-mod/-/style-mod-4.1.3.tgz", + "integrity": "sha512-i/n8VsZydrugj3Iuzll8+x/00GH2vnYsk1eomD8QiRrSAeW6ItbCQDtfXCeJHd0iwiNagqjQkvpvREEPtW3IoQ==", + "license": "MIT" + }, "node_modules/style-to-js": { "version": "1.1.17", "resolved": "https://registry.npmjs.org/style-to-js/-/style-to-js-1.1.17.tgz", @@ -19469,6 +19621,12 @@ "pbf": "^3.2.1" } }, + "node_modules/w3c-keyname": { + "version": "2.2.8", + "resolved": "https://registry.npmjs.org/w3c-keyname/-/w3c-keyname-2.2.8.tgz", + "integrity": "sha512-dpojBhNsCNN7T82Tm7k26A6G9ML3NkhDsnw9n/eoxSRlVBB4CEtIQ/KTCLI2Fwf3ataSXRhYFkQi3SlnFwPvPQ==", + "license": "MIT" + }, "node_modules/w3c-xmlserializer": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz", diff --git a/frontend/package.json b/frontend/package.json index 2817b1ff..68627ff2 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -15,6 +15,10 @@ "dependencies": { "@chakra-ui/icons": "2.1.1", "@chakra-ui/react": "2.8.2", + "@codemirror/language": "^6.12.4", + "@codemirror/legacy-modes": "^6.5.3", + "@codemirror/state": "^6.7.0", + "@codemirror/view": "^6.43.3", "@emotion/react": "11.13.0", "@emotion/styled": "11.13.0", "@stripe/react-stripe-js": "^2.8.0", @@ -24,6 +28,7 @@ "@tanstack/react-router": "^1.48.1", "@types/react-syntax-highlighter": "^15.5.13", "axios": "^1.15.0", + "codemirror": "^6.0.2", "d3-selection": "^3.0.0", "d3-zoom": "^3.0.0", "form-data": "4.0.4", diff --git a/frontend/public/tex/.gitignore b/frontend/public/tex/.gitignore new file mode 100644 index 00000000..60cf1c9d --- /dev/null +++ b/frontend/public/tex/.gitignore @@ -0,0 +1,9 @@ +# Large busytex engine binaries + TeX Live bundles — not committed. +# Fetch with scripts/download-tex-engine.sh. The small MIT glue +# (busytex_worker.js, busytex_pipeline.js) IS committed. +*.data +*.wasm +busytex.js +busytex.patched.js +texlive-basic.js +ubuntu-texlive-*.js diff --git a/frontend/public/tex/busytex.js.stock b/frontend/public/tex/busytex.js.stock new file mode 100644 index 00000000..6742d15a --- /dev/null +++ b/frontend/public/tex/busytex.js.stock @@ -0,0 +1,8546 @@ + +var busytex = (() => { + var _scriptDir = typeof document !== 'undefined' && document.currentScript ? document.currentScript.src : undefined; + if (typeof __filename !== 'undefined') _scriptDir = _scriptDir || __filename; + return ( +function(moduleArg = {}) { + +// include: shell.js +// The Module object: Our interface to the outside world. We import +// and export values on it. There are various ways Module can be used: +// 1. Not defined. We create it here +// 2. A function parameter, function(Module) { ..generated code.. } +// 3. pre-run appended it, var Module = {}; ..generated code.. +// 4. External script tag defines var Module. +// We need to check if Module already exists (e.g. case 3 above). +// Substitution will be replaced with actual code on later stage of the build, +// this way Closure Compiler will not mangle it (e.g. case 4. above). +// Note that if you want to run closure, and also to use Module +// after the generated code, you will need to define var Module = {}; +// before the code. Then that object will be used in the code, and you +// can continue to use Module afterwards as well. +var Module = moduleArg; + +// Set up the promise that indicates the Module is initialized +var readyPromiseResolve, readyPromiseReject; +Module['ready'] = new Promise((resolve, reject) => { + readyPromiseResolve = resolve; + readyPromiseReject = reject; +}); +["_main","_flush_streams","_fflush","onRuntimeInitialized"].forEach((prop) => { + if (!Object.getOwnPropertyDescriptor(Module['ready'], prop)) { + Object.defineProperty(Module['ready'], prop, { + get: () => abort('You are getting ' + prop + ' on the Promise object, instead of the instance. Use .then() to get called back with the instance, see the MODULARIZE docs in src/settings.js'), + set: () => abort('You are setting ' + prop + ' on the Promise object, instead of the instance. Use .then() to get called back with the instance, see the MODULARIZE docs in src/settings.js'), + }); + } +}); + +// --pre-jses are emitted after the Module integration code, so that they can +// refer to Module (if they choose; they can also define Module) + + +// Sometimes an existing Module object exists with properties +// meant to overwrite the default module functionality. Here +// we collect those properties and reapply _after_ we configure +// the current environment's defaults to avoid having to be so +// defensive during initialization. +var moduleOverrides = Object.assign({}, Module); + +var arguments_ = []; +var thisProgram = './this.program'; +var quit_ = (status, toThrow) => { + throw toThrow; +}; + +// Determine the runtime environment we are in. You can customize this by +// setting the ENVIRONMENT setting at compile time (see settings.js). + +// Attempt to auto-detect the environment +var ENVIRONMENT_IS_WEB = typeof window == 'object'; +var ENVIRONMENT_IS_WORKER = typeof importScripts == 'function'; +// N.b. Electron.js environment is simultaneously a NODE-environment, but +// also a web environment. +var ENVIRONMENT_IS_NODE = typeof process == 'object' && typeof process.versions == 'object' && typeof process.versions.node == 'string'; +var ENVIRONMENT_IS_SHELL = !ENVIRONMENT_IS_WEB && !ENVIRONMENT_IS_NODE && !ENVIRONMENT_IS_WORKER; + +if (Module['ENVIRONMENT']) { + throw new Error('Module.ENVIRONMENT has been deprecated. To force the environment, use the ENVIRONMENT compile-time option (for example, -sENVIRONMENT=web or -sENVIRONMENT=node)'); +} + +// `/` should be present at the end if `scriptDirectory` is not empty +var scriptDirectory = ''; +function locateFile(path) { + if (Module['locateFile']) { + return Module['locateFile'](path, scriptDirectory); + } + return scriptDirectory + path; +} + +// Hooks that are implemented differently in different runtime environments. +var read_, + readAsync, + readBinary, + setWindowTitle; + +if (ENVIRONMENT_IS_NODE) { + if (typeof process == 'undefined' || !process.release || process.release.name !== 'node') throw new Error('not compiled for this environment (did you build to HTML and try to run it not on the web, or set ENVIRONMENT to something - like node - and run it someplace else - like on the web?)'); + + var nodeVersion = process.versions.node; + var numericVersion = nodeVersion.split('.').slice(0, 3); + numericVersion = (numericVersion[0] * 10000) + (numericVersion[1] * 100) + (numericVersion[2].split('-')[0] * 1); + var minVersion = 160000; + if (numericVersion < 160000) { + throw new Error('This emscripten-generated code requires node v16.0.0 (detected v' + nodeVersion + ')'); + } + + // `require()` is no-op in an ESM module, use `createRequire()` to construct + // the require()` function. This is only necessary for multi-environment + // builds, `-sENVIRONMENT=node` emits a static import declaration instead. + // TODO: Swap all `require()`'s with `import()`'s? + // These modules will usually be used on Node.js. Load them eagerly to avoid + // the complexity of lazy-loading. + var fs = require('fs'); + var nodePath = require('path'); + + if (ENVIRONMENT_IS_WORKER) { + scriptDirectory = nodePath.dirname(scriptDirectory) + '/'; + } else { + scriptDirectory = __dirname + '/'; + } + +// include: node_shell_read.js +read_ = (filename, binary) => { + // We need to re-wrap `file://` strings to URLs. Normalizing isn't + // necessary in that case, the path should already be absolute. + filename = isFileURI(filename) ? new URL(filename) : nodePath.normalize(filename); + return fs.readFileSync(filename, binary ? undefined : 'utf8'); +}; + +readBinary = (filename) => { + var ret = read_(filename, true); + if (!ret.buffer) { + ret = new Uint8Array(ret); + } + assert(ret.buffer); + return ret; +}; + +readAsync = (filename, onload, onerror, binary = true) => { + // See the comment in the `read_` function. + filename = isFileURI(filename) ? new URL(filename) : nodePath.normalize(filename); + fs.readFile(filename, binary ? undefined : 'utf8', (err, data) => { + if (err) onerror(err); + else onload(binary ? data.buffer : data); + }); +}; +// end include: node_shell_read.js + if (!Module['thisProgram'] && process.argv.length > 1) { + thisProgram = process.argv[1].replace(/\\/g, '/'); + } + + arguments_ = process.argv.slice(2); + + // MODULARIZE will export the module in the proper place outside, we don't need to export here + + quit_ = (status, toThrow) => { + process.exitCode = status; + throw toThrow; + }; + + Module['inspect'] = () => '[Emscripten Module object]'; + +} else +if (ENVIRONMENT_IS_SHELL) { + + if ((typeof process == 'object' && typeof require === 'function') || typeof window == 'object' || typeof importScripts == 'function') throw new Error('not compiled for this environment (did you build to HTML and try to run it not on the web, or set ENVIRONMENT to something - like node - and run it someplace else - like on the web?)'); + + if (typeof read != 'undefined') { + read_ = (f) => { + return read(f); + }; + } + + readBinary = (f) => { + let data; + if (typeof readbuffer == 'function') { + return new Uint8Array(readbuffer(f)); + } + data = read(f, 'binary'); + assert(typeof data == 'object'); + return data; + }; + + readAsync = (f, onload, onerror) => { + setTimeout(() => onload(readBinary(f))); + }; + + if (typeof clearTimeout == 'undefined') { + globalThis.clearTimeout = (id) => {}; + } + + if (typeof setTimeout == 'undefined') { + // spidermonkey lacks setTimeout but we use it above in readAsync. + globalThis.setTimeout = (f) => (typeof f == 'function') ? f() : abort(); + } + + if (typeof scriptArgs != 'undefined') { + arguments_ = scriptArgs; + } else if (typeof arguments != 'undefined') { + arguments_ = arguments; + } + + if (typeof quit == 'function') { + quit_ = (status, toThrow) => { + // Unlike node which has process.exitCode, d8 has no such mechanism. So we + // have no way to set the exit code and then let the program exit with + // that code when it naturally stops running (say, when all setTimeouts + // have completed). For that reason, we must call `quit` - the only way to + // set the exit code - but quit also halts immediately. To increase + // consistency with node (and the web) we schedule the actual quit call + // using a setTimeout to give the current stack and any exception handlers + // a chance to run. This enables features such as addOnPostRun (which + // expected to be able to run code after main returns). + setTimeout(() => { + if (!(toThrow instanceof ExitStatus)) { + let toLog = toThrow; + if (toThrow && typeof toThrow == 'object' && toThrow.stack) { + toLog = [toThrow, toThrow.stack]; + } + err(`exiting due to exception: ${toLog}`); + } + quit(status); + }); + throw toThrow; + }; + } + + if (typeof print != 'undefined') { + // Prefer to use print/printErr where they exist, as they usually work better. + if (typeof console == 'undefined') console = /** @type{!Console} */({}); + console.log = /** @type{!function(this:Console, ...*): undefined} */ (print); + console.warn = console.error = /** @type{!function(this:Console, ...*): undefined} */ (typeof printErr != 'undefined' ? printErr : print); + } + +} else + +// Note that this includes Node.js workers when relevant (pthreads is enabled). +// Node.js workers are detected as a combination of ENVIRONMENT_IS_WORKER and +// ENVIRONMENT_IS_NODE. +if (ENVIRONMENT_IS_WEB || ENVIRONMENT_IS_WORKER) { + if (ENVIRONMENT_IS_WORKER) { // Check worker, not web, since window could be polyfilled + scriptDirectory = self.location.href; + } else if (typeof document != 'undefined' && document.currentScript) { // web + scriptDirectory = document.currentScript.src; + } + // When MODULARIZE, this JS may be executed later, after document.currentScript + // is gone, so we saved it, and we use it here instead of any other info. + if (_scriptDir) { + scriptDirectory = _scriptDir; + } + // blob urls look like blob:http://site.com/etc/etc and we cannot infer anything from them. + // otherwise, slice off the final part of the url to find the script directory. + // if scriptDirectory does not contain a slash, lastIndexOf will return -1, + // and scriptDirectory will correctly be replaced with an empty string. + // If scriptDirectory contains a query (starting with ?) or a fragment (starting with #), + // they are removed because they could contain a slash. + if (scriptDirectory.indexOf('blob:') !== 0) { + scriptDirectory = scriptDirectory.substr(0, scriptDirectory.replace(/[?#].*/, "").lastIndexOf('/')+1); + } else { + scriptDirectory = ''; + } + + if (!(typeof window == 'object' || typeof importScripts == 'function')) throw new Error('not compiled for this environment (did you build to HTML and try to run it not on the web, or set ENVIRONMENT to something - like node - and run it someplace else - like on the web?)'); + + // Differentiate the Web Worker from the Node Worker case, as reading must + // be done differently. + { +// include: web_or_worker_shell_read.js +read_ = (url) => { + var xhr = new XMLHttpRequest(); + xhr.open('GET', url, false); + xhr.send(null); + return xhr.responseText; + } + + if (ENVIRONMENT_IS_WORKER) { + readBinary = (url) => { + var xhr = new XMLHttpRequest(); + xhr.open('GET', url, false); + xhr.responseType = 'arraybuffer'; + xhr.send(null); + return new Uint8Array(/** @type{!ArrayBuffer} */(xhr.response)); + }; + } + + readAsync = (url, onload, onerror) => { + var xhr = new XMLHttpRequest(); + xhr.open('GET', url, true); + xhr.responseType = 'arraybuffer'; + xhr.onload = () => { + if (xhr.status == 200 || (xhr.status == 0 && xhr.response)) { // file URLs can return 0 + onload(xhr.response); + return; + } + onerror(); + }; + xhr.onerror = onerror; + xhr.send(null); + } + +// end include: web_or_worker_shell_read.js + } + + setWindowTitle = (title) => document.title = title; +} else +{ + throw new Error('environment detection error'); +} + +var out = Module['print'] || console.log.bind(console); +var err = Module['printErr'] || console.error.bind(console); + +// Merge back in the overrides +Object.assign(Module, moduleOverrides); +// Free the object hierarchy contained in the overrides, this lets the GC +// reclaim data used e.g. in memoryInitializerRequest, which is a large typed array. +moduleOverrides = null; +checkIncomingModuleAPI(); + +// Emit code to handle expected values on the Module object. This applies Module.x +// to the proper local x. This has two benefits: first, we only emit it if it is +// expected to arrive, and second, by using a local everywhere else that can be +// minified. + +if (Module['arguments']) arguments_ = Module['arguments'];legacyModuleProp('arguments', 'arguments_'); + +if (Module['thisProgram']) thisProgram = Module['thisProgram'];legacyModuleProp('thisProgram', 'thisProgram'); + +if (Module['quit']) quit_ = Module['quit'];legacyModuleProp('quit', 'quit_'); + +// perform assertions in shell.js after we set up out() and err(), as otherwise if an assertion fails it cannot print the message +// Assertions on removed incoming Module JS APIs. +assert(typeof Module['memoryInitializerPrefixURL'] == 'undefined', 'Module.memoryInitializerPrefixURL option was removed, use Module.locateFile instead'); +assert(typeof Module['pthreadMainPrefixURL'] == 'undefined', 'Module.pthreadMainPrefixURL option was removed, use Module.locateFile instead'); +assert(typeof Module['cdInitializerPrefixURL'] == 'undefined', 'Module.cdInitializerPrefixURL option was removed, use Module.locateFile instead'); +assert(typeof Module['filePackagePrefixURL'] == 'undefined', 'Module.filePackagePrefixURL option was removed, use Module.locateFile instead'); +assert(typeof Module['read'] == 'undefined', 'Module.read option was removed (modify read_ in JS)'); +assert(typeof Module['readAsync'] == 'undefined', 'Module.readAsync option was removed (modify readAsync in JS)'); +assert(typeof Module['readBinary'] == 'undefined', 'Module.readBinary option was removed (modify readBinary in JS)'); +assert(typeof Module['setWindowTitle'] == 'undefined', 'Module.setWindowTitle option was removed (modify setWindowTitle in JS)'); +assert(typeof Module['TOTAL_MEMORY'] == 'undefined', 'Module.TOTAL_MEMORY has been renamed Module.INITIAL_MEMORY'); +legacyModuleProp('read', 'read_'); +legacyModuleProp('readAsync', 'readAsync'); +legacyModuleProp('readBinary', 'readBinary'); +legacyModuleProp('setWindowTitle', 'setWindowTitle'); +var IDBFS = 'IDBFS is no longer included by default; build with -lidbfs.js'; +var PROXYFS = 'PROXYFS is no longer included by default; build with -lproxyfs.js'; +var WORKERFS = 'WORKERFS is no longer included by default; build with -lworkerfs.js'; +var NODEFS = 'NODEFS is no longer included by default; build with -lnodefs.js'; + +assert(!ENVIRONMENT_IS_SHELL, "shell environment detected but not enabled at build time. Add 'shell' to `-sENVIRONMENT` to enable."); + + +// end include: shell.js +// include: preamble.js +// === Preamble library stuff === + +// Documentation for the public APIs defined in this file must be updated in: +// site/source/docs/api_reference/preamble.js.rst +// A prebuilt local version of the documentation is available at: +// site/build/text/docs/api_reference/preamble.js.txt +// You can also build docs locally as HTML or other formats in site/ +// An online HTML version (which may be of a different version of Emscripten) +// is up at http://kripken.github.io/emscripten-site/docs/api_reference/preamble.js.html + +var wasmBinary; +if (Module['wasmBinary']) wasmBinary = Module['wasmBinary'];legacyModuleProp('wasmBinary', 'wasmBinary'); +var noExitRuntime = Module['noExitRuntime'] || true;legacyModuleProp('noExitRuntime', 'noExitRuntime'); + +if (typeof WebAssembly != 'object') { + abort('no native wasm support detected'); +} + +// Wasm globals + +var wasmMemory; + +//======================================== +// Runtime essentials +//======================================== + +// whether we are quitting the application. no code should run after this. +// set in exit() and abort() +var ABORT = false; + +// set by exit() and abort(). Passed to 'onExit' handler. +// NOTE: This is also used as the process return code code in shell environments +// but only when noExitRuntime is false. +var EXITSTATUS; + +/** @type {function(*, string=)} */ +function assert(condition, text) { + if (!condition) { + abort('Assertion failed' + (text ? ': ' + text : '')); + } +} + +// We used to include malloc/free by default in the past. Show a helpful error in +// builds with assertions. + +// Memory management + +var HEAP, +/** @type {!Int8Array} */ + HEAP8, +/** @type {!Uint8Array} */ + HEAPU8, +/** @type {!Int16Array} */ + HEAP16, +/** @type {!Uint16Array} */ + HEAPU16, +/** @type {!Int32Array} */ + HEAP32, +/** @type {!Uint32Array} */ + HEAPU32, +/** @type {!Float32Array} */ + HEAPF32, +/** @type {!Float64Array} */ + HEAPF64; + +function updateMemoryViews() { + var b = wasmMemory.buffer; + Module['HEAP8'] = HEAP8 = new Int8Array(b); + Module['HEAP16'] = HEAP16 = new Int16Array(b); + Module['HEAP32'] = HEAP32 = new Int32Array(b); + Module['HEAPU8'] = HEAPU8 = new Uint8Array(b); + Module['HEAPU16'] = HEAPU16 = new Uint16Array(b); + Module['HEAPU32'] = HEAPU32 = new Uint32Array(b); + Module['HEAPF32'] = HEAPF32 = new Float32Array(b); + Module['HEAPF64'] = HEAPF64 = new Float64Array(b); +} + +assert(!Module['STACK_SIZE'], 'STACK_SIZE can no longer be set at runtime. Use -sSTACK_SIZE at link time') + +assert(typeof Int32Array != 'undefined' && typeof Float64Array !== 'undefined' && Int32Array.prototype.subarray != undefined && Int32Array.prototype.set != undefined, + 'JS engine does not provide full typed array support'); + +// If memory is defined in wasm, the user can't provide it, or set INITIAL_MEMORY +assert(!Module['wasmMemory'], 'Use of `wasmMemory` detected. Use -sIMPORTED_MEMORY to define wasmMemory externally'); +assert(!Module['INITIAL_MEMORY'], 'Detected runtime INITIAL_MEMORY setting. Use -sIMPORTED_MEMORY to define wasmMemory dynamically'); + +// include: runtime_init_table.js +// In regular non-RELOCATABLE mode the table is exported +// from the wasm module and this will be assigned once +// the exports are available. +var wasmTable; +// end include: runtime_init_table.js +// include: runtime_stack_check.js +// Initializes the stack cookie. Called at the startup of main and at the startup of each thread in pthreads mode. +function writeStackCookie() { + var max = _emscripten_stack_get_end(); + assert((max & 3) == 0); + // If the stack ends at address zero we write our cookies 4 bytes into the + // stack. This prevents interference with SAFE_HEAP and ASAN which also + // monitor writes to address zero. + if (max == 0) { + max += 4; + } + // The stack grow downwards towards _emscripten_stack_get_end. + // We write cookies to the final two words in the stack and detect if they are + // ever overwritten. + HEAPU32[((max)>>2)] = 0x02135467; + HEAPU32[(((max)+(4))>>2)] = 0x89BACDFE; + // Also test the global address 0 for integrity. + HEAPU32[((0)>>2)] = 1668509029; +} + +function checkStackCookie() { + if (ABORT) return; + var max = _emscripten_stack_get_end(); + // See writeStackCookie(). + if (max == 0) { + max += 4; + } + var cookie1 = HEAPU32[((max)>>2)]; + var cookie2 = HEAPU32[(((max)+(4))>>2)]; + if (cookie1 != 0x02135467 || cookie2 != 0x89BACDFE) { + abort(`Stack overflow! Stack cookie has been overwritten at ${ptrToString(max)}, expected hex dwords 0x89BACDFE and 0x2135467, but received ${ptrToString(cookie2)} ${ptrToString(cookie1)}`); + } + // Also test the global address 0 for integrity. + if (HEAPU32[((0)>>2)] != 0x63736d65 /* 'emsc' */) { + abort('Runtime error: The application has corrupted its heap memory area (address zero)!'); + } +} +// end include: runtime_stack_check.js +// include: runtime_assertions.js +// Endianness check +(function() { + var h16 = new Int16Array(1); + var h8 = new Int8Array(h16.buffer); + h16[0] = 0x6373; + if (h8[0] !== 0x73 || h8[1] !== 0x63) throw 'Runtime error: expected the system to be little-endian! (Run with -sSUPPORT_BIG_ENDIAN to bypass)'; +})(); + +// end include: runtime_assertions.js +var __ATPRERUN__ = []; // functions called before the runtime is initialized +var __ATINIT__ = []; // functions called during startup +var __ATMAIN__ = []; // functions called when main() is to be run +var __ATEXIT__ = []; // functions called during shutdown +var __ATPOSTRUN__ = []; // functions called after the main() is called + +var runtimeInitialized = false; + +var runtimeKeepaliveCounter = 0; + +function keepRuntimeAlive() { + return noExitRuntime || runtimeKeepaliveCounter > 0; +} + +function preRun() { + if (Module['preRun']) { + if (typeof Module['preRun'] == 'function') Module['preRun'] = [Module['preRun']]; + while (Module['preRun'].length) { + addOnPreRun(Module['preRun'].shift()); + } + } + callRuntimeCallbacks(__ATPRERUN__); +} + +function initRuntime() { + assert(!runtimeInitialized); + runtimeInitialized = true; + + checkStackCookie(); + + +if (!Module["noFSInit"] && !FS.init.initialized) + FS.init(); +FS.ignorePermissions = false; + +TTY.init(); +SOCKFS.root = FS.mount(SOCKFS, {}, null); +PIPEFS.root = FS.mount(PIPEFS, {}, null); + callRuntimeCallbacks(__ATINIT__); +} + +function preMain() { + checkStackCookie(); + + callRuntimeCallbacks(__ATMAIN__); +} + +function postRun() { + checkStackCookie(); + + if (Module['postRun']) { + if (typeof Module['postRun'] == 'function') Module['postRun'] = [Module['postRun']]; + while (Module['postRun'].length) { + addOnPostRun(Module['postRun'].shift()); + } + } + + callRuntimeCallbacks(__ATPOSTRUN__); +} + +function addOnPreRun(cb) { + __ATPRERUN__.unshift(cb); +} + +function addOnInit(cb) { + __ATINIT__.unshift(cb); +} + +function addOnPreMain(cb) { + __ATMAIN__.unshift(cb); +} + +function addOnExit(cb) { +} + +function addOnPostRun(cb) { + __ATPOSTRUN__.unshift(cb); +} + +// include: runtime_math.js +// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/imul + +// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/fround + +// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/clz32 + +// https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Math/trunc + +assert(Math.imul, 'This browser does not support Math.imul(), build with LEGACY_VM_SUPPORT or POLYFILL_OLD_MATH_FUNCTIONS to add in a polyfill'); +assert(Math.fround, 'This browser does not support Math.fround(), build with LEGACY_VM_SUPPORT or POLYFILL_OLD_MATH_FUNCTIONS to add in a polyfill'); +assert(Math.clz32, 'This browser does not support Math.clz32(), build with LEGACY_VM_SUPPORT or POLYFILL_OLD_MATH_FUNCTIONS to add in a polyfill'); +assert(Math.trunc, 'This browser does not support Math.trunc(), build with LEGACY_VM_SUPPORT or POLYFILL_OLD_MATH_FUNCTIONS to add in a polyfill'); +// end include: runtime_math.js +// A counter of dependencies for calling run(). If we need to +// do asynchronous work before running, increment this and +// decrement it. Incrementing must happen in a place like +// Module.preRun (used by emcc to add file preloading). +// Note that you can add dependencies in preRun, even though +// it happens right before run - run will be postponed until +// the dependencies are met. +var runDependencies = 0; +var runDependencyWatcher = null; +var dependenciesFulfilled = null; // overridden to take different actions when all run dependencies are fulfilled +var runDependencyTracking = {}; + +function getUniqueRunDependency(id) { + var orig = id; + while (1) { + if (!runDependencyTracking[id]) return id; + id = orig + Math.random(); + } +} + +function addRunDependency(id) { + runDependencies++; + + if (Module['monitorRunDependencies']) { + Module['monitorRunDependencies'](runDependencies); + } + + if (id) { + assert(!runDependencyTracking[id]); + runDependencyTracking[id] = 1; + if (runDependencyWatcher === null && typeof setInterval != 'undefined') { + // Check for missing dependencies every few seconds + runDependencyWatcher = setInterval(() => { + if (ABORT) { + clearInterval(runDependencyWatcher); + runDependencyWatcher = null; + return; + } + var shown = false; + for (var dep in runDependencyTracking) { + if (!shown) { + shown = true; + err('still waiting on run dependencies:'); + } + err('dependency: ' + dep); + } + if (shown) { + err('(end of list)'); + } + }, 10000); + } + } else { + err('warning: run dependency added without ID'); + } +} + +function removeRunDependency(id) { + runDependencies--; + + if (Module['monitorRunDependencies']) { + Module['monitorRunDependencies'](runDependencies); + } + + if (id) { + assert(runDependencyTracking[id]); + delete runDependencyTracking[id]; + } else { + err('warning: run dependency removed without ID'); + } + if (runDependencies == 0) { + if (runDependencyWatcher !== null) { + clearInterval(runDependencyWatcher); + runDependencyWatcher = null; + } + if (dependenciesFulfilled) { + var callback = dependenciesFulfilled; + dependenciesFulfilled = null; + callback(); // can add another dependenciesFulfilled + } + } +} + +/** @param {string|number=} what */ +function abort(what) { + if (Module['onAbort']) { + Module['onAbort'](what); + } + + what = 'Aborted(' + what + ')'; + // TODO(sbc): Should we remove printing and leave it up to whoever + // catches the exception? + err(what); + + ABORT = true; + EXITSTATUS = 1; + + // Use a wasm runtime error, because a JS error might be seen as a foreign + // exception, which means we'd run destructors on it. We need the error to + // simply make the program stop. + // FIXME This approach does not work in Wasm EH because it currently does not assume + // all RuntimeErrors are from traps; it decides whether a RuntimeError is from + // a trap or not based on a hidden field within the object. So at the moment + // we don't have a way of throwing a wasm trap from JS. TODO Make a JS API that + // allows this in the wasm spec. + + // Suppress closure compiler warning here. Closure compiler's builtin extern + // defintion for WebAssembly.RuntimeError claims it takes no arguments even + // though it can. + // TODO(https://github.com/google/closure-compiler/pull/3913): Remove if/when upstream closure gets fixed. + /** @suppress {checkTypes} */ + var e = new WebAssembly.RuntimeError(what); + + readyPromiseReject(e); + // Throw the error whether or not MODULARIZE is set because abort is used + // in code paths apart from instantiation where an exception is expected + // to be thrown when abort is called. + throw e; +} + +// include: memoryprofiler.js +// end include: memoryprofiler.js +// include: URIUtils.js +// Prefix of data URIs emitted by SINGLE_FILE and related options. +var dataURIPrefix = 'data:application/octet-stream;base64,'; + +// Indicates whether filename is a base64 data URI. +function isDataURI(filename) { + // Prefix of data URIs emitted by SINGLE_FILE and related options. + return filename.startsWith(dataURIPrefix); +} + +// Indicates whether filename is delivered via file protocol (as opposed to http/https) +function isFileURI(filename) { + return filename.startsWith('file://'); +} +// end include: URIUtils.js +/** @param {boolean=} fixedasm */ +function createExportWrapper(name, fixedasm) { + return function() { + var displayName = name; + var asm = fixedasm; + if (!fixedasm) { + asm = Module['asm']; + } + assert(runtimeInitialized, 'native function `' + displayName + '` called before runtime initialization'); + if (!asm[name]) { + assert(asm[name], 'exported native function `' + displayName + '` not found'); + } + return asm[name].apply(null, arguments); + }; +} + +// include: runtime_exceptions.js +// end include: runtime_exceptions.js +var wasmBinaryFile; + wasmBinaryFile = 'busytex.wasm'; + if (!isDataURI(wasmBinaryFile)) { + wasmBinaryFile = locateFile(wasmBinaryFile); + } + +function getBinarySync(file) { + if (file == wasmBinaryFile && wasmBinary) { + return new Uint8Array(wasmBinary); + } + if (readBinary) { + return readBinary(file); + } + throw "both async and sync fetching of the wasm failed"; +} + +function getBinaryPromise(binaryFile) { + // If we don't have the binary yet, try to load it asynchronously. + // Fetch has some additional restrictions over XHR, like it can't be used on a file:// url. + // See https://github.com/github/fetch/pull/92#issuecomment-140665932 + // Cordova or Electron apps are typically loaded from a file:// url. + // So use fetch if it is available and the url is not a file, otherwise fall back to XHR. + if (!wasmBinary && (ENVIRONMENT_IS_WEB || ENVIRONMENT_IS_WORKER)) { + if (typeof fetch == 'function' + && !isFileURI(binaryFile) + ) { + return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => { + if (!response['ok']) { + throw "failed to load wasm binary file at '" + binaryFile + "'"; + } + return response['arrayBuffer'](); + }).catch(() => getBinarySync(binaryFile)); + } + else if (readAsync) { + // fetch is not available or url is file => try XHR (readAsync uses XHR internally) + return new Promise((resolve, reject) => { + readAsync(binaryFile, (response) => resolve(new Uint8Array(/** @type{!ArrayBuffer} */(response))), reject) + }); + } + } + + // Otherwise, getBinarySync should be able to get it synchronously + return Promise.resolve().then(() => getBinarySync(binaryFile)); +} + +function instantiateArrayBuffer(binaryFile, imports, receiver) { + return getBinaryPromise(binaryFile).then((binary) => { + return WebAssembly.instantiate(binary, imports); + }).then((instance) => { + return instance; + }).then(receiver, (reason) => { + err('failed to asynchronously prepare wasm: ' + reason); + + // Warn on some common problems. + if (isFileURI(wasmBinaryFile)) { + err('warning: Loading from a file URI (' + wasmBinaryFile + ') is not supported in most browsers. See https://emscripten.org/docs/getting_started/FAQ.html#how-do-i-run-a-local-webserver-for-testing-why-does-my-program-stall-in-downloading-or-preparing'); + } + abort(reason); + }); +} + +function instantiateAsync(binary, binaryFile, imports, callback) { + if (!binary && + typeof WebAssembly.instantiateStreaming == 'function' && + !isDataURI(binaryFile) && + // Don't use streaming for file:// delivered objects in a webview, fetch them synchronously. + !isFileURI(binaryFile) && + // Avoid instantiateStreaming() on Node.js environment for now, as while + // Node.js v18.1.0 implements it, it does not have a full fetch() + // implementation yet. + // + // Reference: + // https://github.com/emscripten-core/emscripten/pull/16917 + !ENVIRONMENT_IS_NODE && + typeof fetch == 'function') { + return fetch(binaryFile, { credentials: 'same-origin' }).then((response) => { + // Suppress closure warning here since the upstream definition for + // instantiateStreaming only allows Promise rather than + // an actual Response. + // TODO(https://github.com/google/closure-compiler/pull/3913): Remove if/when upstream closure is fixed. + /** @suppress {checkTypes} */ + var result = WebAssembly.instantiateStreaming(response, imports); + + return result.then( + callback, + function(reason) { + // We expect the most common failure cause to be a bad MIME type for the binary, + // in which case falling back to ArrayBuffer instantiation should work. + err('wasm streaming compile failed: ' + reason); + err('falling back to ArrayBuffer instantiation'); + return instantiateArrayBuffer(binaryFile, imports, callback); + }); + }); + } + return instantiateArrayBuffer(binaryFile, imports, callback); +} + +// Create the wasm instance. +// Receives the wasm imports, returns the exports. +function createWasm() { + // prepare imports + var info = { + 'env': wasmImports, + 'wasi_snapshot_preview1': wasmImports, + }; + // Load the wasm module and create an instance of using native support in the JS engine. + // handle a generated wasm instance, receiving its exports and + // performing other necessary setup + /** @param {WebAssembly.Module=} module*/ + function receiveInstance(instance, module) { + var exports = instance.exports; + + Module['asm'] = exports; + + wasmMemory = Module['asm']['memory']; + assert(wasmMemory, "memory not found in wasm exports"); + // This assertion doesn't hold when emscripten is run in --post-link + // mode. + // TODO(sbc): Read INITIAL_MEMORY out of the wasm file in post-link mode. + //assert(wasmMemory.buffer.byteLength === 536870912); + updateMemoryViews(); + + wasmTable = Module['asm']['__indirect_function_table']; + assert(wasmTable, "table not found in wasm exports"); + + addOnInit(Module['asm']['__wasm_call_ctors']); + + removeRunDependency('wasm-instantiate'); + return exports; + } + // wait for the pthread pool (if any) + addRunDependency('wasm-instantiate'); + + // Prefer streaming instantiation if available. + // Async compilation can be confusing when an error on the page overwrites Module + // (for example, if the order of elements is wrong, and the one defining Module is + // later), so we save Module and check it later. + var trueModule = Module; + function receiveInstantiationResult(result) { + // 'result' is a ResultObject object which has both the module and instance. + // receiveInstance() will swap in the exports (to Module.asm) so they can be called + assert(Module === trueModule, 'the Module object should not be replaced during async compilation - perhaps the order of HTML elements is wrong?'); + trueModule = null; + // TODO: Due to Closure regression https://github.com/google/closure-compiler/issues/3193, the above line no longer optimizes out down to the following line. + // When the regression is fixed, can restore the above PTHREADS-enabled path. + receiveInstance(result['instance']); + } + + // User shell pages can write their own Module.instantiateWasm = function(imports, successCallback) callback + // to manually instantiate the Wasm module themselves. This allows pages to + // run the instantiation parallel to any other async startup actions they are + // performing. + // Also pthreads and wasm workers initialize the wasm instance through this + // path. + if (Module['instantiateWasm']) { + + try { + return Module['instantiateWasm'](info, receiveInstance); + } catch(e) { + err('Module.instantiateWasm callback failed with error: ' + e); + // If instantiation fails, reject the module ready promise. + readyPromiseReject(e); + } + } + + // If instantiation fails, reject the module ready promise. + instantiateAsync(wasmBinary, wasmBinaryFile, info, receiveInstantiationResult).catch(readyPromiseReject); + return {}; // no exports yet; we'll fill them in later +} + +// Globals used by JS i64 conversions (see makeSetValue) +var tempDouble; +var tempI64; + +// include: runtime_debug.js +function legacyModuleProp(prop, newName) { + if (!Object.getOwnPropertyDescriptor(Module, prop)) { + Object.defineProperty(Module, prop, { + configurable: true, + get() { + abort('Module.' + prop + ' has been replaced with plain ' + newName + ' (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name)'); + } + }); + } +} + +function ignoredModuleProp(prop) { + if (Object.getOwnPropertyDescriptor(Module, prop)) { + abort('`Module.' + prop + '` was supplied but `' + prop + '` not included in INCOMING_MODULE_JS_API'); + } +} + +// forcing the filesystem exports a few things by default +function isExportedByForceFilesystem(name) { + return name === 'FS_createPath' || + name === 'FS_createDataFile' || + name === 'FS_createPreloadedFile' || + name === 'FS_unlink' || + name === 'addRunDependency' || + // The old FS has some functionality that WasmFS lacks. + name === 'FS_createLazyFile' || + name === 'FS_createDevice' || + name === 'removeRunDependency'; +} + +function missingGlobal(sym, msg) { + if (typeof globalThis !== 'undefined') { + Object.defineProperty(globalThis, sym, { + configurable: true, + get() { + warnOnce('`' + sym + '` is not longer defined by emscripten. ' + msg); + return undefined; + } + }); + } +} + +missingGlobal('buffer', 'Please use HEAP8.buffer or wasmMemory.buffer'); + +function missingLibrarySymbol(sym) { + if (typeof globalThis !== 'undefined' && !Object.getOwnPropertyDescriptor(globalThis, sym)) { + Object.defineProperty(globalThis, sym, { + configurable: true, + get() { + // Can't `abort()` here because it would break code that does runtime + // checks. e.g. `if (typeof SDL === 'undefined')`. + var msg = '`' + sym + '` is a library symbol and not included by default; add it to your library.js __deps or to DEFAULT_LIBRARY_FUNCS_TO_INCLUDE on the command line'; + // DEFAULT_LIBRARY_FUNCS_TO_INCLUDE requires the name as it appears in + // library.js, which means $name for a JS name with no prefix, or name + // for a JS name like _name. + var librarySymbol = sym; + if (!librarySymbol.startsWith('_')) { + librarySymbol = '$' + sym; + } + msg += " (e.g. -sDEFAULT_LIBRARY_FUNCS_TO_INCLUDE='" + librarySymbol + "')"; + if (isExportedByForceFilesystem(sym)) { + msg += '. Alternatively, forcing filesystem support (-sFORCE_FILESYSTEM) can export this for you'; + } + warnOnce(msg); + return undefined; + } + }); + } + // Any symbol that is not included from the JS libary is also (by definition) + // not exported on the Module object. + unexportedRuntimeSymbol(sym); +} + +function unexportedRuntimeSymbol(sym) { + if (!Object.getOwnPropertyDescriptor(Module, sym)) { + Object.defineProperty(Module, sym, { + configurable: true, + get() { + var msg = "'" + sym + "' was not exported. add it to EXPORTED_RUNTIME_METHODS (see the Emscripten FAQ)"; + if (isExportedByForceFilesystem(sym)) { + msg += '. Alternatively, forcing filesystem support (-sFORCE_FILESYSTEM) can export this for you'; + } + abort(msg); + } + }); + } +} + +// Used by XXXXX_DEBUG settings to output debug messages. +function dbg(text) { + // TODO(sbc): Make this configurable somehow. Its not always convenient for + // logging to show up as warnings. + console.warn.apply(console, arguments); +} +// end include: runtime_debug.js +// === Body === + +// end include: preamble.js + + /** @constructor */ + function ExitStatus(status) { + this.name = 'ExitStatus'; + this.message = `Program terminated with exit(${status})`; + this.status = status; + } + + var callRuntimeCallbacks = (callbacks) => { + while (callbacks.length > 0) { + // Pass the module as the first argument. + callbacks.shift()(Module); + } + }; + + + /** + * @param {number} ptr + * @param {string} type + */ + function getValue(ptr, type = 'i8') { + if (type.endsWith('*')) type = '*'; + switch (type) { + case 'i1': return HEAP8[((ptr)>>0)]; + case 'i8': return HEAP8[((ptr)>>0)]; + case 'i16': return HEAP16[((ptr)>>1)]; + case 'i32': return HEAP32[((ptr)>>2)]; + case 'i64': abort('to do getValue(i64) use WASM_BIGINT'); + case 'float': return HEAPF32[((ptr)>>2)]; + case 'double': return HEAPF64[((ptr)>>3)]; + case '*': return HEAPU32[((ptr)>>2)]; + default: abort(`invalid type for getValue: ${type}`); + } + } + + var ptrToString = (ptr) => { + assert(typeof ptr === 'number'); + // With CAN_ADDRESS_2GB or MEMORY64, pointers are already unsigned. + ptr >>>= 0; + return '0x' + ptr.toString(16).padStart(8, '0'); + }; + + + /** + * @param {number} ptr + * @param {number} value + * @param {string} type + */ + function setValue(ptr, value, type = 'i8') { + if (type.endsWith('*')) type = '*'; + switch (type) { + case 'i1': HEAP8[((ptr)>>0)] = value; break; + case 'i8': HEAP8[((ptr)>>0)] = value; break; + case 'i16': HEAP16[((ptr)>>1)] = value; break; + case 'i32': HEAP32[((ptr)>>2)] = value; break; + case 'i64': abort('to do setValue(i64) use WASM_BIGINT'); + case 'float': HEAPF32[((ptr)>>2)] = value; break; + case 'double': HEAPF64[((ptr)>>3)] = value; break; + case '*': HEAPU32[((ptr)>>2)] = value; break; + default: abort(`invalid type for setValue: ${type}`); + } + } + + var warnOnce = (text) => { + if (!warnOnce.shown) warnOnce.shown = {}; + if (!warnOnce.shown[text]) { + warnOnce.shown[text] = 1; + if (ENVIRONMENT_IS_NODE) text = 'warning: ' + text; + err(text); + } + }; + + var UTF8Decoder = typeof TextDecoder != 'undefined' ? new TextDecoder('utf8') : undefined; + + /** + * Given a pointer 'idx' to a null-terminated UTF8-encoded string in the given + * array that contains uint8 values, returns a copy of that string as a + * Javascript String object. + * heapOrArray is either a regular array, or a JavaScript typed array view. + * @param {number} idx + * @param {number=} maxBytesToRead + * @return {string} + */ + var UTF8ArrayToString = (heapOrArray, idx, maxBytesToRead) => { + var endIdx = idx + maxBytesToRead; + var endPtr = idx; + // TextDecoder needs to know the byte length in advance, it doesn't stop on + // null terminator by itself. Also, use the length info to avoid running tiny + // strings through TextDecoder, since .subarray() allocates garbage. + // (As a tiny code save trick, compare endPtr against endIdx using a negation, + // so that undefined means Infinity) + while (heapOrArray[endPtr] && !(endPtr >= endIdx)) ++endPtr; + + if (endPtr - idx > 16 && heapOrArray.buffer && UTF8Decoder) { + return UTF8Decoder.decode(heapOrArray.subarray(idx, endPtr)); + } + var str = ''; + // If building with TextDecoder, we have already computed the string length + // above, so test loop end condition against that + while (idx < endPtr) { + // For UTF8 byte structure, see: + // http://en.wikipedia.org/wiki/UTF-8#Description + // https://www.ietf.org/rfc/rfc2279.txt + // https://tools.ietf.org/html/rfc3629 + var u0 = heapOrArray[idx++]; + if (!(u0 & 0x80)) { str += String.fromCharCode(u0); continue; } + var u1 = heapOrArray[idx++] & 63; + if ((u0 & 0xE0) == 0xC0) { str += String.fromCharCode(((u0 & 31) << 6) | u1); continue; } + var u2 = heapOrArray[idx++] & 63; + if ((u0 & 0xF0) == 0xE0) { + u0 = ((u0 & 15) << 12) | (u1 << 6) | u2; + } else { + if ((u0 & 0xF8) != 0xF0) warnOnce('Invalid UTF-8 leading byte ' + ptrToString(u0) + ' encountered when deserializing a UTF-8 string in wasm memory to a JS string!'); + u0 = ((u0 & 7) << 18) | (u1 << 12) | (u2 << 6) | (heapOrArray[idx++] & 63); + } + + if (u0 < 0x10000) { + str += String.fromCharCode(u0); + } else { + var ch = u0 - 0x10000; + str += String.fromCharCode(0xD800 | (ch >> 10), 0xDC00 | (ch & 0x3FF)); + } + } + return str; + }; + + /** + * Given a pointer 'ptr' to a null-terminated UTF8-encoded string in the + * emscripten HEAP, returns a copy of that string as a Javascript String object. + * + * @param {number} ptr + * @param {number=} maxBytesToRead - An optional length that specifies the + * maximum number of bytes to read. You can omit this parameter to scan the + * string until the first 0 byte. If maxBytesToRead is passed, and the string + * at [ptr, ptr+maxBytesToReadr[ contains a null byte in the middle, then the + * string will cut short at that byte index (i.e. maxBytesToRead will not + * produce a string of exact length [ptr, ptr+maxBytesToRead[) N.B. mixing + * frequent uses of UTF8ToString() with and without maxBytesToRead may throw + * JS JIT optimizations off, so it is worth to consider consistently using one + * @return {string} + */ + var UTF8ToString = (ptr, maxBytesToRead) => { + assert(typeof ptr == 'number'); + return ptr ? UTF8ArrayToString(HEAPU8, ptr, maxBytesToRead) : ''; + }; + var ___assert_fail = (condition, filename, line, func) => { + abort(`Assertion failed: ${UTF8ToString(condition)}, at: ` + [filename ? UTF8ToString(filename) : 'unknown filename', line, func ? UTF8ToString(func) : 'unknown function']); + }; + + var wasmTableMirror = []; + var getWasmTableEntry = (funcPtr) => { + var func = wasmTableMirror[funcPtr]; + if (!func) { + if (funcPtr >= wasmTableMirror.length) wasmTableMirror.length = funcPtr + 1; + wasmTableMirror[funcPtr] = func = wasmTable.get(funcPtr); + } + assert(wasmTable.get(funcPtr) == func, "JavaScript-side Wasm function table mirror is out of date!"); + return func; + }; + var ___call_sighandler = (fp, sig) => getWasmTableEntry(fp)(sig); + + /** @constructor */ + function ExceptionInfo(excPtr) { + this.excPtr = excPtr; + this.ptr = excPtr - 24; + + this.set_type = function(type) { + HEAPU32[(((this.ptr)+(4))>>2)] = type; + }; + + this.get_type = function() { + return HEAPU32[(((this.ptr)+(4))>>2)]; + }; + + this.set_destructor = function(destructor) { + HEAPU32[(((this.ptr)+(8))>>2)] = destructor; + }; + + this.get_destructor = function() { + return HEAPU32[(((this.ptr)+(8))>>2)]; + }; + + this.set_caught = function (caught) { + caught = caught ? 1 : 0; + HEAP8[(((this.ptr)+(12))>>0)] = caught; + }; + + this.get_caught = function () { + return HEAP8[(((this.ptr)+(12))>>0)] != 0; + }; + + this.set_rethrown = function (rethrown) { + rethrown = rethrown ? 1 : 0; + HEAP8[(((this.ptr)+(13))>>0)] = rethrown; + }; + + this.get_rethrown = function () { + return HEAP8[(((this.ptr)+(13))>>0)] != 0; + }; + + // Initialize native structure fields. Should be called once after allocated. + this.init = function(type, destructor) { + this.set_adjusted_ptr(0); + this.set_type(type); + this.set_destructor(destructor); + } + + this.set_adjusted_ptr = function(adjustedPtr) { + HEAPU32[(((this.ptr)+(16))>>2)] = adjustedPtr; + }; + + this.get_adjusted_ptr = function() { + return HEAPU32[(((this.ptr)+(16))>>2)]; + }; + + // Get pointer which is expected to be received by catch clause in C++ code. It may be adjusted + // when the pointer is casted to some of the exception object base classes (e.g. when virtual + // inheritance is used). When a pointer is thrown this method should return the thrown pointer + // itself. + this.get_exception_ptr = function() { + // Work around a fastcomp bug, this code is still included for some reason in a build without + // exceptions support. + var isPointer = ___cxa_is_pointer_type(this.get_type()); + if (isPointer) { + return HEAPU32[((this.excPtr)>>2)]; + } + var adjusted = this.get_adjusted_ptr(); + if (adjusted !== 0) return adjusted; + return this.excPtr; + }; + } + + var exceptionLast = 0; + + var uncaughtExceptionCount = 0; + function ___cxa_throw(ptr, type, destructor) { + var info = new ExceptionInfo(ptr); + // Initialize ExceptionInfo content after it was allocated in __cxa_allocate_exception. + info.init(type, destructor); + exceptionLast = ptr; + uncaughtExceptionCount++; + assert(false, 'Exception thrown, but exception catching is not enabled. Compile with -sNO_DISABLE_EXCEPTION_CATCHING or -sEXCEPTION_CATCHING_ALLOWED=[..] to catch.'); + } + + var PATH = { + isAbs:(path) => path.charAt(0) === '/', + splitPath:(filename) => { + var splitPathRe = /^(\/?|)([\s\S]*?)((?:\.{1,2}|[^\/]+?|)(\.[^.\/]*|))(?:[\/]*)$/; + return splitPathRe.exec(filename).slice(1); + }, + normalizeArray:(parts, allowAboveRoot) => { + // if the path tries to go above the root, `up` ends up > 0 + var up = 0; + for (var i = parts.length - 1; i >= 0; i--) { + var last = parts[i]; + if (last === '.') { + parts.splice(i, 1); + } else if (last === '..') { + parts.splice(i, 1); + up++; + } else if (up) { + parts.splice(i, 1); + up--; + } + } + // if the path is allowed to go above the root, restore leading ..s + if (allowAboveRoot) { + for (; up; up--) { + parts.unshift('..'); + } + } + return parts; + }, + normalize:(path) => { + var isAbsolute = PATH.isAbs(path), + trailingSlash = path.substr(-1) === '/'; + // Normalize the path + path = PATH.normalizeArray(path.split('/').filter((p) => !!p), !isAbsolute).join('/'); + if (!path && !isAbsolute) { + path = '.'; + } + if (path && trailingSlash) { + path += '/'; + } + return (isAbsolute ? '/' : '') + path; + }, + dirname:(path) => { + var result = PATH.splitPath(path), + root = result[0], + dir = result[1]; + if (!root && !dir) { + // No dirname whatsoever + return '.'; + } + if (dir) { + // It has a dirname, strip trailing slash + dir = dir.substr(0, dir.length - 1); + } + return root + dir; + }, + basename:(path) => { + // EMSCRIPTEN return '/'' for '/', not an empty string + if (path === '/') return '/'; + path = PATH.normalize(path); + path = path.replace(/\/$/, ""); + var lastSlash = path.lastIndexOf('/'); + if (lastSlash === -1) return path; + return path.substr(lastSlash+1); + }, + join:function() { + var paths = Array.prototype.slice.call(arguments); + return PATH.normalize(paths.join('/')); + }, + join2:(l, r) => { + return PATH.normalize(l + '/' + r); + }, + }; + + var initRandomFill = () => { + if (typeof crypto == 'object' && typeof crypto['getRandomValues'] == 'function') { + // for modern web browsers + return (view) => crypto.getRandomValues(view); + } else + if (ENVIRONMENT_IS_NODE) { + // for nodejs with or without crypto support included + try { + var crypto_module = require('crypto'); + var randomFillSync = crypto_module['randomFillSync']; + if (randomFillSync) { + // nodejs with LTS crypto support + return (view) => crypto_module['randomFillSync'](view); + } + // very old nodejs with the original crypto API + var randomBytes = crypto_module['randomBytes']; + return (view) => ( + view.set(randomBytes(view.byteLength)), + // Return the original view to match modern native implementations. + view + ); + } catch (e) { + // nodejs doesn't have crypto support + } + } + // we couldn't find a proper implementation, as Math.random() is not suitable for /dev/random, see emscripten-core/emscripten/pull/7096 + abort("no cryptographic support found for randomDevice. consider polyfilling it if you want to use something insecure like Math.random(), e.g. put this in a --pre-js: var crypto = { getRandomValues: (array) => { for (var i = 0; i < array.length; i++) array[i] = (Math.random()*256)|0 } };"); + }; + var randomFill = (view) => { + // Lazily init on the first invocation. + return (randomFill = initRandomFill())(view); + }; + + + + var PATH_FS = { + resolve:function() { + var resolvedPath = '', + resolvedAbsolute = false; + for (var i = arguments.length - 1; i >= -1 && !resolvedAbsolute; i--) { + var path = (i >= 0) ? arguments[i] : FS.cwd(); + // Skip empty and invalid entries + if (typeof path != 'string') { + throw new TypeError('Arguments to path.resolve must be strings'); + } else if (!path) { + return ''; // an invalid portion invalidates the whole thing + } + resolvedPath = path + '/' + resolvedPath; + resolvedAbsolute = PATH.isAbs(path); + } + // At this point the path should be resolved to a full absolute path, but + // handle relative paths to be safe (might happen when process.cwd() fails) + resolvedPath = PATH.normalizeArray(resolvedPath.split('/').filter((p) => !!p), !resolvedAbsolute).join('/'); + return ((resolvedAbsolute ? '/' : '') + resolvedPath) || '.'; + }, + relative:(from, to) => { + from = PATH_FS.resolve(from).substr(1); + to = PATH_FS.resolve(to).substr(1); + function trim(arr) { + var start = 0; + for (; start < arr.length; start++) { + if (arr[start] !== '') break; + } + var end = arr.length - 1; + for (; end >= 0; end--) { + if (arr[end] !== '') break; + } + if (start > end) return []; + return arr.slice(start, end - start + 1); + } + var fromParts = trim(from.split('/')); + var toParts = trim(to.split('/')); + var length = Math.min(fromParts.length, toParts.length); + var samePartsLength = length; + for (var i = 0; i < length; i++) { + if (fromParts[i] !== toParts[i]) { + samePartsLength = i; + break; + } + } + var outputParts = []; + for (var i = samePartsLength; i < fromParts.length; i++) { + outputParts.push('..'); + } + outputParts = outputParts.concat(toParts.slice(samePartsLength)); + return outputParts.join('/'); + }, + }; + + + + var FS_stdin_getChar_buffer = []; + + var lengthBytesUTF8 = (str) => { + var len = 0; + for (var i = 0; i < str.length; ++i) { + // Gotcha: charCodeAt returns a 16-bit word that is a UTF-16 encoded code + // unit, not a Unicode code point of the character! So decode + // UTF16->UTF32->UTF8. + // See http://unicode.org/faq/utf_bom.html#utf16-3 + var c = str.charCodeAt(i); // possibly a lead surrogate + if (c <= 0x7F) { + len++; + } else if (c <= 0x7FF) { + len += 2; + } else if (c >= 0xD800 && c <= 0xDFFF) { + len += 4; ++i; + } else { + len += 3; + } + } + return len; + }; + + var stringToUTF8Array = (str, heap, outIdx, maxBytesToWrite) => { + assert(typeof str === 'string'); + // Parameter maxBytesToWrite is not optional. Negative values, 0, null, + // undefined and false each don't write out any bytes. + if (!(maxBytesToWrite > 0)) + return 0; + + var startIdx = outIdx; + var endIdx = outIdx + maxBytesToWrite - 1; // -1 for string null terminator. + for (var i = 0; i < str.length; ++i) { + // Gotcha: charCodeAt returns a 16-bit word that is a UTF-16 encoded code + // unit, not a Unicode code point of the character! So decode + // UTF16->UTF32->UTF8. + // See http://unicode.org/faq/utf_bom.html#utf16-3 + // For UTF8 byte structure, see http://en.wikipedia.org/wiki/UTF-8#Description + // and https://www.ietf.org/rfc/rfc2279.txt + // and https://tools.ietf.org/html/rfc3629 + var u = str.charCodeAt(i); // possibly a lead surrogate + if (u >= 0xD800 && u <= 0xDFFF) { + var u1 = str.charCodeAt(++i); + u = 0x10000 + ((u & 0x3FF) << 10) | (u1 & 0x3FF); + } + if (u <= 0x7F) { + if (outIdx >= endIdx) break; + heap[outIdx++] = u; + } else if (u <= 0x7FF) { + if (outIdx + 1 >= endIdx) break; + heap[outIdx++] = 0xC0 | (u >> 6); + heap[outIdx++] = 0x80 | (u & 63); + } else if (u <= 0xFFFF) { + if (outIdx + 2 >= endIdx) break; + heap[outIdx++] = 0xE0 | (u >> 12); + heap[outIdx++] = 0x80 | ((u >> 6) & 63); + heap[outIdx++] = 0x80 | (u & 63); + } else { + if (outIdx + 3 >= endIdx) break; + if (u > 0x10FFFF) warnOnce('Invalid Unicode code point ' + ptrToString(u) + ' encountered when serializing a JS string to a UTF-8 string in wasm memory! (Valid unicode code points should be in range 0-0x10FFFF).'); + heap[outIdx++] = 0xF0 | (u >> 18); + heap[outIdx++] = 0x80 | ((u >> 12) & 63); + heap[outIdx++] = 0x80 | ((u >> 6) & 63); + heap[outIdx++] = 0x80 | (u & 63); + } + } + // Null-terminate the pointer to the buffer. + heap[outIdx] = 0; + return outIdx - startIdx; + }; + /** @type {function(string, boolean=, number=)} */ + function intArrayFromString(stringy, dontAddNull, length) { + var len = length > 0 ? length : lengthBytesUTF8(stringy)+1; + var u8array = new Array(len); + var numBytesWritten = stringToUTF8Array(stringy, u8array, 0, u8array.length); + if (dontAddNull) u8array.length = numBytesWritten; + return u8array; + } + var FS_stdin_getChar = () => { + if (!FS_stdin_getChar_buffer.length) { + var result = null; + if (ENVIRONMENT_IS_NODE) { + // we will read data by chunks of BUFSIZE + var BUFSIZE = 256; + var buf = Buffer.alloc(BUFSIZE); + var bytesRead = 0; + + // For some reason we must suppress a closure warning here, even though + // fd definitely exists on process.stdin, and is even the proper way to + // get the fd of stdin, + // https://github.com/nodejs/help/issues/2136#issuecomment-523649904 + // This started to happen after moving this logic out of library_tty.js, + // so it is related to the surrounding code in some unclear manner. + /** @suppress {missingProperties} */ + var fd = process.stdin.fd; + + try { + bytesRead = fs.readSync(fd, buf, 0, BUFSIZE, -1); + } catch(e) { + // Cross-platform differences: on Windows, reading EOF throws an exception, but on other OSes, + // reading EOF returns 0. Uniformize behavior by treating the EOF exception to return 0. + if (e.toString().includes('EOF')) bytesRead = 0; + else throw e; + } + + if (bytesRead > 0) { + result = buf.slice(0, bytesRead).toString('utf-8'); + } else { + result = null; + } + } else + if (typeof window != 'undefined' && + typeof window.prompt == 'function') { + // Browser. + result = window.prompt('Input: '); // returns null on cancel + if (result !== null) { + result += '\n'; + } + } else if (typeof readline == 'function') { + // Command line. + result = readline(); + if (result !== null) { + result += '\n'; + } + } + if (!result) { + return null; + } + FS_stdin_getChar_buffer = intArrayFromString(result, true); + } + return FS_stdin_getChar_buffer.shift(); + }; + var TTY = { + ttys:[], + init:function () { + // https://github.com/emscripten-core/emscripten/pull/1555 + // if (ENVIRONMENT_IS_NODE) { + // // currently, FS.init does not distinguish if process.stdin is a file or TTY + // // device, it always assumes it's a TTY device. because of this, we're forcing + // // process.stdin to UTF8 encoding to at least make stdin reading compatible + // // with text files until FS.init can be refactored. + // process.stdin.setEncoding('utf8'); + // } + }, + shutdown:function() { + // https://github.com/emscripten-core/emscripten/pull/1555 + // if (ENVIRONMENT_IS_NODE) { + // // inolen: any idea as to why node -e 'process.stdin.read()' wouldn't exit immediately (with process.stdin being a tty)? + // // isaacs: because now it's reading from the stream, you've expressed interest in it, so that read() kicks off a _read() which creates a ReadReq operation + // // inolen: I thought read() in that case was a synchronous operation that just grabbed some amount of buffered data if it exists? + // // isaacs: it is. but it also triggers a _read() call, which calls readStart() on the handle + // // isaacs: do process.stdin.pause() and i'd think it'd probably close the pending call + // process.stdin.pause(); + // } + }, + register:function(dev, ops) { + TTY.ttys[dev] = { input: [], output: [], ops: ops }; + FS.registerDevice(dev, TTY.stream_ops); + }, + stream_ops:{ + open:function(stream) { + var tty = TTY.ttys[stream.node.rdev]; + if (!tty) { + throw new FS.ErrnoError(43); + } + stream.tty = tty; + stream.seekable = false; + }, + close:function(stream) { + // flush any pending line data + stream.tty.ops.fsync(stream.tty); + }, + fsync:function(stream) { + stream.tty.ops.fsync(stream.tty); + }, + read:function(stream, buffer, offset, length, pos /* ignored */) { + if (!stream.tty || !stream.tty.ops.get_char) { + throw new FS.ErrnoError(60); + } + var bytesRead = 0; + for (var i = 0; i < length; i++) { + var result; + try { + result = stream.tty.ops.get_char(stream.tty); + } catch (e) { + throw new FS.ErrnoError(29); + } + if (result === undefined && bytesRead === 0) { + throw new FS.ErrnoError(6); + } + if (result === null || result === undefined) break; + bytesRead++; + buffer[offset+i] = result; + } + if (bytesRead) { + stream.node.timestamp = Date.now(); + } + return bytesRead; + }, + write:function(stream, buffer, offset, length, pos) { + if (!stream.tty || !stream.tty.ops.put_char) { + throw new FS.ErrnoError(60); + } + try { + for (var i = 0; i < length; i++) { + stream.tty.ops.put_char(stream.tty, buffer[offset+i]); + } + } catch (e) { + throw new FS.ErrnoError(29); + } + if (length) { + stream.node.timestamp = Date.now(); + } + return i; + }, + }, + default_tty_ops:{ + get_char:function(tty) { + return FS_stdin_getChar(); + }, + put_char:function(tty, val) { + if (val === null || val === 10) { + out(UTF8ArrayToString(tty.output, 0)); + tty.output = []; + } else { + if (val != 0) tty.output.push(val); // val == 0 would cut text output off in the middle. + } + }, + fsync:function(tty) { + if (tty.output && tty.output.length > 0) { + out(UTF8ArrayToString(tty.output, 0)); + tty.output = []; + } + }, + ioctl_tcgets:function(tty) { + // typical setting + return { + c_iflag: 25856, + c_oflag: 5, + c_cflag: 191, + c_lflag: 35387, + c_cc: [ + 0x03, 0x1c, 0x7f, 0x15, 0x04, 0x00, 0x01, 0x00, 0x11, 0x13, 0x1a, 0x00, + 0x12, 0x0f, 0x17, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, + ] + }; + }, + ioctl_tcsets:function(tty, optional_actions, data) { + // currently just ignore + return 0; + }, + ioctl_tiocgwinsz:function(tty) { + return [24, 80]; + }, + }, + default_tty1_ops:{ + put_char:function(tty, val) { + if (val === null || val === 10) { + err(UTF8ArrayToString(tty.output, 0)); + tty.output = []; + } else { + if (val != 0) tty.output.push(val); + } + }, + fsync:function(tty) { + if (tty.output && tty.output.length > 0) { + err(UTF8ArrayToString(tty.output, 0)); + tty.output = []; + } + }, + }, + }; + + + var zeroMemory = (address, size) => { + HEAPU8.fill(0, address, address + size); + return address; + }; + + var alignMemory = (size, alignment) => { + assert(alignment, "alignment argument is required"); + return Math.ceil(size / alignment) * alignment; + }; + var mmapAlloc = (size) => { + size = alignMemory(size, 65536); + var ptr = _emscripten_builtin_memalign(65536, size); + if (!ptr) return 0; + return zeroMemory(ptr, size); + }; + var MEMFS = { + ops_table:null, + mount(mount) { + return MEMFS.createNode(null, '/', 16384 | 511 /* 0777 */, 0); + }, + createNode(parent, name, mode, dev) { + if (FS.isBlkdev(mode) || FS.isFIFO(mode)) { + // no supported + throw new FS.ErrnoError(63); + } + if (!MEMFS.ops_table) { + MEMFS.ops_table = { + dir: { + node: { + getattr: MEMFS.node_ops.getattr, + setattr: MEMFS.node_ops.setattr, + lookup: MEMFS.node_ops.lookup, + mknod: MEMFS.node_ops.mknod, + rename: MEMFS.node_ops.rename, + unlink: MEMFS.node_ops.unlink, + rmdir: MEMFS.node_ops.rmdir, + readdir: MEMFS.node_ops.readdir, + symlink: MEMFS.node_ops.symlink + }, + stream: { + llseek: MEMFS.stream_ops.llseek + } + }, + file: { + node: { + getattr: MEMFS.node_ops.getattr, + setattr: MEMFS.node_ops.setattr + }, + stream: { + llseek: MEMFS.stream_ops.llseek, + read: MEMFS.stream_ops.read, + write: MEMFS.stream_ops.write, + allocate: MEMFS.stream_ops.allocate, + mmap: MEMFS.stream_ops.mmap, + msync: MEMFS.stream_ops.msync + } + }, + link: { + node: { + getattr: MEMFS.node_ops.getattr, + setattr: MEMFS.node_ops.setattr, + readlink: MEMFS.node_ops.readlink + }, + stream: {} + }, + chrdev: { + node: { + getattr: MEMFS.node_ops.getattr, + setattr: MEMFS.node_ops.setattr + }, + stream: FS.chrdev_stream_ops + } + }; + } + var node = FS.createNode(parent, name, mode, dev); + if (FS.isDir(node.mode)) { + node.node_ops = MEMFS.ops_table.dir.node; + node.stream_ops = MEMFS.ops_table.dir.stream; + node.contents = {}; + } else if (FS.isFile(node.mode)) { + node.node_ops = MEMFS.ops_table.file.node; + node.stream_ops = MEMFS.ops_table.file.stream; + node.usedBytes = 0; // The actual number of bytes used in the typed array, as opposed to contents.length which gives the whole capacity. + // When the byte data of the file is populated, this will point to either a typed array, or a normal JS array. Typed arrays are preferred + // for performance, and used by default. However, typed arrays are not resizable like normal JS arrays are, so there is a small disk size + // penalty involved for appending file writes that continuously grow a file similar to std::vector capacity vs used -scheme. + node.contents = null; + } else if (FS.isLink(node.mode)) { + node.node_ops = MEMFS.ops_table.link.node; + node.stream_ops = MEMFS.ops_table.link.stream; + } else if (FS.isChrdev(node.mode)) { + node.node_ops = MEMFS.ops_table.chrdev.node; + node.stream_ops = MEMFS.ops_table.chrdev.stream; + } + node.timestamp = Date.now(); + // add the new node to the parent + if (parent) { + parent.contents[name] = node; + parent.timestamp = node.timestamp; + } + return node; + }, + getFileDataAsTypedArray(node) { + if (!node.contents) return new Uint8Array(0); + if (node.contents.subarray) return node.contents.subarray(0, node.usedBytes); // Make sure to not return excess unused bytes. + return new Uint8Array(node.contents); + }, + expandFileStorage(node, newCapacity) { + var prevCapacity = node.contents ? node.contents.length : 0; + if (prevCapacity >= newCapacity) return; // No need to expand, the storage was already large enough. + // Don't expand strictly to the given requested limit if it's only a very small increase, but instead geometrically grow capacity. + // For small filesizes (<1MB), perform size*2 geometric increase, but for large sizes, do a much more conservative size*1.125 increase to + // avoid overshooting the allocation cap by a very large margin. + var CAPACITY_DOUBLING_MAX = 1024 * 1024; + newCapacity = Math.max(newCapacity, (prevCapacity * (prevCapacity < CAPACITY_DOUBLING_MAX ? 2.0 : 1.125)) >>> 0); + if (prevCapacity != 0) newCapacity = Math.max(newCapacity, 256); // At minimum allocate 256b for each file when expanding. + var oldContents = node.contents; + node.contents = new Uint8Array(newCapacity); // Allocate new storage. + if (node.usedBytes > 0) node.contents.set(oldContents.subarray(0, node.usedBytes), 0); // Copy old data over to the new storage. + }, + resizeFileStorage(node, newSize) { + if (node.usedBytes == newSize) return; + if (newSize == 0) { + node.contents = null; // Fully decommit when requesting a resize to zero. + node.usedBytes = 0; + } else { + var oldContents = node.contents; + node.contents = new Uint8Array(newSize); // Allocate new storage. + if (oldContents) { + node.contents.set(oldContents.subarray(0, Math.min(newSize, node.usedBytes))); // Copy old data over to the new storage. + } + node.usedBytes = newSize; + } + }, + node_ops:{ + getattr(node) { + var attr = {}; + // device numbers reuse inode numbers. + attr.dev = FS.isChrdev(node.mode) ? node.id : 1; + attr.ino = node.id; + attr.mode = node.mode; + attr.nlink = 1; + attr.uid = 0; + attr.gid = 0; + attr.rdev = node.rdev; + if (FS.isDir(node.mode)) { + attr.size = 4096; + } else if (FS.isFile(node.mode)) { + attr.size = node.usedBytes; + } else if (FS.isLink(node.mode)) { + attr.size = node.link.length; + } else { + attr.size = 0; + } + attr.atime = new Date(node.timestamp); + attr.mtime = new Date(node.timestamp); + attr.ctime = new Date(node.timestamp); + // NOTE: In our implementation, st_blocks = Math.ceil(st_size/st_blksize), + // but this is not required by the standard. + attr.blksize = 4096; + attr.blocks = Math.ceil(attr.size / attr.blksize); + return attr; + }, + setattr(node, attr) { + if (attr.mode !== undefined) { + node.mode = attr.mode; + } + if (attr.timestamp !== undefined) { + node.timestamp = attr.timestamp; + } + if (attr.size !== undefined) { + MEMFS.resizeFileStorage(node, attr.size); + } + }, + lookup(parent, name) { + throw FS.genericErrors[44]; + }, + mknod(parent, name, mode, dev) { + return MEMFS.createNode(parent, name, mode, dev); + }, + rename(old_node, new_dir, new_name) { + // if we're overwriting a directory at new_name, make sure it's empty. + if (FS.isDir(old_node.mode)) { + var new_node; + try { + new_node = FS.lookupNode(new_dir, new_name); + } catch (e) { + } + if (new_node) { + for (var i in new_node.contents) { + throw new FS.ErrnoError(55); + } + } + } + // do the internal rewiring + delete old_node.parent.contents[old_node.name]; + old_node.parent.timestamp = Date.now() + old_node.name = new_name; + new_dir.contents[new_name] = old_node; + new_dir.timestamp = old_node.parent.timestamp; + old_node.parent = new_dir; + }, + unlink(parent, name) { + delete parent.contents[name]; + parent.timestamp = Date.now(); + }, + rmdir(parent, name) { + var node = FS.lookupNode(parent, name); + for (var i in node.contents) { + throw new FS.ErrnoError(55); + } + delete parent.contents[name]; + parent.timestamp = Date.now(); + }, + readdir(node) { + var entries = ['.', '..']; + for (var key in node.contents) { + if (!node.contents.hasOwnProperty(key)) { + continue; + } + entries.push(key); + } + return entries; + }, + symlink(parent, newname, oldpath) { + var node = MEMFS.createNode(parent, newname, 511 /* 0777 */ | 40960, 0); + node.link = oldpath; + return node; + }, + readlink(node) { + if (!FS.isLink(node.mode)) { + throw new FS.ErrnoError(28); + } + return node.link; + }, + }, + stream_ops:{ + read(stream, buffer, offset, length, position) { + var contents = stream.node.contents; + if (position >= stream.node.usedBytes) return 0; + var size = Math.min(stream.node.usedBytes - position, length); + assert(size >= 0); + if (size > 8 && contents.subarray) { // non-trivial, and typed array + buffer.set(contents.subarray(position, position + size), offset); + } else { + for (var i = 0; i < size; i++) buffer[offset + i] = contents[position + i]; + } + return size; + }, + write(stream, buffer, offset, length, position, canOwn) { + // The data buffer should be a typed array view + assert(!(buffer instanceof ArrayBuffer)); + + if (!length) return 0; + var node = stream.node; + node.timestamp = Date.now(); + + if (buffer.subarray && (!node.contents || node.contents.subarray)) { // This write is from a typed array to a typed array? + if (canOwn) { + assert(position === 0, 'canOwn must imply no weird position inside the file'); + node.contents = buffer.subarray(offset, offset + length); + node.usedBytes = length; + return length; + } else if (node.usedBytes === 0 && position === 0) { // If this is a simple first write to an empty file, do a fast set since we don't need to care about old data. + node.contents = buffer.slice(offset, offset + length); + node.usedBytes = length; + return length; + } else if (position + length <= node.usedBytes) { // Writing to an already allocated and used subrange of the file? + node.contents.set(buffer.subarray(offset, offset + length), position); + return length; + } + } + + // Appending to an existing file and we need to reallocate, or source data did not come as a typed array. + MEMFS.expandFileStorage(node, position+length); + if (node.contents.subarray && buffer.subarray) { + // Use typed array write which is available. + node.contents.set(buffer.subarray(offset, offset + length), position); + } else { + for (var i = 0; i < length; i++) { + node.contents[position + i] = buffer[offset + i]; // Or fall back to manual write if not. + } + } + node.usedBytes = Math.max(node.usedBytes, position + length); + return length; + }, + llseek(stream, offset, whence) { + var position = offset; + if (whence === 1) { + position += stream.position; + } else if (whence === 2) { + if (FS.isFile(stream.node.mode)) { + position += stream.node.usedBytes; + } + } + if (position < 0) { + throw new FS.ErrnoError(28); + } + return position; + }, + allocate(stream, offset, length) { + MEMFS.expandFileStorage(stream.node, offset + length); + stream.node.usedBytes = Math.max(stream.node.usedBytes, offset + length); + }, + mmap(stream, length, position, prot, flags) { + if (!FS.isFile(stream.node.mode)) { + throw new FS.ErrnoError(43); + } + var ptr; + var allocated; + var contents = stream.node.contents; + // Only make a new copy when MAP_PRIVATE is specified. + if (!(flags & 2) && contents.buffer === HEAP8.buffer) { + // We can't emulate MAP_SHARED when the file is not backed by the + // buffer we're mapping to (e.g. the HEAP buffer). + allocated = false; + ptr = contents.byteOffset; + } else { + // Try to avoid unnecessary slices. + if (position > 0 || position + length < contents.length) { + if (contents.subarray) { + contents = contents.subarray(position, position + length); + } else { + contents = Array.prototype.slice.call(contents, position, position + length); + } + } + allocated = true; + ptr = mmapAlloc(length); + if (!ptr) { + throw new FS.ErrnoError(48); + } + HEAP8.set(contents, ptr); + } + return { ptr, allocated }; + }, + msync(stream, buffer, offset, length, mmapFlags) { + MEMFS.stream_ops.write(stream, buffer, 0, length, offset, false); + // should we check if bytesWritten and length are the same? + return 0; + }, + }, + }; + + /** @param {boolean=} noRunDep */ + var asyncLoad = (url, onload, onerror, noRunDep) => { + var dep = !noRunDep ? getUniqueRunDependency(`al ${url}`) : ''; + readAsync(url, (arrayBuffer) => { + assert(arrayBuffer, `Loading data file "${url}" failed (no arrayBuffer).`); + onload(new Uint8Array(arrayBuffer)); + if (dep) removeRunDependency(dep); + }, (event) => { + if (onerror) { + onerror(); + } else { + throw `Loading data file "${url}" failed.`; + } + }); + if (dep) addRunDependency(dep); + }; + + + var preloadPlugins = Module['preloadPlugins'] || []; + function FS_handledByPreloadPlugin(byteArray, fullname, finish, onerror) { + // Ensure plugins are ready. + if (typeof Browser != 'undefined') Browser.init(); + + var handled = false; + preloadPlugins.forEach(function(plugin) { + if (handled) return; + if (plugin['canHandle'](fullname)) { + plugin['handle'](byteArray, fullname, finish, onerror); + handled = true; + } + }); + return handled; + } + function FS_createPreloadedFile(parent, name, url, canRead, canWrite, onload, onerror, dontCreateFile, canOwn, preFinish) { + // TODO we should allow people to just pass in a complete filename instead + // of parent and name being that we just join them anyways + var fullname = name ? PATH_FS.resolve(PATH.join2(parent, name)) : parent; + var dep = getUniqueRunDependency(`cp ${fullname}`); // might have several active requests for the same fullname + function processData(byteArray) { + function finish(byteArray) { + if (preFinish) preFinish(); + if (!dontCreateFile) { + FS.createDataFile(parent, name, byteArray, canRead, canWrite, canOwn); + } + if (onload) onload(); + removeRunDependency(dep); + } + if (FS_handledByPreloadPlugin(byteArray, fullname, finish, () => { + if (onerror) onerror(); + removeRunDependency(dep); + })) { + return; + } + finish(byteArray); + } + addRunDependency(dep); + if (typeof url == 'string') { + asyncLoad(url, (byteArray) => processData(byteArray), onerror); + } else { + processData(url); + } + } + + function FS_modeStringToFlags(str) { + var flagModes = { + 'r': 0, + 'r+': 2, + 'w': 512 | 64 | 1, + 'w+': 512 | 64 | 2, + 'a': 1024 | 64 | 1, + 'a+': 1024 | 64 | 2, + }; + var flags = flagModes[str]; + if (typeof flags == 'undefined') { + throw new Error(`Unknown file open mode: ${str}`); + } + return flags; + } + + function FS_getMode(canRead, canWrite) { + var mode = 0; + if (canRead) mode |= 292 | 73; + if (canWrite) mode |= 146; + return mode; + } + + + + + var ERRNO_MESSAGES = { + 0:"Success", + 1:"Arg list too long", + 2:"Permission denied", + 3:"Address already in use", + 4:"Address not available", + 5:"Address family not supported by protocol family", + 6:"No more processes", + 7:"Socket already connected", + 8:"Bad file number", + 9:"Trying to read unreadable message", + 10:"Mount device busy", + 11:"Operation canceled", + 12:"No children", + 13:"Connection aborted", + 14:"Connection refused", + 15:"Connection reset by peer", + 16:"File locking deadlock error", + 17:"Destination address required", + 18:"Math arg out of domain of func", + 19:"Quota exceeded", + 20:"File exists", + 21:"Bad address", + 22:"File too large", + 23:"Host is unreachable", + 24:"Identifier removed", + 25:"Illegal byte sequence", + 26:"Connection already in progress", + 27:"Interrupted system call", + 28:"Invalid argument", + 29:"I/O error", + 30:"Socket is already connected", + 31:"Is a directory", + 32:"Too many symbolic links", + 33:"Too many open files", + 34:"Too many links", + 35:"Message too long", + 36:"Multihop attempted", + 37:"File or path name too long", + 38:"Network interface is not configured", + 39:"Connection reset by network", + 40:"Network is unreachable", + 41:"Too many open files in system", + 42:"No buffer space available", + 43:"No such device", + 44:"No such file or directory", + 45:"Exec format error", + 46:"No record locks available", + 47:"The link has been severed", + 48:"Not enough core", + 49:"No message of desired type", + 50:"Protocol not available", + 51:"No space left on device", + 52:"Function not implemented", + 53:"Socket is not connected", + 54:"Not a directory", + 55:"Directory not empty", + 56:"State not recoverable", + 57:"Socket operation on non-socket", + 59:"Not a typewriter", + 60:"No such device or address", + 61:"Value too large for defined data type", + 62:"Previous owner died", + 63:"Not super-user", + 64:"Broken pipe", + 65:"Protocol error", + 66:"Unknown protocol", + 67:"Protocol wrong type for socket", + 68:"Math result not representable", + 69:"Read only file system", + 70:"Illegal seek", + 71:"No such process", + 72:"Stale file handle", + 73:"Connection timed out", + 74:"Text file busy", + 75:"Cross-device link", + 100:"Device not a stream", + 101:"Bad font file fmt", + 102:"Invalid slot", + 103:"Invalid request code", + 104:"No anode", + 105:"Block device required", + 106:"Channel number out of range", + 107:"Level 3 halted", + 108:"Level 3 reset", + 109:"Link number out of range", + 110:"Protocol driver not attached", + 111:"No CSI structure available", + 112:"Level 2 halted", + 113:"Invalid exchange", + 114:"Invalid request descriptor", + 115:"Exchange full", + 116:"No data (for no delay io)", + 117:"Timer expired", + 118:"Out of streams resources", + 119:"Machine is not on the network", + 120:"Package not installed", + 121:"The object is remote", + 122:"Advertise error", + 123:"Srmount error", + 124:"Communication error on send", + 125:"Cross mount point (not really error)", + 126:"Given log. name not unique", + 127:"f.d. invalid for this operation", + 128:"Remote address changed", + 129:"Can access a needed shared lib", + 130:"Accessing a corrupted shared lib", + 131:".lib section in a.out corrupted", + 132:"Attempting to link in too many libs", + 133:"Attempting to exec a shared library", + 135:"Streams pipe error", + 136:"Too many users", + 137:"Socket type not supported", + 138:"Not supported", + 139:"Protocol family not supported", + 140:"Can't send after socket shutdown", + 141:"Too many references", + 142:"Host is down", + 148:"No medium (in tape drive)", + 156:"Level 2 not synchronized", + }; + + var ERRNO_CODES = { + }; + + function demangle(func) { + warnOnce('warning: build with -sDEMANGLE_SUPPORT to link in libcxxabi demangling'); + return func; + } + function demangleAll(text) { + var regex = + /\b_Z[\w\d_]+/g; + return text.replace(regex, + function(x) { + var y = demangle(x); + return x === y ? x : (y + ' [' + x + ']'); + }); + } + + + var LZ4 = { + DIR_MODE:16895, + FILE_MODE:33279, + CHUNK_SIZE:-1, + codec:null, + init() { + if (LZ4.codec) return; + LZ4.codec = (function() { + /* + MiniLZ4: Minimal LZ4 block decoding and encoding. + + based off of node-lz4, https://github.com/pierrec/node-lz4 + + ==== + Copyright (c) 2012 Pierre Curto + + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in + all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN + THE SOFTWARE. + ==== + + changes have the same license + */ + + var MiniLZ4 = (function() { + + var exports = {}; + + /** + * Decode a block. Assumptions: input contains all sequences of a + * chunk, output is large enough to receive the decoded data. + * If the output buffer is too small, an error will be thrown. + * If the returned value is negative, an error occured at the returned offset. + * + * @param {ArrayBufferView} input input data + * @param {ArrayBufferView} output output data + * @param {number=} sIdx + * @param {number=} eIdx + * @return {number} number of decoded bytes + * @private + */ + exports.uncompress = function (input, output, sIdx, eIdx) { + sIdx = sIdx || 0 + eIdx = eIdx || (input.length - sIdx) + // Process each sequence in the incoming data + for (var i = sIdx, n = eIdx, j = 0; i < n;) { + var token = input[i++] + + // Literals + var literals_length = (token >> 4) + if (literals_length > 0) { + // length of literals + var l = literals_length + 240 + while (l === 255) { + l = input[i++] + literals_length += l + } + + // Copy the literals + var end = i + literals_length + while (i < end) output[j++] = input[i++] + + // End of buffer? + if (i === n) return j + } + + // Match copy + // 2 bytes offset (little endian) + var offset = input[i++] | (input[i++] << 8) + + // XXX 0 is an invalid offset value + if (offset === 0) return j + if (offset > j) return -(i-2) + + // length of match copy + var match_length = (token & 0xf) + var l = match_length + 240 + while (l === 255) { + l = input[i++] + match_length += l + } + + // Copy the match + var pos = j - offset // position of the match copy in the current output + var end = j + match_length + 4 // minmatch = 4 + while (j < end) output[j++] = output[pos++] + } + + return j + } + + var + maxInputSize = 0x7E000000 + , minMatch = 4 + // uint32() optimization + , hashLog = 16 + , hashShift = (minMatch * 8) - hashLog + , hashSize = 1 << hashLog + + , copyLength = 8 + , lastLiterals = 5 + , mfLimit = copyLength + minMatch + , skipStrength = 6 + + , mlBits = 4 + , mlMask = (1 << mlBits) - 1 + , runBits = 8 - mlBits + , runMask = (1 << runBits) - 1 + + , hasher = /* XXX uint32( */ 2654435761 /* ) */ + + assert(hashShift === 16); + var hashTable = new Int16Array(1<<16); + var empty = new Int16Array(hashTable.length); + + // CompressBound returns the maximum length of a lz4 block, given it's uncompressed length + exports.compressBound = function (isize) { + return isize > maxInputSize + ? 0 + : (isize + (isize/255) + 16) | 0 + } + + /** @param {number=} sIdx + @param {number=} eIdx */ + exports.compress = function (src, dst, sIdx, eIdx) { + hashTable.set(empty); + return compressBlock(src, dst, 0, sIdx || 0, eIdx || dst.length) + } + + function compressBlock (src, dst, pos, sIdx, eIdx) { + // XXX var Hash = uint32() // Reusable unsigned 32 bits integer + var dpos = sIdx + var dlen = eIdx - sIdx + var anchor = 0 + + if (src.length >= maxInputSize) throw new Error("input too large") + + // Minimum of input bytes for compression (LZ4 specs) + if (src.length > mfLimit) { + var n = exports.compressBound(src.length) + if ( dlen < n ) throw Error("output too small: " + dlen + " < " + n) + + var + step = 1 + , findMatchAttempts = (1 << skipStrength) + 3 + // Keep last few bytes incompressible (LZ4 specs): + // last 5 bytes must be literals + , srcLength = src.length - mfLimit + + while (pos + minMatch < srcLength) { + // Find a match + // min match of 4 bytes aka sequence + var sequenceLowBits = src[pos+1]<<8 | src[pos] + var sequenceHighBits = src[pos+3]<<8 | src[pos+2] + // compute hash for the current sequence + var hash = Math.imul(sequenceLowBits | (sequenceHighBits << 16), hasher) >>> hashShift; + /* XXX Hash.fromBits(sequenceLowBits, sequenceHighBits) + .multiply(hasher) + .shiftr(hashShift) + .toNumber() */ + // get the position of the sequence matching the hash + // NB. since 2 different sequences may have the same hash + // it is double-checked below + // do -1 to distinguish between initialized and uninitialized values + var ref = hashTable[hash] - 1 + // save position of current sequence in hash table + hashTable[hash] = pos + 1 + + // first reference or within 64k limit or current sequence !== hashed one: no match + if ( ref < 0 || + ((pos - ref) >>> 16) > 0 || + ( + ((src[ref+3]<<8 | src[ref+2]) != sequenceHighBits) || + ((src[ref+1]<<8 | src[ref]) != sequenceLowBits ) + ) + ) { + // increase step if nothing found within limit + step = findMatchAttempts++ >> skipStrength + pos += step + continue + } + + findMatchAttempts = (1 << skipStrength) + 3 + + // got a match + var literals_length = pos - anchor + var offset = pos - ref + + // minMatch already verified + pos += minMatch + ref += minMatch + + // move to the end of the match (>=minMatch) + var match_length = pos + while (pos < srcLength && src[pos] == src[ref]) { + pos++ + ref++ + } + + // match length + match_length = pos - match_length + + // token + var token = match_length < mlMask ? match_length : mlMask + + // encode literals length + if (literals_length >= runMask) { + // add match length to the token + dst[dpos++] = (runMask << mlBits) + token + for (var len = literals_length - runMask; len > 254; len -= 255) { + dst[dpos++] = 255 + } + dst[dpos++] = len + } else { + // add match length to the token + dst[dpos++] = (literals_length << mlBits) + token + } + + // write literals + for (var i = 0; i < literals_length; i++) { + dst[dpos++] = src[anchor+i] + } + + // encode offset + dst[dpos++] = offset + dst[dpos++] = (offset >> 8) + + // encode match length + if (match_length >= mlMask) { + match_length -= mlMask + while (match_length >= 255) { + match_length -= 255 + dst[dpos++] = 255 + } + + dst[dpos++] = match_length + } + + anchor = pos + } + } + + // cannot compress input + if (anchor == 0) return 0 + + // Write last literals + // encode literals length + literals_length = src.length - anchor + if (literals_length >= runMask) { + // add match length to the token + dst[dpos++] = (runMask << mlBits) + for (var ln = literals_length - runMask; ln > 254; ln -= 255) { + dst[dpos++] = 255 + } + dst[dpos++] = ln + } else { + // add match length to the token + dst[dpos++] = (literals_length << mlBits) + } + + // write literals + pos = anchor + while (pos < src.length) { + dst[dpos++] = src[pos++] + } + + return dpos + } + + exports.CHUNK_SIZE = 2048; // musl libc does readaheads of 1024 bytes, so a multiple of that is a good idea + + exports.compressPackage = function(data, verify) { + if (verify) { + var temp = new Uint8Array(exports.CHUNK_SIZE); + } + // compress the data in chunks + assert(data instanceof ArrayBuffer); + data = new Uint8Array(data); + console.log('compressing package of size ' + data.length); + var compressedChunks = []; + var successes = []; + var offset = 0; + var total = 0; + while (offset < data.length) { + var chunk = data.subarray(offset, offset + exports.CHUNK_SIZE); + //console.log('compress a chunk ' + [offset, total, data.length]); + offset += exports.CHUNK_SIZE; + var bound = exports.compressBound(chunk.length); + var compressed = new Uint8Array(bound); + var compressedSize = exports.compress(chunk, compressed); + if (compressedSize > 0) { + assert(compressedSize <= bound); + compressed = compressed.subarray(0, compressedSize); + compressedChunks.push(compressed); + total += compressedSize; + successes.push(1); + if (verify) { + var back = exports.uncompress(compressed, temp); + assert(back === chunk.length, [back, chunk.length]); + for (var i = 0; i < chunk.length; i++) { + assert(chunk[i] === temp[i]); + } + } + } else { + assert(compressedSize === 0); + // failure to compress :( + compressedChunks.push(chunk); + total += chunk.length; // last chunk may not be the full exports.CHUNK_SIZE size + successes.push(0); + } + } + data = null; // XXX null out pack['data'] too? + var compressedData = { + 'data': new Uint8Array(total + exports.CHUNK_SIZE*2), // store all the compressed data, plus room for two cached decompressed chunk, in one fast array + 'cachedOffset': total, + 'cachedIndexes': [-1, -1], // cache last two blocks, so that reading 1,2,3 + preloading another block won't trigger decompress thrashing + 'cachedChunks': [null, null], + 'offsets': [], // chunk# => start in compressed data + 'sizes': [], + 'successes': successes, // 1 if chunk is compressed + }; + offset = 0; + for (var i = 0; i < compressedChunks.length; i++) { + compressedData['data'].set(compressedChunks[i], offset); + compressedData['offsets'][i] = offset; + compressedData['sizes'][i] = compressedChunks[i].length + offset += compressedChunks[i].length; + } + console.log('compressed package into ' + [compressedData['data'].length]); + assert(offset === total); + return compressedData; + }; + + assert(exports.CHUNK_SIZE < (1 << 15)); // we use 16-bit ints as the type of the hash table, chunk size must be smaller + + return exports; + + })(); + + ; + return MiniLZ4; + })(); + LZ4.CHUNK_SIZE = LZ4.codec.CHUNK_SIZE; + }, + loadPackage(pack, preloadPlugin) { + LZ4.init(); + var compressedData = pack['compressedData']; + if (!compressedData) compressedData = LZ4.codec.compressPackage(pack['data']); + assert(compressedData['cachedIndexes'].length === compressedData['cachedChunks'].length); + for (var i = 0; i < compressedData['cachedIndexes'].length; i++) { + compressedData['cachedIndexes'][i] = -1; + compressedData['cachedChunks'][i] = compressedData['data'].subarray(compressedData['cachedOffset'] + i*LZ4.CHUNK_SIZE, + compressedData['cachedOffset'] + (i+1)*LZ4.CHUNK_SIZE); + assert(compressedData['cachedChunks'][i].length === LZ4.CHUNK_SIZE); + } + pack['metadata'].files.forEach(function(file) { + var dir = PATH.dirname(file.filename); + var name = PATH.basename(file.filename); + FS.createPath('', dir, true, true); + var parent = FS.analyzePath(dir).object; + LZ4.createNode(parent, name, LZ4.FILE_MODE, 0, { + compressedData, + start: file.start, + end: file.end, + }); + }); + // Preload files if necessary. This code is largely similar to + // createPreloadedFile in library_fs.js. However, a main difference here + // is that we only decompress the file if it can be preloaded. + // Abstracting out the common parts seems to be more effort than it is + // worth. + if (preloadPlugin) { + Browser.init(); + pack['metadata'].files.forEach(function(file) { + var handled = false; + var fullname = file.filename; + preloadPlugins.forEach(function(plugin) { + if (handled) return; + if (plugin['canHandle'](fullname)) { + var dep = getUniqueRunDependency('fp ' + fullname); + addRunDependency(dep); + var finish = () => removeRunDependency(dep); + var byteArray = FS.readFile(fullname); + plugin['handle'](byteArray, fullname, finish, finish); + handled = true; + } + }); + }); + } + }, + createNode(parent, name, mode, dev, contents, mtime) { + var node = FS.createNode(parent, name, mode); + node.mode = mode; + node.node_ops = LZ4.node_ops; + node.stream_ops = LZ4.stream_ops; + node.timestamp = (mtime || new Date).getTime(); + assert(LZ4.FILE_MODE !== LZ4.DIR_MODE); + if (mode === LZ4.FILE_MODE) { + node.size = contents.end - contents.start; + node.contents = contents; + } else { + node.size = 4096; + node.contents = {}; + } + if (parent) { + parent.contents[name] = node; + } + return node; + }, + node_ops:{ + getattr(node) { + return { + dev: 1, + ino: node.id, + mode: node.mode, + nlink: 1, + uid: 0, + gid: 0, + rdev: 0, + size: node.size, + atime: new Date(node.timestamp), + mtime: new Date(node.timestamp), + ctime: new Date(node.timestamp), + blksize: 4096, + blocks: Math.ceil(node.size / 4096), + }; + }, + setattr(node, attr) { + if (attr.mode !== undefined) { + node.mode = attr.mode; + } + if (attr.timestamp !== undefined) { + node.timestamp = attr.timestamp; + } + }, + lookup(parent, name) { + throw new FS.ErrnoError(44); + }, + mknod(parent, name, mode, dev) { + throw new FS.ErrnoError(63); + }, + rename(oldNode, newDir, newName) { + throw new FS.ErrnoError(63); + }, + unlink(parent, name) { + throw new FS.ErrnoError(63); + }, + rmdir(parent, name) { + throw new FS.ErrnoError(63); + }, + readdir(node) { + throw new FS.ErrnoError(63); + }, + symlink(parent, newName, oldPath) { + throw new FS.ErrnoError(63); + }, + }, + stream_ops:{ + read(stream, buffer, offset, length, position) { + //out('LZ4 read ' + [offset, length, position]); + length = Math.min(length, stream.node.size - position); + if (length <= 0) return 0; + var contents = stream.node.contents; + var compressedData = contents.compressedData; + var written = 0; + while (written < length) { + var start = contents.start + position + written; // start index in uncompressed data + var desired = length - written; + //out('current read: ' + ['start', start, 'desired', desired]); + var chunkIndex = Math.floor(start / LZ4.CHUNK_SIZE); + var compressedStart = compressedData['offsets'][chunkIndex]; + var compressedSize = compressedData['sizes'][chunkIndex]; + var currChunk; + if (compressedData['successes'][chunkIndex]) { + var found = compressedData['cachedIndexes'].indexOf(chunkIndex); + if (found >= 0) { + currChunk = compressedData['cachedChunks'][found]; + } else { + // decompress the chunk + compressedData['cachedIndexes'].pop(); + compressedData['cachedIndexes'].unshift(chunkIndex); + currChunk = compressedData['cachedChunks'].pop(); + compressedData['cachedChunks'].unshift(currChunk); + if (compressedData['debug']) { + out('decompressing chunk ' + chunkIndex); + Module['decompressedChunks'] = (Module['decompressedChunks'] || 0) + 1; + } + var compressed = compressedData['data'].subarray(compressedStart, compressedStart + compressedSize); + //var t = Date.now(); + var originalSize = LZ4.codec.uncompress(compressed, currChunk); + //out('decompress time: ' + (Date.now() - t)); + if (chunkIndex < compressedData['successes'].length-1) assert(originalSize === LZ4.CHUNK_SIZE); // all but the last chunk must be full-size + } + } else { + // uncompressed + currChunk = compressedData['data'].subarray(compressedStart, compressedStart + LZ4.CHUNK_SIZE); + } + var startInChunk = start % LZ4.CHUNK_SIZE; + var endInChunk = Math.min(startInChunk + desired, LZ4.CHUNK_SIZE); + buffer.set(currChunk.subarray(startInChunk, endInChunk), offset + written); + var currWritten = endInChunk - startInChunk; + written += currWritten; + } + return written; + }, + write(stream, buffer, offset, length, position) { + throw new FS.ErrnoError(29); + }, + llseek(stream, offset, whence) { + var position = offset; + if (whence === 1) { + position += stream.position; + } else if (whence === 2) { + if (FS.isFile(stream.node.mode)) { + position += stream.node.size; + } + } + if (position < 0) { + throw new FS.ErrnoError(28); + } + return position; + }, + }, + }; + var FS = { + root:null, + mounts:[], + devices:{ + }, + streams:[], + nextInode:1, + nameTable:null, + currentPath:"/", + initialized:false, + ignorePermissions:true, + ErrnoError:null, + genericErrors:{ + }, + filesystems:null, + syncFSRequests:0, + lookupPath:(path, opts = {}) => { + path = PATH_FS.resolve(path); + + if (!path) return { path: '', node: null }; + + var defaults = { + follow_mount: true, + recurse_count: 0 + }; + opts = Object.assign(defaults, opts) + + if (opts.recurse_count > 8) { // max recursive lookup of 8 + throw new FS.ErrnoError(32); + } + + // split the absolute path + var parts = path.split('/').filter((p) => !!p); + + // start at the root + var current = FS.root; + var current_path = '/'; + + for (var i = 0; i < parts.length; i++) { + var islast = (i === parts.length-1); + if (islast && opts.parent) { + // stop resolving + break; + } + + current = FS.lookupNode(current, parts[i]); + current_path = PATH.join2(current_path, parts[i]); + + // jump to the mount's root node if this is a mountpoint + if (FS.isMountpoint(current)) { + if (!islast || (islast && opts.follow_mount)) { + current = current.mounted.root; + } + } + + // by default, lookupPath will not follow a symlink if it is the final path component. + // setting opts.follow = true will override this behavior. + if (!islast || opts.follow) { + var count = 0; + while (FS.isLink(current.mode)) { + var link = FS.readlink(current_path); + current_path = PATH_FS.resolve(PATH.dirname(current_path), link); + + var lookup = FS.lookupPath(current_path, { recurse_count: opts.recurse_count + 1 }); + current = lookup.node; + + if (count++ > 40) { // limit max consecutive symlinks to 40 (SYMLOOP_MAX). + throw new FS.ErrnoError(32); + } + } + } + } + + return { path: current_path, node: current }; + }, + getPath:(node) => { + var path; + while (true) { + if (FS.isRoot(node)) { + var mount = node.mount.mountpoint; + if (!path) return mount; + return mount[mount.length-1] !== '/' ? `${mount}/${path}` : mount + path; + } + path = path ? `${node.name}/${path}` : node.name; + node = node.parent; + } + }, + hashName:(parentid, name) => { + var hash = 0; + + for (var i = 0; i < name.length; i++) { + hash = ((hash << 5) - hash + name.charCodeAt(i)) | 0; + } + return ((parentid + hash) >>> 0) % FS.nameTable.length; + }, + hashAddNode:(node) => { + var hash = FS.hashName(node.parent.id, node.name); + node.name_next = FS.nameTable[hash]; + FS.nameTable[hash] = node; + }, + hashRemoveNode:(node) => { + var hash = FS.hashName(node.parent.id, node.name); + if (FS.nameTable[hash] === node) { + FS.nameTable[hash] = node.name_next; + } else { + var current = FS.nameTable[hash]; + while (current) { + if (current.name_next === node) { + current.name_next = node.name_next; + break; + } + current = current.name_next; + } + } + }, + lookupNode:(parent, name) => { + var errCode = FS.mayLookup(parent); + if (errCode) { + throw new FS.ErrnoError(errCode, parent); + } + var hash = FS.hashName(parent.id, name); + for (var node = FS.nameTable[hash]; node; node = node.name_next) { + var nodeName = node.name; + if (node.parent.id === parent.id && nodeName === name) { + return node; + } + } + // if we failed to find it in the cache, call into the VFS + return FS.lookup(parent, name); + }, + createNode:(parent, name, mode, rdev) => { + assert(typeof parent == 'object') + var node = new FS.FSNode(parent, name, mode, rdev); + + FS.hashAddNode(node); + + return node; + }, + destroyNode:(node) => { + FS.hashRemoveNode(node); + }, + isRoot:(node) => { + return node === node.parent; + }, + isMountpoint:(node) => { + return !!node.mounted; + }, + isFile:(mode) => { + return (mode & 61440) === 32768; + }, + isDir:(mode) => { + return (mode & 61440) === 16384; + }, + isLink:(mode) => { + return (mode & 61440) === 40960; + }, + isChrdev:(mode) => { + return (mode & 61440) === 8192; + }, + isBlkdev:(mode) => { + return (mode & 61440) === 24576; + }, + isFIFO:(mode) => { + return (mode & 61440) === 4096; + }, + isSocket:(mode) => { + return (mode & 49152) === 49152; + }, + flagsToPermissionString:(flag) => { + var perms = ['r', 'w', 'rw'][flag & 3]; + if ((flag & 512)) { + perms += 'w'; + } + return perms; + }, + nodePermissions:(node, perms) => { + if (FS.ignorePermissions) { + return 0; + } + // return 0 if any user, group or owner bits are set. + if (perms.includes('r') && !(node.mode & 292)) { + return 2; + } else if (perms.includes('w') && !(node.mode & 146)) { + return 2; + } else if (perms.includes('x') && !(node.mode & 73)) { + return 2; + } + return 0; + }, + mayLookup:(dir) => { + var errCode = FS.nodePermissions(dir, 'x'); + if (errCode) return errCode; + if (!dir.node_ops.lookup) return 2; + return 0; + }, + mayCreate:(dir, name) => { + try { + var node = FS.lookupNode(dir, name); + return 20; + } catch (e) { + } + return FS.nodePermissions(dir, 'wx'); + }, + mayDelete:(dir, name, isdir) => { + var node; + try { + node = FS.lookupNode(dir, name); + } catch (e) { + return e.errno; + } + var errCode = FS.nodePermissions(dir, 'wx'); + if (errCode) { + return errCode; + } + if (isdir) { + if (!FS.isDir(node.mode)) { + return 54; + } + if (FS.isRoot(node) || FS.getPath(node) === FS.cwd()) { + return 10; + } + } else { + if (FS.isDir(node.mode)) { + return 31; + } + } + return 0; + }, + mayOpen:(node, flags) => { + if (!node) { + return 44; + } + if (FS.isLink(node.mode)) { + return 32; + } else if (FS.isDir(node.mode)) { + if (FS.flagsToPermissionString(flags) !== 'r' || // opening for write + (flags & 512)) { // TODO: check for O_SEARCH? (== search for dir only) + return 31; + } + } + return FS.nodePermissions(node, FS.flagsToPermissionString(flags)); + }, + MAX_OPEN_FDS:4096, + nextfd:() => { + for (var fd = 0; fd <= FS.MAX_OPEN_FDS; fd++) { + if (!FS.streams[fd]) { + return fd; + } + } + throw new FS.ErrnoError(33); + }, + getStreamChecked:(fd) => { + var stream = FS.getStream(fd); + if (!stream) { + throw new FS.ErrnoError(8); + } + return stream; + }, + getStream:(fd) => FS.streams[fd], + createStream:(stream, fd = -1) => { + if (!FS.FSStream) { + FS.FSStream = /** @constructor */ function() { + this.shared = { }; + }; + FS.FSStream.prototype = {}; + Object.defineProperties(FS.FSStream.prototype, { + object: { + /** @this {FS.FSStream} */ + get() { return this.node; }, + /** @this {FS.FSStream} */ + set(val) { this.node = val; } + }, + isRead: { + /** @this {FS.FSStream} */ + get() { return (this.flags & 2097155) !== 1; } + }, + isWrite: { + /** @this {FS.FSStream} */ + get() { return (this.flags & 2097155) !== 0; } + }, + isAppend: { + /** @this {FS.FSStream} */ + get() { return (this.flags & 1024); } + }, + flags: { + /** @this {FS.FSStream} */ + get() { return this.shared.flags; }, + /** @this {FS.FSStream} */ + set(val) { this.shared.flags = val; }, + }, + position : { + /** @this {FS.FSStream} */ + get() { return this.shared.position; }, + /** @this {FS.FSStream} */ + set(val) { this.shared.position = val; }, + }, + }); + } + // clone it, so we can return an instance of FSStream + stream = Object.assign(new FS.FSStream(), stream); + if (fd == -1) { + fd = FS.nextfd(); + } + stream.fd = fd; + FS.streams[fd] = stream; + return stream; + }, + closeStream:(fd) => { + FS.streams[fd] = null; + }, + chrdev_stream_ops:{ + open:(stream) => { + var device = FS.getDevice(stream.node.rdev); + // override node's stream ops with the device's + stream.stream_ops = device.stream_ops; + // forward the open call + if (stream.stream_ops.open) { + stream.stream_ops.open(stream); + } + }, + llseek:() => { + throw new FS.ErrnoError(70); + }, + }, + major:(dev) => ((dev) >> 8), + minor:(dev) => ((dev) & 0xff), + makedev:(ma, mi) => ((ma) << 8 | (mi)), + registerDevice:(dev, ops) => { + FS.devices[dev] = { stream_ops: ops }; + }, + getDevice:(dev) => FS.devices[dev], + getMounts:(mount) => { + var mounts = []; + var check = [mount]; + + while (check.length) { + var m = check.pop(); + + mounts.push(m); + + check.push.apply(check, m.mounts); + } + + return mounts; + }, + syncfs:(populate, callback) => { + if (typeof populate == 'function') { + callback = populate; + populate = false; + } + + FS.syncFSRequests++; + + if (FS.syncFSRequests > 1) { + err(`warning: ${FS.syncFSRequests} FS.syncfs operations in flight at once, probably just doing extra work`); + } + + var mounts = FS.getMounts(FS.root.mount); + var completed = 0; + + function doCallback(errCode) { + assert(FS.syncFSRequests > 0); + FS.syncFSRequests--; + return callback(errCode); + } + + function done(errCode) { + if (errCode) { + if (!done.errored) { + done.errored = true; + return doCallback(errCode); + } + return; + } + if (++completed >= mounts.length) { + doCallback(null); + } + }; + + // sync all mounts + mounts.forEach((mount) => { + if (!mount.type.syncfs) { + return done(null); + } + mount.type.syncfs(mount, populate, done); + }); + }, + mount:(type, opts, mountpoint) => { + if (typeof type == 'string') { + // The filesystem was not included, and instead we have an error + // message stored in the variable. + throw type; + } + var root = mountpoint === '/'; + var pseudo = !mountpoint; + var node; + + if (root && FS.root) { + throw new FS.ErrnoError(10); + } else if (!root && !pseudo) { + var lookup = FS.lookupPath(mountpoint, { follow_mount: false }); + + mountpoint = lookup.path; // use the absolute path + node = lookup.node; + + if (FS.isMountpoint(node)) { + throw new FS.ErrnoError(10); + } + + if (!FS.isDir(node.mode)) { + throw new FS.ErrnoError(54); + } + } + + var mount = { + type, + opts, + mountpoint, + mounts: [] + }; + + // create a root node for the fs + var mountRoot = type.mount(mount); + mountRoot.mount = mount; + mount.root = mountRoot; + + if (root) { + FS.root = mountRoot; + } else if (node) { + // set as a mountpoint + node.mounted = mount; + + // add the new mount to the current mount's children + if (node.mount) { + node.mount.mounts.push(mount); + } + } + + return mountRoot; + }, + unmount:(mountpoint) => { + var lookup = FS.lookupPath(mountpoint, { follow_mount: false }); + + if (!FS.isMountpoint(lookup.node)) { + throw new FS.ErrnoError(28); + } + + // destroy the nodes for this mount, and all its child mounts + var node = lookup.node; + var mount = node.mounted; + var mounts = FS.getMounts(mount); + + Object.keys(FS.nameTable).forEach((hash) => { + var current = FS.nameTable[hash]; + + while (current) { + var next = current.name_next; + + if (mounts.includes(current.mount)) { + FS.destroyNode(current); + } + + current = next; + } + }); + + // no longer a mountpoint + node.mounted = null; + + // remove this mount from the child mounts + var idx = node.mount.mounts.indexOf(mount); + assert(idx !== -1); + node.mount.mounts.splice(idx, 1); + }, + lookup:(parent, name) => { + return parent.node_ops.lookup(parent, name); + }, + mknod:(path, mode, dev) => { + var lookup = FS.lookupPath(path, { parent: true }); + var parent = lookup.node; + var name = PATH.basename(path); + if (!name || name === '.' || name === '..') { + throw new FS.ErrnoError(28); + } + var errCode = FS.mayCreate(parent, name); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + if (!parent.node_ops.mknod) { + throw new FS.ErrnoError(63); + } + return parent.node_ops.mknod(parent, name, mode, dev); + }, + create:(path, mode) => { + mode = mode !== undefined ? mode : 438 /* 0666 */; + mode &= 4095; + mode |= 32768; + return FS.mknod(path, mode, 0); + }, + mkdir:(path, mode) => { + mode = mode !== undefined ? mode : 511 /* 0777 */; + mode &= 511 | 512; + mode |= 16384; + return FS.mknod(path, mode, 0); + }, + mkdirTree:(path, mode) => { + var dirs = path.split('/'); + var d = ''; + for (var i = 0; i < dirs.length; ++i) { + if (!dirs[i]) continue; + d += '/' + dirs[i]; + try { + FS.mkdir(d, mode); + } catch(e) { + if (e.errno != 20) throw e; + } + } + }, + mkdev:(path, mode, dev) => { + if (typeof dev == 'undefined') { + dev = mode; + mode = 438 /* 0666 */; + } + mode |= 8192; + return FS.mknod(path, mode, dev); + }, + symlink:(oldpath, newpath) => { + if (!PATH_FS.resolve(oldpath)) { + throw new FS.ErrnoError(44); + } + var lookup = FS.lookupPath(newpath, { parent: true }); + var parent = lookup.node; + if (!parent) { + throw new FS.ErrnoError(44); + } + var newname = PATH.basename(newpath); + var errCode = FS.mayCreate(parent, newname); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + if (!parent.node_ops.symlink) { + throw new FS.ErrnoError(63); + } + return parent.node_ops.symlink(parent, newname, oldpath); + }, + rename:(old_path, new_path) => { + var old_dirname = PATH.dirname(old_path); + var new_dirname = PATH.dirname(new_path); + var old_name = PATH.basename(old_path); + var new_name = PATH.basename(new_path); + // parents must exist + var lookup, old_dir, new_dir; + + // let the errors from non existant directories percolate up + lookup = FS.lookupPath(old_path, { parent: true }); + old_dir = lookup.node; + lookup = FS.lookupPath(new_path, { parent: true }); + new_dir = lookup.node; + + if (!old_dir || !new_dir) throw new FS.ErrnoError(44); + // need to be part of the same mount + if (old_dir.mount !== new_dir.mount) { + throw new FS.ErrnoError(75); + } + // source must exist + var old_node = FS.lookupNode(old_dir, old_name); + // old path should not be an ancestor of the new path + var relative = PATH_FS.relative(old_path, new_dirname); + if (relative.charAt(0) !== '.') { + throw new FS.ErrnoError(28); + } + // new path should not be an ancestor of the old path + relative = PATH_FS.relative(new_path, old_dirname); + if (relative.charAt(0) !== '.') { + throw new FS.ErrnoError(55); + } + // see if the new path already exists + var new_node; + try { + new_node = FS.lookupNode(new_dir, new_name); + } catch (e) { + // not fatal + } + // early out if nothing needs to change + if (old_node === new_node) { + return; + } + // we'll need to delete the old entry + var isdir = FS.isDir(old_node.mode); + var errCode = FS.mayDelete(old_dir, old_name, isdir); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + // need delete permissions if we'll be overwriting. + // need create permissions if new doesn't already exist. + errCode = new_node ? + FS.mayDelete(new_dir, new_name, isdir) : + FS.mayCreate(new_dir, new_name); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + if (!old_dir.node_ops.rename) { + throw new FS.ErrnoError(63); + } + if (FS.isMountpoint(old_node) || (new_node && FS.isMountpoint(new_node))) { + throw new FS.ErrnoError(10); + } + // if we are going to change the parent, check write permissions + if (new_dir !== old_dir) { + errCode = FS.nodePermissions(old_dir, 'w'); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + } + // remove the node from the lookup hash + FS.hashRemoveNode(old_node); + // do the underlying fs rename + try { + old_dir.node_ops.rename(old_node, new_dir, new_name); + } catch (e) { + throw e; + } finally { + // add the node back to the hash (in case node_ops.rename + // changed its name) + FS.hashAddNode(old_node); + } + }, + rmdir:(path) => { + var lookup = FS.lookupPath(path, { parent: true }); + var parent = lookup.node; + var name = PATH.basename(path); + var node = FS.lookupNode(parent, name); + var errCode = FS.mayDelete(parent, name, true); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + if (!parent.node_ops.rmdir) { + throw new FS.ErrnoError(63); + } + if (FS.isMountpoint(node)) { + throw new FS.ErrnoError(10); + } + parent.node_ops.rmdir(parent, name); + FS.destroyNode(node); + }, + readdir:(path) => { + var lookup = FS.lookupPath(path, { follow: true }); + var node = lookup.node; + if (!node.node_ops.readdir) { + throw new FS.ErrnoError(54); + } + return node.node_ops.readdir(node); + }, + unlink:(path) => { + var lookup = FS.lookupPath(path, { parent: true }); + var parent = lookup.node; + if (!parent) { + throw new FS.ErrnoError(44); + } + var name = PATH.basename(path); + var node = FS.lookupNode(parent, name); + var errCode = FS.mayDelete(parent, name, false); + if (errCode) { + // According to POSIX, we should map EISDIR to EPERM, but + // we instead do what Linux does (and we must, as we use + // the musl linux libc). + throw new FS.ErrnoError(errCode); + } + if (!parent.node_ops.unlink) { + throw new FS.ErrnoError(63); + } + if (FS.isMountpoint(node)) { + throw new FS.ErrnoError(10); + } + parent.node_ops.unlink(parent, name); + FS.destroyNode(node); + }, + readlink:(path) => { + var lookup = FS.lookupPath(path); + var link = lookup.node; + if (!link) { + throw new FS.ErrnoError(44); + } + if (!link.node_ops.readlink) { + throw new FS.ErrnoError(28); + } + return PATH_FS.resolve(FS.getPath(link.parent), link.node_ops.readlink(link)); + }, + stat:(path, dontFollow) => { + var lookup = FS.lookupPath(path, { follow: !dontFollow }); + var node = lookup.node; + if (!node) { + throw new FS.ErrnoError(44); + } + if (!node.node_ops.getattr) { + throw new FS.ErrnoError(63); + } + return node.node_ops.getattr(node); + }, + lstat:(path) => { + return FS.stat(path, true); + }, + chmod:(path, mode, dontFollow) => { + var node; + if (typeof path == 'string') { + var lookup = FS.lookupPath(path, { follow: !dontFollow }); + node = lookup.node; + } else { + node = path; + } + if (!node.node_ops.setattr) { + throw new FS.ErrnoError(63); + } + node.node_ops.setattr(node, { + mode: (mode & 4095) | (node.mode & ~4095), + timestamp: Date.now() + }); + }, + lchmod:(path, mode) => { + FS.chmod(path, mode, true); + }, + fchmod:(fd, mode) => { + var stream = FS.getStreamChecked(fd); + FS.chmod(stream.node, mode); + }, + chown:(path, uid, gid, dontFollow) => { + var node; + if (typeof path == 'string') { + var lookup = FS.lookupPath(path, { follow: !dontFollow }); + node = lookup.node; + } else { + node = path; + } + if (!node.node_ops.setattr) { + throw new FS.ErrnoError(63); + } + node.node_ops.setattr(node, { + timestamp: Date.now() + // we ignore the uid / gid for now + }); + }, + lchown:(path, uid, gid) => { + FS.chown(path, uid, gid, true); + }, + fchown:(fd, uid, gid) => { + var stream = FS.getStreamChecked(fd); + FS.chown(stream.node, uid, gid); + }, + truncate:(path, len) => { + if (len < 0) { + throw new FS.ErrnoError(28); + } + var node; + if (typeof path == 'string') { + var lookup = FS.lookupPath(path, { follow: true }); + node = lookup.node; + } else { + node = path; + } + if (!node.node_ops.setattr) { + throw new FS.ErrnoError(63); + } + if (FS.isDir(node.mode)) { + throw new FS.ErrnoError(31); + } + if (!FS.isFile(node.mode)) { + throw new FS.ErrnoError(28); + } + var errCode = FS.nodePermissions(node, 'w'); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + node.node_ops.setattr(node, { + size: len, + timestamp: Date.now() + }); + }, + ftruncate:(fd, len) => { + var stream = FS.getStreamChecked(fd); + if ((stream.flags & 2097155) === 0) { + throw new FS.ErrnoError(28); + } + FS.truncate(stream.node, len); + }, + utime:(path, atime, mtime) => { + var lookup = FS.lookupPath(path, { follow: true }); + var node = lookup.node; + node.node_ops.setattr(node, { + timestamp: Math.max(atime, mtime) + }); + }, + open:(path, flags, mode) => { + if (path === "") { + throw new FS.ErrnoError(44); + } + flags = typeof flags == 'string' ? FS_modeStringToFlags(flags) : flags; + mode = typeof mode == 'undefined' ? 438 /* 0666 */ : mode; + if ((flags & 64)) { + mode = (mode & 4095) | 32768; + } else { + mode = 0; + } + var node; + if (typeof path == 'object') { + node = path; + } else { + path = PATH.normalize(path); + try { + var lookup = FS.lookupPath(path, { + follow: !(flags & 131072) + }); + node = lookup.node; + } catch (e) { + // ignore + } + } + // perhaps we need to create the node + var created = false; + if ((flags & 64)) { + if (node) { + // if O_CREAT and O_EXCL are set, error out if the node already exists + if ((flags & 128)) { + throw new FS.ErrnoError(20); + } + } else { + // node doesn't exist, try to create it + node = FS.mknod(path, mode, 0); + created = true; + } + } + if (!node) { + throw new FS.ErrnoError(44); + } + // can't truncate a device + if (FS.isChrdev(node.mode)) { + flags &= ~512; + } + // if asked only for a directory, then this must be one + if ((flags & 65536) && !FS.isDir(node.mode)) { + throw new FS.ErrnoError(54); + } + // check permissions, if this is not a file we just created now (it is ok to + // create and write to a file with read-only permissions; it is read-only + // for later use) + if (!created) { + var errCode = FS.mayOpen(node, flags); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + } + // do truncation if necessary + if ((flags & 512) && !created) { + FS.truncate(node, 0); + } + // we've already handled these, don't pass down to the underlying vfs + flags &= ~(128 | 512 | 131072); + + // register the stream with the filesystem + var stream = FS.createStream({ + node, + path: FS.getPath(node), // we want the absolute path to the node + flags, + seekable: true, + position: 0, + stream_ops: node.stream_ops, + // used by the file family libc calls (fopen, fwrite, ferror, etc.) + ungotten: [], + error: false + }); + // call the new stream's open function + if (stream.stream_ops.open) { + stream.stream_ops.open(stream); + } + if (Module['logReadFiles'] && !(flags & 1)) { + if (!FS.readFiles) FS.readFiles = {}; + if (!(path in FS.readFiles)) { + FS.readFiles[path] = 1; + } + } + return stream; + }, + close:(stream) => { + if (FS.isClosed(stream)) { + throw new FS.ErrnoError(8); + } + if (stream.getdents) stream.getdents = null; // free readdir state + try { + if (stream.stream_ops.close) { + stream.stream_ops.close(stream); + } + } catch (e) { + throw e; + } finally { + FS.closeStream(stream.fd); + } + stream.fd = null; + }, + isClosed:(stream) => { + return stream.fd === null; + }, + llseek:(stream, offset, whence) => { + if (FS.isClosed(stream)) { + throw new FS.ErrnoError(8); + } + if (!stream.seekable || !stream.stream_ops.llseek) { + throw new FS.ErrnoError(70); + } + if (whence != 0 && whence != 1 && whence != 2) { + throw new FS.ErrnoError(28); + } + stream.position = stream.stream_ops.llseek(stream, offset, whence); + stream.ungotten = []; + return stream.position; + }, + read:(stream, buffer, offset, length, position) => { + assert(offset >= 0); + if (length < 0 || position < 0) { + throw new FS.ErrnoError(28); + } + if (FS.isClosed(stream)) { + throw new FS.ErrnoError(8); + } + if ((stream.flags & 2097155) === 1) { + throw new FS.ErrnoError(8); + } + if (FS.isDir(stream.node.mode)) { + throw new FS.ErrnoError(31); + } + if (!stream.stream_ops.read) { + throw new FS.ErrnoError(28); + } + var seeking = typeof position != 'undefined'; + if (!seeking) { + position = stream.position; + } else if (!stream.seekable) { + throw new FS.ErrnoError(70); + } + var bytesRead = stream.stream_ops.read(stream, buffer, offset, length, position); + if (!seeking) stream.position += bytesRead; + return bytesRead; + }, + write:(stream, buffer, offset, length, position, canOwn) => { + assert(offset >= 0); + if (length < 0 || position < 0) { + throw new FS.ErrnoError(28); + } + if (FS.isClosed(stream)) { + throw new FS.ErrnoError(8); + } + if ((stream.flags & 2097155) === 0) { + throw new FS.ErrnoError(8); + } + if (FS.isDir(stream.node.mode)) { + throw new FS.ErrnoError(31); + } + if (!stream.stream_ops.write) { + throw new FS.ErrnoError(28); + } + if (stream.seekable && stream.flags & 1024) { + // seek to the end before writing in append mode + FS.llseek(stream, 0, 2); + } + var seeking = typeof position != 'undefined'; + if (!seeking) { + position = stream.position; + } else if (!stream.seekable) { + throw new FS.ErrnoError(70); + } + var bytesWritten = stream.stream_ops.write(stream, buffer, offset, length, position, canOwn); + if (!seeking) stream.position += bytesWritten; + return bytesWritten; + }, + allocate:(stream, offset, length) => { + if (FS.isClosed(stream)) { + throw new FS.ErrnoError(8); + } + if (offset < 0 || length <= 0) { + throw new FS.ErrnoError(28); + } + if ((stream.flags & 2097155) === 0) { + throw new FS.ErrnoError(8); + } + if (!FS.isFile(stream.node.mode) && !FS.isDir(stream.node.mode)) { + throw new FS.ErrnoError(43); + } + if (!stream.stream_ops.allocate) { + throw new FS.ErrnoError(138); + } + stream.stream_ops.allocate(stream, offset, length); + }, + mmap:(stream, length, position, prot, flags) => { + // User requests writing to file (prot & PROT_WRITE != 0). + // Checking if we have permissions to write to the file unless + // MAP_PRIVATE flag is set. According to POSIX spec it is possible + // to write to file opened in read-only mode with MAP_PRIVATE flag, + // as all modifications will be visible only in the memory of + // the current process. + if ((prot & 2) !== 0 + && (flags & 2) === 0 + && (stream.flags & 2097155) !== 2) { + throw new FS.ErrnoError(2); + } + if ((stream.flags & 2097155) === 1) { + throw new FS.ErrnoError(2); + } + if (!stream.stream_ops.mmap) { + throw new FS.ErrnoError(43); + } + return stream.stream_ops.mmap(stream, length, position, prot, flags); + }, + msync:(stream, buffer, offset, length, mmapFlags) => { + assert(offset >= 0); + if (!stream.stream_ops.msync) { + return 0; + } + return stream.stream_ops.msync(stream, buffer, offset, length, mmapFlags); + }, + munmap:(stream) => 0, + ioctl:(stream, cmd, arg) => { + if (!stream.stream_ops.ioctl) { + throw new FS.ErrnoError(59); + } + return stream.stream_ops.ioctl(stream, cmd, arg); + }, + readFile:(path, opts = {}) => { + opts.flags = opts.flags || 0; + opts.encoding = opts.encoding || 'binary'; + if (opts.encoding !== 'utf8' && opts.encoding !== 'binary') { + throw new Error(`Invalid encoding type "${opts.encoding}"`); + } + var ret; + var stream = FS.open(path, opts.flags); + var stat = FS.stat(path); + var length = stat.size; + var buf = new Uint8Array(length); + FS.read(stream, buf, 0, length, 0); + if (opts.encoding === 'utf8') { + ret = UTF8ArrayToString(buf, 0); + } else if (opts.encoding === 'binary') { + ret = buf; + } + FS.close(stream); + return ret; + }, + writeFile:(path, data, opts = {}) => { + opts.flags = opts.flags || 577; + var stream = FS.open(path, opts.flags, opts.mode); + if (typeof data == 'string') { + var buf = new Uint8Array(lengthBytesUTF8(data)+1); + var actualNumBytes = stringToUTF8Array(data, buf, 0, buf.length); + FS.write(stream, buf, 0, actualNumBytes, undefined, opts.canOwn); + } else if (ArrayBuffer.isView(data)) { + FS.write(stream, data, 0, data.byteLength, undefined, opts.canOwn); + } else { + throw new Error('Unsupported data type'); + } + FS.close(stream); + }, + cwd:() => FS.currentPath, + chdir:(path) => { + var lookup = FS.lookupPath(path, { follow: true }); + if (lookup.node === null) { + throw new FS.ErrnoError(44); + } + if (!FS.isDir(lookup.node.mode)) { + throw new FS.ErrnoError(54); + } + var errCode = FS.nodePermissions(lookup.node, 'x'); + if (errCode) { + throw new FS.ErrnoError(errCode); + } + FS.currentPath = lookup.path; + }, + createDefaultDirectories:() => { + FS.mkdir('/tmp'); + FS.mkdir('/home'); + FS.mkdir('/home/web_user'); + }, + createDefaultDevices:() => { + // create /dev + FS.mkdir('/dev'); + // setup /dev/null + FS.registerDevice(FS.makedev(1, 3), { + read: () => 0, + write: (stream, buffer, offset, length, pos) => length, + }); + FS.mkdev('/dev/null', FS.makedev(1, 3)); + // setup /dev/tty and /dev/tty1 + // stderr needs to print output using err() rather than out() + // so we register a second tty just for it. + TTY.register(FS.makedev(5, 0), TTY.default_tty_ops); + TTY.register(FS.makedev(6, 0), TTY.default_tty1_ops); + FS.mkdev('/dev/tty', FS.makedev(5, 0)); + FS.mkdev('/dev/tty1', FS.makedev(6, 0)); + // setup /dev/[u]random + // use a buffer to avoid overhead of individual crypto calls per byte + var randomBuffer = new Uint8Array(1024), randomLeft = 0; + var randomByte = () => { + if (randomLeft === 0) { + randomLeft = randomFill(randomBuffer).byteLength; + } + return randomBuffer[--randomLeft]; + }; + FS.createDevice('/dev', 'random', randomByte); + FS.createDevice('/dev', 'urandom', randomByte); + // we're not going to emulate the actual shm device, + // just create the tmp dirs that reside in it commonly + FS.mkdir('/dev/shm'); + FS.mkdir('/dev/shm/tmp'); + }, + createSpecialDirectories:() => { + // create /proc/self/fd which allows /proc/self/fd/6 => readlink gives the + // name of the stream for fd 6 (see test_unistd_ttyname) + FS.mkdir('/proc'); + var proc_self = FS.mkdir('/proc/self'); + FS.mkdir('/proc/self/fd'); + FS.mount({ + mount: () => { + var node = FS.createNode(proc_self, 'fd', 16384 | 511 /* 0777 */, 73); + node.node_ops = { + lookup: (parent, name) => { + var fd = +name; + var stream = FS.getStreamChecked(fd); + var ret = { + parent: null, + mount: { mountpoint: 'fake' }, + node_ops: { readlink: () => stream.path }, + }; + ret.parent = ret; // make it look like a simple root node + return ret; + } + }; + return node; + } + }, {}, '/proc/self/fd'); + }, + createStandardStreams:() => { + // TODO deprecate the old functionality of a single + // input / output callback and that utilizes FS.createDevice + // and instead require a unique set of stream ops + + // by default, we symlink the standard streams to the + // default tty devices. however, if the standard streams + // have been overwritten we create a unique device for + // them instead. + if (Module['stdin']) { + FS.createDevice('/dev', 'stdin', Module['stdin']); + } else { + FS.symlink('/dev/tty', '/dev/stdin'); + } + if (Module['stdout']) { + FS.createDevice('/dev', 'stdout', null, Module['stdout']); + } else { + FS.symlink('/dev/tty', '/dev/stdout'); + } + if (Module['stderr']) { + FS.createDevice('/dev', 'stderr', null, Module['stderr']); + } else { + FS.symlink('/dev/tty1', '/dev/stderr'); + } + + // open default streams for the stdin, stdout and stderr devices + var stdin = FS.open('/dev/stdin', 0); + var stdout = FS.open('/dev/stdout', 1); + var stderr = FS.open('/dev/stderr', 1); + assert(stdin.fd === 0, `invalid handle for stdin (${stdin.fd})`); + assert(stdout.fd === 1, `invalid handle for stdout (${stdout.fd})`); + assert(stderr.fd === 2, `invalid handle for stderr (${stderr.fd})`); + }, + ensureErrnoError:() => { + if (FS.ErrnoError) return; + FS.ErrnoError = /** @this{Object} */ function ErrnoError(errno, node) { + // We set the `name` property to be able to identify `FS.ErrnoError` + // - the `name` is a standard ECMA-262 property of error objects. Kind of good to have it anyway. + // - when using PROXYFS, an error can come from an underlying FS + // as different FS objects have their own FS.ErrnoError each, + // the test `err instanceof FS.ErrnoError` won't detect an error coming from another filesystem, causing bugs. + // we'll use the reliable test `err.name == "ErrnoError"` instead + this.name = 'ErrnoError'; + this.node = node; + this.setErrno = /** @this{Object} */ function(errno) { + this.errno = errno; + for (var key in ERRNO_CODES) { + if (ERRNO_CODES[key] === errno) { + this.code = key; + break; + } + } + }; + this.setErrno(errno); + this.message = ERRNO_MESSAGES[errno]; + + // Try to get a maximally helpful stack trace. On Node.js, getting Error.stack + // now ensures it shows what we want. + if (this.stack) { + // Define the stack property for Node.js 4, which otherwise errors on the next line. + Object.defineProperty(this, "stack", { value: (new Error).stack, writable: true }); + this.stack = demangleAll(this.stack); + } + }; + FS.ErrnoError.prototype = new Error(); + FS.ErrnoError.prototype.constructor = FS.ErrnoError; + // Some errors may happen quite a bit, to avoid overhead we reuse them (and suffer a lack of stack info) + [44].forEach((code) => { + FS.genericErrors[code] = new FS.ErrnoError(code); + FS.genericErrors[code].stack = ''; + }); + }, + staticInit:() => { + FS.ensureErrnoError(); + + FS.nameTable = new Array(4096); + + FS.mount(MEMFS, {}, '/'); + + FS.createDefaultDirectories(); + FS.createDefaultDevices(); + FS.createSpecialDirectories(); + + FS.filesystems = { + 'MEMFS': MEMFS, + }; + }, + init:(input, output, error) => { + assert(!FS.init.initialized, 'FS.init was previously called. If you want to initialize later with custom parameters, remove any earlier calls (note that one is automatically added to the generated code)'); + FS.init.initialized = true; + + FS.ensureErrnoError(); + + // Allow Module.stdin etc. to provide defaults, if none explicitly passed to us here + Module['stdin'] = input || Module['stdin']; + Module['stdout'] = output || Module['stdout']; + Module['stderr'] = error || Module['stderr']; + + FS.createStandardStreams(); + }, + quit:() => { + FS.init.initialized = false; + // force-flush all streams, so we get musl std streams printed out + _fflush(0); + // close all of our streams + for (var i = 0; i < FS.streams.length; i++) { + var stream = FS.streams[i]; + if (!stream) { + continue; + } + FS.close(stream); + } + }, + findObject:(path, dontResolveLastLink) => { + var ret = FS.analyzePath(path, dontResolveLastLink); + if (!ret.exists) { + return null; + } + return ret.object; + }, + analyzePath:(path, dontResolveLastLink) => { + // operate from within the context of the symlink's target + try { + var lookup = FS.lookupPath(path, { follow: !dontResolveLastLink }); + path = lookup.path; + } catch (e) { + } + var ret = { + isRoot: false, exists: false, error: 0, name: null, path: null, object: null, + parentExists: false, parentPath: null, parentObject: null + }; + try { + var lookup = FS.lookupPath(path, { parent: true }); + ret.parentExists = true; + ret.parentPath = lookup.path; + ret.parentObject = lookup.node; + ret.name = PATH.basename(path); + lookup = FS.lookupPath(path, { follow: !dontResolveLastLink }); + ret.exists = true; + ret.path = lookup.path; + ret.object = lookup.node; + ret.name = lookup.node.name; + ret.isRoot = lookup.path === '/'; + } catch (e) { + ret.error = e.errno; + }; + return ret; + }, + createPath:(parent, path, canRead, canWrite) => { + parent = typeof parent == 'string' ? parent : FS.getPath(parent); + var parts = path.split('/').reverse(); + while (parts.length) { + var part = parts.pop(); + if (!part) continue; + var current = PATH.join2(parent, part); + try { + FS.mkdir(current); + } catch (e) { + // ignore EEXIST + } + parent = current; + } + return current; + }, + createFile:(parent, name, properties, canRead, canWrite) => { + var path = PATH.join2(typeof parent == 'string' ? parent : FS.getPath(parent), name); + var mode = FS_getMode(canRead, canWrite); + return FS.create(path, mode); + }, + createDataFile:(parent, name, data, canRead, canWrite, canOwn) => { + var path = name; + if (parent) { + parent = typeof parent == 'string' ? parent : FS.getPath(parent); + path = name ? PATH.join2(parent, name) : parent; + } + var mode = FS_getMode(canRead, canWrite); + var node = FS.create(path, mode); + if (data) { + if (typeof data == 'string') { + var arr = new Array(data.length); + for (var i = 0, len = data.length; i < len; ++i) arr[i] = data.charCodeAt(i); + data = arr; + } + // make sure we can write to the file + FS.chmod(node, mode | 146); + var stream = FS.open(node, 577); + FS.write(stream, data, 0, data.length, 0, canOwn); + FS.close(stream); + FS.chmod(node, mode); + } + return node; + }, + createDevice:(parent, name, input, output) => { + var path = PATH.join2(typeof parent == 'string' ? parent : FS.getPath(parent), name); + var mode = FS_getMode(!!input, !!output); + if (!FS.createDevice.major) FS.createDevice.major = 64; + var dev = FS.makedev(FS.createDevice.major++, 0); + // Create a fake device that a set of stream ops to emulate + // the old behavior. + FS.registerDevice(dev, { + open: (stream) => { + stream.seekable = false; + }, + close: (stream) => { + // flush any pending line data + if (output && output.buffer && output.buffer.length) { + output(10); + } + }, + read: (stream, buffer, offset, length, pos /* ignored */) => { + var bytesRead = 0; + for (var i = 0; i < length; i++) { + var result; + try { + result = input(); + } catch (e) { + throw new FS.ErrnoError(29); + } + if (result === undefined && bytesRead === 0) { + throw new FS.ErrnoError(6); + } + if (result === null || result === undefined) break; + bytesRead++; + buffer[offset+i] = result; + } + if (bytesRead) { + stream.node.timestamp = Date.now(); + } + return bytesRead; + }, + write: (stream, buffer, offset, length, pos) => { + for (var i = 0; i < length; i++) { + try { + output(buffer[offset+i]); + } catch (e) { + throw new FS.ErrnoError(29); + } + } + if (length) { + stream.node.timestamp = Date.now(); + } + return i; + } + }); + return FS.mkdev(path, mode, dev); + }, + forceLoadFile:(obj) => { + if (obj.isDevice || obj.isFolder || obj.link || obj.contents) return true; + if (typeof XMLHttpRequest != 'undefined') { + throw new Error("Lazy loading should have been performed (contents set) in createLazyFile, but it was not. Lazy loading only works in web workers. Use --embed-file or --preload-file in emcc on the main thread."); + } else if (read_) { + // Command-line. + try { + // WARNING: Can't read binary files in V8's d8 or tracemonkey's js, as + // read() will try to parse UTF8. + obj.contents = intArrayFromString(read_(obj.url), true); + obj.usedBytes = obj.contents.length; + } catch (e) { + throw new FS.ErrnoError(29); + } + } else { + throw new Error('Cannot load without read() or XMLHttpRequest.'); + } + }, + createLazyFile:(parent, name, url, canRead, canWrite) => { + // Lazy chunked Uint8Array (implements get and length from Uint8Array). Actual getting is abstracted away for eventual reuse. + /** @constructor */ + function LazyUint8Array() { + this.lengthKnown = false; + this.chunks = []; // Loaded chunks. Index is the chunk number + } + LazyUint8Array.prototype.get = /** @this{Object} */ function LazyUint8Array_get(idx) { + if (idx > this.length-1 || idx < 0) { + return undefined; + } + var chunkOffset = idx % this.chunkSize; + var chunkNum = (idx / this.chunkSize)|0; + return this.getter(chunkNum)[chunkOffset]; + }; + LazyUint8Array.prototype.setDataGetter = function LazyUint8Array_setDataGetter(getter) { + this.getter = getter; + }; + LazyUint8Array.prototype.cacheLength = function LazyUint8Array_cacheLength() { + // Find length + var xhr = new XMLHttpRequest(); + xhr.open('HEAD', url, false); + xhr.send(null); + if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status); + var datalength = Number(xhr.getResponseHeader("Content-length")); + var header; + var hasByteServing = (header = xhr.getResponseHeader("Accept-Ranges")) && header === "bytes"; + var usesGzip = (header = xhr.getResponseHeader("Content-Encoding")) && header === "gzip"; + + var chunkSize = 1024*1024; // Chunk size in bytes + + if (!hasByteServing) chunkSize = datalength; + + // Function to get a range from the remote URL. + var doXHR = (from, to) => { + if (from > to) throw new Error("invalid range (" + from + ", " + to + ") or no bytes requested!"); + if (to > datalength-1) throw new Error("only " + datalength + " bytes available! programmer error!"); + + // TODO: Use mozResponseArrayBuffer, responseStream, etc. if available. + var xhr = new XMLHttpRequest(); + xhr.open('GET', url, false); + if (datalength !== chunkSize) xhr.setRequestHeader("Range", "bytes=" + from + "-" + to); + + // Some hints to the browser that we want binary data. + xhr.responseType = 'arraybuffer'; + if (xhr.overrideMimeType) { + xhr.overrideMimeType('text/plain; charset=x-user-defined'); + } + + xhr.send(null); + if (!(xhr.status >= 200 && xhr.status < 300 || xhr.status === 304)) throw new Error("Couldn't load " + url + ". Status: " + xhr.status); + if (xhr.response !== undefined) { + return new Uint8Array(/** @type{Array} */(xhr.response || [])); + } + return intArrayFromString(xhr.responseText || '', true); + }; + var lazyArray = this; + lazyArray.setDataGetter((chunkNum) => { + var start = chunkNum * chunkSize; + var end = (chunkNum+1) * chunkSize - 1; // including this byte + end = Math.min(end, datalength-1); // if datalength-1 is selected, this is the last block + if (typeof lazyArray.chunks[chunkNum] == 'undefined') { + lazyArray.chunks[chunkNum] = doXHR(start, end); + } + if (typeof lazyArray.chunks[chunkNum] == 'undefined') throw new Error('doXHR failed!'); + return lazyArray.chunks[chunkNum]; + }); + + if (usesGzip || !datalength) { + // if the server uses gzip or doesn't supply the length, we have to download the whole file to get the (uncompressed) length + chunkSize = datalength = 1; // this will force getter(0)/doXHR do download the whole file + datalength = this.getter(0).length; + chunkSize = datalength; + out("LazyFiles on gzip forces download of the whole file when length is accessed"); + } + + this._length = datalength; + this._chunkSize = chunkSize; + this.lengthKnown = true; + }; + if (typeof XMLHttpRequest != 'undefined') { + if (!ENVIRONMENT_IS_WORKER) throw 'Cannot do synchronous binary XHRs outside webworkers in modern browsers. Use --embed-file or --preload-file in emcc'; + var lazyArray = new LazyUint8Array(); + Object.defineProperties(lazyArray, { + length: { + get: /** @this{Object} */ function() { + if (!this.lengthKnown) { + this.cacheLength(); + } + return this._length; + } + }, + chunkSize: { + get: /** @this{Object} */ function() { + if (!this.lengthKnown) { + this.cacheLength(); + } + return this._chunkSize; + } + } + }); + + var properties = { isDevice: false, contents: lazyArray }; + } else { + var properties = { isDevice: false, url: url }; + } + + var node = FS.createFile(parent, name, properties, canRead, canWrite); + // This is a total hack, but I want to get this lazy file code out of the + // core of MEMFS. If we want to keep this lazy file concept I feel it should + // be its own thin LAZYFS proxying calls to MEMFS. + if (properties.contents) { + node.contents = properties.contents; + } else if (properties.url) { + node.contents = null; + node.url = properties.url; + } + // Add a function that defers querying the file size until it is asked the first time. + Object.defineProperties(node, { + usedBytes: { + get: /** @this {FSNode} */ function() { return this.contents.length; } + } + }); + // override each stream op with one that tries to force load the lazy file first + var stream_ops = {}; + var keys = Object.keys(node.stream_ops); + keys.forEach((key) => { + var fn = node.stream_ops[key]; + stream_ops[key] = function forceLoadLazyFile() { + FS.forceLoadFile(node); + return fn.apply(null, arguments); + }; + }); + function writeChunks(stream, buffer, offset, length, position) { + var contents = stream.node.contents; + if (position >= contents.length) + return 0; + var size = Math.min(contents.length - position, length); + assert(size >= 0); + if (contents.slice) { // normal array + for (var i = 0; i < size; i++) { + buffer[offset + i] = contents[position + i]; + } + } else { + for (var i = 0; i < size; i++) { // LazyUint8Array from sync binary XHR + buffer[offset + i] = contents.get(position + i); + } + } + return size; + } + // use a custom read function + stream_ops.read = (stream, buffer, offset, length, position) => { + FS.forceLoadFile(node); + return writeChunks(stream, buffer, offset, length, position) + }; + // use a custom mmap function + stream_ops.mmap = (stream, length, position, prot, flags) => { + FS.forceLoadFile(node); + var ptr = mmapAlloc(length); + if (!ptr) { + throw new FS.ErrnoError(48); + } + writeChunks(stream, HEAP8, ptr, length, position); + return { ptr, allocated: true }; + }; + node.stream_ops = stream_ops; + return node; + }, + absolutePath:() => { + abort('FS.absolutePath has been removed; use PATH_FS.resolve instead'); + }, + createFolder:() => { + abort('FS.createFolder has been removed; use FS.mkdir instead'); + }, + createLink:() => { + abort('FS.createLink has been removed; use FS.symlink instead'); + }, + joinPath:() => { + abort('FS.joinPath has been removed; use PATH.join instead'); + }, + mmapAlloc:() => { + abort('FS.mmapAlloc has been replaced by the top level function mmapAlloc'); + }, + standardizePath:() => { + abort('FS.standardizePath has been removed; use PATH.normalize instead'); + }, + }; + + var SYSCALLS = { + DEFAULT_POLLMASK:5, + calculateAt:function(dirfd, path, allowEmpty) { + if (PATH.isAbs(path)) { + return path; + } + // relative path + var dir; + if (dirfd === -100) { + dir = FS.cwd(); + } else { + var dirstream = SYSCALLS.getStreamFromFD(dirfd); + dir = dirstream.path; + } + if (path.length == 0) { + if (!allowEmpty) { + throw new FS.ErrnoError(44);; + } + return dir; + } + return PATH.join2(dir, path); + }, + doStat:function(func, path, buf) { + try { + var stat = func(path); + } catch (e) { + if (e && e.node && PATH.normalize(path) !== PATH.normalize(FS.getPath(e.node))) { + // an error occurred while trying to look up the path; we should just report ENOTDIR + return -54; + } + throw e; + } + HEAP32[((buf)>>2)] = stat.dev; + HEAP32[(((buf)+(4))>>2)] = stat.mode; + HEAPU32[(((buf)+(8))>>2)] = stat.nlink; + HEAP32[(((buf)+(12))>>2)] = stat.uid; + HEAP32[(((buf)+(16))>>2)] = stat.gid; + HEAP32[(((buf)+(20))>>2)] = stat.rdev; + (tempI64 = [stat.size>>>0,(tempDouble=stat.size,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((buf)+(24))>>2)] = tempI64[0],HEAP32[(((buf)+(28))>>2)] = tempI64[1]); + HEAP32[(((buf)+(32))>>2)] = 4096; + HEAP32[(((buf)+(36))>>2)] = stat.blocks; + var atime = stat.atime.getTime(); + var mtime = stat.mtime.getTime(); + var ctime = stat.ctime.getTime(); + (tempI64 = [Math.floor(atime / 1000)>>>0,(tempDouble=Math.floor(atime / 1000),(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((buf)+(40))>>2)] = tempI64[0],HEAP32[(((buf)+(44))>>2)] = tempI64[1]); + HEAPU32[(((buf)+(48))>>2)] = (atime % 1000) * 1000; + (tempI64 = [Math.floor(mtime / 1000)>>>0,(tempDouble=Math.floor(mtime / 1000),(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((buf)+(56))>>2)] = tempI64[0],HEAP32[(((buf)+(60))>>2)] = tempI64[1]); + HEAPU32[(((buf)+(64))>>2)] = (mtime % 1000) * 1000; + (tempI64 = [Math.floor(ctime / 1000)>>>0,(tempDouble=Math.floor(ctime / 1000),(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((buf)+(72))>>2)] = tempI64[0],HEAP32[(((buf)+(76))>>2)] = tempI64[1]); + HEAPU32[(((buf)+(80))>>2)] = (ctime % 1000) * 1000; + (tempI64 = [stat.ino>>>0,(tempDouble=stat.ino,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((buf)+(88))>>2)] = tempI64[0],HEAP32[(((buf)+(92))>>2)] = tempI64[1]); + return 0; + }, + doMsync:function(addr, stream, len, flags, offset) { + if (!FS.isFile(stream.node.mode)) { + throw new FS.ErrnoError(43); + } + if (flags & 2) { + // MAP_PRIVATE calls need not to be synced back to underlying fs + return 0; + } + var buffer = HEAPU8.slice(addr, addr + len); + FS.msync(stream, buffer, offset, len, flags); + }, + varargs:undefined, + get() { + assert(SYSCALLS.varargs != undefined); + SYSCALLS.varargs += 4; + var ret = HEAP32[(((SYSCALLS.varargs)-(4))>>2)]; + return ret; + }, + getStr(ptr) { + var ret = UTF8ToString(ptr); + return ret; + }, + getStreamFromFD:function(fd) { + var stream = FS.getStreamChecked(fd); + return stream; + }, + }; + function ___syscall__newselect(nfds, readfds, writefds, exceptfds, timeout) { + try { + + // readfds are supported, + // writefds checks socket open status + // exceptfds not supported + // timeout is always 0 - fully async + assert(nfds <= 64, 'nfds must be less than or equal to 64'); // fd sets have 64 bits // TODO: this could be 1024 based on current musl headers + assert(!exceptfds, 'exceptfds not supported'); + + var total = 0; + + var srcReadLow = (readfds ? HEAP32[((readfds)>>2)] : 0), + srcReadHigh = (readfds ? HEAP32[(((readfds)+(4))>>2)] : 0); + var srcWriteLow = (writefds ? HEAP32[((writefds)>>2)] : 0), + srcWriteHigh = (writefds ? HEAP32[(((writefds)+(4))>>2)] : 0); + var srcExceptLow = (exceptfds ? HEAP32[((exceptfds)>>2)] : 0), + srcExceptHigh = (exceptfds ? HEAP32[(((exceptfds)+(4))>>2)] : 0); + + var dstReadLow = 0, + dstReadHigh = 0; + var dstWriteLow = 0, + dstWriteHigh = 0; + var dstExceptLow = 0, + dstExceptHigh = 0; + + var allLow = (readfds ? HEAP32[((readfds)>>2)] : 0) | + (writefds ? HEAP32[((writefds)>>2)] : 0) | + (exceptfds ? HEAP32[((exceptfds)>>2)] : 0); + var allHigh = (readfds ? HEAP32[(((readfds)+(4))>>2)] : 0) | + (writefds ? HEAP32[(((writefds)+(4))>>2)] : 0) | + (exceptfds ? HEAP32[(((exceptfds)+(4))>>2)] : 0); + + var check = function(fd, low, high, val) { + return (fd < 32 ? (low & val) : (high & val)); + }; + + for (var fd = 0; fd < nfds; fd++) { + var mask = 1 << (fd % 32); + if (!(check(fd, allLow, allHigh, mask))) { + continue; // index isn't in the set + } + + var stream = SYSCALLS.getStreamFromFD(fd); + + var flags = SYSCALLS.DEFAULT_POLLMASK; + + if (stream.stream_ops.poll) { + var timeoutInMillis = -1; + if (timeout) { + var tv_sec = (readfds ? HEAP32[((timeout)>>2)] : 0), + tv_usec = (readfds ? HEAP32[(((timeout)+(8))>>2)] : 0); + timeoutInMillis = (tv_sec + tv_usec / 1000000) * 1000; + } + flags = stream.stream_ops.poll(stream, timeoutInMillis); + } + + if ((flags & 1) && check(fd, srcReadLow, srcReadHigh, mask)) { + fd < 32 ? (dstReadLow = dstReadLow | mask) : (dstReadHigh = dstReadHigh | mask); + total++; + } + if ((flags & 4) && check(fd, srcWriteLow, srcWriteHigh, mask)) { + fd < 32 ? (dstWriteLow = dstWriteLow | mask) : (dstWriteHigh = dstWriteHigh | mask); + total++; + } + if ((flags & 2) && check(fd, srcExceptLow, srcExceptHigh, mask)) { + fd < 32 ? (dstExceptLow = dstExceptLow | mask) : (dstExceptHigh = dstExceptHigh | mask); + total++; + } + } + + if (readfds) { + HEAP32[((readfds)>>2)] = dstReadLow; + HEAP32[(((readfds)+(4))>>2)] = dstReadHigh; + } + if (writefds) { + HEAP32[((writefds)>>2)] = dstWriteLow; + HEAP32[(((writefds)+(4))>>2)] = dstWriteHigh; + } + if (exceptfds) { + HEAP32[((exceptfds)>>2)] = dstExceptLow; + HEAP32[(((exceptfds)+(4))>>2)] = dstExceptHigh; + } + + return total; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + var SOCKFS = { + mount(mount) { + // If Module['websocket'] has already been defined (e.g. for configuring + // the subprotocol/url) use that, if not initialise it to a new object. + Module['websocket'] = (Module['websocket'] && + ('object' === typeof Module['websocket'])) ? Module['websocket'] : {}; + + // Add the Event registration mechanism to the exported websocket configuration + // object so we can register network callbacks from native JavaScript too. + // For more documentation see system/include/emscripten/emscripten.h + Module['websocket']._callbacks = {}; + Module['websocket']['on'] = /** @this{Object} */ function(event, callback) { + if ('function' === typeof callback) { + this._callbacks[event] = callback; + } + return this; + }; + + Module['websocket'].emit = /** @this{Object} */ function(event, param) { + if ('function' === typeof this._callbacks[event]) { + this._callbacks[event].call(this, param); + } + }; + + // If debug is enabled register simple default logging callbacks for each Event. + + return FS.createNode(null, '/', 16384 | 511 /* 0777 */, 0); + }, + createSocket(family, type, protocol) { + type &= ~526336; // Some applications may pass it; it makes no sense for a single process. + var streaming = type == 1; + if (streaming && protocol && protocol != 6) { + throw new FS.ErrnoError(66); // if SOCK_STREAM, must be tcp or 0. + } + + // create our internal socket structure + var sock = { + family, + type, + protocol, + server: null, + error: null, // Used in getsockopt for SOL_SOCKET/SO_ERROR test + peers: {}, + pending: [], + recv_queue: [], + sock_ops: SOCKFS.websocket_sock_ops + }; + + // create the filesystem node to store the socket structure + var name = SOCKFS.nextname(); + var node = FS.createNode(SOCKFS.root, name, 49152, 0); + node.sock = sock; + + // and the wrapping stream that enables library functions such + // as read and write to indirectly interact with the socket + var stream = FS.createStream({ + path: name, + node, + flags: 2, + seekable: false, + stream_ops: SOCKFS.stream_ops + }); + + // map the new stream to the socket structure (sockets have a 1:1 + // relationship with a stream) + sock.stream = stream; + + return sock; + }, + getSocket(fd) { + var stream = FS.getStream(fd); + if (!stream || !FS.isSocket(stream.node.mode)) { + return null; + } + return stream.node.sock; + }, + stream_ops:{ + poll(stream) { + var sock = stream.node.sock; + return sock.sock_ops.poll(sock); + }, + ioctl(stream, request, varargs) { + var sock = stream.node.sock; + return sock.sock_ops.ioctl(sock, request, varargs); + }, + read(stream, buffer, offset, length, position /* ignored */) { + var sock = stream.node.sock; + var msg = sock.sock_ops.recvmsg(sock, length); + if (!msg) { + // socket is closed + return 0; + } + buffer.set(msg.buffer, offset); + return msg.buffer.length; + }, + write(stream, buffer, offset, length, position /* ignored */) { + var sock = stream.node.sock; + return sock.sock_ops.sendmsg(sock, buffer, offset, length); + }, + close(stream) { + var sock = stream.node.sock; + sock.sock_ops.close(sock); + }, + }, + nextname() { + if (!SOCKFS.nextname.current) { + SOCKFS.nextname.current = 0; + } + return 'socket[' + (SOCKFS.nextname.current++) + ']'; + }, + websocket_sock_ops:{ + createPeer(sock, addr, port) { + var ws; + + if (typeof addr == 'object') { + ws = addr; + addr = null; + port = null; + } + + if (ws) { + // for sockets that've already connected (e.g. we're the server) + // we can inspect the _socket property for the address + if (ws._socket) { + addr = ws._socket.remoteAddress; + port = ws._socket.remotePort; + } + // if we're just now initializing a connection to the remote, + // inspect the url property + else { + var result = /ws[s]?:\/\/([^:]+):(\d+)/.exec(ws.url); + if (!result) { + throw new Error('WebSocket URL must be in the format ws(s)://address:port'); + } + addr = result[1]; + port = parseInt(result[2], 10); + } + } else { + // create the actual websocket object and connect + try { + // runtimeConfig gets set to true if WebSocket runtime configuration is available. + var runtimeConfig = (Module['websocket'] && ('object' === typeof Module['websocket'])); + + // The default value is 'ws://' the replace is needed because the compiler replaces '//' comments with '#' + // comments without checking context, so we'd end up with ws:#, the replace swaps the '#' for '//' again. + var url = 'ws:#'.replace('#', '//'); + + if (runtimeConfig) { + if ('string' === typeof Module['websocket']['url']) { + url = Module['websocket']['url']; // Fetch runtime WebSocket URL config. + } + } + + if (url === 'ws://' || url === 'wss://') { // Is the supplied URL config just a prefix, if so complete it. + var parts = addr.split('/'); + url = url + parts[0] + ":" + port + "/" + parts.slice(1).join('/'); + } + + // Make the WebSocket subprotocol (Sec-WebSocket-Protocol) default to binary if no configuration is set. + var subProtocols = 'binary'; // The default value is 'binary' + + if (runtimeConfig) { + if ('string' === typeof Module['websocket']['subprotocol']) { + subProtocols = Module['websocket']['subprotocol']; // Fetch runtime WebSocket subprotocol config. + } + } + + // The default WebSocket options + var opts = undefined; + + if (subProtocols !== 'null') { + // The regex trims the string (removes spaces at the beginning and end, then splits the string by + // , into an Array. Whitespace removal is important for Websockify and ws. + subProtocols = subProtocols.replace(/^ +| +$/g,"").split(/ *, */); + + opts = subProtocols; + } + + // some webservers (azure) does not support subprotocol header + if (runtimeConfig && null === Module['websocket']['subprotocol']) { + subProtocols = 'null'; + opts = undefined; + } + + // If node we use the ws library. + var WebSocketConstructor; + if (ENVIRONMENT_IS_NODE) { + WebSocketConstructor = /** @type{(typeof WebSocket)} */(require('ws')); + } else + { + WebSocketConstructor = WebSocket; + } + ws = new WebSocketConstructor(url, opts); + ws.binaryType = 'arraybuffer'; + } catch (e) { + throw new FS.ErrnoError(23); + } + } + + var peer = { + addr, + port, + socket: ws, + dgram_send_queue: [] + }; + + SOCKFS.websocket_sock_ops.addPeer(sock, peer); + SOCKFS.websocket_sock_ops.handlePeerEvents(sock, peer); + + // if this is a bound dgram socket, send the port number first to allow + // us to override the ephemeral port reported to us by remotePort on the + // remote end. + if (sock.type === 2 && typeof sock.sport != 'undefined') { + peer.dgram_send_queue.push(new Uint8Array([ + 255, 255, 255, 255, + 'p'.charCodeAt(0), 'o'.charCodeAt(0), 'r'.charCodeAt(0), 't'.charCodeAt(0), + ((sock.sport & 0xff00) >> 8) , (sock.sport & 0xff) + ])); + } + + return peer; + }, + getPeer(sock, addr, port) { + return sock.peers[addr + ':' + port]; + }, + addPeer(sock, peer) { + sock.peers[peer.addr + ':' + peer.port] = peer; + }, + removePeer(sock, peer) { + delete sock.peers[peer.addr + ':' + peer.port]; + }, + handlePeerEvents(sock, peer) { + var first = true; + + var handleOpen = function () { + + Module['websocket'].emit('open', sock.stream.fd); + + try { + var queued = peer.dgram_send_queue.shift(); + while (queued) { + peer.socket.send(queued); + queued = peer.dgram_send_queue.shift(); + } + } catch (e) { + // not much we can do here in the way of proper error handling as we've already + // lied and said this data was sent. shut it down. + peer.socket.close(); + } + }; + + function handleMessage(data) { + if (typeof data == 'string') { + var encoder = new TextEncoder(); // should be utf-8 + data = encoder.encode(data); // make a typed array from the string + } else { + assert(data.byteLength !== undefined); // must receive an ArrayBuffer + if (data.byteLength == 0) { + // An empty ArrayBuffer will emit a pseudo disconnect event + // as recv/recvmsg will return zero which indicates that a socket + // has performed a shutdown although the connection has not been disconnected yet. + return; + } + data = new Uint8Array(data); // make a typed array view on the array buffer + } + + // if this is the port message, override the peer's port with it + var wasfirst = first; + first = false; + if (wasfirst && + data.length === 10 && + data[0] === 255 && data[1] === 255 && data[2] === 255 && data[3] === 255 && + data[4] === 'p'.charCodeAt(0) && data[5] === 'o'.charCodeAt(0) && data[6] === 'r'.charCodeAt(0) && data[7] === 't'.charCodeAt(0)) { + // update the peer's port and it's key in the peer map + var newport = ((data[8] << 8) | data[9]); + SOCKFS.websocket_sock_ops.removePeer(sock, peer); + peer.port = newport; + SOCKFS.websocket_sock_ops.addPeer(sock, peer); + return; + } + + sock.recv_queue.push({ addr: peer.addr, port: peer.port, data: data }); + Module['websocket'].emit('message', sock.stream.fd); + }; + + if (ENVIRONMENT_IS_NODE) { + peer.socket.on('open', handleOpen); + peer.socket.on('message', function(data, isBinary) { + if (!isBinary) { + return; + } + handleMessage((new Uint8Array(data)).buffer); // copy from node Buffer -> ArrayBuffer + }); + peer.socket.on('close', function() { + Module['websocket'].emit('close', sock.stream.fd); + }); + peer.socket.on('error', function(error) { + // Although the ws library may pass errors that may be more descriptive than + // ECONNREFUSED they are not necessarily the expected error code e.g. + // ENOTFOUND on getaddrinfo seems to be node.js specific, so using ECONNREFUSED + // is still probably the most useful thing to do. + sock.error = 14; // Used in getsockopt for SOL_SOCKET/SO_ERROR test. + Module['websocket'].emit('error', [sock.stream.fd, sock.error, 'ECONNREFUSED: Connection refused']); + // don't throw + }); + } else { + peer.socket.onopen = handleOpen; + peer.socket.onclose = function() { + Module['websocket'].emit('close', sock.stream.fd); + }; + peer.socket.onmessage = function peer_socket_onmessage(event) { + handleMessage(event.data); + }; + peer.socket.onerror = function(error) { + // The WebSocket spec only allows a 'simple event' to be thrown on error, + // so we only really know as much as ECONNREFUSED. + sock.error = 14; // Used in getsockopt for SOL_SOCKET/SO_ERROR test. + Module['websocket'].emit('error', [sock.stream.fd, sock.error, 'ECONNREFUSED: Connection refused']); + }; + } + }, + poll(sock) { + if (sock.type === 1 && sock.server) { + // listen sockets should only say they're available for reading + // if there are pending clients. + return sock.pending.length ? (64 | 1) : 0; + } + + var mask = 0; + var dest = sock.type === 1 ? // we only care about the socket state for connection-based sockets + SOCKFS.websocket_sock_ops.getPeer(sock, sock.daddr, sock.dport) : + null; + + if (sock.recv_queue.length || + !dest || // connection-less sockets are always ready to read + (dest && dest.socket.readyState === dest.socket.CLOSING) || + (dest && dest.socket.readyState === dest.socket.CLOSED)) { // let recv return 0 once closed + mask |= (64 | 1); + } + + if (!dest || // connection-less sockets are always ready to write + (dest && dest.socket.readyState === dest.socket.OPEN)) { + mask |= 4; + } + + if ((dest && dest.socket.readyState === dest.socket.CLOSING) || + (dest && dest.socket.readyState === dest.socket.CLOSED)) { + mask |= 16; + } + + return mask; + }, + ioctl(sock, request, arg) { + switch (request) { + case 21531: + var bytes = 0; + if (sock.recv_queue.length) { + bytes = sock.recv_queue[0].data.length; + } + HEAP32[((arg)>>2)] = bytes; + return 0; + default: + return 28; + } + }, + close(sock) { + // if we've spawned a listen server, close it + if (sock.server) { + try { + sock.server.close(); + } catch (e) { + } + sock.server = null; + } + // close any peer connections + var peers = Object.keys(sock.peers); + for (var i = 0; i < peers.length; i++) { + var peer = sock.peers[peers[i]]; + try { + peer.socket.close(); + } catch (e) { + } + SOCKFS.websocket_sock_ops.removePeer(sock, peer); + } + return 0; + }, + bind(sock, addr, port) { + if (typeof sock.saddr != 'undefined' || typeof sock.sport != 'undefined') { + throw new FS.ErrnoError(28); // already bound + } + sock.saddr = addr; + sock.sport = port; + // in order to emulate dgram sockets, we need to launch a listen server when + // binding on a connection-less socket + // note: this is only required on the server side + if (sock.type === 2) { + // close the existing server if it exists + if (sock.server) { + sock.server.close(); + sock.server = null; + } + // swallow error operation not supported error that occurs when binding in the + // browser where this isn't supported + try { + sock.sock_ops.listen(sock, 0); + } catch (e) { + if (!(e.name === 'ErrnoError')) throw e; + if (e.errno !== 138) throw e; + } + } + }, + connect(sock, addr, port) { + if (sock.server) { + throw new FS.ErrnoError(138); + } + + // TODO autobind + // if (!sock.addr && sock.type == 2) { + // } + + // early out if we're already connected / in the middle of connecting + if (typeof sock.daddr != 'undefined' && typeof sock.dport != 'undefined') { + var dest = SOCKFS.websocket_sock_ops.getPeer(sock, sock.daddr, sock.dport); + if (dest) { + if (dest.socket.readyState === dest.socket.CONNECTING) { + throw new FS.ErrnoError(7); + } else { + throw new FS.ErrnoError(30); + } + } + } + + // add the socket to our peer list and set our + // destination address / port to match + var peer = SOCKFS.websocket_sock_ops.createPeer(sock, addr, port); + sock.daddr = peer.addr; + sock.dport = peer.port; + + // always "fail" in non-blocking mode + throw new FS.ErrnoError(26); + }, + listen(sock, backlog) { + if (!ENVIRONMENT_IS_NODE) { + throw new FS.ErrnoError(138); + } + if (sock.server) { + throw new FS.ErrnoError(28); // already listening + } + var WebSocketServer = require('ws').Server; + var host = sock.saddr; + sock.server = new WebSocketServer({ + host, + port: sock.sport + // TODO support backlog + }); + Module['websocket'].emit('listen', sock.stream.fd); // Send Event with listen fd. + + sock.server.on('connection', function(ws) { + if (sock.type === 1) { + var newsock = SOCKFS.createSocket(sock.family, sock.type, sock.protocol); + + // create a peer on the new socket + var peer = SOCKFS.websocket_sock_ops.createPeer(newsock, ws); + newsock.daddr = peer.addr; + newsock.dport = peer.port; + + // push to queue for accept to pick up + sock.pending.push(newsock); + Module['websocket'].emit('connection', newsock.stream.fd); + } else { + // create a peer on the listen socket so calling sendto + // with the listen socket and an address will resolve + // to the correct client + SOCKFS.websocket_sock_ops.createPeer(sock, ws); + Module['websocket'].emit('connection', sock.stream.fd); + } + }); + sock.server.on('close', function() { + Module['websocket'].emit('close', sock.stream.fd); + sock.server = null; + }); + sock.server.on('error', function(error) { + // Although the ws library may pass errors that may be more descriptive than + // ECONNREFUSED they are not necessarily the expected error code e.g. + // ENOTFOUND on getaddrinfo seems to be node.js specific, so using EHOSTUNREACH + // is still probably the most useful thing to do. This error shouldn't + // occur in a well written app as errors should get trapped in the compiled + // app's own getaddrinfo call. + sock.error = 23; // Used in getsockopt for SOL_SOCKET/SO_ERROR test. + Module['websocket'].emit('error', [sock.stream.fd, sock.error, 'EHOSTUNREACH: Host is unreachable']); + // don't throw + }); + }, + accept(listensock) { + if (!listensock.server || !listensock.pending.length) { + throw new FS.ErrnoError(28); + } + var newsock = listensock.pending.shift(); + newsock.stream.flags = listensock.stream.flags; + return newsock; + }, + getname(sock, peer) { + var addr, port; + if (peer) { + if (sock.daddr === undefined || sock.dport === undefined) { + throw new FS.ErrnoError(53); + } + addr = sock.daddr; + port = sock.dport; + } else { + // TODO saddr and sport will be set for bind()'d UDP sockets, but what + // should we be returning for TCP sockets that've been connect()'d? + addr = sock.saddr || 0; + port = sock.sport || 0; + } + return { addr, port }; + }, + sendmsg(sock, buffer, offset, length, addr, port) { + if (sock.type === 2) { + // connection-less sockets will honor the message address, + // and otherwise fall back to the bound destination address + if (addr === undefined || port === undefined) { + addr = sock.daddr; + port = sock.dport; + } + // if there was no address to fall back to, error out + if (addr === undefined || port === undefined) { + throw new FS.ErrnoError(17); + } + } else { + // connection-based sockets will only use the bound + addr = sock.daddr; + port = sock.dport; + } + + // find the peer for the destination address + var dest = SOCKFS.websocket_sock_ops.getPeer(sock, addr, port); + + // early out if not connected with a connection-based socket + if (sock.type === 1) { + if (!dest || dest.socket.readyState === dest.socket.CLOSING || dest.socket.readyState === dest.socket.CLOSED) { + throw new FS.ErrnoError(53); + } else if (dest.socket.readyState === dest.socket.CONNECTING) { + throw new FS.ErrnoError(6); + } + } + + // create a copy of the incoming data to send, as the WebSocket API + // doesn't work entirely with an ArrayBufferView, it'll just send + // the entire underlying buffer + if (ArrayBuffer.isView(buffer)) { + offset += buffer.byteOffset; + buffer = buffer.buffer; + } + + var data; + data = buffer.slice(offset, offset + length); + + // if we're emulating a connection-less dgram socket and don't have + // a cached connection, queue the buffer to send upon connect and + // lie, saying the data was sent now. + if (sock.type === 2) { + if (!dest || dest.socket.readyState !== dest.socket.OPEN) { + // if we're not connected, open a new connection + if (!dest || dest.socket.readyState === dest.socket.CLOSING || dest.socket.readyState === dest.socket.CLOSED) { + dest = SOCKFS.websocket_sock_ops.createPeer(sock, addr, port); + } + dest.dgram_send_queue.push(data); + return length; + } + } + + try { + // send the actual data + dest.socket.send(data); + return length; + } catch (e) { + throw new FS.ErrnoError(28); + } + }, + recvmsg(sock, length) { + // http://pubs.opengroup.org/onlinepubs/7908799/xns/recvmsg.html + if (sock.type === 1 && sock.server) { + // tcp servers should not be recv()'ing on the listen socket + throw new FS.ErrnoError(53); + } + + var queued = sock.recv_queue.shift(); + if (!queued) { + if (sock.type === 1) { + var dest = SOCKFS.websocket_sock_ops.getPeer(sock, sock.daddr, sock.dport); + + if (!dest) { + // if we have a destination address but are not connected, error out + throw new FS.ErrnoError(53); + } + if (dest.socket.readyState === dest.socket.CLOSING || dest.socket.readyState === dest.socket.CLOSED) { + // return null if the socket has closed + return null; + } + // else, our socket is in a valid state but truly has nothing available + throw new FS.ErrnoError(6); + } + throw new FS.ErrnoError(6); + } + + // queued.data will be an ArrayBuffer if it's unadulterated, but if it's + // requeued TCP data it'll be an ArrayBufferView + var queuedLength = queued.data.byteLength || queued.data.length; + var queuedOffset = queued.data.byteOffset || 0; + var queuedBuffer = queued.data.buffer || queued.data; + var bytesRead = Math.min(length, queuedLength); + var res = { + buffer: new Uint8Array(queuedBuffer, queuedOffset, bytesRead), + addr: queued.addr, + port: queued.port + }; + + // push back any unread data for TCP connections + if (sock.type === 1 && bytesRead < queuedLength) { + var bytesRemaining = queuedLength - bytesRead; + queued.data = new Uint8Array(queuedBuffer, queuedOffset + bytesRead, bytesRemaining); + sock.recv_queue.unshift(queued); + } + + return res; + }, + }, + }; + + function getSocketFromFD(fd) { + var socket = SOCKFS.getSocket(fd); + if (!socket) throw new FS.ErrnoError(8); + return socket; + } + + var setErrNo = (value) => { + HEAP32[((___errno_location())>>2)] = value; + return value; + }; + var Sockets = { + BUFFER_SIZE:10240, + MAX_BUFFER_SIZE:10485760, + nextFd:1, + fds:{ + }, + nextport:1, + maxport:65535, + peer:null, + connections:{ + }, + portmap:{ + }, + localAddr:4261412874, + addrPool:[33554442,50331658,67108874,83886090,100663306,117440522,134217738,150994954,167772170,184549386,201326602,218103818,234881034], + }; + + var inetPton4 = (str) => { + var b = str.split('.'); + for (var i = 0; i < 4; i++) { + var tmp = Number(b[i]); + if (isNaN(tmp)) return null; + b[i] = tmp; + } + return (b[0] | (b[1] << 8) | (b[2] << 16) | (b[3] << 24)) >>> 0; + }; + + + /** @suppress {checkTypes} */ + var jstoi_q = (str) => parseInt(str); + var inetPton6 = (str) => { + var words; + var w, offset, z, i; + /* http://home.deds.nl/~aeron/regex/ */ + var valid6regx = /^((?=.*::)(?!.*::.+::)(::)?([\dA-F]{1,4}:(:|\b)|){5}|([\dA-F]{1,4}:){6})((([\dA-F]{1,4}((?!\3)::|:\b|$))|(?!\2\3)){2}|(((2[0-4]|1\d|[1-9])?\d|25[0-5])\.?\b){4})$/i + var parts = []; + if (!valid6regx.test(str)) { + return null; + } + if (str === "::") { + return [0, 0, 0, 0, 0, 0, 0, 0]; + } + // Z placeholder to keep track of zeros when splitting the string on ":" + if (str.startsWith("::")) { + str = str.replace("::", "Z:"); // leading zeros case + } else { + str = str.replace("::", ":Z:"); + } + + if (str.indexOf(".") > 0) { + // parse IPv4 embedded stress + str = str.replace(new RegExp('[.]', 'g'), ":"); + words = str.split(":"); + words[words.length-4] = jstoi_q(words[words.length-4]) + jstoi_q(words[words.length-3])*256; + words[words.length-3] = jstoi_q(words[words.length-2]) + jstoi_q(words[words.length-1])*256; + words = words.slice(0, words.length-2); + } else { + words = str.split(":"); + } + + offset = 0; z = 0; + for (w=0; w < words.length; w++) { + if (typeof words[w] == 'string') { + if (words[w] === 'Z') { + // compressed zeros - write appropriate number of zero words + for (z = 0; z < (8 - words.length+1); z++) { + parts[w+z] = 0; + } + offset = z-1; + } else { + // parse hex to field to 16-bit value and write it in network byte-order + parts[w+offset] = _htons(parseInt(words[w],16)); + } + } else { + // parsed IPv4 words + parts[w+offset] = words[w]; + } + } + return [ + (parts[1] << 16) | parts[0], + (parts[3] << 16) | parts[2], + (parts[5] << 16) | parts[4], + (parts[7] << 16) | parts[6] + ]; + }; + + + /** @param {number=} addrlen */ + var writeSockaddr = (sa, family, addr, port, addrlen) => { + switch (family) { + case 2: + addr = inetPton4(addr); + zeroMemory(sa, 16); + if (addrlen) { + HEAP32[((addrlen)>>2)] = 16; + } + HEAP16[((sa)>>1)] = family; + HEAP32[(((sa)+(4))>>2)] = addr; + HEAP16[(((sa)+(2))>>1)] = _htons(port); + break; + case 10: + addr = inetPton6(addr); + zeroMemory(sa, 28); + if (addrlen) { + HEAP32[((addrlen)>>2)] = 28; + } + HEAP32[((sa)>>2)] = family; + HEAP32[(((sa)+(8))>>2)] = addr[0]; + HEAP32[(((sa)+(12))>>2)] = addr[1]; + HEAP32[(((sa)+(16))>>2)] = addr[2]; + HEAP32[(((sa)+(20))>>2)] = addr[3]; + HEAP16[(((sa)+(2))>>1)] = _htons(port); + break; + default: + return 5; + } + return 0; + }; + + + var DNS = { + address_map:{ + id:1, + addrs:{ + }, + names:{ + }, + }, + lookup_name:(name) => { + // If the name is already a valid ipv4 / ipv6 address, don't generate a fake one. + var res = inetPton4(name); + if (res !== null) { + return name; + } + res = inetPton6(name); + if (res !== null) { + return name; + } + + // See if this name is already mapped. + var addr; + + if (DNS.address_map.addrs[name]) { + addr = DNS.address_map.addrs[name]; + } else { + var id = DNS.address_map.id++; + assert(id < 65535, 'exceeded max address mappings of 65535'); + + addr = '172.29.' + (id & 0xff) + '.' + (id & 0xff00); + + DNS.address_map.names[addr] = name; + DNS.address_map.addrs[name] = addr; + } + + return addr; + }, + lookup_addr:(addr) => { + if (DNS.address_map.names[addr]) { + return DNS.address_map.names[addr]; + } + + return null; + }, + }; + + function ___syscall_accept4(fd, addr, addrlen, flags, d1, d2) { + try { + + var sock = getSocketFromFD(fd); + var newsock = sock.sock_ops.accept(sock); + if (addr) { + var errno = writeSockaddr(addr, newsock.family, DNS.lookup_name(newsock.daddr), newsock.dport, addrlen); + assert(!errno); + } + return newsock.stream.fd; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + var inetNtop4 = (addr) => { + return (addr & 0xff) + '.' + ((addr >> 8) & 0xff) + '.' + ((addr >> 16) & 0xff) + '.' + ((addr >> 24) & 0xff) + }; + + + var inetNtop6 = (ints) => { + // ref: http://www.ietf.org/rfc/rfc2373.txt - section 2.5.4 + // Format for IPv4 compatible and mapped 128-bit IPv6 Addresses + // 128-bits are split into eight 16-bit words + // stored in network byte order (big-endian) + // | 80 bits | 16 | 32 bits | + // +-----------------------------------------------------------------+ + // | 10 bytes | 2 | 4 bytes | + // +--------------------------------------+--------------------------+ + // + 5 words | 1 | 2 words | + // +--------------------------------------+--------------------------+ + // |0000..............................0000|0000| IPv4 ADDRESS | (compatible) + // +--------------------------------------+----+---------------------+ + // |0000..............................0000|FFFF| IPv4 ADDRESS | (mapped) + // +--------------------------------------+----+---------------------+ + var str = ""; + var word = 0; + var longest = 0; + var lastzero = 0; + var zstart = 0; + var len = 0; + var i = 0; + var parts = [ + ints[0] & 0xffff, + (ints[0] >> 16), + ints[1] & 0xffff, + (ints[1] >> 16), + ints[2] & 0xffff, + (ints[2] >> 16), + ints[3] & 0xffff, + (ints[3] >> 16) + ]; + + // Handle IPv4-compatible, IPv4-mapped, loopback and any/unspecified addresses + + var hasipv4 = true; + var v4part = ""; + // check if the 10 high-order bytes are all zeros (first 5 words) + for (i = 0; i < 5; i++) { + if (parts[i] !== 0) { hasipv4 = false; break; } + } + + if (hasipv4) { + // low-order 32-bits store an IPv4 address (bytes 13 to 16) (last 2 words) + v4part = inetNtop4(parts[6] | (parts[7] << 16)); + // IPv4-mapped IPv6 address if 16-bit value (bytes 11 and 12) == 0xFFFF (6th word) + if (parts[5] === -1) { + str = "::ffff:"; + str += v4part; + return str; + } + // IPv4-compatible IPv6 address if 16-bit value (bytes 11 and 12) == 0x0000 (6th word) + if (parts[5] === 0) { + str = "::"; + //special case IPv6 addresses + if (v4part === "0.0.0.0") v4part = ""; // any/unspecified address + if (v4part === "0.0.0.1") v4part = "1";// loopback address + str += v4part; + return str; + } + } + + // Handle all other IPv6 addresses + + // first run to find the longest contiguous zero words + for (word = 0; word < 8; word++) { + if (parts[word] === 0) { + if (word - lastzero > 1) { + len = 0; + } + lastzero = word; + len++; + } + if (len > longest) { + longest = len; + zstart = word - longest + 1; + } + } + + for (word = 0; word < 8; word++) { + if (longest > 1) { + // compress contiguous zeros - to produce "::" + if (parts[word] === 0 && word >= zstart && word < (zstart + longest) ) { + if (word === zstart) { + str += ":"; + if (zstart === 0) str += ":"; //leading zeros case + } + continue; + } + } + // converts 16-bit words from big-endian to little-endian before converting to hex string + str += Number(_ntohs(parts[word] & 0xffff)).toString(16); + str += word < 7 ? ":" : ""; + } + return str; + }; + + var readSockaddr = (sa, salen) => { + // family / port offsets are common to both sockaddr_in and sockaddr_in6 + var family = HEAP16[((sa)>>1)]; + var port = _ntohs(HEAPU16[(((sa)+(2))>>1)]); + var addr; + + switch (family) { + case 2: + if (salen !== 16) { + return { errno: 28 }; + } + addr = HEAP32[(((sa)+(4))>>2)]; + addr = inetNtop4(addr); + break; + case 10: + if (salen !== 28) { + return { errno: 28 }; + } + addr = [ + HEAP32[(((sa)+(8))>>2)], + HEAP32[(((sa)+(12))>>2)], + HEAP32[(((sa)+(16))>>2)], + HEAP32[(((sa)+(20))>>2)] + ]; + addr = inetNtop6(addr); + break; + default: + return { errno: 5 }; + } + + return { family: family, addr: addr, port: port }; + }; + + + /** @param {boolean=} allowNull */ + function getSocketAddress(addrp, addrlen, allowNull) { + if (allowNull && addrp === 0) return null; + var info = readSockaddr(addrp, addrlen); + if (info.errno) throw new FS.ErrnoError(info.errno); + info.addr = DNS.lookup_addr(info.addr) || info.addr; + return info; + } + + function ___syscall_bind(fd, addr, addrlen, d1, d2, d3) { + try { + + var sock = getSocketFromFD(fd); + var info = getSocketAddress(addr, addrlen); + sock.sock_ops.bind(sock, info.addr, info.port); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_chdir(path) { + try { + + path = SYSCALLS.getStr(path); + FS.chdir(path); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_chmod(path, mode) { + try { + + path = SYSCALLS.getStr(path); + FS.chmod(path, mode); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + function ___syscall_connect(fd, addr, addrlen, d1, d2, d3) { + try { + + var sock = getSocketFromFD(fd); + var info = getSocketAddress(addr, addrlen); + sock.sock_ops.connect(sock, info.addr, info.port); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_dup(fd) { + try { + + var old = SYSCALLS.getStreamFromFD(fd); + return FS.createStream(old).fd; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_dup3(fd, newfd, flags) { + try { + + var old = SYSCALLS.getStreamFromFD(fd); + assert(!flags); + if (old.fd === newfd) return -28; + var existing = FS.getStream(newfd); + if (existing) FS.close(existing); + return FS.createStream(old, newfd).fd; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_faccessat(dirfd, path, amode, flags) { + try { + + path = SYSCALLS.getStr(path); + assert(flags === 0); + path = SYSCALLS.calculateAt(dirfd, path); + if (amode & ~7) { + // need a valid mode + return -28; + } + var lookup = FS.lookupPath(path, { follow: true }); + var node = lookup.node; + if (!node) { + return -44; + } + var perms = ''; + if (amode & 4) perms += 'r'; + if (amode & 2) perms += 'w'; + if (amode & 1) perms += 'x'; + if (perms /* otherwise, they've just passed F_OK */ && FS.nodePermissions(node, perms)) { + return -2; + } + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + function ___syscall_fcntl64(fd, cmd, varargs) { + SYSCALLS.varargs = varargs; + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + switch (cmd) { + case 0: { + var arg = SYSCALLS.get(); + if (arg < 0) { + return -28; + } + var newStream; + newStream = FS.createStream(stream, arg); + return newStream.fd; + } + case 1: + case 2: + return 0; // FD_CLOEXEC makes no sense for a single process. + case 3: + return stream.flags; + case 4: { + var arg = SYSCALLS.get(); + stream.flags |= arg; + return 0; + } + case 5: + /* case 5: Currently in musl F_GETLK64 has same value as F_GETLK, so omitted to avoid duplicate case blocks. If that changes, uncomment this */ { + + var arg = SYSCALLS.get(); + var offset = 0; + // We're always unlocked. + HEAP16[(((arg)+(offset))>>1)] = 2; + return 0; + } + case 6: + case 7: + /* case 6: Currently in musl F_SETLK64 has same value as F_SETLK, so omitted to avoid duplicate case blocks. If that changes, uncomment this */ + /* case 7: Currently in musl F_SETLKW64 has same value as F_SETLKW, so omitted to avoid duplicate case blocks. If that changes, uncomment this */ + + + return 0; // Pretend that the locking is successful. + case 16: + case 8: + return -28; // These are for sockets. We don't have them fully implemented yet. + case 9: + // musl trusts getown return values, due to a bug where they must be, as they overlap with errors. just return -1 here, so fcntl() returns that, and we set errno ourselves. + setErrNo(28); + return -1; + default: { + return -28; + } + } + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_fstat64(fd, buf) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + return SYSCALLS.doStat(FS.stat, stream.path, buf); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_statfs64(path, size, buf) { + try { + + path = SYSCALLS.getStr(path); + assert(size === 64); + // NOTE: None of the constants here are true. We're just returning safe and + // sane values. + HEAP32[(((buf)+(4))>>2)] = 4096; + HEAP32[(((buf)+(40))>>2)] = 4096; + HEAP32[(((buf)+(8))>>2)] = 1000000; + HEAP32[(((buf)+(12))>>2)] = 500000; + HEAP32[(((buf)+(16))>>2)] = 500000; + HEAP32[(((buf)+(20))>>2)] = FS.nextInode; + HEAP32[(((buf)+(24))>>2)] = 1000000; + HEAP32[(((buf)+(28))>>2)] = 42; + HEAP32[(((buf)+(44))>>2)] = 2; // ST_NOSUID + HEAP32[(((buf)+(36))>>2)] = 255; + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_fstatfs64(fd, size, buf) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + return ___syscall_statfs64(0, size, buf); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + var stringToUTF8 = (str, outPtr, maxBytesToWrite) => { + assert(typeof maxBytesToWrite == 'number', 'stringToUTF8(str, outPtr, maxBytesToWrite) is missing the third parameter that specifies the length of the output buffer!'); + return stringToUTF8Array(str, HEAPU8,outPtr, maxBytesToWrite); + }; + + function ___syscall_getcwd(buf, size) { + try { + + if (size === 0) return -28; + var cwd = FS.cwd(); + var cwdLengthInBytes = lengthBytesUTF8(cwd) + 1; + if (size < cwdLengthInBytes) return -68; + stringToUTF8(cwd, buf, size); + return cwdLengthInBytes; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + function ___syscall_getdents64(fd, dirp, count) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd) + if (!stream.getdents) { + stream.getdents = FS.readdir(stream.path); + } + + var struct_size = 280; + var pos = 0; + var off = FS.llseek(stream, 0, 1); + + var idx = Math.floor(off / struct_size); + + while (idx < stream.getdents.length && pos + struct_size <= count) { + var id; + var type; + var name = stream.getdents[idx]; + if (name === '.') { + id = stream.node.id; + type = 4; // DT_DIR + } + else if (name === '..') { + var lookup = FS.lookupPath(stream.path, { parent: true }); + id = lookup.node.id; + type = 4; // DT_DIR + } + else { + var child = FS.lookupNode(stream.node, name); + id = child.id; + type = FS.isChrdev(child.mode) ? 2 : // DT_CHR, character device. + FS.isDir(child.mode) ? 4 : // DT_DIR, directory. + FS.isLink(child.mode) ? 10 : // DT_LNK, symbolic link. + 8; // DT_REG, regular file. + } + assert(id); + (tempI64 = [id>>>0,(tempDouble=id,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[((dirp + pos)>>2)] = tempI64[0],HEAP32[(((dirp + pos)+(4))>>2)] = tempI64[1]); + (tempI64 = [(idx + 1) * struct_size>>>0,(tempDouble=(idx + 1) * struct_size,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[(((dirp + pos)+(8))>>2)] = tempI64[0],HEAP32[(((dirp + pos)+(12))>>2)] = tempI64[1]); + HEAP16[(((dirp + pos)+(16))>>1)] = 280; + HEAP8[(((dirp + pos)+(18))>>0)] = type; + stringToUTF8(name, dirp + pos + 19, 256); + pos += struct_size; + idx += 1; + } + FS.llseek(stream, idx * struct_size, 0); + return pos; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + + function ___syscall_getpeername(fd, addr, addrlen, d1, d2, d3) { + try { + + var sock = getSocketFromFD(fd); + if (!sock.daddr) { + return -53; // The socket is not connected. + } + var errno = writeSockaddr(addr, sock.family, DNS.lookup_name(sock.daddr), sock.dport, addrlen); + assert(!errno); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + + function ___syscall_getsockname(fd, addr, addrlen, d1, d2, d3) { + try { + + var sock = getSocketFromFD(fd); + // TODO: sock.saddr should never be undefined, see TODO in websocket_sock_ops.getname + var errno = writeSockaddr(addr, sock.family, DNS.lookup_name(sock.saddr || '0.0.0.0'), sock.sport, addrlen); + assert(!errno); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + function ___syscall_getsockopt(fd, level, optname, optval, optlen, d1) { + try { + + var sock = getSocketFromFD(fd); + // Minimal getsockopt aimed at resolving https://github.com/emscripten-core/emscripten/issues/2211 + // so only supports SOL_SOCKET with SO_ERROR. + if (level === 1) { + if (optname === 4) { + HEAP32[((optval)>>2)] = sock.error; + HEAP32[((optlen)>>2)] = 4; + sock.error = null; // Clear the error (The SO_ERROR option obtains and then clears this field). + return 0; + } + } + return -50; // The option is unknown at the level indicated. + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_ioctl(fd, op, varargs) { + SYSCALLS.varargs = varargs; + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + switch (op) { + case 21509: { + if (!stream.tty) return -59; + return 0; + } + case 21505: { + if (!stream.tty) return -59; + if (stream.tty.ops.ioctl_tcgets) { + var termios = stream.tty.ops.ioctl_tcgets(stream); + var argp = SYSCALLS.get(); + HEAP32[((argp)>>2)] = termios.c_iflag || 0; + HEAP32[(((argp)+(4))>>2)] = termios.c_oflag || 0; + HEAP32[(((argp)+(8))>>2)] = termios.c_cflag || 0; + HEAP32[(((argp)+(12))>>2)] = termios.c_lflag || 0; + for (var i = 0; i < 32; i++) { + HEAP8[(((argp + i)+(17))>>0)] = termios.c_cc[i] || 0; + } + return 0; + } + return 0; + } + case 21510: + case 21511: + case 21512: { + if (!stream.tty) return -59; + return 0; // no-op, not actually adjusting terminal settings + } + case 21506: + case 21507: + case 21508: { + if (!stream.tty) return -59; + if (stream.tty.ops.ioctl_tcsets) { + var argp = SYSCALLS.get(); + var c_iflag = HEAP32[((argp)>>2)]; + var c_oflag = HEAP32[(((argp)+(4))>>2)]; + var c_cflag = HEAP32[(((argp)+(8))>>2)]; + var c_lflag = HEAP32[(((argp)+(12))>>2)]; + var c_cc = [] + for (var i = 0; i < 32; i++) { + c_cc.push(HEAP8[(((argp + i)+(17))>>0)]); + } + return stream.tty.ops.ioctl_tcsets(stream.tty, op, { c_iflag, c_oflag, c_cflag, c_lflag, c_cc }); + } + return 0; // no-op, not actually adjusting terminal settings + } + case 21519: { + if (!stream.tty) return -59; + var argp = SYSCALLS.get(); + HEAP32[((argp)>>2)] = 0; + return 0; + } + case 21520: { + if (!stream.tty) return -59; + return -28; // not supported + } + case 21531: { + var argp = SYSCALLS.get(); + return FS.ioctl(stream, op, argp); + } + case 21523: { + // TODO: in theory we should write to the winsize struct that gets + // passed in, but for now musl doesn't read anything on it + if (!stream.tty) return -59; + if (stream.tty.ops.ioctl_tiocgwinsz) { + var winsize = stream.tty.ops.ioctl_tiocgwinsz(stream.tty); + var argp = SYSCALLS.get(); + HEAP16[((argp)>>1)] = winsize[0]; + HEAP16[(((argp)+(2))>>1)] = winsize[1]; + } + return 0; + } + case 21524: { + // TODO: technically, this ioctl call should change the window size. + // but, since emscripten doesn't have any concept of a terminal window + // yet, we'll just silently throw it away as we do TIOCGWINSZ + if (!stream.tty) return -59; + return 0; + } + case 21515: { + if (!stream.tty) return -59; + return 0; + } + default: return -28; // not supported + } + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + function ___syscall_listen(fd, backlog) { + try { + + var sock = getSocketFromFD(fd); + sock.sock_ops.listen(sock, backlog); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_lstat64(path, buf) { + try { + + path = SYSCALLS.getStr(path); + return SYSCALLS.doStat(FS.lstat, path, buf); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_mkdirat(dirfd, path, mode) { + try { + + path = SYSCALLS.getStr(path); + path = SYSCALLS.calculateAt(dirfd, path); + // remove a trailing slash, if one - /a/b/ has basename of '', but + // we want to create b in the context of this function + path = PATH.normalize(path); + if (path[path.length-1] === '/') path = path.substr(0, path.length-1); + FS.mkdir(path, mode, 0); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_newfstatat(dirfd, path, buf, flags) { + try { + + path = SYSCALLS.getStr(path); + var nofollow = flags & 256; + var allowEmpty = flags & 4096; + flags = flags & (~6400); + assert(!flags, `unknown flags in __syscall_newfstatat: ${flags}`); + path = SYSCALLS.calculateAt(dirfd, path, allowEmpty); + return SYSCALLS.doStat(nofollow ? FS.lstat : FS.stat, path, buf); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_openat(dirfd, path, flags, varargs) { + SYSCALLS.varargs = varargs; + try { + + path = SYSCALLS.getStr(path); + path = SYSCALLS.calculateAt(dirfd, path); + var mode = varargs ? SYSCALLS.get() : 0; + return FS.open(path, flags, mode).fd; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + var PIPEFS = { + BUCKET_BUFFER_SIZE:8192, + mount(mount) { + // Do not pollute the real root directory or its child nodes with pipes + // Looks like it is OK to create another pseudo-root node not linked to the FS.root hierarchy this way + return FS.createNode(null, '/', 16384 | 511 /* 0777 */, 0); + }, + createPipe() { + var pipe = { + buckets: [], + // refcnt 2 because pipe has a read end and a write end. We need to be + // able to read from the read end after write end is closed. + refcnt : 2, + }; + + pipe.buckets.push({ + buffer: new Uint8Array(PIPEFS.BUCKET_BUFFER_SIZE), + offset: 0, + roffset: 0 + }); + + var rName = PIPEFS.nextname(); + var wName = PIPEFS.nextname(); + var rNode = FS.createNode(PIPEFS.root, rName, 4096, 0); + var wNode = FS.createNode(PIPEFS.root, wName, 4096, 0); + + rNode.pipe = pipe; + wNode.pipe = pipe; + + var readableStream = FS.createStream({ + path: rName, + node: rNode, + flags: 0, + seekable: false, + stream_ops: PIPEFS.stream_ops + }); + rNode.stream = readableStream; + + var writableStream = FS.createStream({ + path: wName, + node: wNode, + flags: 1, + seekable: false, + stream_ops: PIPEFS.stream_ops + }); + wNode.stream = writableStream; + + return { + readable_fd: readableStream.fd, + writable_fd: writableStream.fd + }; + }, + stream_ops:{ + poll(stream) { + var pipe = stream.node.pipe; + + if ((stream.flags & 2097155) === 1) { + return (256 | 4); + } + if (pipe.buckets.length > 0) { + for (var i = 0; i < pipe.buckets.length; i++) { + var bucket = pipe.buckets[i]; + if (bucket.offset - bucket.roffset > 0) { + return (64 | 1); + } + } + } + + return 0; + }, + ioctl(stream, request, varargs) { + return 28; + }, + fsync(stream) { + return 28; + }, + read(stream, buffer, offset, length, position /* ignored */) { + var pipe = stream.node.pipe; + var currentLength = 0; + + for (var i = 0; i < pipe.buckets.length; i++) { + var bucket = pipe.buckets[i]; + currentLength += bucket.offset - bucket.roffset; + } + + assert(buffer instanceof ArrayBuffer || ArrayBuffer.isView(buffer)); + var data = buffer.subarray(offset, offset + length); + + if (length <= 0) { + return 0; + } + if (currentLength == 0) { + // Behave as if the read end is always non-blocking + throw new FS.ErrnoError(6); + } + var toRead = Math.min(currentLength, length); + + var totalRead = toRead; + var toRemove = 0; + + for (var i = 0; i < pipe.buckets.length; i++) { + var currBucket = pipe.buckets[i]; + var bucketSize = currBucket.offset - currBucket.roffset; + + if (toRead <= bucketSize) { + var tmpSlice = currBucket.buffer.subarray(currBucket.roffset, currBucket.offset); + if (toRead < bucketSize) { + tmpSlice = tmpSlice.subarray(0, toRead); + currBucket.roffset += toRead; + } else { + toRemove++; + } + data.set(tmpSlice); + break; + } else { + var tmpSlice = currBucket.buffer.subarray(currBucket.roffset, currBucket.offset); + data.set(tmpSlice); + data = data.subarray(tmpSlice.byteLength); + toRead -= tmpSlice.byteLength; + toRemove++; + } + } + + if (toRemove && toRemove == pipe.buckets.length) { + // Do not generate excessive garbage in use cases such as + // write several bytes, read everything, write several bytes, read everything... + toRemove--; + pipe.buckets[toRemove].offset = 0; + pipe.buckets[toRemove].roffset = 0; + } + + pipe.buckets.splice(0, toRemove); + + return totalRead; + }, + write(stream, buffer, offset, length, position /* ignored */) { + var pipe = stream.node.pipe; + + assert(buffer instanceof ArrayBuffer || ArrayBuffer.isView(buffer)); + var data = buffer.subarray(offset, offset + length); + + var dataLen = data.byteLength; + if (dataLen <= 0) { + return 0; + } + + var currBucket = null; + + if (pipe.buckets.length == 0) { + currBucket = { + buffer: new Uint8Array(PIPEFS.BUCKET_BUFFER_SIZE), + offset: 0, + roffset: 0 + }; + pipe.buckets.push(currBucket); + } else { + currBucket = pipe.buckets[pipe.buckets.length - 1]; + } + + assert(currBucket.offset <= PIPEFS.BUCKET_BUFFER_SIZE); + + var freeBytesInCurrBuffer = PIPEFS.BUCKET_BUFFER_SIZE - currBucket.offset; + if (freeBytesInCurrBuffer >= dataLen) { + currBucket.buffer.set(data, currBucket.offset); + currBucket.offset += dataLen; + return dataLen; + } else if (freeBytesInCurrBuffer > 0) { + currBucket.buffer.set(data.subarray(0, freeBytesInCurrBuffer), currBucket.offset); + currBucket.offset += freeBytesInCurrBuffer; + data = data.subarray(freeBytesInCurrBuffer, data.byteLength); + } + + var numBuckets = (data.byteLength / PIPEFS.BUCKET_BUFFER_SIZE) | 0; + var remElements = data.byteLength % PIPEFS.BUCKET_BUFFER_SIZE; + + for (var i = 0; i < numBuckets; i++) { + var newBucket = { + buffer: new Uint8Array(PIPEFS.BUCKET_BUFFER_SIZE), + offset: PIPEFS.BUCKET_BUFFER_SIZE, + roffset: 0 + }; + pipe.buckets.push(newBucket); + newBucket.buffer.set(data.subarray(0, PIPEFS.BUCKET_BUFFER_SIZE)); + data = data.subarray(PIPEFS.BUCKET_BUFFER_SIZE, data.byteLength); + } + + if (remElements > 0) { + var newBucket = { + buffer: new Uint8Array(PIPEFS.BUCKET_BUFFER_SIZE), + offset: data.byteLength, + roffset: 0 + }; + pipe.buckets.push(newBucket); + newBucket.buffer.set(data); + } + + return dataLen; + }, + close(stream) { + var pipe = stream.node.pipe; + pipe.refcnt--; + if (pipe.refcnt === 0) { + pipe.buckets = null; + } + }, + }, + nextname() { + if (!PIPEFS.nextname.current) { + PIPEFS.nextname.current = 0; + } + return 'pipe[' + (PIPEFS.nextname.current++) + ']'; + }, + }; + + function ___syscall_pipe(fdPtr) { + try { + + if (fdPtr == 0) { + throw new FS.ErrnoError(21); + } + + var res = PIPEFS.createPipe(); + + HEAP32[((fdPtr)>>2)] = res.readable_fd; + HEAP32[(((fdPtr)+(4))>>2)] = res.writable_fd; + + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + function ___syscall_readlinkat(dirfd, path, buf, bufsize) { + try { + + path = SYSCALLS.getStr(path); + path = SYSCALLS.calculateAt(dirfd, path); + if (bufsize <= 0) return -28; + var ret = FS.readlink(path); + + var len = Math.min(bufsize, lengthBytesUTF8(ret)); + var endChar = HEAP8[buf+len]; + stringToUTF8(ret, buf, bufsize+1); + // readlink is one of the rare functions that write out a C string, but does never append a null to the output buffer(!) + // stringToUTF8() always appends a null byte, so restore the character under the null byte after the write. + HEAP8[buf+len] = endChar; + return len; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + + function ___syscall_recvfrom(fd, buf, len, flags, addr, addrlen) { + try { + + var sock = getSocketFromFD(fd); + var msg = sock.sock_ops.recvmsg(sock, len); + if (!msg) return 0; // socket is closed + if (addr) { + var errno = writeSockaddr(addr, sock.family, DNS.lookup_name(msg.addr), msg.port, addrlen); + assert(!errno); + } + HEAPU8.set(msg.buffer, buf); + return msg.buffer.byteLength; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_renameat(olddirfd, oldpath, newdirfd, newpath) { + try { + + oldpath = SYSCALLS.getStr(oldpath); + newpath = SYSCALLS.getStr(newpath); + oldpath = SYSCALLS.calculateAt(olddirfd, oldpath); + newpath = SYSCALLS.calculateAt(newdirfd, newpath); + FS.rename(oldpath, newpath); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_rmdir(path) { + try { + + path = SYSCALLS.getStr(path); + FS.rmdir(path); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + + function ___syscall_sendto(fd, message, length, flags, addr, addr_len) { + try { + + var sock = getSocketFromFD(fd); + var dest = getSocketAddress(addr, addr_len, true); + if (!dest) { + // send, no address provided + return FS.write(sock.stream, HEAP8,message, length); + } + // sendto an address + return sock.sock_ops.sendmsg(sock, HEAP8,message, length, dest.addr, dest.port); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + + function ___syscall_socket(domain, type, protocol) { + try { + + var sock = SOCKFS.createSocket(domain, type, protocol); + assert(sock.stream.fd < 64); // XXX ? select() assumes socket fd values are in 0..63 + return sock.stream.fd; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_stat64(path, buf) { + try { + + path = SYSCALLS.getStr(path); + return SYSCALLS.doStat(FS.stat, path, buf); + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_symlink(target, linkpath) { + try { + + target = SYSCALLS.getStr(target); + linkpath = SYSCALLS.getStr(linkpath); + FS.symlink(target, linkpath); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function ___syscall_unlinkat(dirfd, path, flags) { + try { + + path = SYSCALLS.getStr(path); + path = SYSCALLS.calculateAt(dirfd, path); + if (flags === 0) { + FS.unlink(path); + } else if (flags === 512) { + FS.rmdir(path); + } else { + abort('Invalid flags passed to unlinkat'); + } + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + function readI53FromI64(ptr) { + return HEAPU32[ptr>>2] + HEAP32[ptr+4>>2] * 4294967296; + } + + function ___syscall_utimensat(dirfd, path, times, flags) { + try { + + path = SYSCALLS.getStr(path); + assert(flags === 0); + path = SYSCALLS.calculateAt(dirfd, path, true); + if (!times) { + var atime = Date.now(); + var mtime = atime; + } else { + var seconds = readI53FromI64(times); + var nanoseconds = HEAP32[(((times)+(8))>>2)]; + atime = (seconds*1000) + (nanoseconds/(1000*1000)); + times += 16; + seconds = readI53FromI64(times); + nanoseconds = HEAP32[(((times)+(8))>>2)]; + mtime = (seconds*1000) + (nanoseconds/(1000*1000)); + } + FS.utime(path, atime, mtime); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + } + + var nowIsMonotonic = true;; + var __emscripten_get_now_is_monotonic = () => nowIsMonotonic; + + var __emscripten_throw_longjmp = () => { + throw Infinity; + }; + + function convertI32PairToI53Checked(lo, hi) { + assert(lo == (lo >>> 0) || lo == (lo|0)); // lo should either be a i32 or a u32 + assert(hi === (hi|0)); // hi should be a i32 + return ((hi + 0x200000) >>> 0 < 0x400001 - !!lo) ? (lo >>> 0) + hi * 4294967296 : NaN; + } + function __gmtime_js(time_low, time_high,tmPtr) { + var time = convertI32PairToI53Checked(time_low, time_high);; + + + var date = new Date(time * 1000); + HEAP32[((tmPtr)>>2)] = date.getUTCSeconds(); + HEAP32[(((tmPtr)+(4))>>2)] = date.getUTCMinutes(); + HEAP32[(((tmPtr)+(8))>>2)] = date.getUTCHours(); + HEAP32[(((tmPtr)+(12))>>2)] = date.getUTCDate(); + HEAP32[(((tmPtr)+(16))>>2)] = date.getUTCMonth(); + HEAP32[(((tmPtr)+(20))>>2)] = date.getUTCFullYear()-1900; + HEAP32[(((tmPtr)+(24))>>2)] = date.getUTCDay(); + var start = Date.UTC(date.getUTCFullYear(), 0, 1, 0, 0, 0, 0); + var yday = ((date.getTime() - start) / (1000 * 60 * 60 * 24))|0; + HEAP32[(((tmPtr)+(28))>>2)] = yday; + ; + } + + var isLeapYear = (year) => { + return year%4 === 0 && (year%100 !== 0 || year%400 === 0); + }; + + var MONTH_DAYS_LEAP_CUMULATIVE = [0,31,60,91,121,152,182,213,244,274,305,335]; + + var MONTH_DAYS_REGULAR_CUMULATIVE = [0,31,59,90,120,151,181,212,243,273,304,334]; + var ydayFromDate = (date) => { + var leap = isLeapYear(date.getFullYear()); + var monthDaysCumulative = (leap ? MONTH_DAYS_LEAP_CUMULATIVE : MONTH_DAYS_REGULAR_CUMULATIVE); + var yday = monthDaysCumulative[date.getMonth()] + date.getDate() - 1; // -1 since it's days since Jan 1 + + return yday; + }; + + function __localtime_js(time_low, time_high,tmPtr) { + var time = convertI32PairToI53Checked(time_low, time_high);; + + + var date = new Date(time*1000); + HEAP32[((tmPtr)>>2)] = date.getSeconds(); + HEAP32[(((tmPtr)+(4))>>2)] = date.getMinutes(); + HEAP32[(((tmPtr)+(8))>>2)] = date.getHours(); + HEAP32[(((tmPtr)+(12))>>2)] = date.getDate(); + HEAP32[(((tmPtr)+(16))>>2)] = date.getMonth(); + HEAP32[(((tmPtr)+(20))>>2)] = date.getFullYear()-1900; + HEAP32[(((tmPtr)+(24))>>2)] = date.getDay(); + + var yday = ydayFromDate(date)|0; + HEAP32[(((tmPtr)+(28))>>2)] = yday; + HEAP32[(((tmPtr)+(36))>>2)] = -(date.getTimezoneOffset() * 60); + + // Attention: DST is in December in South, and some regions don't have DST at all. + var start = new Date(date.getFullYear(), 0, 1); + var summerOffset = new Date(date.getFullYear(), 6, 1).getTimezoneOffset(); + var winterOffset = start.getTimezoneOffset(); + var dst = (summerOffset != winterOffset && date.getTimezoneOffset() == Math.min(winterOffset, summerOffset))|0; + HEAP32[(((tmPtr)+(32))>>2)] = dst; + ; + } + + + + var __mktime_js = function(tmPtr) { + + var ret = (() => { + var date = new Date(HEAP32[(((tmPtr)+(20))>>2)] + 1900, + HEAP32[(((tmPtr)+(16))>>2)], + HEAP32[(((tmPtr)+(12))>>2)], + HEAP32[(((tmPtr)+(8))>>2)], + HEAP32[(((tmPtr)+(4))>>2)], + HEAP32[((tmPtr)>>2)], + 0); + + // There's an ambiguous hour when the time goes back; the tm_isdst field is + // used to disambiguate it. Date() basically guesses, so we fix it up if it + // guessed wrong, or fill in tm_isdst with the guess if it's -1. + var dst = HEAP32[(((tmPtr)+(32))>>2)]; + var guessedOffset = date.getTimezoneOffset(); + var start = new Date(date.getFullYear(), 0, 1); + var summerOffset = new Date(date.getFullYear(), 6, 1).getTimezoneOffset(); + var winterOffset = start.getTimezoneOffset(); + var dstOffset = Math.min(winterOffset, summerOffset); // DST is in December in South + if (dst < 0) { + // Attention: some regions don't have DST at all. + HEAP32[(((tmPtr)+(32))>>2)] = Number(summerOffset != winterOffset && dstOffset == guessedOffset); + } else if ((dst > 0) != (dstOffset == guessedOffset)) { + var nonDstOffset = Math.max(winterOffset, summerOffset); + var trueOffset = dst > 0 ? dstOffset : nonDstOffset; + // Don't try setMinutes(date.getMinutes() + ...) -- it's messed up. + date.setTime(date.getTime() + (trueOffset - guessedOffset)*60000); + } + + HEAP32[(((tmPtr)+(24))>>2)] = date.getDay(); + var yday = ydayFromDate(date)|0; + HEAP32[(((tmPtr)+(28))>>2)] = yday; + // To match expected behavior, update fields from date + HEAP32[((tmPtr)>>2)] = date.getSeconds(); + HEAP32[(((tmPtr)+(4))>>2)] = date.getMinutes(); + HEAP32[(((tmPtr)+(8))>>2)] = date.getHours(); + HEAP32[(((tmPtr)+(12))>>2)] = date.getDate(); + HEAP32[(((tmPtr)+(16))>>2)] = date.getMonth(); + HEAP32[(((tmPtr)+(20))>>2)] = date.getYear(); + + return date.getTime() / 1000; + })(); + return (setTempRet0((tempDouble=ret,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)), ret>>>0); + }; + + + + + + + function __mmap_js(len,prot,flags,fd,offset_low, offset_high,allocated,addr) { + var offset = convertI32PairToI53Checked(offset_low, offset_high);; + + + try { + + if (isNaN(offset)) return 61; + var stream = SYSCALLS.getStreamFromFD(fd); + var res = FS.mmap(stream, len, offset, prot, flags); + var ptr = res.ptr; + HEAP32[((allocated)>>2)] = res.allocated; + HEAPU32[((addr)>>2)] = ptr; + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + ; + } + + + + + function __munmap_js(addr,len,prot,flags,fd,offset_low, offset_high) { + var offset = convertI32PairToI53Checked(offset_low, offset_high);; + + + try { + + if (isNaN(offset)) return 61; + var stream = SYSCALLS.getStreamFromFD(fd); + if (prot & 2) { + SYSCALLS.doMsync(addr, stream, len, flags, offset); + } + FS.munmap(stream); + // implicitly return 0 + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return -e.errno; + } + ; + } + + + + var stringToNewUTF8 = (str) => { + var size = lengthBytesUTF8(str) + 1; + var ret = _malloc(size); + if (ret) stringToUTF8(str, ret, size); + return ret; + }; + var __tzset_js = (timezone, daylight, tzname) => { + // TODO: Use (malleable) environment variables instead of system settings. + var currentYear = new Date().getFullYear(); + var winter = new Date(currentYear, 0, 1); + var summer = new Date(currentYear, 6, 1); + var winterOffset = winter.getTimezoneOffset(); + var summerOffset = summer.getTimezoneOffset(); + + // Local standard timezone offset. Local standard time is not adjusted for daylight savings. + // This code uses the fact that getTimezoneOffset returns a greater value during Standard Time versus Daylight Saving Time (DST). + // Thus it determines the expected output during Standard Time, and it compares whether the output of the given date the same (Standard) or less (DST). + var stdTimezoneOffset = Math.max(winterOffset, summerOffset); + + // timezone is specified as seconds west of UTC ("The external variable + // `timezone` shall be set to the difference, in seconds, between + // Coordinated Universal Time (UTC) and local standard time."), the same + // as returned by stdTimezoneOffset. + // See http://pubs.opengroup.org/onlinepubs/009695399/functions/tzset.html + HEAPU32[((timezone)>>2)] = stdTimezoneOffset * 60; + + HEAP32[((daylight)>>2)] = Number(winterOffset != summerOffset); + + function extractZone(date) { + var match = date.toTimeString().match(/\(([A-Za-z ]+)\)$/); + return match ? match[1] : "GMT"; + }; + var winterName = extractZone(winter); + var summerName = extractZone(summer); + var winterNamePtr = stringToNewUTF8(winterName); + var summerNamePtr = stringToNewUTF8(summerName); + if (summerOffset < winterOffset) { + // Northern hemisphere + HEAPU32[((tzname)>>2)] = winterNamePtr; + HEAPU32[(((tzname)+(4))>>2)] = summerNamePtr; + } else { + HEAPU32[((tzname)>>2)] = summerNamePtr; + HEAPU32[(((tzname)+(4))>>2)] = winterNamePtr; + } + }; + + var _abort = () => { + abort('native code called abort()'); + }; + + var _emscripten_console_error = (str) => { + assert(typeof str == 'number'); + console.error(UTF8ToString(str)); + }; + + function _emscripten_date_now() { + return Date.now(); + } + + var getHeapMax = () => + HEAPU8.length; + var _emscripten_get_heap_max = () => getHeapMax(); + + var _emscripten_get_now; + // Modern environment where performance.now() is supported: + // N.B. a shorter form "_emscripten_get_now = performance.now;" is + // unfortunately not allowed even in current browsers (e.g. FF Nightly 75). + _emscripten_get_now = () => performance.now(); + ; + + var _emscripten_memcpy_big = (dest, src, num) => HEAPU8.copyWithin(dest, src, src + num); + + + var abortOnCannotGrowMemory = (requestedSize) => { + abort(`Cannot enlarge memory arrays to size ${requestedSize} bytes (OOM). Either (1) compile with -sINITIAL_MEMORY=X with X higher than the current value ${HEAP8.length}, (2) compile with -sALLOW_MEMORY_GROWTH which allows increasing the size at runtime, or (3) if you want malloc to return NULL (0) instead of this abort, compile with -sABORTING_MALLOC=0`); + }; + var _emscripten_resize_heap = (requestedSize) => { + var oldSize = HEAPU8.length; + // With CAN_ADDRESS_2GB or MEMORY64, pointers are already unsigned. + requestedSize >>>= 0; + abortOnCannotGrowMemory(requestedSize); + }; + + var ENV = { + }; + + var getExecutableName = () => { + return thisProgram || './this.program'; + }; + var getEnvStrings = () => { + if (!getEnvStrings.strings) { + // Default values. + // Browser language detection #8751 + var lang = ((typeof navigator == 'object' && navigator.languages && navigator.languages[0]) || 'C').replace('-', '_') + '.UTF-8'; + var env = { + 'USER': 'web_user', + 'LOGNAME': 'web_user', + 'PATH': '/', + 'PWD': '/', + 'HOME': '/home/web_user', + 'LANG': lang, + '_': getExecutableName() + }; + // Apply the user-provided values, if any. + for (var x in ENV) { + // x is a key in ENV; if ENV[x] is undefined, that means it was + // explicitly set to be so. We allow user code to do that to + // force variables with default values to remain unset. + if (ENV[x] === undefined) delete env[x]; + else env[x] = ENV[x]; + } + var strings = []; + for (var x in env) { + strings.push(`${x}=${env[x]}`); + } + getEnvStrings.strings = strings; + } + return getEnvStrings.strings; + }; + + var stringToAscii = (str, buffer) => { + for (var i = 0; i < str.length; ++i) { + assert(str.charCodeAt(i) === (str.charCodeAt(i) & 0xff)); + HEAP8[((buffer++)>>0)] = str.charCodeAt(i); + } + // Null-terminate the string + HEAP8[((buffer)>>0)] = 0; + }; + + var _environ_get = (__environ, environ_buf) => { + var bufSize = 0; + getEnvStrings().forEach(function(string, i) { + var ptr = environ_buf + bufSize; + HEAPU32[(((__environ)+(i*4))>>2)] = ptr; + stringToAscii(string, ptr); + bufSize += string.length + 1; + }); + return 0; + }; + + + var _environ_sizes_get = (penviron_count, penviron_buf_size) => { + var strings = getEnvStrings(); + HEAPU32[((penviron_count)>>2)] = strings.length; + var bufSize = 0; + strings.forEach(function(string) { + bufSize += string.length + 1; + }); + HEAPU32[((penviron_buf_size)>>2)] = bufSize; + return 0; + }; + + + var _proc_exit = (code) => { + EXITSTATUS = code; + if (!keepRuntimeAlive()) { + if (Module['onExit']) Module['onExit'](code); + ABORT = true; + } + quit_(code, new ExitStatus(code)); + }; + /** @suppress {duplicate } */ + /** @param {boolean|number=} implicit */ + var exitJS = (status, implicit) => { + EXITSTATUS = status; + + checkUnflushedContent(); + + // if exit() was called explicitly, warn the user if the runtime isn't actually being shut down + if (keepRuntimeAlive() && !implicit) { + var msg = `program exited (with status: ${status}), but keepRuntimeAlive() is set (counter=${runtimeKeepaliveCounter}) due to an async operation, so halting execution but not exiting the runtime or preventing further async execution (you can use emscripten_force_exit, if you want to force a true shutdown)`; + readyPromiseReject(msg); + err(msg); + } + + _proc_exit(status); + }; + var _exit = exitJS; + + function _fd_close(fd) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + FS.close(stream); + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return e.errno; + } + } + + /** @param {number=} offset */ + var doReadv = (stream, iov, iovcnt, offset) => { + var ret = 0; + for (var i = 0; i < iovcnt; i++) { + var ptr = HEAPU32[((iov)>>2)]; + var len = HEAPU32[(((iov)+(4))>>2)]; + iov += 8; + var curr = FS.read(stream, HEAP8,ptr, len, offset); + if (curr < 0) return -1; + ret += curr; + if (curr < len) break; // nothing more to read + if (typeof offset !== 'undefined') { + offset += curr; + } + } + return ret; + }; + + function _fd_read(fd, iov, iovcnt, pnum) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + var num = doReadv(stream, iov, iovcnt); + HEAPU32[((pnum)>>2)] = num; + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return e.errno; + } + } + + + function _fd_seek(fd,offset_low, offset_high,whence,newOffset) { + var offset = convertI32PairToI53Checked(offset_low, offset_high);; + + + try { + + if (isNaN(offset)) return 61; + var stream = SYSCALLS.getStreamFromFD(fd); + FS.llseek(stream, offset, whence); + (tempI64 = [stream.position>>>0,(tempDouble=stream.position,(+(Math.abs(tempDouble))) >= 1.0 ? (tempDouble > 0.0 ? (+(Math.floor((tempDouble)/4294967296.0)))>>>0 : (~~((+(Math.ceil((tempDouble - +(((~~(tempDouble)))>>>0))/4294967296.0)))))>>>0) : 0)], HEAP32[((newOffset)>>2)] = tempI64[0],HEAP32[(((newOffset)+(4))>>2)] = tempI64[1]); + if (stream.getdents && offset === 0 && whence === 0) stream.getdents = null; // reset readdir state + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return e.errno; + } + ; + } + + function _fd_sync(fd) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + if (stream.stream_ops && stream.stream_ops.fsync) { + return stream.stream_ops.fsync(stream); + } + return 0; // we can't do anything synchronously; the in-memory FS is already synced to + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return e.errno; + } + } + + /** @param {number=} offset */ + var doWritev = (stream, iov, iovcnt, offset) => { + var ret = 0; + for (var i = 0; i < iovcnt; i++) { + var ptr = HEAPU32[((iov)>>2)]; + var len = HEAPU32[(((iov)+(4))>>2)]; + iov += 8; + var curr = FS.write(stream, HEAP8,ptr, len, offset); + if (curr < 0) return -1; + ret += curr; + if (typeof offset !== 'undefined') { + offset += curr; + } + } + return ret; + }; + + function _fd_write(fd, iov, iovcnt, pnum) { + try { + + var stream = SYSCALLS.getStreamFromFD(fd); + var num = doWritev(stream, iov, iovcnt); + HEAPU32[((pnum)>>2)] = num; + return 0; + } catch (e) { + if (typeof FS == 'undefined' || !(e.name === 'ErrnoError')) throw e; + return e.errno; + } + } + + + + + + + + + + + var _getaddrinfo = (node, service, hint, out) => { + // Note getaddrinfo currently only returns a single addrinfo with ai_next defaulting to NULL. When NULL + // hints are specified or ai_family set to AF_UNSPEC or ai_socktype or ai_protocol set to 0 then we + // really should provide a linked list of suitable addrinfo values. + var addrs = []; + var canon = null; + var addr = 0; + var port = 0; + var flags = 0; + var family = 0; + var type = 0; + var proto = 0; + var ai, last; + + function allocaddrinfo(family, type, proto, canon, addr, port) { + var sa, salen, ai; + var errno; + + salen = family === 10 ? + 28 : + 16; + addr = family === 10 ? + inetNtop6(addr) : + inetNtop4(addr); + sa = _malloc(salen); + errno = writeSockaddr(sa, family, addr, port); + assert(!errno); + + ai = _malloc(32); + HEAP32[(((ai)+(4))>>2)] = family; + HEAP32[(((ai)+(8))>>2)] = type; + HEAP32[(((ai)+(12))>>2)] = proto; + HEAPU32[(((ai)+(24))>>2)] = canon; + HEAPU32[(((ai)+(20))>>2)] = sa; + if (family === 10) { + HEAP32[(((ai)+(16))>>2)] = 28; + } else { + HEAP32[(((ai)+(16))>>2)] = 16; + } + HEAP32[(((ai)+(28))>>2)] = 0; + + return ai; + } + + if (hint) { + flags = HEAP32[((hint)>>2)]; + family = HEAP32[(((hint)+(4))>>2)]; + type = HEAP32[(((hint)+(8))>>2)]; + proto = HEAP32[(((hint)+(12))>>2)]; + } + if (type && !proto) { + proto = type === 2 ? 17 : 6; + } + if (!type && proto) { + type = proto === 17 ? 2 : 1; + } + + // If type or proto are set to zero in hints we should really be returning multiple addrinfo values, but for + // now default to a TCP STREAM socket so we can at least return a sensible addrinfo given NULL hints. + if (proto === 0) { + proto = 6; + } + if (type === 0) { + type = 1; + } + + if (!node && !service) { + return -2; + } + if (flags & ~(1|2|4| + 1024|8|16|32)) { + return -1; + } + if (hint !== 0 && (HEAP32[((hint)>>2)] & 2) && !node) { + return -1; + } + if (flags & 32) { + // TODO + return -2; + } + if (type !== 0 && type !== 1 && type !== 2) { + return -7; + } + if (family !== 0 && family !== 2 && family !== 10) { + return -6; + } + + if (service) { + service = UTF8ToString(service); + port = parseInt(service, 10); + + if (isNaN(port)) { + if (flags & 1024) { + return -2; + } + // TODO support resolving well-known service names from: + // http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt + return -8; + } + } + + if (!node) { + if (family === 0) { + family = 2; + } + if ((flags & 1) === 0) { + if (family === 2) { + addr = _htonl(2130706433); + } else { + addr = [0, 0, 0, 1]; + } + } + ai = allocaddrinfo(family, type, proto, null, addr, port); + HEAPU32[((out)>>2)] = ai; + return 0; + } + + // + // try as a numeric address + // + node = UTF8ToString(node); + addr = inetPton4(node); + if (addr !== null) { + // incoming node is a valid ipv4 address + if (family === 0 || family === 2) { + family = 2; + } + else if (family === 10 && (flags & 8)) { + addr = [0, 0, _htonl(0xffff), addr]; + family = 10; + } else { + return -2; + } + } else { + addr = inetPton6(node); + if (addr !== null) { + // incoming node is a valid ipv6 address + if (family === 0 || family === 10) { + family = 10; + } else { + return -2; + } + } + } + if (addr != null) { + ai = allocaddrinfo(family, type, proto, node, addr, port); + HEAPU32[((out)>>2)] = ai; + return 0; + } + if (flags & 4) { + return -2; + } + + // + // try as a hostname + // + // resolve the hostname to a temporary fake address + node = DNS.lookup_name(node); + addr = inetPton4(node); + if (family === 0) { + family = 2; + } else if (family === 10) { + addr = [0, 0, _htonl(0xffff), addr]; + } + ai = allocaddrinfo(family, type, proto, null, addr, port); + HEAPU32[((out)>>2)] = ai; + return 0; + }; + + + + + + var getHostByName = (name) => { + // generate hostent + var ret = _malloc(20); // XXX possibly leaked, as are others here + var nameBuf = stringToNewUTF8(name); + HEAPU32[((ret)>>2)] = nameBuf; + var aliasesBuf = _malloc(4); + HEAPU32[((aliasesBuf)>>2)] = 0; + HEAPU32[(((ret)+(4))>>2)] = aliasesBuf; + var afinet = 2; + HEAP32[(((ret)+(8))>>2)] = afinet; + HEAP32[(((ret)+(12))>>2)] = 4; + var addrListBuf = _malloc(12); + HEAPU32[((addrListBuf)>>2)] = addrListBuf+8; + HEAPU32[(((addrListBuf)+(4))>>2)] = 0; + HEAP32[(((addrListBuf)+(8))>>2)] = inetPton4(DNS.lookup_name(name)); + HEAPU32[(((ret)+(16))>>2)] = addrListBuf; + return ret; + }; + + + var _gethostbyaddr = (addr, addrlen, type) => { + if (type !== 2) { + setErrNo(5); + // TODO: set h_errno + return null; + } + addr = HEAP32[((addr)>>2)]; // addr is in_addr + var host = inetNtop4(addr); + var lookup = DNS.lookup_addr(host); + if (lookup) { + host = lookup; + } + return getHostByName(host); + }; + + + var _gethostbyname = (name) => { + return getHostByName(UTF8ToString(name)); + }; + + + + + var _getnameinfo = (sa, salen, node, nodelen, serv, servlen, flags) => { + var info = readSockaddr(sa, salen); + if (info.errno) { + return -6; + } + var port = info.port; + var addr = info.addr; + + var overflowed = false; + + if (node && nodelen) { + var lookup; + if ((flags & 1) || !(lookup = DNS.lookup_addr(addr))) { + if (flags & 8) { + return -2; + } + } else { + addr = lookup; + } + var numBytesWrittenExclNull = stringToUTF8(addr, node, nodelen); + + if (numBytesWrittenExclNull+1 >= nodelen) { + overflowed = true; + } + } + + if (serv && servlen) { + port = '' + port; + var numBytesWrittenExclNull = stringToUTF8(port, serv, servlen); + + if (numBytesWrittenExclNull+1 >= servlen) { + overflowed = true; + } + } + + if (overflowed) { + // Note: even when we overflow, getnameinfo() is specced to write out the truncated results. + return -12; + } + + return 0; + }; + + /** @type {function(...*):?} */ + function _getpass( + ) { + err('missing function: getpass'); abort(-1); + } + + + + var arraySum = (array, index) => { + var sum = 0; + for (var i = 0; i <= index; sum += array[i++]) { + // no-op + } + return sum; + }; + + + var MONTH_DAYS_LEAP = [31,29,31,30,31,30,31,31,30,31,30,31]; + + var MONTH_DAYS_REGULAR = [31,28,31,30,31,30,31,31,30,31,30,31]; + var addDays = (date, days) => { + var newDate = new Date(date.getTime()); + while (days > 0) { + var leap = isLeapYear(newDate.getFullYear()); + var currentMonth = newDate.getMonth(); + var daysInCurrentMonth = (leap ? MONTH_DAYS_LEAP : MONTH_DAYS_REGULAR)[currentMonth]; + + if (days > daysInCurrentMonth-newDate.getDate()) { + // we spill over to next month + days -= (daysInCurrentMonth-newDate.getDate()+1); + newDate.setDate(1); + if (currentMonth < 11) { + newDate.setMonth(currentMonth+1) + } else { + newDate.setMonth(0); + newDate.setFullYear(newDate.getFullYear()+1); + } + } else { + // we stay in current month + newDate.setDate(newDate.getDate()+days); + return newDate; + } + } + + return newDate; + }; + + + + + var writeArrayToMemory = (array, buffer) => { + assert(array.length >= 0, 'writeArrayToMemory array must have a length (should be an array or typed array)') + HEAP8.set(array, buffer); + }; + + var _strftime = (s, maxsize, format, tm) => { + // size_t strftime(char *restrict s, size_t maxsize, const char *restrict format, const struct tm *restrict timeptr); + // http://pubs.opengroup.org/onlinepubs/009695399/functions/strftime.html + + var tm_zone = HEAP32[(((tm)+(40))>>2)]; + + var date = { + tm_sec: HEAP32[((tm)>>2)], + tm_min: HEAP32[(((tm)+(4))>>2)], + tm_hour: HEAP32[(((tm)+(8))>>2)], + tm_mday: HEAP32[(((tm)+(12))>>2)], + tm_mon: HEAP32[(((tm)+(16))>>2)], + tm_year: HEAP32[(((tm)+(20))>>2)], + tm_wday: HEAP32[(((tm)+(24))>>2)], + tm_yday: HEAP32[(((tm)+(28))>>2)], + tm_isdst: HEAP32[(((tm)+(32))>>2)], + tm_gmtoff: HEAP32[(((tm)+(36))>>2)], + tm_zone: tm_zone ? UTF8ToString(tm_zone) : '' + }; + + var pattern = UTF8ToString(format); + + // expand format + var EXPANSION_RULES_1 = { + '%c': '%a %b %d %H:%M:%S %Y', // Replaced by the locale's appropriate date and time representation - e.g., Mon Aug 3 14:02:01 2013 + '%D': '%m/%d/%y', // Equivalent to %m / %d / %y + '%F': '%Y-%m-%d', // Equivalent to %Y - %m - %d + '%h': '%b', // Equivalent to %b + '%r': '%I:%M:%S %p', // Replaced by the time in a.m. and p.m. notation + '%R': '%H:%M', // Replaced by the time in 24-hour notation + '%T': '%H:%M:%S', // Replaced by the time + '%x': '%m/%d/%y', // Replaced by the locale's appropriate date representation + '%X': '%H:%M:%S', // Replaced by the locale's appropriate time representation + // Modified Conversion Specifiers + '%Ec': '%c', // Replaced by the locale's alternative appropriate date and time representation. + '%EC': '%C', // Replaced by the name of the base year (period) in the locale's alternative representation. + '%Ex': '%m/%d/%y', // Replaced by the locale's alternative date representation. + '%EX': '%H:%M:%S', // Replaced by the locale's alternative time representation. + '%Ey': '%y', // Replaced by the offset from %EC (year only) in the locale's alternative representation. + '%EY': '%Y', // Replaced by the full alternative year representation. + '%Od': '%d', // Replaced by the day of the month, using the locale's alternative numeric symbols, filled as needed with leading zeros if there is any alternative symbol for zero; otherwise, with leading characters. + '%Oe': '%e', // Replaced by the day of the month, using the locale's alternative numeric symbols, filled as needed with leading characters. + '%OH': '%H', // Replaced by the hour (24-hour clock) using the locale's alternative numeric symbols. + '%OI': '%I', // Replaced by the hour (12-hour clock) using the locale's alternative numeric symbols. + '%Om': '%m', // Replaced by the month using the locale's alternative numeric symbols. + '%OM': '%M', // Replaced by the minutes using the locale's alternative numeric symbols. + '%OS': '%S', // Replaced by the seconds using the locale's alternative numeric symbols. + '%Ou': '%u', // Replaced by the weekday as a number in the locale's alternative representation (Monday=1). + '%OU': '%U', // Replaced by the week number of the year (Sunday as the first day of the week, rules corresponding to %U ) using the locale's alternative numeric symbols. + '%OV': '%V', // Replaced by the week number of the year (Monday as the first day of the week, rules corresponding to %V ) using the locale's alternative numeric symbols. + '%Ow': '%w', // Replaced by the number of the weekday (Sunday=0) using the locale's alternative numeric symbols. + '%OW': '%W', // Replaced by the week number of the year (Monday as the first day of the week) using the locale's alternative numeric symbols. + '%Oy': '%y', // Replaced by the year (offset from %C ) using the locale's alternative numeric symbols. + }; + for (var rule in EXPANSION_RULES_1) { + pattern = pattern.replace(new RegExp(rule, 'g'), EXPANSION_RULES_1[rule]); + } + + var WEEKDAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday']; + var MONTHS = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']; + + function leadingSomething(value, digits, character) { + var str = typeof value == 'number' ? value.toString() : (value || ''); + while (str.length < digits) { + str = character[0]+str; + } + return str; + } + + function leadingNulls(value, digits) { + return leadingSomething(value, digits, '0'); + } + + function compareByDay(date1, date2) { + function sgn(value) { + return value < 0 ? -1 : (value > 0 ? 1 : 0); + } + + var compare; + if ((compare = sgn(date1.getFullYear()-date2.getFullYear())) === 0) { + if ((compare = sgn(date1.getMonth()-date2.getMonth())) === 0) { + compare = sgn(date1.getDate()-date2.getDate()); + } + } + return compare; + } + + function getFirstWeekStartDate(janFourth) { + switch (janFourth.getDay()) { + case 0: // Sunday + return new Date(janFourth.getFullYear()-1, 11, 29); + case 1: // Monday + return janFourth; + case 2: // Tuesday + return new Date(janFourth.getFullYear(), 0, 3); + case 3: // Wednesday + return new Date(janFourth.getFullYear(), 0, 2); + case 4: // Thursday + return new Date(janFourth.getFullYear(), 0, 1); + case 5: // Friday + return new Date(janFourth.getFullYear()-1, 11, 31); + case 6: // Saturday + return new Date(janFourth.getFullYear()-1, 11, 30); + } + } + + function getWeekBasedYear(date) { + var thisDate = addDays(new Date(date.tm_year+1900, 0, 1), date.tm_yday); + + var janFourthThisYear = new Date(thisDate.getFullYear(), 0, 4); + var janFourthNextYear = new Date(thisDate.getFullYear()+1, 0, 4); + + var firstWeekStartThisYear = getFirstWeekStartDate(janFourthThisYear); + var firstWeekStartNextYear = getFirstWeekStartDate(janFourthNextYear); + + if (compareByDay(firstWeekStartThisYear, thisDate) <= 0) { + // this date is after the start of the first week of this year + if (compareByDay(firstWeekStartNextYear, thisDate) <= 0) { + return thisDate.getFullYear()+1; + } + return thisDate.getFullYear(); + } + return thisDate.getFullYear()-1; + } + + var EXPANSION_RULES_2 = { + '%a': (date) => WEEKDAYS[date.tm_wday].substring(0,3) , + '%A': (date) => WEEKDAYS[date.tm_wday], + '%b': (date) => MONTHS[date.tm_mon].substring(0,3), + '%B': (date) => MONTHS[date.tm_mon], + '%C': (date) => { + var year = date.tm_year+1900; + return leadingNulls((year/100)|0,2); + }, + '%d': (date) => leadingNulls(date.tm_mday, 2), + '%e': (date) => leadingSomething(date.tm_mday, 2, ' '), + '%g': (date) => { + // %g, %G, and %V give values according to the ISO 8601:2000 standard week-based year. + // In this system, weeks begin on a Monday and week 1 of the year is the week that includes + // January 4th, which is also the week that includes the first Thursday of the year, and + // is also the first week that contains at least four days in the year. + // If the first Monday of January is the 2nd, 3rd, or 4th, the preceding days are part of + // the last week of the preceding year; thus, for Saturday 2nd January 1999, + // %G is replaced by 1998 and %V is replaced by 53. If December 29th, 30th, + // or 31st is a Monday, it and any following days are part of week 1 of the following year. + // Thus, for Tuesday 30th December 1997, %G is replaced by 1998 and %V is replaced by 01. + + return getWeekBasedYear(date).toString().substring(2); + }, + '%G': (date) => getWeekBasedYear(date), + '%H': (date) => leadingNulls(date.tm_hour, 2), + '%I': (date) => { + var twelveHour = date.tm_hour; + if (twelveHour == 0) twelveHour = 12; + else if (twelveHour > 12) twelveHour -= 12; + return leadingNulls(twelveHour, 2); + }, + '%j': (date) => { + // Day of the year (001-366) + return leadingNulls(date.tm_mday + arraySum(isLeapYear(date.tm_year+1900) ? MONTH_DAYS_LEAP : MONTH_DAYS_REGULAR, date.tm_mon-1), 3); + }, + '%m': (date) => leadingNulls(date.tm_mon+1, 2), + '%M': (date) => leadingNulls(date.tm_min, 2), + '%n': () => '\n', + '%p': (date) => { + if (date.tm_hour >= 0 && date.tm_hour < 12) { + return 'AM'; + } + return 'PM'; + }, + '%S': (date) => leadingNulls(date.tm_sec, 2), + '%t': () => '\t', + '%u': (date) => date.tm_wday || 7, + '%U': (date) => { + var days = date.tm_yday + 7 - date.tm_wday; + return leadingNulls(Math.floor(days / 7), 2); + }, + '%V': (date) => { + // Replaced by the week number of the year (Monday as the first day of the week) + // as a decimal number [01,53]. If the week containing 1 January has four + // or more days in the new year, then it is considered week 1. + // Otherwise, it is the last week of the previous year, and the next week is week 1. + // Both January 4th and the first Thursday of January are always in week 1. [ tm_year, tm_wday, tm_yday] + var val = Math.floor((date.tm_yday + 7 - (date.tm_wday + 6) % 7 ) / 7); + // If 1 Jan is just 1-3 days past Monday, the previous week + // is also in this year. + if ((date.tm_wday + 371 - date.tm_yday - 2) % 7 <= 2) { + val++; + } + if (!val) { + val = 52; + // If 31 December of prev year a Thursday, or Friday of a + // leap year, then the prev year has 53 weeks. + var dec31 = (date.tm_wday + 7 - date.tm_yday - 1) % 7; + if (dec31 == 4 || (dec31 == 5 && isLeapYear(date.tm_year%400-1))) { + val++; + } + } else if (val == 53) { + // If 1 January is not a Thursday, and not a Wednesday of a + // leap year, then this year has only 52 weeks. + var jan1 = (date.tm_wday + 371 - date.tm_yday) % 7; + if (jan1 != 4 && (jan1 != 3 || !isLeapYear(date.tm_year))) + val = 1; + } + return leadingNulls(val, 2); + }, + '%w': (date) => date.tm_wday, + '%W': (date) => { + var days = date.tm_yday + 7 - ((date.tm_wday + 6) % 7); + return leadingNulls(Math.floor(days / 7), 2); + }, + '%y': (date) => { + // Replaced by the last two digits of the year as a decimal number [00,99]. [ tm_year] + return (date.tm_year+1900).toString().substring(2); + }, + // Replaced by the year as a decimal number (for example, 1997). [ tm_year] + '%Y': (date) => date.tm_year+1900, + '%z': (date) => { + // Replaced by the offset from UTC in the ISO 8601:2000 standard format ( +hhmm or -hhmm ). + // For example, "-0430" means 4 hours 30 minutes behind UTC (west of Greenwich). + var off = date.tm_gmtoff; + var ahead = off >= 0; + off = Math.abs(off) / 60; + // convert from minutes into hhmm format (which means 60 minutes = 100 units) + off = (off / 60)*100 + (off % 60); + return (ahead ? '+' : '-') + String("0000" + off).slice(-4); + }, + '%Z': (date) => date.tm_zone, + '%%': () => '%' + }; + + // Replace %% with a pair of NULLs (which cannot occur in a C string), then + // re-inject them after processing. + pattern = pattern.replace(/%%/g, '\0\0') + for (var rule in EXPANSION_RULES_2) { + if (pattern.includes(rule)) { + pattern = pattern.replace(new RegExp(rule, 'g'), EXPANSION_RULES_2[rule](date)); + } + } + pattern = pattern.replace(/\0\0/g, '%') + + var bytes = intArrayFromString(pattern, false); + if (bytes.length > maxsize) { + return 0; + } + + writeArrayToMemory(bytes, s); + return bytes.length-1; + }; + + + var _system = (command) => { + if (ENVIRONMENT_IS_NODE) { + if (!command) return 1; // shell is available + + var cmdstr = UTF8ToString(command); + if (!cmdstr.length) return 0; // this is what glibc seems to do (shell works test?) + + var cp = require('child_process'); + var ret = cp.spawnSync(cmdstr, [], {shell:true, stdio:'inherit'}); + + var _W_EXITCODE = (ret, sig) => ((ret) << 8 | (sig)); + + // this really only can happen if process is killed by signal + if (ret.status === null) { + // sadly node doesn't expose such function + var signalToNumber = (sig) => { + // implement only the most common ones, and fallback to SIGINT + switch (sig) { + case 'SIGHUP': return 1; + case 'SIGINT': return 2; + case 'SIGQUIT': return 3; + case 'SIGFPE': return 8; + case 'SIGKILL': return 9; + case 'SIGALRM': return 14; + case 'SIGTERM': return 15; + } + return 2; // SIGINT + } + return _W_EXITCODE(0, signalToNumber(ret.signal)); + } + + return _W_EXITCODE(ret.status, 0); + } + // int system(const char *command); + // http://pubs.opengroup.org/onlinepubs/000095399/functions/system.html + // Can't call external programs. + if (!command) return 0; // no shell available + setErrNo(52); + return -1; + }; + + + var handleException = (e) => { + // Certain exception types we do not treat as errors since they are used for + // internal control flow. + // 1. ExitStatus, which is thrown by exit() + // 2. "unwind", which is thrown by emscripten_unwind_to_js_event_loop() and others + // that wish to return to JS event loop. + if (e instanceof ExitStatus || e == 'unwind') { + return EXITSTATUS; + } + checkStackCookie(); + if (e instanceof WebAssembly.RuntimeError) { + if (_emscripten_stack_get_current() <= 0) { + err('Stack overflow detected. You can try increasing -sSTACK_SIZE (currently set to 65536)'); + } + } + quit_(1, e); + }; + + + var stringToUTF8OnStack = (str) => { + var size = lengthBytesUTF8(str) + 1; + var ret = stackAlloc(size); + stringToUTF8(str, ret, size); + return ret; + }; + + + + + + + + + + var FSNode = /** @constructor */ function(parent, name, mode, rdev) { + if (!parent) { + parent = this; // root node sets parent to itself + } + this.parent = parent; + this.mount = parent.mount; + this.mounted = null; + this.id = FS.nextInode++; + this.name = name; + this.mode = mode; + this.node_ops = {}; + this.stream_ops = {}; + this.rdev = rdev; + }; + var readMode = 292/*292*/ | 73/*73*/; + var writeMode = 146/*146*/; + Object.defineProperties(FSNode.prototype, { + read: { + get: /** @this{FSNode} */function() { + return (this.mode & readMode) === readMode; + }, + set: /** @this{FSNode} */function(val) { + val ? this.mode |= readMode : this.mode &= ~readMode; + } + }, + write: { + get: /** @this{FSNode} */function() { + return (this.mode & writeMode) === writeMode; + }, + set: /** @this{FSNode} */function(val) { + val ? this.mode |= writeMode : this.mode &= ~writeMode; + } + }, + isFolder: { + get: /** @this{FSNode} */function() { + return FS.isDir(this.mode); + } + }, + isDevice: { + get: /** @this{FSNode} */function() { + return FS.isChrdev(this.mode); + } + } + }); + FS.FSNode = FSNode; + FS.createPreloadedFile = FS_createPreloadedFile; + FS.staticInit();Module["FS_createPath"] = FS.createPath;Module["FS_createDataFile"] = FS.createDataFile;Module["FS_createPreloadedFile"] = FS.createPreloadedFile;Module["FS_unlink"] = FS.unlink;Module["FS_createLazyFile"] = FS.createLazyFile;Module["FS_createDevice"] = FS.createDevice;; +ERRNO_CODES = { + 'EPERM': 63, + 'ENOENT': 44, + 'ESRCH': 71, + 'EINTR': 27, + 'EIO': 29, + 'ENXIO': 60, + 'E2BIG': 1, + 'ENOEXEC': 45, + 'EBADF': 8, + 'ECHILD': 12, + 'EAGAIN': 6, + 'EWOULDBLOCK': 6, + 'ENOMEM': 48, + 'EACCES': 2, + 'EFAULT': 21, + 'ENOTBLK': 105, + 'EBUSY': 10, + 'EEXIST': 20, + 'EXDEV': 75, + 'ENODEV': 43, + 'ENOTDIR': 54, + 'EISDIR': 31, + 'EINVAL': 28, + 'ENFILE': 41, + 'EMFILE': 33, + 'ENOTTY': 59, + 'ETXTBSY': 74, + 'EFBIG': 22, + 'ENOSPC': 51, + 'ESPIPE': 70, + 'EROFS': 69, + 'EMLINK': 34, + 'EPIPE': 64, + 'EDOM': 18, + 'ERANGE': 68, + 'ENOMSG': 49, + 'EIDRM': 24, + 'ECHRNG': 106, + 'EL2NSYNC': 156, + 'EL3HLT': 107, + 'EL3RST': 108, + 'ELNRNG': 109, + 'EUNATCH': 110, + 'ENOCSI': 111, + 'EL2HLT': 112, + 'EDEADLK': 16, + 'ENOLCK': 46, + 'EBADE': 113, + 'EBADR': 114, + 'EXFULL': 115, + 'ENOANO': 104, + 'EBADRQC': 103, + 'EBADSLT': 102, + 'EDEADLOCK': 16, + 'EBFONT': 101, + 'ENOSTR': 100, + 'ENODATA': 116, + 'ETIME': 117, + 'ENOSR': 118, + 'ENONET': 119, + 'ENOPKG': 120, + 'EREMOTE': 121, + 'ENOLINK': 47, + 'EADV': 122, + 'ESRMNT': 123, + 'ECOMM': 124, + 'EPROTO': 65, + 'EMULTIHOP': 36, + 'EDOTDOT': 125, + 'EBADMSG': 9, + 'ENOTUNIQ': 126, + 'EBADFD': 127, + 'EREMCHG': 128, + 'ELIBACC': 129, + 'ELIBBAD': 130, + 'ELIBSCN': 131, + 'ELIBMAX': 132, + 'ELIBEXEC': 133, + 'ENOSYS': 52, + 'ENOTEMPTY': 55, + 'ENAMETOOLONG': 37, + 'ELOOP': 32, + 'EOPNOTSUPP': 138, + 'EPFNOSUPPORT': 139, + 'ECONNRESET': 15, + 'ENOBUFS': 42, + 'EAFNOSUPPORT': 5, + 'EPROTOTYPE': 67, + 'ENOTSOCK': 57, + 'ENOPROTOOPT': 50, + 'ESHUTDOWN': 140, + 'ECONNREFUSED': 14, + 'EADDRINUSE': 3, + 'ECONNABORTED': 13, + 'ENETUNREACH': 40, + 'ENETDOWN': 38, + 'ETIMEDOUT': 73, + 'EHOSTDOWN': 142, + 'EHOSTUNREACH': 23, + 'EINPROGRESS': 26, + 'EALREADY': 7, + 'EDESTADDRREQ': 17, + 'EMSGSIZE': 35, + 'EPROTONOSUPPORT': 66, + 'ESOCKTNOSUPPORT': 137, + 'EADDRNOTAVAIL': 4, + 'ENETRESET': 39, + 'EISCONN': 30, + 'ENOTCONN': 53, + 'ETOOMANYREFS': 141, + 'EUSERS': 136, + 'EDQUOT': 19, + 'ESTALE': 72, + 'ENOTSUP': 138, + 'ENOMEDIUM': 148, + 'EILSEQ': 25, + 'EOVERFLOW': 61, + 'ECANCELED': 11, + 'ENOTRECOVERABLE': 56, + 'EOWNERDEAD': 62, + 'ESTRPIPE': 135, + };; +function checkIncomingModuleAPI() { + ignoredModuleProp('fetchSettings'); +} +var wasmImports = { + __assert_fail: ___assert_fail, + __call_sighandler: ___call_sighandler, + __cxa_throw: ___cxa_throw, + __syscall__newselect: ___syscall__newselect, + __syscall_accept4: ___syscall_accept4, + __syscall_bind: ___syscall_bind, + __syscall_chdir: ___syscall_chdir, + __syscall_chmod: ___syscall_chmod, + __syscall_connect: ___syscall_connect, + __syscall_dup: ___syscall_dup, + __syscall_dup3: ___syscall_dup3, + __syscall_faccessat: ___syscall_faccessat, + __syscall_fcntl64: ___syscall_fcntl64, + __syscall_fstat64: ___syscall_fstat64, + __syscall_fstatfs64: ___syscall_fstatfs64, + __syscall_getcwd: ___syscall_getcwd, + __syscall_getdents64: ___syscall_getdents64, + __syscall_getpeername: ___syscall_getpeername, + __syscall_getsockname: ___syscall_getsockname, + __syscall_getsockopt: ___syscall_getsockopt, + __syscall_ioctl: ___syscall_ioctl, + __syscall_listen: ___syscall_listen, + __syscall_lstat64: ___syscall_lstat64, + __syscall_mkdirat: ___syscall_mkdirat, + __syscall_newfstatat: ___syscall_newfstatat, + __syscall_openat: ___syscall_openat, + __syscall_pipe: ___syscall_pipe, + __syscall_readlinkat: ___syscall_readlinkat, + __syscall_recvfrom: ___syscall_recvfrom, + __syscall_renameat: ___syscall_renameat, + __syscall_rmdir: ___syscall_rmdir, + __syscall_sendto: ___syscall_sendto, + __syscall_socket: ___syscall_socket, + __syscall_stat64: ___syscall_stat64, + __syscall_symlink: ___syscall_symlink, + __syscall_unlinkat: ___syscall_unlinkat, + __syscall_utimensat: ___syscall_utimensat, + _emscripten_get_now_is_monotonic: __emscripten_get_now_is_monotonic, + _emscripten_throw_longjmp: __emscripten_throw_longjmp, + _gmtime_js: __gmtime_js, + _localtime_js: __localtime_js, + _mktime_js: __mktime_js, + _mmap_js: __mmap_js, + _munmap_js: __munmap_js, + _tzset_js: __tzset_js, + abort: _abort, + emscripten_console_error: _emscripten_console_error, + emscripten_date_now: _emscripten_date_now, + emscripten_get_heap_max: _emscripten_get_heap_max, + emscripten_get_now: _emscripten_get_now, + emscripten_memcpy_big: _emscripten_memcpy_big, + emscripten_resize_heap: _emscripten_resize_heap, + environ_get: _environ_get, + environ_sizes_get: _environ_sizes_get, + exit: _exit, + fd_close: _fd_close, + fd_read: _fd_read, + fd_seek: _fd_seek, + fd_sync: _fd_sync, + fd_write: _fd_write, + getaddrinfo: _getaddrinfo, + gethostbyaddr: _gethostbyaddr, + gethostbyname: _gethostbyname, + getnameinfo: _getnameinfo, + getpass: _getpass, + invoke_di: invoke_di, + invoke_i: invoke_i, + invoke_id: invoke_id, + invoke_ii: invoke_ii, + invoke_iii: invoke_iii, + invoke_iiid: invoke_iiid, + invoke_iiii: invoke_iiii, + invoke_iiiii: invoke_iiiii, + invoke_iiiiiii: invoke_iiiiiii, + invoke_v: invoke_v, + invoke_vi: invoke_vi, + invoke_vid: invoke_vid, + invoke_vidd: invoke_vidd, + invoke_vii: invoke_vii, + invoke_viii: invoke_viii, + invoke_viiii: invoke_viiii, + proc_exit: _proc_exit, + strftime: _strftime, + system: _system +}; +var asm = createWasm(); +/** @type {function(...*):?} */ +var ___wasm_call_ctors = createExportWrapper("__wasm_call_ctors"); +/** @type {function(...*):?} */ +var _flush_streams = Module['_flush_streams'] = createExportWrapper("flush_streams"); +/** @type {function(...*):?} */ +var _fflush = Module['_fflush'] = createExportWrapper("fflush"); +/** @type {function(...*):?} */ +var _main = Module['_main'] = createExportWrapper("__main_argc_argv"); +/** @type {function(...*):?} */ +var _free = createExportWrapper("free"); +/** @type {function(...*):?} */ +var ___errno_location = createExportWrapper("__errno_location"); +/** @type {function(...*):?} */ +var _malloc = createExportWrapper("malloc"); +/** @type {function(...*):?} */ +var setTempRet0 = createExportWrapper("setTempRet0"); +/** @type {function(...*):?} */ +var _htonl = createExportWrapper("htonl"); +/** @type {function(...*):?} */ +var _htons = createExportWrapper("htons"); +/** @type {function(...*):?} */ +var _emscripten_builtin_memalign = createExportWrapper("emscripten_builtin_memalign"); +/** @type {function(...*):?} */ +var _ntohs = createExportWrapper("ntohs"); +/** @type {function(...*):?} */ +var _setThrew = createExportWrapper("setThrew"); +/** @type {function(...*):?} */ +var _emscripten_stack_init = function() { + return (_emscripten_stack_init = Module['asm']['emscripten_stack_init']).apply(null, arguments); +}; + +/** @type {function(...*):?} */ +var _emscripten_stack_get_free = function() { + return (_emscripten_stack_get_free = Module['asm']['emscripten_stack_get_free']).apply(null, arguments); +}; + +/** @type {function(...*):?} */ +var _emscripten_stack_get_base = function() { + return (_emscripten_stack_get_base = Module['asm']['emscripten_stack_get_base']).apply(null, arguments); +}; + +/** @type {function(...*):?} */ +var _emscripten_stack_get_end = function() { + return (_emscripten_stack_get_end = Module['asm']['emscripten_stack_get_end']).apply(null, arguments); +}; + +/** @type {function(...*):?} */ +var stackSave = createExportWrapper("stackSave"); +/** @type {function(...*):?} */ +var stackRestore = createExportWrapper("stackRestore"); +/** @type {function(...*):?} */ +var stackAlloc = createExportWrapper("stackAlloc"); +/** @type {function(...*):?} */ +var _emscripten_stack_get_current = function() { + return (_emscripten_stack_get_current = Module['asm']['emscripten_stack_get_current']).apply(null, arguments); +}; + +/** @type {function(...*):?} */ +var ___cxa_is_pointer_type = createExportWrapper("__cxa_is_pointer_type"); +/** @type {function(...*):?} */ +var dynCall_jiji = Module['dynCall_jiji'] = createExportWrapper("dynCall_jiji"); +/** @type {function(...*):?} */ +var dynCall_ji = Module['dynCall_ji'] = createExportWrapper("dynCall_ji"); +/** @type {function(...*):?} */ +var dynCall_iiji = Module['dynCall_iiji'] = createExportWrapper("dynCall_iiji"); +/** @type {function(...*):?} */ +var dynCall_iijjiii = Module['dynCall_iijjiii'] = createExportWrapper("dynCall_iijjiii"); +/** @type {function(...*):?} */ +var dynCall_vijjjii = Module['dynCall_vijjjii'] = createExportWrapper("dynCall_vijjjii"); +/** @type {function(...*):?} */ +var dynCall_viji = Module['dynCall_viji'] = createExportWrapper("dynCall_viji"); +/** @type {function(...*):?} */ +var dynCall_iijiji = Module['dynCall_iijiji'] = createExportWrapper("dynCall_iijiji"); + +function invoke_iii(index,a1,a2) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1,a2); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_iiiii(index,a1,a2,a3,a4) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1,a2,a3,a4); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_vii(index,a1,a2) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1,a2); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_ii(index,a1) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_iiii(index,a1,a2,a3) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1,a2,a3); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_id(index,a1) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_i(index) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_viii(index,a1,a2,a3) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1,a2,a3); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_vi(index,a1) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_iiiiiii(index,a1,a2,a3,a4,a5,a6) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1,a2,a3,a4,a5,a6); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_viiii(index,a1,a2,a3,a4) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1,a2,a3,a4); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_vidd(index,a1,a2,a3) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1,a2,a3); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_v(index) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_di(index,a1) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_vid(index,a1,a2) { + var sp = stackSave(); + try { + getWasmTableEntry(index)(a1,a2); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + +function invoke_iiid(index,a1,a2,a3) { + var sp = stackSave(); + try { + return getWasmTableEntry(index)(a1,a2,a3); + } catch(e) { + stackRestore(sp); + if (e !== e+0) throw e; + _setThrew(1, 0); + } +} + + +// include: postamble.js +// === Auto-generated postamble setup entry stuff === + +// include: base64Utils.js +// Converts a string of base64 into a byte array. +// Throws error on invalid input. +function intArrayFromBase64(s) { + if (typeof ENVIRONMENT_IS_NODE != 'undefined' && ENVIRONMENT_IS_NODE) { + var buf = Buffer.from(s, 'base64'); + return new Uint8Array(buf['buffer'], buf['byteOffset'], buf['byteLength']); + } + + try { + var decoded = atob(s); + var bytes = new Uint8Array(decoded.length); + for (var i = 0 ; i < decoded.length ; ++i) { + bytes[i] = decoded.charCodeAt(i); + } + return bytes; + } catch (_) { + throw new Error('Converting base64 string to bytes failed.'); + } +} + +// If filename is a base64 data URI, parses and returns data (Buffer on node, +// Uint8Array otherwise). If filename is not a base64 data URI, returns undefined. +function tryParseAsDataURI(filename) { + if (!isDataURI(filename)) { + return; + } + + return intArrayFromBase64(filename.slice(dataURIPrefix.length)); +} +// end include: base64Utils.js +Module['addRunDependency'] = addRunDependency; +Module['removeRunDependency'] = removeRunDependency; +Module['FS_createPath'] = FS.createPath; +Module['FS_createDataFile'] = FS.createDataFile; +Module['FS_createLazyFile'] = FS.createLazyFile; +Module['FS_createDevice'] = FS.createDevice; +Module['FS_unlink'] = FS.unlink; +Module['callMain'] = callMain; +Module['ENV'] = ENV; +Module['PATH'] = PATH; +Module['FS_createPreloadedFile'] = FS.createPreloadedFile; +Module['FS'] = FS; +Module['LZ4'] = LZ4; +var missingLibrarySymbols = [ + 'writeI53ToI64', + 'writeI53ToI64Clamped', + 'writeI53ToI64Signaling', + 'writeI53ToU64Clamped', + 'writeI53ToU64Signaling', + 'readI53FromU64', + 'convertI32PairToI53', + 'convertU32PairToI53', + 'growMemory', + 'traverseStack', + 'getCallstack', + 'emscriptenLog', + 'convertPCtoSourceLocation', + 'readEmAsmArgs', + 'jstoi_s', + 'listenOnce', + 'autoResumeAudioContext', + 'dynCallLegacy', + 'getDynCaller', + 'dynCall', + 'runtimeKeepalivePush', + 'runtimeKeepalivePop', + 'callUserCallback', + 'maybeExit', + 'safeSetTimeout', + 'asmjsMangle', + 'handleAllocatorInit', + 'HandleAllocator', + 'getNativeTypeSize', + 'STACK_SIZE', + 'STACK_ALIGN', + 'POINTER_SIZE', + 'ASSERTIONS', + 'getCFunc', + 'ccall', + 'cwrap', + 'uleb128Encode', + 'sigToWasmTypes', + 'generateFuncType', + 'convertJsFunctionToWasm', + 'getEmptyTableSlot', + 'updateTableMap', + 'getFunctionAddress', + 'addFunction', + 'removeFunction', + 'reallyNegative', + 'unSign', + 'strLen', + 'reSign', + 'formatString', + 'intArrayToString', + 'AsciiToString', + 'UTF16ToString', + 'stringToUTF16', + 'lengthBytesUTF16', + 'UTF32ToString', + 'stringToUTF32', + 'lengthBytesUTF32', + 'registerKeyEventCallback', + 'maybeCStringToJsString', + 'findEventTarget', + 'findCanvasEventTarget', + 'getBoundingClientRect', + 'fillMouseEventData', + 'registerMouseEventCallback', + 'registerWheelEventCallback', + 'registerUiEventCallback', + 'registerFocusEventCallback', + 'fillDeviceOrientationEventData', + 'registerDeviceOrientationEventCallback', + 'fillDeviceMotionEventData', + 'registerDeviceMotionEventCallback', + 'screenOrientation', + 'fillOrientationChangeEventData', + 'registerOrientationChangeEventCallback', + 'fillFullscreenChangeEventData', + 'registerFullscreenChangeEventCallback', + 'JSEvents_requestFullscreen', + 'JSEvents_resizeCanvasForFullscreen', + 'registerRestoreOldStyle', + 'hideEverythingExceptGivenElement', + 'restoreHiddenElements', + 'setLetterbox', + 'softFullscreenResizeWebGLRenderTarget', + 'doRequestFullscreen', + 'fillPointerlockChangeEventData', + 'registerPointerlockChangeEventCallback', + 'registerPointerlockErrorEventCallback', + 'requestPointerLock', + 'fillVisibilityChangeEventData', + 'registerVisibilityChangeEventCallback', + 'registerTouchEventCallback', + 'fillGamepadEventData', + 'registerGamepadEventCallback', + 'registerBeforeUnloadEventCallback', + 'fillBatteryEventData', + 'battery', + 'registerBatteryEventCallback', + 'setCanvasElementSize', + 'getCanvasElementSize', + 'jsStackTrace', + 'stackTrace', + 'checkWasiClock', + 'wasiRightsToMuslOFlags', + 'wasiOFlagsToMuslOFlags', + 'createDyncallWrapper', + 'setImmediateWrapped', + 'clearImmediateWrapped', + 'polyfillSetImmediate', + 'getPromise', + 'makePromise', + 'idsToPromises', + 'makePromiseCallback', + 'setMainLoop', + '_setNetworkCallback', + 'heapObjectForWebGLType', + 'heapAccessShiftForWebGLHeap', + 'webgl_enable_ANGLE_instanced_arrays', + 'webgl_enable_OES_vertex_array_object', + 'webgl_enable_WEBGL_draw_buffers', + 'webgl_enable_WEBGL_multi_draw', + 'emscriptenWebGLGet', + 'computeUnpackAlignedImageSize', + 'colorChannelsInGlTextureFormat', + 'emscriptenWebGLGetTexPixelData', + '__glGenObject', + 'emscriptenWebGLGetUniform', + 'webglGetUniformLocation', + 'webglPrepareUniformLocationsBeforeFirstUse', + 'webglGetLeftBracePos', + 'emscriptenWebGLGetVertexAttrib', + '__glGetActiveAttribOrUniform', + 'writeGLArray', + 'registerWebGlEventCallback', + 'runAndAbortIfError', + 'SDL_unicode', + 'SDL_ttfContext', + 'SDL_audio', + 'GLFW_Window', + 'ALLOC_NORMAL', + 'ALLOC_STACK', + 'allocate', + 'writeStringToMemory', + 'writeAsciiToMemory', +]; +missingLibrarySymbols.forEach(missingLibrarySymbol) + +var unexportedSymbols = [ + 'run', + 'addOnPreRun', + 'addOnInit', + 'addOnPreMain', + 'addOnExit', + 'addOnPostRun', + 'FS_createFolder', + 'FS_createLink', + 'out', + 'err', + 'abort', + 'keepRuntimeAlive', + 'wasmMemory', + 'stackAlloc', + 'stackSave', + 'stackRestore', + 'getTempRet0', + 'setTempRet0', + 'writeStackCookie', + 'checkStackCookie', + 'readI53FromI64', + 'convertI32PairToI53Checked', + 'ptrToString', + 'zeroMemory', + 'exitJS', + 'getHeapMax', + 'abortOnCannotGrowMemory', + 'MONTH_DAYS_REGULAR', + 'MONTH_DAYS_LEAP', + 'MONTH_DAYS_REGULAR_CUMULATIVE', + 'MONTH_DAYS_LEAP_CUMULATIVE', + 'isLeapYear', + 'ydayFromDate', + 'arraySum', + 'addDays', + 'ERRNO_CODES', + 'ERRNO_MESSAGES', + 'setErrNo', + 'inetPton4', + 'inetNtop4', + 'inetPton6', + 'inetNtop6', + 'readSockaddr', + 'writeSockaddr', + 'DNS', + 'getHostByName', + 'Protocols', + 'Sockets', + 'initRandomFill', + 'randomFill', + 'timers', + 'warnOnce', + 'UNWIND_CACHE', + 'readEmAsmArgsArray', + 'jstoi_q', + 'getExecutableName', + 'handleException', + 'asyncLoad', + 'alignMemory', + 'mmapAlloc', + 'freeTableIndexes', + 'functionsInTableMap', + 'setValue', + 'getValue', + 'PATH_FS', + 'UTF8Decoder', + 'UTF8ArrayToString', + 'UTF8ToString', + 'stringToUTF8Array', + 'stringToUTF8', + 'lengthBytesUTF8', + 'intArrayFromString', + 'stringToAscii', + 'UTF16Decoder', + 'stringToNewUTF8', + 'stringToUTF8OnStack', + 'writeArrayToMemory', + 'JSEvents', + 'specialHTMLTargets', + 'currentFullscreenStrategy', + 'restoreOldWindowedStyle', + 'demangle', + 'demangleAll', + 'ExitStatus', + 'getEnvStrings', + 'doReadv', + 'doWritev', + 'promiseMap', + 'uncaughtExceptionCount', + 'exceptionLast', + 'exceptionCaught', + 'ExceptionInfo', + 'Browser', + 'wget', + 'SYSCALLS', + 'getSocketFromFD', + 'getSocketAddress', + 'preloadPlugins', + 'FS_modeStringToFlags', + 'FS_getMode', + 'FS_stdin_getChar_buffer', + 'FS_stdin_getChar', + 'MEMFS', + 'TTY', + 'PIPEFS', + 'SOCKFS', + 'tempFixedLengthArray', + 'miniTempWebGLFloatBuffers', + 'miniTempWebGLIntBuffers', + 'GL', + 'emscripten_webgl_power_preferences', + 'AL', + 'GLUT', + 'EGL', + 'GLEW', + 'IDBStore', + 'SDL', + 'SDL_gfx', + 'GLFW', + 'allocateUTF8', + 'allocateUTF8OnStack', +]; +unexportedSymbols.forEach(unexportedRuntimeSymbol); + + + +var calledRun; + +dependenciesFulfilled = function runCaller() { + // If run has never been called, and we should call run (INVOKE_RUN is true, and Module.noInitialRun is not false) + if (!calledRun) run(); + if (!calledRun) dependenciesFulfilled = runCaller; // try this again later, after new deps are fulfilled +}; + +function callMain(args = []) { + assert(runDependencies == 0, 'cannot call main when async dependencies remain! (listen on Module["onRuntimeInitialized"])'); + assert(__ATPRERUN__.length == 0, 'cannot call main when preRun functions remain to be called'); + + var entryFunction = _main; + + args.unshift(thisProgram); + + var argc = args.length; + var argv = stackAlloc((argc + 1) * 4); + var argv_ptr = argv >> 2; + args.forEach((arg) => { + HEAP32[argv_ptr++] = stringToUTF8OnStack(arg); + }); + HEAP32[argv_ptr] = 0; + + try { + + var ret = entryFunction(argc, argv); + + // if we're not running an evented main loop, it's time to exit + exitJS(ret, /* implicit = */ true); + return ret; + } + catch (e) { + return handleException(e); + } +} + +function stackCheckInit() { + // This is normally called automatically during __wasm_call_ctors but need to + // get these values before even running any of the ctors so we call it redundantly + // here. + _emscripten_stack_init(); + // TODO(sbc): Move writeStackCookie to native to to avoid this. + writeStackCookie(); +} + +function run(args = arguments_) { + + if (runDependencies > 0) { + return; + } + + stackCheckInit(); + + preRun(); + + // a preRun added a dependency, run will be called later + if (runDependencies > 0) { + return; + } + + function doRun() { + // run may have just been called through dependencies being fulfilled just in this very frame, + // or while the async setStatus time below was happening + if (calledRun) return; + calledRun = true; + Module['calledRun'] = true; + + if (ABORT) return; + + initRuntime(); + + preMain(); + + readyPromiseResolve(Module); + if (Module['onRuntimeInitialized']) Module['onRuntimeInitialized'](); + + if (shouldRunNow) callMain(args); + + postRun(); + } + + if (Module['setStatus']) { + Module['setStatus']('Running...'); + setTimeout(function() { + setTimeout(function() { + Module['setStatus'](''); + }, 1); + doRun(); + }, 1); + } else + { + doRun(); + } + checkStackCookie(); +} + +function checkUnflushedContent() { + // Compiler settings do not allow exiting the runtime, so flushing + // the streams is not possible. but in ASSERTIONS mode we check + // if there was something to flush, and if so tell the user they + // should request that the runtime be exitable. + // Normally we would not even include flush() at all, but in ASSERTIONS + // builds we do so just for this check, and here we see if there is any + // content to flush, that is, we check if there would have been + // something a non-ASSERTIONS build would have not seen. + // How we flush the streams depends on whether we are in SYSCALLS_REQUIRE_FILESYSTEM=0 + // mode (which has its own special function for this; otherwise, all + // the code is inside libc) + var oldOut = out; + var oldErr = err; + var has = false; + out = err = (x) => { + has = true; + } + try { // it doesn't matter if it fails + _fflush(0); + // also flush in the JS FS layer + ['stdout', 'stderr'].forEach(function(name) { + var info = FS.analyzePath('/dev/' + name); + if (!info) return; + var stream = info.object; + var rdev = stream.rdev; + var tty = TTY.ttys[rdev]; + if (tty && tty.output && tty.output.length) { + has = true; + } + }); + } catch(e) {} + out = oldOut; + err = oldErr; + if (has) { + warnOnce('stdio streams had content in them that was not flushed. you should set EXIT_RUNTIME to 1 (see the Emscripten FAQ), or make sure to emit a newline when you printf etc.'); + } +} + +if (Module['preInit']) { + if (typeof Module['preInit'] == 'function') Module['preInit'] = [Module['preInit']]; + while (Module['preInit'].length > 0) { + Module['preInit'].pop()(); + } +} + +// shouldRunNow refers to calling main(), not run(). +var shouldRunNow = false; + +if (Module['noInitialRun']) shouldRunNow = false; + +run(); + + +// end include: postamble.js + + + return moduleArg.ready +} + +); +})(); +if (typeof exports === 'object' && typeof module === 'object') + module.exports = busytex; +else if (typeof define === 'function' && define['amd']) + define([], () => busytex); diff --git a/frontend/public/tex/busytex.wasm.stock b/frontend/public/tex/busytex.wasm.stock new file mode 100644 index 00000000..5d47cbbb Binary files /dev/null and b/frontend/public/tex/busytex.wasm.stock differ diff --git a/frontend/public/tex/busytex_pipeline.js b/frontend/public/tex/busytex_pipeline.js new file mode 100644 index 00000000..aa91d017 --- /dev/null +++ b/frontend/public/tex/busytex_pipeline.js @@ -0,0 +1,645 @@ +//TODO: what happens if creating another pipeline (waiting data error?) +//TODO: put texlive into /opt/texlive/2020 or ~/.texlive2020? +//TODO: configure fontconfig to use /etc/fonts + +/* +xdvipdfmx:warning: Color stack underflow. Just ignore. +xdvipdfmx:warning: Color stack underflow. Just ignore. +xdvipdfmx:warning: Color stack underflow. Just ignore. +xdvipdfmx:warning: Color stack underflow. Just ignore. +xdvipdfmx:warning: Color stack underflow. Just ignore. + +kpathsea: Running mktexpk --mfmode / --bdpi 600 --mag 1+264/600 --dpi 864 ec-qhvr +kpathsea: fork(): Function not implemented +kpathsea: Appending font creation commands to missfont.log. +xdvipdfmx:warning: Could not locate a virtual/physical font for TFM "ec-qhvr". +xdvipdfmx:warning: >> There are no valid font mapping entry for this font. +xdvipdfmx:warning: >> Font file name "ec-qhvr" was assumed but failed to locate that font. +xdvipdfmx:fatal: Cannot proceed without .vf or "physical" font for PDF output... + +No output PDF file written. +*/ + +class BusytexDataPackageResolver +{ + constructor(data_packages_js, texmf_system = [], texmf_local = [], remap = { + config : null, + firstaid : 'latex-firstaid', + hyphen : null, + jknapltx : null, + latexconfig : null, + pdfwin : null, + plweb : 'pl', + symbol : null, + syntax : null, + third : null, + twoup : null, + zapfding : null + }) + { + this.regex_usepackage = /\\usepackage(\[.*?\])?\{(.+?)\}/g; // [2019/06/21 v0.4.0 A clean LaTeX style for thesis documents] : {source: 'local', used: false} + this.regex_providespackage = /\\ProvidesPackage\{(.+?)\}(\[.*?\])?/g; + this.basename = path => path.slice(path.lastIndexOf('/') + 1); + this.dirname = path => path.slice(0, path.lastIndexOf('/')); + this.isfile = path => this.basename(path).includes('.'); + + this.msgs = []; + this.data_packages_js = data_packages_js; + this.data_packages = data_packages_js.map(data_package_js => [data_package_js, fetch(data_package_js).then(r => r.text()).then( data_package_js_script => new Set(Array.from(data_package_js_script.matchAll(this.regex_providespackage)).map(groups => groups[1].toLowerCase().trim())))]); + console.log('BusytexDataPackageResolver', this.data_packages); + this.remap = remap; + this.texmf_local_texmfdist_tex = texmf_local .map(t => t + '/texmf-dist/tex/'); + this.texmf_system_texmfdist_tex = texmf_system.map(t => t + '/texmf-dist/tex/'); + this.texmf_texmfdist_tex = [...this.texmf_system_texmfdist_tex, ...this.texmf_local_texmfdist_tex]; + this.data_packages_cache = null; + } + + async resolve_data_packages() + { + const values = await Promise.all(this.data_packages.map(([k, v]) => v)); + return this.data_packages.map(([k, v], i) => [k, Array.from(values[i]).sort()]); + } + + cache_data_packages() + { + if(!this.data_packages_cache) + this.data_packages_cache = Promise.all(this.data_packages_js.map(data_package_js => fetch(data_package_js.replace('.js', '.data'), {mode : 'no-cors'}))); + } + + extract_tex_package_name(path, contents = '') + { + // implicitly excludes /.../temxf-dist/{fonts,bibtex} + // cat urls.txt | while read URL; do echo $(curl -sI ${URL%$'\r'} | head -n 1 | cut -d' ' -f2) $URL; done | grep 404 | sort | uniq + + // https://ctan.org/tex-archive/macros/latex/required/graphics, graphicx + + const splitrootdir = path => { const splitted = path.split('/'); return [splitted[0], splitted.slice(1).join('/') ]; }; + + if(!path.endsWith('.sty')) + return null; + + const basename = this.basename(path); + let tex_package_name = basename.slice(0, basename.length - '.sty'.length); + if(this.isfile(path)) + { + const prefix = this.texmf_texmfdist_tex.find(t => path.startsWith(t)); + if(contents) + { + const tex_packages = contents.split('\n').filter(l => l.trim().startsWith('\\ProvidesPackage')).map(l => Array.from(l.matchAll(this.regex_providespackage)).filter(groups => groups.length >= 2).map(groups => groups[0] ) ).flat(); + if(tex_packages.length > 0) + tex_package_name = tex_packages[0]; + } + else if(prefix) + { + tex_package_name = splitrootdir(splitrootdir(path.slice(prefix.length))[1])[0]; + this.msgs.push([tex_package_name, path, 'https://ctan.org/pkg/' + tex_package_name]); + + if(tex_package_name in this.remap) + tex_package_name = this.remap[tex_package_name]; + } + } + + return tex_package_name; + } + + async resolve(files, main_tex_path, data_packages_js = null) + { + const tex_packages = files.filter(f => typeof(f.contents) == 'string' && f.path == main_tex_path).map(f => f.contents.split('\n').filter(l => l.trim().startsWith('\\usepackage')).map(l => Array.from(l.matchAll(this.regex_usepackage)).filter(groups => groups.length >= 2).map(groups => groups.pop().split(',') ) )).flat().flat().flat(); + + const tex_packages_local = new Set(files.filter(f => this.texmf_local_texmfdist_tex.some(t => f.path.startsWith(t)) || f.path.endsWith('.sty')).map(f => this.extract_tex_package_name(f.path, typeof(f.contents) == 'string' ? f.contents : '')).filter(f => f)); + + const tex_packages_to_resolve = tex_packages.filter(tex_package => !tex_packages_local.has(tex_package)); + + const resolved = Object.fromEntries(tex_packages.map(tex_package => ([tex_package, {used: true, source : null}]))); + for(const tex_package of tex_packages_local) + { + resolved[tex_package] = resolved[tex_package] || {}; + resolved[tex_package].source = 'local'; + resolved[tex_package].used = resolved[tex_package].used || false; + } + + let update_data_packages_js = false; + const tex_packages_not_resolved = []; + let data_packages = []; + + if(data_packages_js === null) + { + update_data_packages_js = true; + data_packages = this.data_packages; + data_packages_js = new Set(); + } + else + { + update_data_packages_js = false; + data_packages = this.data_packages.filter(([data_package_js, tex_packages]) => data_packages_js.includes(data_package_js)); + } + + for(const tex_package of tex_packages_to_resolve) + { + for(const [data_package_js, tex_packages] of [...data_packages, [null, null]]) + { + if(tex_packages !== null && (await tex_packages).has(tex_package)) + { + resolved[tex_package].source = data_package_js; + + if(update_data_packages_js) + data_packages_js.add(data_package_js); + break; + } + } + } + + return resolved; + } +} + + +class BusytexBibtexResolver +{ + resolve (files, bib_tex_commands = ['\\bibliography', '\\printbibliography']) + { + return files.some(f => f.path.endsWith('.tex') && typeof(f.contents) == 'string' && bib_tex_commands.some(b => f.contents.includes(b))); + // files.some(({path, contents}) => contents != null && path.endsWith('.bib')); + } +} + +class BusytexPipeline +{ + static texmf_system = ['/texlive', '/texmf']; + static VerboseSilent = 'silent'; + static VerboseInfo = 'info'; + static VerboseDebug = 'debug'; + + //FIXME begin: have to do static to execute LZ4 data packages: https://github.com/emscripten-core/emscripten/issues/12347 + static preRun = []; + static calledRun = false; + static data_packages = []; + static locateFile(remote_package_name) + { + return BusytexPipeline.data_packages.map(data_package_js => data_package_js.replace('.js', '.data')).find(data_file => data_file.endsWith(remote_package_name)); + } + //FIXME end + + static ScriptLoaderDocument(src) + { + return new Promise((resolve, reject) => + { + let s = self.document.createElement('script'); + s.src = src; + s.onload = resolve; + s.onerror = reject; + self.document.head.appendChild(s); + }); + } + + static ScriptLoaderRequire(src) + { + return new Promise(resolve => self.require([src], resolve)); + } + + static ScriptLoaderWorker(src) + { + return Promise.resolve(self.importScripts(src)); + } + + load_package(data_package_js) + { + if(data_package_js in this.data_package_promises) + return this.data_package_promises[data_package_js]; + + BusytexPipeline.calledRun = false; + BusytexPipeline.data_packages.push(data_package_js); + const promise = this.script_loader(data_package_js); + this.data_package_promises[data_package_js] = promise; + return promise; + } + + constructor(busytex_js, busytex_wasm, data_packages_js, preload_data_packages_js, texmf_local, print, on_initialized, preload, script_loader, calkit_texmf_endpoint) + { + this.print = text => {console.log(text); print(text); }; + this.preload = preload; + this.script_loader = script_loader; + // Calkit: on a local kpathsea miss the patched engine fetches the file + // from this texmf proxy (empty => hook no-ops, stock behaviour). + this.calkit_texmf_endpoint = calkit_texmf_endpoint || ''; + + this.project_dir = '/home/web_user/project_dir'; + this.bin_busytex = '/bin/busytex'; + this.fmt = { + pdftex : '/texlive/texmf-dist/texmf-var/web2c/pdftex/pdflatex.fmt', + xetex : '/texlive/texmf-dist/texmf-var/web2c/xetex/xelatex.fmt', + luahbtex: '/texlive/texmf-dist/texmf-var/web2c/luahbtex/luahblatex.fmt', + luatex : '/texlive/texmf-dist/texmf-var/web2c/luahbtex/lualatex.fmt', + }; + this.dir_texmfdist = [...BusytexPipeline.texmf_system, ...texmf_local].map(texmf => texmf + '/texmf-dist').join(':'); + this.dir_texmfvar = '/texlive/texmf-dist/texmf-var'; + this.dir_cnf = '/texlive/texmf-dist/web2c'; + this.dir_fontconfig = '/etc/fonts'; + this.texmflog = '/tmp/texmf.log'; + this.missfontlog = 'missfont.log'; // http://tug.ctan.org/info/tex-font-errors-cheatsheet/tex-font-cheatsheet.pdf + + this.verbose_args = + { + [BusytexPipeline.VerboseSilent] : { + pdftex : [], + xetex : [], + luatex : [], + luahbtex : [], + bibtex8 : [], + xdvipdfmx : [], + }, + [BusytexPipeline.VerboseInfo] : { + pdftex : ['-kpathsea-debug', '32'], + xetex : ['-kpathsea-debug', '32'], + luatex : ['-kpathsea-debug', '32'], + luahbtex : ['-kpathsea-debug', '32'], + xdvipdfmx : ['--kpathsea-debug','32', '-v'], + bibtex8 : ['--debug', 'search'], + }, + [BusytexPipeline.VerboseDebug] : { + pdftex : ['-kpathsea-debug', '63', '-recorder'], + xetex : ['-kpathsea-debug', '63', '-recorder'], + luatex : ['-kpathsea-debug', '63', '-recorder', '--debug-format'], + luahbtex : ['-kpathsea-debug', '63', '-recorder', '--debug-format'], + xdvipdfmx : ['--kpathsea-debug','63', '-vv'], + bibtex8 : ['--debug', 'all'], + }, + }; + this.supported_drivers = ['xetex_bibtex8_dvipdfmx', 'pdftex_bibtex8', 'luahbtex_bibtex8', 'luatex_bibtex8']; + + this.error_messages_fatal = ['Fatal error occurred', 'That was a fatal error', ':fatal:', '! Undefined control sequence.', 'undefined old font command']; + this.error_messages_all = this.error_messages_fatal.concat(['no output PDF file produced', 'No pages of output.']); + + this.env = { + TEXMFDIST : this.dir_texmfdist, + TEXMFVAR : this.dir_texmfvar, + TEXMFCNF : this.dir_cnf, + TEXMFLOG : this.texmflog, + FONTCONFIG_PATH : this.dir_fontconfig + }; + + this.remove = (FS, log_path) => FS.analyzePath(log_path).exists ? FS.unlink(log_path) : null; + this.read_all_text = (FS, log_path) => FS.analyzePath(log_path).exists ? FS.readFile(log_path, {encoding : 'utf8'}).trim() : ''; + this.read_all_bytes = (FS, pdf_path) =>FS.analyzePath(pdf_path).exists ? FS.readFile(pdf_path, {encoding: 'binary'}) : new Uint8Array(); + this.mkdir_p = (FS, PATH, dirpath, dirs = new Set()) => + { + if(!dirs.has(dirpath)) + { + this.mkdir_p(FS, PATH, PATH.dirname(dirpath), dirs); + FS.mkdir(dirpath); + dirs.add(dirpath); + } + }; + + this.bibtex_resolver = new BusytexBibtexResolver(); + this.data_package_resolver = new BusytexDataPackageResolver(data_packages_js, BusytexPipeline.texmf_system, texmf_local); + this.wasm_module_promise = fetch(busytex_wasm).then(WebAssembly.compileStreaming); + this.mem_header_size = 2 ** 26; + + this.em_module_promise = this.script_loader(busytex_js); + BusytexPipeline.data_packages = []; + this.data_package_promises = {}; + this.preload_data_packages_js = preload_data_packages_js; + for(const data_package_js of this.preload_data_packages_js) + this.load_package(data_package_js); + this.Module = this.reload_module_if_needed(this.preload !== false, this.env, this.project_dir, this.preload_data_packages_js); + + this.on_initialized = null; + this.on_initialized_promise = new Promise(resolve => (this.on_initialized = resolve)); + this.on_initialized_promise_notification = this.on_initialized_promise.then(on_initialized); + + this.on_initialization_error = null; + } + + terminate() + { + this.Module = null; + } + + async reload_module_if_needed(cond, env, project_dir, data_packages_js) + { + if(cond) + { + return this.reload_module(env, project_dir, data_packages_js, true); + } + else if(this.Module) + { + const Module = await this.Module; + const enabled_packages_js = Module.data_packages_js; + const new_data_packages_js = data_packages_js.filter(data_package_js => !enabled_packages_js.includes(data_package_js)); + + if(new_data_packages_js.length > 0) + { + return this.reload_module(env, project_dir, Array.from(enabled_packages_js).concat(Array.from(new_data_packages_js)), false); + } + + return Module; + } + } + + async reload_module(env, project_dir, data_packages_js = [], report_applet_versions = false) + { + const data_packages_js_promise = data_packages_js.map(data_package_js => this.load_package(data_package_js)); + const [em_module, wasm_module] = await Promise.all([this.em_module_promise, WebAssembly.compileStreaming ? this.wasm_module_promise : this.wasm_module_promise.then(r => r.arrayBuffer()), ...data_packages_js_promise]); + const {print, init_env} = this; + + const pre_run_packages = Module => () => + { + Object.setPrototypeOf(BusytexPipeline, Module); + + for(const preRun of BusytexPipeline.preRun) + { + if(Module.preRuns.includes(preRun)) + continue; + + preRun(); + Module.preRuns.push(preRun); + } + } + + const Module = + { + thisProgram : this.bin_busytex, + noInitialRun : true, + totalDependencies: 0, + prefix : '', + preRuns : [], + data_packages_js : data_packages_js, + pre_run_packages : pre_run_packages, + // Calkit: read by the patched engine's kpse_remote_fetch hook. + calkitTexmfEndpoint : this.calkit_texmf_endpoint, + + preRun : [() => { Object.assign(Module.ENV, env); Module.FS.mkdir(project_dir); self.LZ4 = Module.LZ4; }, () => pre_run_packages(Module)()], + + instantiateWasm(imports, successCallback) + { + WebAssembly.instantiate(wasm_module, imports).then(output => successCallback(WebAssembly.compileStreaming ? output : output.instance)).catch(err => {throw new Error('Error while initializing BusyTex!\n\n' + err.toString())}); + return {}; + }, + + do_print : true, + output_stdout : '', + print(text) + { + text = (arguments.length > 1 ? Array.prototype.slice.call(arguments).join(' ') : text) || ''; + Module.output_stdout += text + '\n'; + if(Module.do_print) + Module.setStatus(Module.thisProgram + ' stdout: ' + text); + }, + output_stderr : '', + printErr(text) + { + text = (arguments.length > 1 ? Array.prototype.slice.call(arguments).join(' ') : text) || ''; + Module.output_stderr += text + '\n'; + Module.setStatus(Module.thisProgram + ' stderr: ' + text); + }, + + setPrefix(text) + { + this.prefix = text; + }, + + setStatus(text) + { + if(this.do_print) + print(text); + }, + + monitorRunDependencies(left) + { + this.totalDependencies = Math.max(this.totalDependencies, left); + Module.setStatus(left ? 'Preparing... (' + (this.totalDependencies-left) + '/' + this.totalDependencies + ')' : 'All downloads complete.'); + }, + + callMainWithRedirects(args = [], print = false) + { + const Module = this; + Module.do_print = print; + Module.output_stdout = ''; + Module.output_stderr = ''; + Module.setPrefix(args[0]); + const exit_code = Module.callMain(args); + Module._flush_streams(); + + return { exit_code : exit_code, stdout : Module.output_stdout, stderr : Module.output_stderr }; + } + }; + + const initialized_module = await busytex(Module); + + if(!(this.mem_header_size % 4 == 0 && initialized_module.HEAP32.slice(this.mem_header_size / 4).every(x => x == 0))) + throw new Error(`Memory header size [${this.mem_header_size}] must be divisible by 4, and remaining memory must be zero`); + + if(report_applet_versions) + { + const applets = initialized_module.callMainWithRedirects().stdout.split('\n').filter(line => line.length > 0); + initialized_module.applet_versions = Object.fromEntries(applets.map(applet => ([applet, applet != 'makeindex' ? initialized_module.callMainWithRedirects([applet, '--version']).stdout : 'makeindex does not support --version' ]))); + // TODO: exception here not caught? + this.on_initialized(initialized_module.applet_versions); + } + else + initialized_module.applet_versions = {}; + + return initialized_module; + } + + async compile(files, main_tex_path, bibtex, verbose, driver, data_packages_js = []) + { + if(!this.supported_drivers.includes(driver)) + throw new Error(`Driver [${driver}] is not supported, only [${this.supported_drivers}] are supported`); + this.print(`New compilation started: [${main_tex_path}]`); + + if(bibtex === null) + bibtex = this.bibtex_resolver.resolve(files); + + const resolved = await this.data_package_resolver.resolve(files, main_tex_path, data_packages_js); + const filter_map = (f, return_tex_package = true) => Object.entries(resolved).filter(([tex_package, v]) => f(v)).map(([tex_package, v]) => return_tex_package ? tex_package : v.source); + + data_packages_js = Array.from(new Set(filter_map(v => v.used && v.source != 'local' && v.source != null, false))).sort(); + + const tex_packages_not_resolved = filter_map(v => v.source == null); + + const fmt_packages_list = packages => '[' + (packages ? packages.toString().replaceAll(',', ', ') : '') + ']'; + + console.log('resolved', resolved); + console.log('tex_packages_not_resolved', tex_packages_not_resolved); + console.log('data_packages_js', data_packages_js); + + this.print('TeX packages: ' + fmt_packages_list(filter_map(v => v.used))); + this.print('TeX packages local: ' + fmt_packages_list(filter_map(v => v.source == 'local'))); + this.print('TeX packages unresolved: ' + fmt_packages_list(tex_packages_not_resolved)); + this.print('TeX packages unresolved (in local or preloaded): ' + fmt_packages_list(filter_map(v => v.used && (v.source != 'local' && !this.preload_data_packages_js.includes(v.source))))); + + this.print('Data packages used (preloaded): ' + fmt_packages_list(this.preload_data_packages_js)); + this.print('Data packages used (not preloaded): ' + fmt_packages_list(Array.from(new Set(filter_map(v => v.used && v.source != 'local' && v.source != null && !this.preload_data_packages_js.includes(v.source), false))).sort())); + this.print('Data packages used: ' + fmt_packages_list(data_packages_js)); + + if(tex_packages_not_resolved.length > 0) + { + // TODO: replace by regular return? override? + // throw new Error('Not resolved TeX packages: ' + tex_packages_not_resolved.join(', ')); + + data_packages_js = this.data_package_resolver.data_packages_js; + this.print('Because of unresolved TeX packages, enabling all available data packages: ' + data_packages_js.sort().toString()); + } + + this.Module = this.reload_module_if_needed(this.Module == null, this.env, this.project_dir, data_packages_js); + + const Module = await this.Module; + const {FS, PATH} = Module; + + const tex_path = PATH.basename(main_tex_path), dirname = PATH.dirname(main_tex_path); + + const [xdv_path, pdf_path, log_path, aux_path, blg_path, bbl_path] = ['.xdv', '.pdf', '.log', '.aux', '.blg', '.bbl'].map(ext => tex_path.replace('.tex', ext)); + + const xetex = ['xelatex' , '--no-shell-escape', '--interaction=batchmode', '--halt-on-error', '--no-pdf' , '--fmt', this.fmt.xetex , tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).xetex); + const pdftex = ['pdflatex', '--no-shell-escape', '--interaction=nonstopmode', '--halt-on-error', '--output-format=pdf', '--fmt', this.fmt.pdftex, tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).pdftex); + const pdftex_not_final = ['pdflatex', '--no-shell-escape', '--interaction=batchmode', '--halt-on-error', '--fmt', this.fmt.pdftex, tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).pdftex); + + const luahbtex = ['luahblatex', '--no-shell-escape', '--interaction=nonstopmode', '--halt-on-error', '--output-format=pdf', '--fmt', this.fmt.luahbtex, '--nosocket', tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).luahbtex); + const luahbtex_not_final = ['luahblatex', '--no-shell-escape', '--interaction=nonstopmode', '--halt-on-error', '--no-pdf', '--fmt', this.fmt.luahbtex, '--nosocket', tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).luahbtex); + + const luatex = ['lualatex', '--no-shell-escape', '--interaction=nonstopmode', '--halt-on-error', '--output-format=pdf', '--fmt', this.fmt.luatex, '--nosocket', tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).luahbtex); + const luatex_not_final = ['lualatex', '--no-shell-escape', '--interaction=nonstopmode', '--halt-on-error', '--no-pdf', '--fmt', this.fmt.luatex, '--nosocket', tex_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).luahbtex); + + const bibtex8 = ['bibtex8', '--8bit', aux_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).bibtex8); + + const xdvipdfmx = ['xdvipdfmx', '-o', pdf_path, xdv_path].concat((this.verbose_args[verbose] || this.verbose_args[BusytexPipeline.VerboseSilent]).xdvipdfmx); + + if(FS.analyzePath(this.project_dir).object.mount.mountpoint == this.project_dir) + FS.unmount(this.project_dir); + FS.mount(FS.filesystems.MEMFS, {}, this.project_dir); + + let dirs = new Set(['/', this.project_dir]); + for(const {path, contents} of files.sort((lhs, rhs) => lhs['path'] < rhs['path'] ? -1 : 1)) + { + const absolute_path = PATH.join(this.project_dir, path); + if(contents == null) + this.mkdir_p(FS, PATH, absolute_path, dirs); + else + { + this.mkdir_p(FS, PATH, PATH.dirname(absolute_path), dirs); + FS.writeFile(absolute_path, contents); + } + } + + const source_dir = PATH.join(this.project_dir, dirname); + FS.chdir(source_dir); + + let cmds = []; + if(driver == 'xetex_bibtex8_dvipdfmx') + { + cmds = bibtex ? + [ + [xetex, this.error_messages_fatal, false], + [bibtex8, this.error_messages_fatal, true], + [xetex, this.error_messages_fatal, true], + [xetex, this.error_messages_all, true], + [xdvipdfmx, this.error_messages_all, false] + ] : + [ + [xetex, this.error_messages_all, false], + [xdvipdfmx, this.error_messages_all, false] + ]; + } + else if(driver == 'pdftex_bibtex8') + { + cmds = bibtex ? + [ + [pdftex_not_final, this.error_messages_fatal, false], + [bibtex8, this.error_messages_fatal, true], + [pdftex_not_final, this.error_messages_fatal, true], + [pdftex, this.error_messages_all, false] + ] : + [ + [pdftex, this.error_messages_all] + ]; + } + else if(driver == 'luahbtex_bibtex8') + { + cmds = bibtex ? + [ + [luahbtex, this.error_messages_fatal, false], + [bibtex8 , this.error_messages_fatal, true], + [luahbtex, this.error_messages_fatal, true], + [luahbtex, this.error_messages_all, true] + ] : + [ + [luahbtex, this.error_messages_all, false] + ]; + } + else if(driver == 'luatex_bibtex8') + { + cmds = bibtex ? + [ + [luatex, this.error_messages_fatal, false], + [bibtex8,this.error_messages_fatal, true], + [luatex, this.error_messages_fatal, true], + [luatex, this.error_messages_all, false] + ] : + [ + [luatex, this.error_messages_all, false] + ]; + } + + let exit_code = 0, stdout = '', stderr = '', log = '', aux = ''; + let skip = false; + const mem_header = Uint8Array.from(Module.HEAPU8.slice(0, this.mem_header_size)); + const logs = []; + for(const [cmd, error_messages, can_skip] of cmds) + { + if(can_skip && skip) + continue; + + const is_bibtex = cmd[0].startsWith('bibtex'); + const cmd_log_path = is_bibtex ? blg_path : log_path; + const cmd_aux_path = is_bibtex ? bbl_path : aux_path; + + this.remove(FS, this.texmflog); + this.remove(FS, this.missfontlog); + this.remove(FS, cmd_log_path); + + this.print('$ busytex ' + cmd.join(' ')); + ({exit_code, stdout, stderr} = Module.callMainWithRedirects(cmd, verbose != BusytexPipeline.VerboseSilent)); + + Module.HEAPU8.fill(0); + Module.HEAPU8.set(mem_header); + + this.print('$ echo $?'); + this.print(`${exit_code}\n`); + + if(is_bibtex && this.read_all_text(FS, bbl_path).trim() == '' && (exit_code == 0 || exit_code == 2)) + { + skip = true; + this.print('$ # bibtex found no citation commands, skipping extra calls'); + } + + aux = this.read_all_text(FS, cmd_aux_path); + log = this.read_all_text(FS, cmd_log_path); + exit_code = stdout.trim() ? (error_messages.some(err => stdout.includes(err)) ? exit_code : 0) : exit_code; + + logs.push({ + cmd : cmd.join(' '), + texmflog : (verbose == BusytexPipeline.VerboseInfo || verbose == BusytexPipeline.VerboseDebug) ? this.read_all_text(FS, this.texmflog) : '', + missfontlog : (verbose == BusytexPipeline.VerboseInfo || verbose == BusytexPipeline.VerboseDebug) ? this.read_all_text(FS, this.missfontlog) : '', + log : log.trim(), + aux : aux.trim(), + stdout : stdout.trim(), + stderr : stderr.trim(), + exit_code : exit_code + }); + + if(exit_code != 0) + break; + } + + console.log('LOGS', logs); + + const pdf = exit_code == 0 ? this.read_all_bytes(FS, pdf_path) : null; + const logcat = logs.map(({cmd, texmflog, missfontlog, log, exit_code, stdout, stderr}) => ([`$ ${cmd}`, `EXITCODE: ${exit_code}`, '', 'TEXMFLOG:', texmflog, '==', 'MISSFONTLOG:', missfontlog, '==', 'LOG:', log, '==', 'STDOUT:', stdout, '==', 'STDERR:', stderr, '======'].join('\n'))).join('\n\n'); + + this.Module = this.preload == false ? null : this.Module; + + return {pdf : pdf, log : logcat, exit_code : exit_code, logs : logs}; + } +} diff --git a/frontend/public/tex/busytex_worker.js b/frontend/public/tex/busytex_worker.js new file mode 100644 index 00000000..75aec2ca --- /dev/null +++ b/frontend/public/tex/busytex_worker.js @@ -0,0 +1,31 @@ +importScripts('busytex_pipeline.js'); + +self.pipeline = null; + +onmessage = async ({data : {files, main_tex_path, bibtex, busytex_wasm, busytex_js, preload_data_packages_js, data_packages_js, texmf_local, preload, verbose, driver, calkit_texmf_endpoint}}) => +{ + // TODO: cache data packages from here? https://developer.mozilla.org/en-US/docs/Web/API/Cache + + if(busytex_wasm && busytex_js && preload_data_packages_js) + { + try + { + self.pipeline = new BusytexPipeline(busytex_js, busytex_wasm, data_packages_js, preload_data_packages_js, texmf_local, msg => postMessage({print : msg}), applet_versions => postMessage({ initialized : applet_versions }), preload, BusytexPipeline.ScriptLoaderWorker, calkit_texmf_endpoint); + } + catch(err) + { + postMessage({exception: 'Exception during initialization: ' + err.toString() + '\nStack:\n' + err.stack}); + } + } + else if(files && self.pipeline) + { + try + { + postMessage(await self.pipeline.compile(files, main_tex_path, bibtex, verbose, driver, data_packages_js)) + } + catch(err) + { + postMessage({exception: 'Exception during compilation: ' + err.toString() + '\nStack:\n' + err.stack}); + } + } +}; diff --git a/frontend/scripts/download-tex-engine.sh b/frontend/scripts/download-tex-engine.sh new file mode 100644 index 00000000..3005fe44 --- /dev/null +++ b/frontend/scripts/download-tex-engine.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Fetch the large busytex WASM engine binaries into frontend/public/tex/. +# The small MIT JS glue (busytex_worker.js, busytex_pipeline.js) is committed; +# these big binaries are gitignored. Engine = upstream MIT busytex/busytex +# (TeX Live 2023). See LATEX_EDITOR_PLAN.md §0/§8.1. +# +# For production, host these on a CDN/object storage and set VITE_TEX_ENGINE_URL +# instead of serving them from public/. +set -euo pipefail + +REPO="busytex/busytex" +REL="build_wasm_4499aa69fd3cf77ad86a47287d9a5193cf5ad993_7936974349_1" +DIR="$(cd "$(dirname "$0")/.." && pwd)/public/tex" +mkdir -p "$DIR" + +echo "Downloading busytex engine binaries -> $DIR" +gh release download "$REL" --repo "$REPO" --dir "$DIR" --clobber \ + --pattern busytex.wasm \ + --pattern busytex.js \ + --pattern texlive-basic.data \ + --pattern texlive-basic.js \ + --pattern "ubuntu-texlive-latex-base.data" \ + --pattern "ubuntu-texlive-latex-base.js" \ + --pattern "ubuntu-texlive-latex-recommended.data" \ + --pattern "ubuntu-texlive-latex-recommended.js" \ + --pattern "ubuntu-texlive-latex-extra.data" \ + --pattern "ubuntu-texlive-latex-extra.js" \ + --pattern "ubuntu-texlive-science.data" \ + --pattern "ubuntu-texlive-science.js" \ + --pattern "ubuntu-texlive-fonts-recommended.data" \ + --pattern "ubuntu-texlive-fonts-recommended.js" + +echo "Done." diff --git a/frontend/src/client/schemas.gen.ts b/frontend/src/client/schemas.gen.ts index 759b5481..09d09fdf 100644 --- a/frontend/src/client/schemas.gen.ts +++ b/frontend/src/client/schemas.gen.ts @@ -7,7 +7,14 @@ export const AccountPublicSchema = { title: "Name", }, github_name: { - type: "string", + anyOf: [ + { + type: "string", + }, + { + type: "null", + }, + ], title: "Github Name", }, display_name: { @@ -372,6 +379,17 @@ export const Body_projects_put_project_contentsSchema = { format: "binary", title: "File", }, + message: { + anyOf: [ + { + type: "string", + }, + { + type: "null", + }, + ], + title: "Message", + }, }, type: "object", required: ["file"], @@ -1704,7 +1722,14 @@ export const FileLockSchema = { title: "User Id", }, user_github_username: { - type: "string", + anyOf: [ + { + type: "string", + }, + { + type: "null", + }, + ], title: "User Github Username", readOnly: true, }, @@ -3739,7 +3764,14 @@ export const ProjectCommentSchema = { title: "Git Rev", }, user_github_username: { - type: "string", + anyOf: [ + { + type: "string", + }, + { + type: "null", + }, + ], title: "User Github Username", readOnly: true, }, @@ -3870,6 +3902,196 @@ export const ProjectCommentPostSchema = { title: "ProjectCommentPost", } as const +export const ProjectInvitationCreatedSchema = { + properties: { + id: { + type: "string", + format: "uuid", + title: "Id", + }, + role_name: { + type: "string", + title: "Role Name", + }, + created: { + type: "string", + format: "date-time", + title: "Created", + }, + expires: { + anyOf: [ + { + type: "string", + format: "date-time", + }, + { + type: "null", + }, + ], + title: "Expires", + }, + max_uses: { + anyOf: [ + { + type: "integer", + }, + { + type: "null", + }, + ], + title: "Max Uses", + }, + use_count: { + type: "integer", + title: "Use Count", + }, + revoked: { + type: "boolean", + title: "Revoked", + }, + token: { + type: "string", + title: "Token", + }, + url: { + type: "string", + title: "Url", + }, + }, + type: "object", + required: [ + "id", + "role_name", + "created", + "expires", + "max_uses", + "use_count", + "revoked", + "token", + "url", + ], + title: "ProjectInvitationCreated", +} as const + +export const ProjectInvitationPostSchema = { + properties: { + role: { + type: "string", + enum: ["read", "write", "admin"], + title: "Role", + default: "write", + }, + expires_days: { + anyOf: [ + { + type: "integer", + maximum: 365, + minimum: 1, + }, + { + type: "null", + }, + ], + title: "Expires Days", + }, + max_uses: { + anyOf: [ + { + type: "integer", + minimum: 1, + }, + { + type: "null", + }, + ], + title: "Max Uses", + }, + }, + type: "object", + title: "ProjectInvitationPost", +} as const + +export const ProjectInvitationPublicSchema = { + properties: { + id: { + type: "string", + format: "uuid", + title: "Id", + }, + role_name: { + type: "string", + title: "Role Name", + }, + created: { + type: "string", + format: "date-time", + title: "Created", + }, + expires: { + anyOf: [ + { + type: "string", + format: "date-time", + }, + { + type: "null", + }, + ], + title: "Expires", + }, + max_uses: { + anyOf: [ + { + type: "integer", + }, + { + type: "null", + }, + ], + title: "Max Uses", + }, + use_count: { + type: "integer", + title: "Use Count", + }, + revoked: { + type: "boolean", + title: "Revoked", + }, + }, + type: "object", + required: [ + "id", + "role_name", + "created", + "expires", + "max_uses", + "use_count", + "revoked", + ], + title: "ProjectInvitationPublic", +} as const + +export const ProjectInvitationRedeemedSchema = { + properties: { + owner_name: { + type: "string", + title: "Owner Name", + }, + project_name: { + type: "string", + title: "Project Name", + }, + role_name: { + type: "string", + title: "Role Name", + }, + }, + type: "object", + required: ["owner_name", "project_name", "role_name"], + title: "ProjectInvitationRedeemed", +} as const + export const ProjectOptionalExtendedSchema = { properties: { name: { @@ -6759,8 +6981,15 @@ export const UserCreateSchema = { title: "Account Name", }, github_username: { - type: "string", - maxLength: 64, + anyOf: [ + { + type: "string", + maxLength: 64, + }, + { + type: "null", + }, + ], title: "Github Username", }, }, @@ -6805,7 +7034,14 @@ export const UserPublicSchema = { title: "Id", }, github_username: { - type: "string", + anyOf: [ + { + type: "string", + }, + { + type: "null", + }, + ], title: "Github Username", }, subscription: { diff --git a/frontend/src/client/sdk.gen.ts b/frontend/src/client/sdk.gen.ts index bbc6dae3..ddb4a23b 100644 --- a/frontend/src/client/sdk.gen.ts +++ b/frontend/src/client/sdk.gen.ts @@ -151,6 +151,14 @@ import type { PutProjectCollaboratorResponse, DeleteProjectCollaboratorData, DeleteProjectCollaboratorResponse, + PostProjectInvitationData, + PostProjectInvitationResponse, + GetProjectInvitationsData, + GetProjectInvitationsResponse, + DeleteProjectInvitationData, + DeleteProjectInvitationResponse, + PostProjectInvitationRedemptionData, + PostProjectInvitationRedemptionResponse, GetProjectIssuesData, GetProjectIssuesResponse, PostProjectIssueData, @@ -2315,6 +2323,110 @@ export class ProjectsService { }) } + /** + * Post Project Invitation + * Create a shareable invite link granting native project membership. + * + * The raw token is returned only here; the DB stores its hash. Invites can + * grant up to admin, never ownership. + * @param data The data for the request. + * @param data.ownerName + * @param data.projectName + * @param data.requestBody + * @returns ProjectInvitationCreated Successful Response + * @throws ApiError + */ + public static postProjectInvitation( + data: PostProjectInvitationData, + ): CancelablePromise { + return __request(OpenAPI, { + method: "POST", + url: "/projects/{owner_name}/{project_name}/invitations", + path: { + owner_name: data.ownerName, + project_name: data.projectName, + }, + body: data.requestBody, + mediaType: "application/json", + errors: { + 422: "Validation Error", + }, + }) + } + + /** + * Get Project Invitations + * @param data The data for the request. + * @param data.ownerName + * @param data.projectName + * @returns ProjectInvitationPublic Successful Response + * @throws ApiError + */ + public static getProjectInvitations( + data: GetProjectInvitationsData, + ): CancelablePromise { + return __request(OpenAPI, { + method: "GET", + url: "/projects/{owner_name}/{project_name}/invitations", + path: { + owner_name: data.ownerName, + project_name: data.projectName, + }, + errors: { + 422: "Validation Error", + }, + }) + } + + /** + * Delete Project Invitation + * @param data The data for the request. + * @param data.ownerName + * @param data.projectName + * @param data.invitationId + * @returns Message Successful Response + * @throws ApiError + */ + public static deleteProjectInvitation( + data: DeleteProjectInvitationData, + ): CancelablePromise { + return __request(OpenAPI, { + method: "DELETE", + url: "/projects/{owner_name}/{project_name}/invitations/{invitation_id}", + path: { + owner_name: data.ownerName, + project_name: data.projectName, + invitation_id: data.invitationId, + }, + errors: { + 422: "Validation Error", + }, + }) + } + + /** + * Post Project Invitation Redemption + * Redeem an invite link, granting the current user native membership. + * @param data The data for the request. + * @param data.token + * @returns ProjectInvitationRedeemed Successful Response + * @throws ApiError + */ + public static postProjectInvitationRedemption( + data: PostProjectInvitationRedemptionData, + ): CancelablePromise { + return __request(OpenAPI, { + method: "POST", + url: "/project-invitations/{token}", + path: { + token: data.token, + }, + errors: { + 422: "Validation Error", + }, + }) + } + /** * Get Project Issues * @param data The data for the request. @@ -3594,7 +3706,10 @@ export class UsersService { /** * Register User - * Create new user without the need to be logged in. + * Create a new user with email + password, without a GitHub account. + * + * Such users can collaborate on projects (e.g. via invite links) but cannot + * own projects until git hosting is decoupled from GitHub. * @param data The data for the request. * @param data.requestBody * @returns UserPublic Successful Response diff --git a/frontend/src/client/types.gen.ts b/frontend/src/client/types.gen.ts index 6fde0b1b..1825d188 100644 --- a/frontend/src/client/types.gen.ts +++ b/frontend/src/client/types.gen.ts @@ -17,7 +17,7 @@ export type _ContentsItemBase = { export type AccountPublic = { name: string - github_name: string + github_name: string | null display_name: string kind: "user" | "org" role?: "self" | "read" | "write" | "admin" | "owner" | null @@ -106,6 +106,7 @@ export type kind3 = export type Body_projects_put_project_contents = { file: Blob | File + message?: string | null } export type Collaborator = { @@ -391,7 +392,7 @@ export type FileLock = { path: string created?: string user_id: string - readonly user_github_username: string + readonly user_github_username: string | null readonly user_email: string } @@ -844,7 +845,7 @@ export type ProjectComment = { resolved?: string | null git_ref?: string | null git_rev?: string | null - readonly user_github_username: string + readonly user_github_username: string | null readonly user_full_name: string | null readonly user_email: string } @@ -863,6 +864,42 @@ export type ProjectCommentPost = { git_ref?: string | null } +export type ProjectInvitationCreated = { + id: string + role_name: string + created: string + expires: string | null + max_uses: number | null + use_count: number + revoked: boolean + token: string + url: string +} + +export type ProjectInvitationPost = { + role?: "read" | "write" | "admin" + expires_days?: number | null + max_uses?: number | null +} + +export type role2 = "read" | "write" | "admin" + +export type ProjectInvitationPublic = { + id: string + role_name: string + created: string + expires: string | null + max_uses: number | null + use_count: number + revoked: boolean +} + +export type ProjectInvitationRedeemed = { + owner_name: string + project_name: string + role_name: string +} + export type ProjectOptionalExtended = { name: string title: string @@ -1423,7 +1460,7 @@ export type UserCreate = { full_name?: string | null password: string account_name?: string | null - github_username?: string + github_username?: string | null } export type UserPublic = { @@ -1432,7 +1469,7 @@ export type UserPublic = { is_superuser?: boolean full_name?: string | null id: string - github_username: string + github_username: string | null subscription: UserSubscription | null } @@ -2081,6 +2118,35 @@ export type DeleteProjectCollaboratorData = { export type DeleteProjectCollaboratorResponse = Message +export type PostProjectInvitationData = { + ownerName: string + projectName: string + requestBody: ProjectInvitationPost +} + +export type PostProjectInvitationResponse = ProjectInvitationCreated + +export type GetProjectInvitationsData = { + ownerName: string + projectName: string +} + +export type GetProjectInvitationsResponse = Array + +export type DeleteProjectInvitationData = { + invitationId: string + ownerName: string + projectName: string +} + +export type DeleteProjectInvitationResponse = Message + +export type PostProjectInvitationRedemptionData = { + token: string +} + +export type PostProjectInvitationRedemptionResponse = ProjectInvitationRedeemed + export type GetProjectIssuesData = { ownerName: string page?: number diff --git a/frontend/src/components/Projects/InviteLinks.tsx b/frontend/src/components/Projects/InviteLinks.tsx new file mode 100644 index 00000000..2f1fa71c --- /dev/null +++ b/frontend/src/components/Projects/InviteLinks.tsx @@ -0,0 +1,343 @@ +import { + Badge, + Box, + Button, + Code, + Flex, + FormControl, + FormLabel, + HStack, + Heading, + IconButton, + Input, + Modal, + ModalBody, + ModalCloseButton, + ModalContent, + ModalFooter, + ModalHeader, + ModalOverlay, + Select, + SkeletonText, + Table, + TableContainer, + Tbody, + Td, + Text, + Th, + Thead, + Tr, + useDisclosure, +} from "@chakra-ui/react" +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query" +import { useState } from "react" +import { type SubmitHandler, useForm } from "react-hook-form" +import { FiCopy, FiTrash } from "react-icons/fi" + +import { + type ProjectInvitationCreated, + type ProjectInvitationPost, + ProjectsService, +} from "../../client" +import type { ApiError } from "../../client/core/ApiError" +import useCustomToast from "../../hooks/useCustomToast" +import { handleError } from "../../lib/errors" + +interface InviteLinksProps { + ownerName: string + projectName: string +} + +interface CreateInviteForm { + role: "read" | "write" | "admin" + expires_days: string + max_uses: string +} + +function invitationStatus(invite: { + revoked: boolean + expires: string | null + max_uses: number | null + use_count: number +}): { label: string; color: string } { + if (invite.revoked) { + return { label: "Revoked", color: "red" } + } + if (invite.expires && new Date(invite.expires) < new Date()) { + return { label: "Expired", color: "gray" } + } + if (invite.max_uses !== null && invite.use_count >= invite.max_uses) { + return { label: "Used up", color: "gray" } + } + return { label: "Active", color: "green" } +} + +const CreateInviteModal = ({ + ownerName, + projectName, + isOpen, + onClose, + onCreated, +}: InviteLinksProps & { + isOpen: boolean + onClose: () => void + onCreated: (invite: ProjectInvitationCreated) => void +}) => { + const queryClient = useQueryClient() + const showToast = useCustomToast() + const { + register, + handleSubmit, + reset, + formState: { errors, isSubmitting }, + } = useForm({ + mode: "onBlur", + defaultValues: { role: "write", expires_days: "", max_uses: "" }, + }) + + const mutation = useMutation({ + mutationFn: (data: CreateInviteForm) => { + const requestBody: ProjectInvitationPost = { + role: data.role, + expires_days: data.expires_days ? Number(data.expires_days) : null, + max_uses: data.max_uses ? Number(data.max_uses) : null, + } + return ProjectsService.postProjectInvitation({ + ownerName, + projectName, + requestBody, + }) + }, + onSuccess: (invite) => { + showToast("Success!", "Invite link created.", "success") + reset() + onClose() + onCreated(invite) + }, + onError: (err: ApiError) => { + handleError(err, showToast) + }, + onSettled: () => { + queryClient.invalidateQueries({ + queryKey: ["projects", ownerName, projectName, "invitations"], + }) + }, + }) + + const onSubmit: SubmitHandler = (data) => { + mutation.mutate(data) + } + + return ( + + + + Create invite link + + + + Access level + + + + Expires in (days) + + + + Max uses + + + + + + + + + + ) +} + +const InviteLinks = ({ ownerName, projectName }: InviteLinksProps) => { + const queryClient = useQueryClient() + const showToast = useCustomToast() + const createModal = useDisclosure() + const [created, setCreated] = useState(null) + const { isPending, data: invitations } = useQuery({ + queryKey: ["projects", ownerName, projectName, "invitations"], + queryFn: () => + ProjectsService.getProjectInvitations({ ownerName, projectName }), + }) + + const revokeMutation = useMutation({ + mutationFn: (invitationId: string) => + ProjectsService.deleteProjectInvitation({ + ownerName, + projectName, + invitationId, + }), + onSuccess: () => { + showToast("Success!", "Invite link revoked.", "success") + }, + onError: (err: ApiError) => { + handleError(err, showToast) + }, + onSettled: () => { + queryClient.invalidateQueries({ + queryKey: ["projects", ownerName, projectName, "invitations"], + }) + }, + }) + + const copyLink = (url: string) => { + navigator.clipboard.writeText(url) + showToast("Copied", "Invite link copied to clipboard.", "success") + } + + return ( + + + Invite links + + + + Share a link to let people join this project — including collaborators + without a GitHub account. + + {created && ( + + + New invite link (copy it now — it won't be shown again): + + + + {created.url} + + } + onClick={() => copyLink(created.url)} + /> + + + )} + + + + + + + + + + + + {isPending ? ( + + + {new Array(5).fill(null).map((_, index) => ( + + ))} + + + ) : ( + + {invitations?.length ? ( + invitations.map((invite) => { + const status = invitationStatus(invite) + return ( + + + + + + + + ) + }) + ) : ( + + + + )} + + )} +
AccessStatusUsesExpiresActions
+ +
{invite.role_name} + {status.label} + + {invite.use_count} + {invite.max_uses !== null + ? ` / ${invite.max_uses}` + : ""} + + {invite.expires + ? new Date(invite.expires).toLocaleDateString() + : "Never"} + + } + variant="ghost" + color="ui.danger" + isDisabled={invite.revoked} + isLoading={ + revokeMutation.isPending && + revokeMutation.variables === invite.id + } + onClick={() => revokeMutation.mutate(invite.id)} + /> +
+ No invite links yet. +
+ + + + ) +} + +export default InviteLinks diff --git a/frontend/src/components/Publications/LatexEditor.tsx b/frontend/src/components/Publications/LatexEditor.tsx new file mode 100644 index 00000000..7283569b --- /dev/null +++ b/frontend/src/components/Publications/LatexEditor.tsx @@ -0,0 +1,558 @@ +import { + Badge, + Box, + Button, + Collapse, + Flex, + FormLabel, + HStack, + Input, + Modal, + ModalBody, + ModalCloseButton, + ModalContent, + ModalFooter, + ModalHeader, + ModalOverlay, + Spinner, + Switch, + Text, + VStack, + useDisclosure, +} from "@chakra-ui/react" +import { StreamLanguage } from "@codemirror/language" +import { stex } from "@codemirror/legacy-modes/mode/stex" +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query" +import { EditorView, basicSetup } from "codemirror" +import { type MutableRefObject, useEffect, useRef, useState } from "react" + +import { ProjectsService } from "../../client" +import type { ApiError } from "../../client/core/ApiError" +import useCustomToast from "../../hooks/useCustomToast" +import { + LatexCompiler, + type LatexFile, + findMissingPackages, +} from "../../lib/latexCompiler" +import { loadLatexProject } from "../../lib/latexProject" +import { handleError } from "../../lib/errors" +import PdfDocumentViewer from "../Common/PdfDocumentViewer" + +interface LatexEditorProps { + isOpen: boolean + onClose: () => void + ownerName: string + projectName: string + texPath: string + // The publication's pipeline-stage deps, if any — used to load figures and + // other inputs that live outside the .tex's own directory. + deps?: string[] | null +} + +// Display a repo path relative to the main file's directory, surfacing `../` +// for files that live above the paper directory. +function relativeTo(fromDir: string, to: string): string { + const fromParts = fromDir ? fromDir.split("/") : [] + const toParts = to.split("/") + let i = 0 + while ( + i < fromParts.length && + i < toParts.length - 1 && + fromParts[i] === toParts[i] + ) { + i++ + } + return "../".repeat(fromParts.length - i) + toParts.slice(i).join("/") +} + +function EditorPane({ + initialDoc, + viewRef, + onChange, +}: { + initialDoc: string + viewRef: MutableRefObject + onChange: (text: string) => void +}) { + const ref = useRef(null) + useEffect(() => { + if (!ref.current) { + return + } + const view = new EditorView({ + doc: initialDoc, + extensions: [ + basicSetup, + StreamLanguage.define(stex), + EditorView.lineWrapping, + EditorView.updateListener.of((u) => { + if (u.docChanged) { + onChange(u.state.doc.toString()) + } + }), + ], + parent: ref.current, + }) + viewRef.current = view + return () => { + view.destroy() + viewRef.current = null + } + }, []) + return +} + +const LatexEditor = ({ + isOpen, + onClose, + ownerName, + projectName, + texPath, + deps, +}: LatexEditorProps) => { + // Files are keyed by their full repo path so relative refs (e.g. + // \includegraphics{../figures/x.png}) resolve against the real layout. + const viewRef = useRef(null) + const compilerRef = useRef(null) + const buffersRef = useRef>(new Map()) + const binariesRef = useRef>(new Map()) + const initializedRef = useRef(false) + const compilingRef = useRef(false) + const pendingCompileRef = useRef(false) + const compileTimerRef = useRef(null) + const showToast = useCustomToast() + const queryClient = useQueryClient() + const logPanel = useDisclosure() + const commitModal = useDisclosure() + const [textPaths, setTextPaths] = useState([]) + const [activePath, setActivePath] = useState(texPath) + const [mainPath, setMainPath] = useState(texPath) + const [ready, setReady] = useState(false) + const [dirty, setDirty] = useState>(new Set()) + const dirtyRef = useRef(dirty) + dirtyRef.current = dirty + const [log, setLog] = useState("") + const [status, setStatus] = useState("") + const [pdfUrl, setPdfUrl] = useState(null) + const [compiling, setCompiling] = useState(false) + const [autoCompile, setAutoCompile] = useState(true) + const [commitMessage, setCommitMessage] = useState("") + + const { data: projectFiles } = useQuery({ + queryKey: ["projects", ownerName, projectName, "latex-project", texPath], + queryFn: () => loadLatexProject(ownerName, projectName, texPath, deps), + enabled: isOpen, + staleTime: 0, + }) + + useEffect(() => { + if (!projectFiles || initializedRef.current) { + return + } + initializedRef.current = true + const texts: string[] = [] + for (const f of projectFiles) { + if (f.kind === "text") { + buffersRef.current.set(f.path, f.text ?? "") + texts.push(f.path) + } else if (f.bytes) { + binariesRef.current.set(f.path, f.bytes) + } + } + texts.sort() + // Resolve the main file robustly: prefer the publication's path if it is a + // real root document (has both \documentclass and \begin{document}, so a + // short stub/wrapper isn't picked), else the first such .tex. + const looksMain = (p: string) => { + const c = buffersRef.current.get(p) ?? "" + return c.includes("\\documentclass") && c.includes("\\begin{document}") + } + let main = texPath + if (!texts.includes(main) || !looksMain(main)) { + main = texts.find(looksMain) ?? texts[0] ?? texPath + } + setTextPaths(texts) + setMainPath(main) + setActivePath(main) + setReady(true) + }, [projectFiles, texPath]) + + useEffect(() => { + return () => { + if (compileTimerRef.current) { + window.clearTimeout(compileTimerRef.current) + } + compilerRef.current?.terminate() + compilerRef.current = null + if (pdfUrl) { + URL.revokeObjectURL(pdfUrl) + } + } + }, []) + + const compile = async () => { + // Serialize compiles; if one is requested while another runs, recompile + // once it finishes (so the preview reflects the latest edits). + if (compilingRef.current) { + pendingCompileRef.current = true + return + } + compilingRef.current = true + setCompiling(true) + setLog("") + setStatus("Loading engine & compiling…") + try { + if (!compilerRef.current) { + compilerRef.current = new LatexCompiler({ + onLog: (line) => setLog((l) => `${l}${line}\n`), + }) + } + const files: LatexFile[] = [] + for (const [path, text] of buffersRef.current) { + files.push({ path, contents: text }) + } + for (const [path, bytes] of binariesRef.current) { + files.push({ path, contents: bytes }) + } + const result = await compilerRef.current.compile(files, mainPath) + // exit_code can be 0 with an empty PDF (busytex returns an empty array + // when no PDF was written) — treat that as a failure, not a blank preview. + if (result.exitCode === 0 && result.pdf && result.pdf.byteLength > 0) { + const blob = new Blob([result.pdf], { type: "application/pdf" }) + setPdfUrl((prev) => { + if (prev) { + URL.revokeObjectURL(prev) + } + return URL.createObjectURL(blob) + }) + setStatus("Compiled ✓") + } else { + const missing = findMissingPackages(result.log) + setStatus( + missing.length > 0 + ? `Missing from the in-browser TeX bundle: ${missing.join(", ")}` + : result.exitCode === 0 + ? "Compiled, but no PDF was produced — see log" + : `Compile failed (exit ${result.exitCode})`, + ) + setLog(result.log) + logPanel.onOpen() + } + } catch (e) { + setStatus("Engine error") + setLog(String(e)) + logPanel.onOpen() + } finally { + compilingRef.current = false + setCompiling(false) + if (pendingCompileRef.current) { + pendingCompileRef.current = false + compile() + } + } + } + + const scheduleCompile = () => { + if (!autoCompile) { + return + } + if (compileTimerRef.current) { + window.clearTimeout(compileTimerRef.current) + } + compileTimerRef.current = window.setTimeout(() => compile(), 1500) + } + + const markDirty = (path: string, text: string) => { + buffersRef.current.set(path, text) + setDirty((d) => (d.has(path) ? d : new Set(d).add(path))) + scheduleCompile() + } + + const saveMutation = useMutation({ + mutationFn: async (message: string) => { + for (const repoPath of dirtyRef.current) { + const text = buffersRef.current.get(repoPath) ?? "" + const file = new File([text], repoPath.split("/").pop() || repoPath, { + type: "text/plain", + }) + await ProjectsService.putProjectContents({ + ownerName, + projectName, + path: repoPath, + contentLength: file.size, + formData: { file, message: message || null }, + }) + } + }, + onSuccess: () => { + setDirty(new Set()) + setCommitMessage("") + commitModal.onClose() + showToast("Saved", "Your changes were committed.", "success") + queryClient.invalidateQueries({ + queryKey: ["projects", ownerName, projectName], + }) + }, + onError: (err: ApiError) => { + handleError(err, showToast) + }, + }) + + // Ctrl/Cmd+S (and the Save button) ask for a commit message before saving. + const requestSave = () => { + if (dirtyRef.current.size > 0) { + commitModal.onOpen() + } + } + + useEffect(() => { + if (ready && autoCompile) { + compile() + } + // Compile once on launch when auto-compile is on. + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [ready]) + + useEffect(() => { + if (!isOpen) { + return + } + const handler = (e: KeyboardEvent) => { + if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === "s") { + e.preventDefault() + requestSave() + } + } + document.addEventListener("keydown", handler, true) + return () => document.removeEventListener("keydown", handler, true) + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [isOpen]) + + const handleClose = () => { + if (dirty.size > 0 && !window.confirm("Discard unsaved changes?")) { + return + } + onClose() + } + + const mainDir = mainPath.includes("/") + ? mainPath.slice(0, mainPath.lastIndexOf("/")) + : "" + const displayPath = (p: string) => relativeTo(mainDir, p) + + return ( + <> + + + + + {displayPath(activePath || mainPath)} + {dirty.size > 0 && ( + + {dirty.size} unsaved + + )} + + + setAutoCompile(e.target.checked)} + /> + + Auto + + + + + + {status} + + + + Draft preview — not the published PDF + + + + + {!ready ? ( + + + + ) : ( + + + + Files + + + {textPaths.map((p) => ( + + ))} + {[...binariesRef.current.keys()].sort().map((p) => ( + + {displayPath(p)} + + ))} + + + + markDirty(activePath, text)} + /> + + + {pdfUrl ? ( + + ) : ( + + No preview yet. + + Click "Compile preview" to render the PDF. + + + )} + + + + {log || "(no output)"} + + + + + )} + + + + + + { + e.preventDefault() + saveMutation.mutate(commitMessage) + }} + > + Describe your change + + + setCommitMessage(e.target.value)} + placeholder="Ex: Add paragraph about the boundary conditions" + /> + + + + + + + + + ) +} + +export default LatexEditor diff --git a/frontend/src/lib/latexCompiler.ts b/frontend/src/lib/latexCompiler.ts new file mode 100644 index 00000000..c6767d91 --- /dev/null +++ b/frontend/src/lib/latexCompiler.ts @@ -0,0 +1,176 @@ +// Thin client-side wrapper around the (MIT) busytex WASM worker for in-browser +// LaTeX compilation. This is our own loader (Path 1) — no TeXlyre code. The +// engine binaries are served from VITE_TEX_ENGINE_URL (default same-origin +// /tex). Compilation is preview-only; see LATEX_EDITOR_PLAN.md §3.1. + +export interface LatexFile { + path: string + contents: string | Uint8Array | null +} + +export interface CompileResult { + pdf: Uint8Array | null + log: string + exitCode: number +} + +// busytex drivers: pdftex_bibtex8 | xetex_bibtex8_dvipdfmx | luahbtex_bibtex8 +const DRIVER = "pdftex_bibtex8" +// Eagerly loaded base filesystem. +const PRELOAD_PACKAGES = ["texlive-basic.js"] +// All available bundles; the engine resolves \usepackage names against these +// and loads the needed .data on demand. Bundles are a subset of full TeX Live; +// anything absent (e.g. sectsty, revtex) is fetched on demand from the texmf +// proxy by the patched engine when VITE_TEXMF_PROXY is set. See +// spikes/busytex-remote-fetch and LATEX_EDITOR_PLAN.md. +const DATA_PACKAGES = [ + "texlive-basic.js", + "ubuntu-texlive-latex-base.js", + "ubuntu-texlive-latex-recommended.js", + "ubuntu-texlive-latex-extra.js", + "ubuntu-texlive-science.js", + "ubuntu-texlive-fonts-recommended.js", +] + +const ENGINE_BASE = (import.meta.env.VITE_TEX_ENGINE_URL || "/tex").replace( + /\/$/, + "", +) + +// Bump when the engine binaries or worker glue change. The ~30 MB busytex.wasm +// is aggressively cached by the browser; without a version query, a rebuilt +// engine (e.g. the remote-fetch/font patches) won't be picked up until a hard +// refresh. Appended to the worker + engine URLs to bust the HTTP cache. +const ENGINE_VERSION = "2026-07-02-remote-fetch" +const V = `?v=${ENGINE_VERSION}` + +// Self-hosted texmf proxy. When set, the patched busytex engine fetches any TeX +// file missing from the bundled subset on demand (one compile, exact filenames), +// giving full TeX Live coverage. Empty => engine falls back to stock behaviour +// (missing package => friendly error). See spikes/busytex-remote-fetch. +const TEXMF_PROXY = (import.meta.env.VITE_TEXMF_PROXY || "").replace(/\/$/, "") + +type Pending = { + resolve: (r: CompileResult) => void + reject: (e: Error) => void +} + +// Pull "File `foo.sty' not found" package names out of a TeX log so the UI can +// explain that a package isn't in the in-browser bundle. +export function findMissingPackages(log: string): string[] { + const out = new Set() + const re = /File `([^']+\.(?:sty|cls))' not found/g + let m = re.exec(log) + while (m !== null) { + out.add(m[1]) + m = re.exec(log) + } + return [...out] +} + +export class LatexCompiler { + private worker: Worker | null = null + private ready: Promise | null = null + private pending: Pending | null = null + private onLog?: (line: string) => void + + constructor(opts: { onLog?: (line: string) => void } = {}) { + this.onLog = opts.onLog + } + + // Lazily spin up the worker and wait for the engine to initialize. + init(): Promise { + if (this.ready) { + return this.ready + } + this.ready = new Promise((resolve, reject) => { + const worker = new Worker(`${ENGINE_BASE}/busytex_worker.js${V}`) + this.worker = worker + worker.onmessage = ({ data }) => { + if (data.print !== undefined) { + this.onLog?.(data.print) + return + } + if (data.initialized !== undefined) { + resolve() + return + } + if (data.exception !== undefined) { + const err = new Error(data.exception) + if (this.pending) { + this.pending.reject(err) + this.pending = null + } else { + reject(err) + } + return + } + // Otherwise: a compile result. + if (this.pending) { + this.pending.resolve({ + pdf: data.pdf ?? null, + log: data.log ?? "", + exitCode: data.exit_code ?? -1, + }) + this.pending = null + } + } + worker.onerror = (e) => { + const err = new Error(`Engine worker error: ${e.message}`) + if (this.pending) { + this.pending.reject(err) + this.pending = null + } else { + reject(err) + } + } + // Data-package and wasm paths are resolved relative to the worker's + // location unless absolute, so pass absolute engine URLs. + worker.postMessage({ + busytex_wasm: `${ENGINE_BASE}/busytex.wasm${V}`, + busytex_js: `${ENGINE_BASE}/busytex.js${V}`, + preload_data_packages_js: PRELOAD_PACKAGES.map( + (p) => `${ENGINE_BASE}/${p}`, + ), + data_packages_js: DATA_PACKAGES.map((p) => `${ENGINE_BASE}/${p}`), + texmf_local: [], + preload: true, + calkit_texmf_endpoint: TEXMF_PROXY, + }) + }) + return this.ready + } + + async compile( + files: LatexFile[], + mainTexPath: string, + ): Promise { + await this.init() + if (this.pending) { + throw new Error("A compilation is already in progress") + } + return new Promise((resolve, reject) => { + this.pending = { resolve, reject } + this.worker?.postMessage({ + files, + main_tex_path: mainTexPath, + // Force a single pdflatex pass. The bibtex multi-pass reuses one WASM + // module instance, which makes pdftex assert on the 2nd run + // (pdfinitmapfile). Single-pass means a preview always renders; + // bibliography citations stay unresolved until we have a build that + // supports multi-pass (or per-pass module recreation). Preview-only. + bibtex: false, + verbose: "silent", + driver: DRIVER, + data_packages_js: DATA_PACKAGES.map((p) => `${ENGINE_BASE}/${p}`), + }) + }) + } + + terminate(): void { + this.worker?.terminate() + this.worker = null + this.ready = null + this.pending = null + } +} diff --git a/frontend/src/lib/latexProject.ts b/frontend/src/lib/latexProject.ts new file mode 100644 index 00000000..141d2704 --- /dev/null +++ b/frontend/src/lib/latexProject.ts @@ -0,0 +1,172 @@ +// Load the files a publication needs to compile in the browser. Driven by the +// publication's pipeline-stage deps when available (the authoritative list, +// including figures outside the .tex's own directory and DVC-tracked outputs), +// falling back to scanning the .tex's directory. Files are seeded at their full +// repo paths so relative refs like \includegraphics{../figures/x.png} resolve. +import { ProjectsService } from "../client" +import { decodeBase64Utf8 } from "./strings" + +const TEXT_EXT = new Set([ + "tex", + "bib", + "cls", + "sty", + "bst", + "bbl", + "ltx", + "def", + "clo", + "cfg", +]) +const IMG_EXT = new Set(["png", "jpg", "jpeg", "pdf", "eps", "gif"]) +const SKIP_PREFIXES = [".calkit/", ".git/", ".dvc/"] +const MAX_FILES = 150 +const MAX_DEPTH = 5 + +export interface ProjectFile { + path: string + kind: "text" | "binary" + text?: string + bytes?: Uint8Array +} + +function ext(p: string): string { + const i = p.lastIndexOf(".") + return i < 0 ? "" : p.slice(i + 1).toLowerCase() +} + +function relevant(name: string): boolean { + const e = ext(name) + return TEXT_EXT.has(e) || IMG_EXT.has(e) +} + +function base64ToBytes(b64: string): Uint8Array { + const bin = atob(b64) + const out = new Uint8Array(bin.length) + for (let i = 0; i < bin.length; i++) { + out[i] = bin.charCodeAt(i) + } + return out +} + +async function listDir( + ownerName: string, + projectName: string, + dir: string, + depth: number, + acc: Set, +): Promise { + if (depth > MAX_DEPTH || acc.size >= MAX_FILES) { + return + } + let res: Awaited> + try { + res = await ProjectsService.getProjectContents({ + ownerName, + projectName, + path: dir || undefined, + }) + } catch { + return + } + for (const item of res.dir_items ?? []) { + if (acc.size >= MAX_FILES) { + break + } + if (item.type === "dir") { + await listDir(ownerName, projectName, item.path, depth + 1, acc) + } else if (relevant(item.name)) { + acc.add(item.path) + } + } +} + +// Expand a pipeline-stage dep (a file or a directory) into concrete file paths. +async function expandDep( + ownerName: string, + projectName: string, + dep: string, + acc: Set, +): Promise { + if (SKIP_PREFIXES.some((p) => dep.startsWith(p)) || dep.startsWith(".")) { + return + } + let res: Awaited> + try { + res = await ProjectsService.getProjectContents({ + ownerName, + projectName, + path: dep, + }) + } catch { + return + } + if (res.type === "dir") { + await listDir(ownerName, projectName, dep, 0, acc) + } else if (relevant(dep)) { + acc.add(dep) + } +} + +async function fetchOne( + ownerName: string, + projectName: string, + path: string, +): Promise { + let res: Awaited> + try { + res = await ProjectsService.getProjectContents({ + ownerName, + projectName, + path, + }) + } catch { + return null + } + if (TEXT_EXT.has(ext(path))) { + // Files over the API's inline-content size limit come back as a signed + // URL with no `content` — fetch the text so large sources aren't empty. + let text = "" + if (res.content) { + text = decodeBase64Utf8(res.content) + } else if (res.url) { + text = await (await fetch(res.url)).text() + } + return { path, kind: "text", text } + } + let bytes: Uint8Array | null = null + if (res.content) { + bytes = base64ToBytes(res.content) + } else if (res.url) { + bytes = new Uint8Array(await (await fetch(res.url)).arrayBuffer()) + } + if (!bytes) { + return null + } + return { path, kind: "binary", bytes } +} + +export async function loadLatexProject( + ownerName: string, + projectName: string, + texPath: string, + deps?: string[] | null, +): Promise { + const paths = new Set([texPath]) + if (deps && deps.length > 0) { + for (const dep of deps) { + await expandDep(ownerName, projectName, dep, paths) + } + } else { + const dir = texPath.includes("/") + ? texPath.slice(0, texPath.lastIndexOf("/")) + : "" + await listDir(ownerName, projectName, dir, 0, paths) + } + const files = await Promise.all( + [...paths].map((p) => + fetchOne(ownerName, projectName, p).catch(() => null), + ), + ) + return files.filter((f): f is ProjectFile => f !== null) +} diff --git a/frontend/src/routeTree.gen.ts b/frontend/src/routeTree.gen.ts index d346055e..005c1afc 100644 --- a/frontend/src/routeTree.gen.ts +++ b/frontend/src/routeTree.gen.ts @@ -12,6 +12,7 @@ import { createFileRoute } from '@tanstack/react-router' import { Route as rootRouteImport } from './routes/__root' import { Route as ZenodoAuthRouteImport } from './routes/zenodo-auth' +import { Route as SignupRouteImport } from './routes/signup' import { Route as ResetPasswordRouteImport } from './routes/reset-password' import { Route as RecoverPasswordRouteImport } from './routes/recover-password' import { Route as GoogleAuthRouteImport } from './routes/google-auth' @@ -20,6 +21,7 @@ import { Route as LayoutRouteImport } from './routes/_layout' import { Route as LoginIndexRouteImport } from './routes/login/index' import { Route as LayoutIndexRouteImport } from './routes/_layout/index' import { Route as LoginDeviceRouteImport } from './routes/login/device' +import { Route as JoinTokenRouteImport } from './routes/join/$token' import { Route as LayoutSettingsRouteImport } from './routes/_layout/settings' import { Route as LayoutProjectsRouteImport } from './routes/_layout/projects' import { Route as LayoutOrgsRouteImport } from './routes/_layout/orgs' @@ -55,6 +57,11 @@ const ZenodoAuthRoute = ZenodoAuthRouteImport.update({ path: '/zenodo-auth', getParentRoute: () => rootRouteImport, } as any) +const SignupRoute = SignupRouteImport.update({ + id: '/signup', + path: '/signup', + getParentRoute: () => rootRouteImport, +} as any) const ResetPasswordRoute = ResetPasswordRouteImport.update({ id: '/reset-password', path: '/reset-password', @@ -94,6 +101,11 @@ const LoginDeviceRoute = LoginDeviceRouteImport.update({ path: '/login/device', getParentRoute: () => rootRouteImport, } as any) +const JoinTokenRoute = JoinTokenRouteImport.update({ + id: '/join/$token', + path: '/join/$token', + getParentRoute: () => rootRouteImport, +} as any) const LayoutSettingsRoute = LayoutSettingsRouteImport.update({ id: '/settings', path: '/settings', @@ -248,6 +260,7 @@ export interface FileRoutesByFullPath { '/google-auth': typeof GoogleAuthRoute '/recover-password': typeof RecoverPasswordRoute '/reset-password': typeof ResetPasswordRoute + '/signup': typeof SignupRoute '/zenodo-auth': typeof ZenodoAuthRoute '/admin': typeof LayoutAdminRoute '/datasets': typeof LayoutDatasetsRoute @@ -255,6 +268,7 @@ export interface FileRoutesByFullPath { '/orgs': typeof LayoutOrgsRoute '/projects': typeof LayoutProjectsRoute '/settings': typeof LayoutSettingsRoute + '/join/$token': typeof JoinTokenRoute '/login/device': typeof LoginDeviceRoute '/': typeof LayoutIndexRoute '/login': typeof LoginIndexRoute @@ -283,6 +297,7 @@ export interface FileRoutesByTo { '/google-auth': typeof GoogleAuthRoute '/recover-password': typeof RecoverPasswordRoute '/reset-password': typeof ResetPasswordRoute + '/signup': typeof SignupRoute '/zenodo-auth': typeof ZenodoAuthRoute '/admin': typeof LayoutAdminRoute '/datasets': typeof LayoutDatasetsRoute @@ -290,6 +305,7 @@ export interface FileRoutesByTo { '/orgs': typeof LayoutOrgsRoute '/projects': typeof LayoutProjectsRoute '/settings': typeof LayoutSettingsRoute + '/join/$token': typeof JoinTokenRoute '/login/device': typeof LoginDeviceRoute '/': typeof LayoutIndexRoute '/login': typeof LoginIndexRoute @@ -319,6 +335,7 @@ export interface FileRoutesById { '/google-auth': typeof GoogleAuthRoute '/recover-password': typeof RecoverPasswordRoute '/reset-password': typeof ResetPasswordRoute + '/signup': typeof SignupRoute '/zenodo-auth': typeof ZenodoAuthRoute '/_layout/admin': typeof LayoutAdminRoute '/_layout/datasets': typeof LayoutDatasetsRoute @@ -326,6 +343,7 @@ export interface FileRoutesById { '/_layout/orgs': typeof LayoutOrgsRoute '/_layout/projects': typeof LayoutProjectsRoute '/_layout/settings': typeof LayoutSettingsRoute + '/join/$token': typeof JoinTokenRoute '/login/device': typeof LoginDeviceRoute '/_layout/': typeof LayoutIndexRoute '/login/': typeof LoginIndexRoute @@ -357,6 +375,7 @@ export interface FileRouteTypes { | '/google-auth' | '/recover-password' | '/reset-password' + | '/signup' | '/zenodo-auth' | '/admin' | '/datasets' @@ -364,6 +383,7 @@ export interface FileRouteTypes { | '/orgs' | '/projects' | '/settings' + | '/join/$token' | '/login/device' | '/' | '/login' @@ -392,6 +412,7 @@ export interface FileRouteTypes { | '/google-auth' | '/recover-password' | '/reset-password' + | '/signup' | '/zenodo-auth' | '/admin' | '/datasets' @@ -399,6 +420,7 @@ export interface FileRouteTypes { | '/orgs' | '/projects' | '/settings' + | '/join/$token' | '/login/device' | '/' | '/login' @@ -427,6 +449,7 @@ export interface FileRouteTypes { | '/google-auth' | '/recover-password' | '/reset-password' + | '/signup' | '/zenodo-auth' | '/_layout/admin' | '/_layout/datasets' @@ -434,6 +457,7 @@ export interface FileRouteTypes { | '/_layout/orgs' | '/_layout/projects' | '/_layout/settings' + | '/join/$token' | '/login/device' | '/_layout/' | '/login/' @@ -465,7 +489,9 @@ export interface RootRouteChildren { GoogleAuthRoute: typeof GoogleAuthRoute RecoverPasswordRoute: typeof RecoverPasswordRoute ResetPasswordRoute: typeof ResetPasswordRoute + SignupRoute: typeof SignupRoute ZenodoAuthRoute: typeof ZenodoAuthRoute + JoinTokenRoute: typeof JoinTokenRoute LoginDeviceRoute: typeof LoginDeviceRoute LoginIndexRoute: typeof LoginIndexRoute } @@ -479,6 +505,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof ZenodoAuthRouteImport parentRoute: typeof rootRouteImport } + '/signup': { + id: '/signup' + path: '/signup' + fullPath: '/signup' + preLoaderRoute: typeof SignupRouteImport + parentRoute: typeof rootRouteImport + } '/reset-password': { id: '/reset-password' path: '/reset-password' @@ -535,6 +568,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof LoginDeviceRouteImport parentRoute: typeof rootRouteImport } + '/join/$token': { + id: '/join/$token' + path: '/join/$token' + fullPath: '/join/$token' + preLoaderRoute: typeof JoinTokenRouteImport + parentRoute: typeof rootRouteImport + } '/_layout/settings': { id: '/_layout/settings' path: '/settings' @@ -832,7 +872,9 @@ const rootRouteChildren: RootRouteChildren = { GoogleAuthRoute: GoogleAuthRoute, RecoverPasswordRoute: RecoverPasswordRoute, ResetPasswordRoute: ResetPasswordRoute, + SignupRoute: SignupRoute, ZenodoAuthRoute: ZenodoAuthRoute, + JoinTokenRoute: JoinTokenRoute, LoginDeviceRoute: LoginDeviceRoute, LoginIndexRoute: LoginIndexRoute, } diff --git a/frontend/src/routes/_layout.tsx b/frontend/src/routes/_layout.tsx index ef691eec..5672e7ee 100644 --- a/frontend/src/routes/_layout.tsx +++ b/frontend/src/routes/_layout.tsx @@ -1,15 +1,15 @@ -import { Flex, Box, Container, Link, Button } from "@chakra-ui/react" -import LoadingSpinner from "../components/Common/LoadingSpinner" -import { Outlet, createFileRoute } from "@tanstack/react-router" +import { Box, Button, Container, Flex, Link } from "@chakra-ui/react" import { useQuery } from "@tanstack/react-query" +import { Outlet, createFileRoute } from "@tanstack/react-router" import mixpanel from "mixpanel-browser" +import LoadingSpinner from "../components/Common/LoadingSpinner" -import useAuth from "../hooks/useAuth" +import { UsersService } from "../client" import Topbar from "../components/Common/Topbar" import PickSubscription from "../components/UserSettings/PickSubscription" -import { UsersService } from "../client" -import { appName } from "../lib/core" +import useAuth from "../hooks/useAuth" import { isAuthenticationError } from "../lib/auth" +import { appName } from "../lib/core" export const Route = createFileRoute("/_layout")({ component: Layout, @@ -38,12 +38,15 @@ function Layout() { $plan_name: user.subscription?.plan_name, }) } + // GitHub-less users (email/Google signups) don't have — and can't install — + // the GitHub App, so the install gate only applies to GitHub users. + const isGithubUser = Boolean(user?.github_username) const ghAppInstalledQuery = useQuery({ queryKey: ["user", "github-app-installations"], queryFn: () => UsersService.getUserGithubAppInstallations(), refetchOnWindowFocus: false, refetchOnMount: false, - enabled: Boolean(user), + enabled: Boolean(user) && isGithubUser, retry: (failureCount, error: any) => { if (isAuthenticationError(error)) return false return failureCount < 1 @@ -54,16 +57,18 @@ function Layout() { logout() } } - // Check that the user has at least one installation + // Check that GitHub users have at least one installation const ghAppNotInstalled = - user && ghAppInstalledQuery.data && !ghAppInstalledQuery.data.total_count + isGithubUser && + ghAppInstalledQuery.data && + !ghAppInstalledQuery.data.total_count if (ghAppNotInstalled) { location.href = `https://github.com/apps/${appName}/installations/new` } return ( - {isLoading || (user && ghAppInstalledQuery.isPending) ? ( + {isLoading || (isGithubUser && ghAppInstalledQuery.isPending) ? ( ) : ( <> diff --git a/frontend/src/routes/_layout/$accountName/$projectName/_layout/collaborators.tsx b/frontend/src/routes/_layout/$accountName/$projectName/_layout/collaborators.tsx index e1d62f67..ce56c40c 100644 --- a/frontend/src/routes/_layout/$accountName/$projectName/_layout/collaborators.tsx +++ b/frontend/src/routes/_layout/$accountName/$projectName/_layout/collaborators.tsx @@ -24,6 +24,7 @@ import { useQuery } from "@tanstack/react-query" import Navbar from "../../../../../components/Common/Navbar" import AddCollaborator from "../../../../../components/Projects/AddCollaborator" +import InviteLinks from "../../../../../components/Projects/InviteLinks" import { ProjectsService } from "../../../../../client" import useAuth from "../../../../../hooks/useAuth" import Delete from "../../../../../components/Common/DeleteAlert" @@ -154,6 +155,7 @@ function Collaborators() { )} + ) } diff --git a/frontend/src/routes/_layout/$accountName/$projectName/_layout/publications.tsx b/frontend/src/routes/_layout/$accountName/$projectName/_layout/publications.tsx index 5c29d091..9cf7ba5a 100644 --- a/frontend/src/routes/_layout/$accountName/$projectName/_layout/publications.tsx +++ b/frontend/src/routes/_layout/$accountName/$projectName/_layout/publications.tsx @@ -43,6 +43,7 @@ import CommentsPanel, { import LoadingSpinner from "../../../../../components/Common/LoadingSpinner" import PageMenu from "../../../../../components/Common/PageMenu" import ImportOverleaf from "../../../../../components/Publications/ImportOverleaf" +import LatexEditor from "../../../../../components/Publications/LatexEditor" import NewPublication from "../../../../../components/Publications/NewPublication" import PdfAnnotator, { commentToHighlight, @@ -89,6 +90,13 @@ function PubInfo({ const secBgColor = useColorModeValue("ui.secondary", "ui.darkSlate") const showToast = useCustomToast() const queryClient = useQueryClient() + const latexEditor = useDisclosure() + // Derive the LaTeX source path from the publication output path + // (e.g. paper.pdf -> paper.tex). Phase-1 heuristic; a later version can + // resolve the source from the publication's pipeline-stage deps. + const texPath = publication.path + ? publication.path.replace(/\.[^/.]+$/, ".tex") + : null const overleafSyncMutation = useMutation({ mutationFn: () => @@ -124,6 +132,29 @@ function PubInfo({ Info + {userHasWriteAccess && texPath && ( + <> + + {latexEditor.isOpen && ( + + )} + + )} Title: diff --git a/frontend/src/routes/join/$token.tsx b/frontend/src/routes/join/$token.tsx new file mode 100644 index 00000000..e5734ed7 --- /dev/null +++ b/frontend/src/routes/join/$token.tsx @@ -0,0 +1,70 @@ +import { Container, Spinner, Text, VStack } from "@chakra-ui/react" +import { useMutation } from "@tanstack/react-query" +import { createFileRoute, useNavigate } from "@tanstack/react-router" +import { useEffect, useRef } from "react" + +import { ProjectsService } from "../../client" +import type { ApiError } from "../../client/core/ApiError" +import { isLoggedIn } from "../../hooks/useAuth" +import useCustomToast from "../../hooks/useCustomToast" +import { handleError } from "../../lib/errors" + +export const Route = createFileRoute("/join/$token")({ + component: Join, +}) + +function Join() { + const { token } = Route.useParams() + const navigate = useNavigate() + const showToast = useCustomToast() + const ran = useRef(false) + + const mutation = useMutation({ + mutationFn: () => + ProjectsService.postProjectInvitationRedemption({ token }), + onSuccess: (data) => { + showToast( + "You're in!", + `You now have ${data.role_name} access to this project.`, + "success", + ) + navigate({ + to: "/$accountName/$projectName", + params: { + accountName: data.owner_name, + projectName: data.project_name, + }, + }) + }, + onError: (err: ApiError) => { + handleError(err, showToast) + }, + }) + + useEffect(() => { + if (ran.current) { + return + } + ran.current = true + if (!isLoggedIn()) { + // Send unauthenticated visitors to sign up, then return here to redeem. + localStorage.setItem("post_login_redirect", `/join/${token}`) + navigate({ to: "/signup" }) + return + } + mutation.mutate() + }, []) + + return ( + + + + + {mutation.isError + ? "This invite link is invalid or has expired." + : "Joining project…"} + + + + ) +} diff --git a/frontend/src/routes/login/index.tsx b/frontend/src/routes/login/index.tsx index fe8ac4fb..cd46734a 100644 --- a/frontend/src/routes/login/index.tsx +++ b/frontend/src/routes/login/index.tsx @@ -1,10 +1,26 @@ -import { Button, Container, Image, Text, Link } from "@chakra-ui/react" -import { createFileRoute, redirect } from "@tanstack/react-router" +import { + Button, + Container, + Divider, + FormControl, + FormErrorMessage, + HStack, + Image, + Input, + Link, + Text, +} from "@chakra-ui/react" +import { + Link as RouterLink, + createFileRoute, + redirect, +} from "@tanstack/react-router" -import { z } from "zod" -import { useEffect, useRef } from "react" import mixpanel from "mixpanel-browser" +import { useEffect, useRef } from "react" +import { type SubmitHandler, useForm } from "react-hook-form" import { FaGithub } from "react-icons/fa" +import { z } from "zod" import Logo from "/assets/images/calkit-no-bg.svg" import useAuth, { isLoggedIn } from "../../hooks/useAuth" @@ -34,10 +50,24 @@ function generateOAuthState(): string { return Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join("") } +interface EmailLoginForm { + username: string + password: string +} + function Login() { - const { loginGitHubMutation } = useAuth() + const { loginGitHubMutation, loginMutation, error, resetError } = useAuth() const { code: ghAuthCode, state: ghAuthStateRecv } = Route.useSearch() const isMounted = useRef(false) + const { + register, + handleSubmit, + formState: { isSubmitting }, + } = useForm({ mode: "onBlur" }) + const onEmailLogin: SubmitHandler = (data) => { + resetError() + loginMutation.mutate(data) + } const clientId = import.meta.env.VITE_GH_CLIENT_ID const getGitHubRedirectUri = () => { @@ -100,6 +130,43 @@ function Login() { > Sign in with GitHub + + + + or + + + +
+ + + + + + {error && {error}} + + +
+ + New to Calkit?{" "} + + Create an account + + Learn more diff --git a/frontend/src/routes/signup.tsx b/frontend/src/routes/signup.tsx new file mode 100644 index 00000000..71ae2ccd --- /dev/null +++ b/frontend/src/routes/signup.tsx @@ -0,0 +1,154 @@ +import { + Button, + Container, + FormControl, + FormErrorMessage, + FormLabel, + Image, + Input, + Link, + Text, +} from "@chakra-ui/react" +import { useMutation } from "@tanstack/react-query" +import { + Link as RouterLink, + createFileRoute, + redirect, + useNavigate, +} from "@tanstack/react-router" +import { type SubmitHandler, useForm } from "react-hook-form" + +import Logo from "/assets/images/calkit-no-bg.svg" +import { LoginService, UsersService } from "../client" +import type { ApiError } from "../client/core/ApiError" +import { isLoggedIn } from "../hooks/useAuth" +import useCustomToast from "../hooks/useCustomToast" +import { popPostLoginRedirect, storeTokens } from "../lib/auth" +import { handleError } from "../lib/errors" + +export const Route = createFileRoute("/signup")({ + component: SignUp, + beforeLoad: async () => { + if (isLoggedIn()) { + const stored = popPostLoginRedirect() + throw redirect({ to: stored || "/" }) + } + }, +}) + +interface SignUpForm { + full_name: string + email: string + password: string +} + +function SignUp() { + const navigate = useNavigate() + const showToast = useCustomToast() + const { + register, + handleSubmit, + formState: { errors, isSubmitting }, + } = useForm({ mode: "onBlur" }) + + const mutation = useMutation({ + mutationFn: async (data: SignUpForm) => { + await UsersService.registerUser({ + requestBody: { + email: data.email, + password: data.password, + full_name: data.full_name, + }, + }) + const resp = await LoginService.accessToken({ + formData: { username: data.email, password: data.password }, + }) + storeTokens(resp.access_token, resp.refresh_token) + }, + onSuccess: () => { + showToast( + "Welcome to Calkit!", + "Your account has been created.", + "success", + ) + const redirectTo = popPostLoginRedirect() + navigate({ to: redirectTo || "/" }) + }, + onError: (err: ApiError) => { + handleError(err, showToast) + }, + }) + + const onSubmit: SubmitHandler = (data) => { + mutation.mutate(data) + } + + return ( + + Logo + + Create your account + +
+ + Name + + {errors.full_name && ( + {errors.full_name.message} + )} + + + Email + + {errors.email && ( + {errors.email.message} + )} + + + Password + + {errors.password && ( + {errors.password.message} + )} + + +
+ + Already have an account?{" "} + + Sign in + + + + ) +} diff --git a/spikes/busytex-remote-fetch/README.md b/spikes/busytex-remote-fetch/README.md new file mode 100644 index 00000000..eab0b9b6 --- /dev/null +++ b/spikes/busytex-remote-fetch/README.md @@ -0,0 +1,121 @@ +# Scaffold: busytex + in-engine remote-fetch (kpathsea) hook + +The licensing-clean path from the package-proxy spike (`../latex-package-proxy/`): +patch **busytex** (MIT, TeX Live 2023) with our **own** kpathsea hook so the WASM +engine fetches missing `.cls`/`.sty`/etc. on demand from a self-hosted texmf +proxy — in **one** compile, with exact filenames and no log-parsing. SwiftLaTeX's +engine has this hook but is **AGPL-3.0**; this is a clean-room reimplementation, +so the result stays MIT. + +## Status: BUILT ✅ + +`build.sh` ran green under `emscripten/emsdk:3.1.43` and produced a working +engine — **`busytex.js` (~297 KB) + `busytex.wasm` (~30 MB)** — with our hook +embedded (`calkitTexmfEndpoint` / `__calkitCache` in the JS, `_malloc` / +`stringToUTF8` / `lengthBytesUTF8` / `UTF8ToString` exported). Remaining: +copy the binaries into `frontend/public/tex/`, set `Module.calkitTexmfEndpoint`, +and test end-to-end against boom-paper (single-pass on-demand fetch). + +## Design: pure-C kpathsea + EM_JS only in the engine + +The hook is split in two so **every** binary links, not just the engine: + +- **kpathsea stays pure C.** `apply_patch.py` adds, in `tex-file.c`: (1) the + call inside `kpathsea_find_file` on a local miss, and (2) a real, always-defined + `kpse_remote_fetch` that just delegates through a function pointer + `kpse_remote_fetch_hook` (default `NULL`). No EM_JS here. +- **The browser fetch (EM_JS) lives only in `busytex.c`** (`remote_fetch.c` is + appended to it), and a `constructor` installs it into the hook pointer at + engine startup. + +### Fonts: intercept `mktex*` before it forks + +Bitmap-font (PK) and metric (TFM) lookups do **not** go through +`kpathsea_find_file` — they use `kpathsea_path_search`, and on a miss kpathsea +tries to **generate** the file by forking a `mktexpk`/`mktextfm` script. WASM has +no `fork()`, so this is fatal (e.g. boom-paper's `tctt1000` TS1 font: +`Font tctt1000 at 600 not found`). So `apply_patch.py` also patches +`tex-make.c`: `kpathsea_make_tex()` first tries `kpse_remote_fetch()` for the +file the script *would* produce (`.pk`, `.tfm`, …) and only +falls back to the (doomed) fork if that misses. The **proxy** generates these on +demand — `proxy-server.py` runs `kpsewhich -mktex=pk -mktex=tfm`, where `fork()` +works. Verified: the previously-fatal `tctt1000.600pk` is fetched and the doc +compiles. + +Why: `EM_JS` makes a symbol a JS *import*. busytex builds ~6 **standalone applet +executables** (kpsewhich, bibtex8, …) whose link steps pull in `libkpathsea` and +**reject** a JS-import symbol. Keeping kpathsea pure C makes `kpse_remote_fetch` +a real defined wasm symbol everywhere; the applets get a NULL-pointer no-op +(stock behaviour), and only the engine carries the fetch. This was the crux — +an earlier `#include "remote_fetch.c"` into `busytex.c` (def only in the engine) +and a later EM_JS-in-kpathsea both failed applet links; the indirection fixes it. + +## Files + +| File | What it is | Status | +|---|---|---| +| `apply_patch.py` | kpathsea patch: `tex-file.c` (call site + pure-C indirection) **and** `tex-make.c` (font/metric gen → remote fetch). Takes the kpathsea dir. | **validated**, idempotent; produced the built engine | +| `remote_fetch.c` | engine-side EM_JS fetch + constructor (appended to `busytex.c`) | **built into the engine** | +| `build.sh` | clone busytex → download-native → unpack → patch → append hook → extend exports → `make wasm` | **ran green** | +| `tex-file.patch` | reference call-site diff (human-readable); `apply_patch.py` is authoritative | reference only | + +## How it works + +1. TeX asks kpathsea for a file (e.g. `revtex4-1.cls`). The normal local search + (`kpathsea_find_file` → `kpathsea_find_file_generic`) returns `NULL` because + it's not in the bundled texmf. +2. The patched `kpathsea_find_file` calls `kpse_remote_fetch(name, format)`, + which delegates to `kpse_remote_fetch_hook` (installed by the engine). +3. The engine hook (EM_JS) does a **synchronous** `XMLHttpRequest` to + `Module.calkitTexmfEndpoint + "/f/"`, writes the bytes into MEMFS at + `/calkit-remote/`, and returns that path. Sync XHR is allowed because + busytex runs in a Web Worker → **no Asyncify needed**. Hits and misses are + cached on `Module.__calkitCache` so `\IfFileExists` probes don't spam the + proxy. +4. kpathsea returns the path; TeX opens it and continues. One compile. + +## Build (needs the busytex toolchain; multi-hour TeX Live compile) + +```sh +# Inside the pinned Emscripten image (amd64): +docker run --rm --platform linux/amd64 \ + -v "$PWD:/scaffold:ro" -v /tmp/busytex-remote-build:/work \ + emscripten/emsdk:3.1.43 \ + bash -c 'SCAFFOLD=/scaffold /scaffold/build.sh /work' +# -> /tmp/busytex-remote-build/busytex/build/wasm/busytex.{js,wasm} +``` + +Notes learned the hard way: install `file` (needed by `make download-native`); +build via `make wasm` (not the final-link target alone — it needs the applet +prerequisites); and **redirect `make` output to a file** — piping its very large +output through `docker logs` can break the pipe (`make: write error: stdout`) +and abort an otherwise-successful build. + +## Wiring into the frontend (after a successful build) + +1. Replace the engine binaries in `frontend/public/tex/` with the patched + `busytex.{js,wasm}` (keep the worker/pipeline glue). +2. In `frontend/src/lib/latexCompiler.ts`, set the endpoint on the worker before + init, e.g. post `Module.calkitTexmfEndpoint = import.meta.env.VITE_TEXMF_PROXY` + (or have the worker assign it onto `Module`). +3. **Delete the iterative loop idea entirely** — missing files now resolve inside + a single `compile()`. The Phase-2 "Missing from the in-browser TeX bundle: X" + message becomes a rare fallback (proxy 404 / offline). + +## Texmf proxy (the server side) + +Reuse `../latex-package-proxy/proxy-server.py` (a `kpsewhich`-backed +`texlive/texlive` container, `GET /f/`). For production: host it ourselves, +CORS-restrict to the app origin, cache aggressively, pin a TeX Live version +(ideally 2023 to match the engine). Cheap and entirely ours. + +## Open items before this ships + +- Run the build; confirm the unpack target and that the EM_JS helpers + (`stringToUTF8`, `lengthBytesUTF8`) aren't dead-code-eliminated (the export + list in `build.sh` guards this). +- Verify on **boom-paper** end-to-end: expect a single compile, ~100 files + pulled once, then cached. +- Decide proxy hosting + a TeX Live version pin; add an HTTP cache (CDN) in front. +- Path-qualified lookups: the hook handles basenames only; check whether any + real lookups arrive path-qualified and extend if so. diff --git a/spikes/busytex-remote-fetch/apply_patch.py b/spikes/busytex-remote-fetch/apply_patch.py new file mode 100644 index 00000000..1ef24ab3 --- /dev/null +++ b/spikes/busytex-remote-fetch/apply_patch.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Patch kpathsea with the Calkit remote-fetch indirection (tex-file.c + tex-make.c). + +All edits idempotent. tex-file.c (file lookups): + + 1. In kpathsea_find_file(): on a local miss (ret == NULL), call + kpse_remote_fetch(). This is the single function TeX uses to locate + .tex/.sty/.cls/etc., so it covers the on-demand case in one compile. + + 2. Append a PURE-C definition of kpse_remote_fetch that just delegates through + a function pointer (kpse_remote_fetch_hook, default NULL). No EM_JS here. + +tex-make.c (font/metric generation): + + 3. In kpathsea_make_tex(): before forking a mktex* script (impossible under + WASM — no fork()), fetch the file it WOULD generate (e.g. tctt1000.600pk) + from the remote proxy, which runs a full TeX Live and can generate it. + PK/GF glyph lookups never hit kpathsea_find_file, so edit (1) can't cover + them; this does. + +Why pure C in kpathsea: EM_JS makes a symbol a JS import, and busytex builds +several standalone applet executables (kpsewhich, bibtex8, ...) whose link steps +reject a JS-import symbol pulled in via libkpathsea. Keeping kpathsea pure C +means kpse_remote_fetch is a real, defined wasm symbol everywhere; the actual +browser fetch (EM_JS) lives in busytex.c (remote_fetch.c) and is installed into +the hook pointer only in the engine. See README.md. + +More robust than a context diff: matches the functions and edits them, so it +survives line drift across TeX Live point releases. + +Usage: apply_patch.py path/to/texk/kpathsea (dir, or the tex-file.c inside it) +""" +import os +import sys + +FUNC_SIG = "kpathsea_find_file (kpathsea kpse, const_string name," +OLD = " string ret = *ret_list;\n free (ret_list);\n return ret;\n}" +NEW = ( + " string ret = *ret_list;\n" + " free (ret_list);\n" + " if (ret == NULL)\n" + " ret = kpse_remote_fetch (name, format);\n" + " return ret;\n}" +) +# Forward declaration, inserted before the function's return-type line so the +# call above compiles. +DECL = ( + "/* Calkit: remote-fetch fallback; defined at end of file (indirection) and\n" + " installed by the engine (remote_fetch.c -> busytex.c). */\n" + "extern string kpse_remote_fetch (const_string name,\n" + " kpse_file_format_type format);\n\n" +) +# Pure-C definition appended at end of file: delegate through a hook pointer so +# every binary linking libkpathsea resolves the symbol; the browser fetch is +# installed into the pointer by the engine only. +DEFN = """ +/* Calkit: remote-fetch indirection (see spikes/busytex-remote-fetch, MIT). + kpathsea stays pure C here so every binary that links libkpathsea.a — the + standalone applets (kpsewhich, bibtex8, ...) AND the busytex engine — resolves + kpse_remote_fetch as a real, defined wasm symbol. The actual browser fetch is + an EM_JS function in busytex.c, installed into this hook pointer at engine + startup; when the pointer is NULL (any standalone applet) the call is a + harmless no-op that preserves stock kpathsea behaviour. */ +char *(*kpse_remote_fetch_hook) (const char *name, int format) = 0; + +string +kpse_remote_fetch (const_string name, kpse_file_format_type format) +{ + if (kpse_remote_fetch_hook == 0) + return 0; + return kpse_remote_fetch_hook ((const char *) name, (int) format); +} +""" + + +# --- tex-make.c: intercept font/metric generation before it forks ------------ +MK_INC_OLD = "#include \n" +MK_INC_NEW = "#include \n#include \n" +MK_OLD = ( + " args[argnum++] = xstrdup(base);\n" + " args[argnum] = NULL;\n\n" + " ret = maketex (kpse, format, args);" +) +MK_NEW = ( + " args[argnum++] = xstrdup(base);\n" + " args[argnum] = NULL;\n\n" + " /* Calkit: mktex* scripts fork(), which WASM can't do, so the engine's\n" + " on-the-fly font/metric generation fails fatally. Before forking, try to\n" + " fetch the file the script WOULD generate from the remote texmf proxy (a\n" + " full TeX Live that can generate it). Reuses the kpse_remote_fetch hook\n" + " the engine installs; a NULL hook (standalone applets) just skips this. */\n" + " {\n" + " extern string kpse_remote_fetch (const_string name,\n" + " kpse_file_format_type format);\n" + " string ck_target = NULL;\n" + " if (format == kpse_pk_format || format == kpse_gf_format) {\n" + " string ck_dpi = kpathsea_var_value (kpse, \"KPATHSEA_DPI\");\n" + " if (ck_dpi) {\n" + " ck_target = concatn (base, \".\", ck_dpi,\n" + " format == kpse_pk_format ? \"pk\" : \"gf\",\n" + " (char *) NULL);\n" + " free (ck_dpi);\n" + " }\n" + " } else if (format == kpse_tfm_format) {\n" + " ck_target = concat (base, \".tfm\");\n" + " }\n" + " if (ck_target) {\n" + " string ck_fetched = kpse_remote_fetch (ck_target, format);\n" + " free (ck_target);\n" + " if (ck_fetched) {\n" + " for (argnum = 0; args[argnum] != NULL; argnum++)\n" + " free (args[argnum]);\n" + " free (args);\n" + " return ck_fetched;\n" + " }\n" + " }\n" + " }\n\n" + " ret = maketex (kpse, format, args);" +) + + +def patch_tex_file(path: str) -> int: + src = open(path).read() + if "kpse_remote_fetch" in src: + print(f"{path}: already patched") + return 0 + if FUNC_SIG not in src or OLD not in src: + print(f"ERROR: could not locate kpathsea_find_file in {path}", file=sys.stderr) + return 1 + marker = "\nstring\n" + FUNC_SIG + src = src.replace(marker, "\n" + DECL + "string\n" + FUNC_SIG, 1) + src = src.replace(OLD, NEW, 1) + src = src.rstrip("\n") + "\n" + DEFN + open(path, "w").write(src) + print(f"{path}: patched (call + pure-C indirection)") + return 0 + + +def patch_tex_make(path: str) -> int: + src = open(path).read() + if "kpse_remote_fetch" in src: + print(f"{path}: already patched") + return 0 + if MK_OLD not in src or MK_INC_OLD not in src: + print(f"ERROR: could not locate kpathsea_make_tex in {path}", file=sys.stderr) + return 1 + src = src.replace(MK_INC_OLD, MK_INC_NEW, 1) + src = src.replace(MK_OLD, MK_NEW, 1) + open(path, "w").write(src) + print(f"{path}: patched (font/metric make_tex -> remote fetch)") + return 0 + + +def main(arg: str) -> int: + d = arg if os.path.isdir(arg) else os.path.dirname(arg) + rc = patch_tex_file(os.path.join(d, "tex-file.c")) + if rc == 0: + rc = patch_tex_make(os.path.join(d, "tex-make.c")) + return rc + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1])) diff --git a/spikes/busytex-remote-fetch/build.sh b/spikes/busytex-remote-fetch/build.sh new file mode 100644 index 00000000..cc900ba9 --- /dev/null +++ b/spikes/busytex-remote-fetch/build.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +# Build a busytex WASM engine patched with the Calkit remote-fetch kpathsea hook. +# Produces build/wasm/busytex.{js,wasm} that fetches missing TeX files on demand +# from a texmf proxy (set Module.calkitTexmfEndpoint before init). +# +# STATUS: RAN GREEN. This is the exact sequence that produced a working +# busytex.js (~297 KB) + busytex.wasm (~30 MB) with our hook embedded, under +# emscripten/emsdk:3.1.43 (amd64). Design: kpathsea stays pure C with a NULL +# function pointer (apply_patch.py); the EM_JS browser fetch lives only in +# busytex.c (remote_fetch.c) and is installed into that pointer at engine +# startup — so every standalone applet (kpsewhich, bibtex8, ...) links cleanly. +# +# Toolchain: run inside `emscripten/emsdk:3.1.43`. Extra apt deps below. +# docker run --rm --platform linux/amd64 -v "$PWD:/scaffold:ro" -v /tmp/bt:/work \ +# emscripten/emsdk:3.1.43 bash -c 'SCAFFOLD=/scaffold /scaffold/build.sh /work' +set -euo pipefail +HERE="${SCAFFOLD:-$(cd "$(dirname "$0")" && pwd)}" +WORK="${1:-/tmp/busytex-remote-build}" +# Pinned busytex native-binary release (kpathsea/web2c helpers used during the +# WASM build). Matches the tree this recipe was validated against. +NATIVE="build_native_ff0318af379bd80fb72b9b928d4744b5d9c9077d_12853073565_1" + +mkdir -p "$WORK"; cd "$WORK" + +# 0. Build deps (Debian/emsdk image). `file` is needed by make download-native. +if command -v apt-get >/dev/null 2>&1; then + export DEBIAN_FRONTEND=noninteractive + apt-get update -qq + apt-get install -y -qq gperf p7zip-full icu-devtools wget git python3 ca-certificates file >/dev/null +fi + +# 1. busytex source (MIT). TeX Live 2023 is downloaded by its Makefile. +[ -d busytex ] || git clone --depth 1 https://github.com/busytex/busytex +cd busytex + +# 2. Prebuilt native helpers (ctangle/otangle/web2c/...), then unpack + prep the +# TeX Live source tree so kpathsea can be patched before it compiles. +echo "=== [1/4] download native binaries ===" +make URLRELEASE="https://github.com/busytex/busytex/releases/download/$NATIVE" download-native +echo "=== [2/4] fetch + prepare TeX Live source ===" +make source/texlive.txt build/versions.txt + +# 3. Patch kpathsea: tex-file.c (file lookups, pure-C indirection) AND +# tex-make.c (font/metric generation -> remote fetch instead of fork). +echo "=== [3/4] apply Calkit patch + engine hook ===" +python3 "$HERE/apply_patch.py" source/texlive/texk/kpathsea + +# 4. Append the EM_JS fetch + constructor to busytex.c (engine-only TU). +grep -q 'calkit_remote_fetch_js' busytex.c || { printf '\n'; cat "$HERE/remote_fetch.c"; } >> busytex.c + +# 5. Export the runtime helpers the EM_JS hook needs (FS is already exported). +python3 - <<'PY' +mk = open("Makefile").read() +mk = mk.replace( + '-sEXPORTED_FUNCTIONS=\'["_main", "_flush_streams"]\'', + '-sEXPORTED_FUNCTIONS=\'["_main", "_flush_streams", "_malloc"]\'', +) +mk = mk.replace( + '-sEXPORTED_RUNTIME_METHODS=\'["callMain", "FS", "ENV", "LZ4", "PATH"]\'', + '-sEXPORTED_RUNTIME_METHODS=\'["callMain", "FS", "ENV", "LZ4", "PATH", ' + '"stringToUTF8", "lengthBytesUTF8", "UTF8ToString"]\'', +) +open("Makefile", "w").write(mk) +print("Makefile: exports extended for the remote-fetch hook") +PY + +# 6. Build the wasm engine (long — compiles TeX Live). Redirect to a FILE: +# make floods stdout, and piping it through docker logs can break the pipe +# ('make: write error: stdout') and abort an otherwise-fine build. +echo "=== [4/4] make wasm (log: $WORK/busytex/build-wasm.log) ===" +make MAKEFLAGS=-j4 wasm > build-wasm.log 2>&1 + +ls -la build/wasm/busytex.js build/wasm/busytex.wasm +echo "Done. Engine: $WORK/busytex/build/wasm/busytex.{js,wasm}" +echo "Wire-up: set Module.calkitTexmfEndpoint to the texmf proxy before init;" +echo "no iterative loop needed — missing files resolve in a single compile." diff --git a/spikes/busytex-remote-fetch/remote_fetch.c b/spikes/busytex-remote-fetch/remote_fetch.c new file mode 100644 index 00000000..50555477 --- /dev/null +++ b/spikes/busytex-remote-fetch/remote_fetch.c @@ -0,0 +1,74 @@ +/* Calkit remote texmf fetch (clean-room, MIT) — ENGINE side. + * + * This snippet is appended to busytex's own busytex.c, so it is linked ONLY + * into the final busytex engine, never into the standalone applets (kpsewhich, + * bibtex8, ...). That matters: EM_JS defines a JS-import symbol, and several of + * busytex's standalone applet link steps reject an undefined/JS-import symbol + * pulled in transitively via libkpathsea. Keeping the EM_JS here — plus a + * pure-C indirection in kpathsea (see apply_patch.py) — means every binary + * links cleanly and only the engine carries the browser fetch. + * + * Mechanism: kpathsea, on a local miss, calls its kpse_remote_fetch(), which + * delegates through kpse_remote_fetch_hook. The constructor below installs + * calkit_remote_fetch_js into that hook at engine startup. The fetch is a + * *synchronous* XHR — busytex runs in a Web Worker where sync XHR is permitted, + * so no Emscripten Asyncify is required. Binary files are read byte-exact via + * the classic "x-user-defined" charset trick. Hits AND misses are memoised on + * Module.__calkitCache so a package's many \IfFileExists probes don't hammer + * the proxy. Clean-room reimplementation of the SwiftLaTeX technique; none of + * their (AGPL) code is used, so the result stays MIT. + * + * Set Module.calkitTexmfEndpoint (e.g. the latex-package-proxy) before init. + */ +#include + +extern char *(*kpse_remote_fetch_hook) (const char *name, int format); + +EM_JS(char *, calkit_remote_fetch_js, (const char *name_ptr, int format), { + var name = UTF8ToString(name_ptr); + /* Only basename lookups; reject path traversal. */ + if (!name || name.indexOf("..") >= 0 || name.indexOf("/") >= 0) return 0; + if (typeof Module === "undefined" || !Module.calkitTexmfEndpoint) return 0; + Module.__calkitCache = Module.__calkitCache || {}; + var cached = Module.__calkitCache[name]; + if (cached === null) return 0; /* known-missing: don't re-ask */ + var path; + if (cached === undefined) { + path = null; + try { + var ep = Module.calkitTexmfEndpoint; + if (ep.charAt(ep.length - 1) === "/") ep = ep.substr(0, ep.length - 1); + var url = ep + "/f/" + encodeURIComponent(name); + var xhr = new XMLHttpRequest(); + xhr.open("GET", url, false); /* synchronous */ + xhr.overrideMimeType("text/plain; charset=x-user-defined"); + xhr.send(); + if (xhr.status === 200) { + var s = xhr.responseText; + var bytes = new Uint8Array(s.length); + for (var i = 0; i < s.length; i++) bytes[i] = s.charCodeAt(i) & 0xff; + var dir = "/calkit-remote"; + try { FS.mkdir(dir); } catch (e) {} + path = dir + "/" + name; + FS.writeFile(path, bytes); + } + } catch (e) { + path = null; + } + Module.__calkitCache[name] = path; + } else { + path = cached; + } + if (!path) return 0; + var len = lengthBytesUTF8(path) + 1; + var ptr = _malloc(len); + stringToUTF8(path, ptr, len); + return ptr; /* TeX frees / owns this string */ +}); + +/* Install the hook into kpathsea at engine startup. */ +__attribute__((constructor)) +static void calkit_install_remote_fetch(void) +{ + kpse_remote_fetch_hook = calkit_remote_fetch_js; +} diff --git a/spikes/busytex-remote-fetch/tex-file.patch b/spikes/busytex-remote-fetch/tex-file.patch new file mode 100644 index 00000000..499faa7c --- /dev/null +++ b/spikes/busytex-remote-fetch/tex-file.patch @@ -0,0 +1,34 @@ +Calkit: remote-fetch fallback in kpathsea's public find-file entry point. + +When the normal local search returns NULL (file not in the bundled texmf), +fall back to kpse_remote_fetch() (remote_fetch.c), which pulls the file from +the texmf proxy into MEMFS and returns its path. This is the single function +TeX uses to locate .tex/.sty/.cls/etc., so it covers the on-demand case. + +Target: texlive-source (TeX Live 2023) texk/kpathsea/tex-file.c +Apply from the texlive source root: patch -p1 --fuzz=3 < tex-file.patch +(build.sh also ships a Python fallback that inserts by matching the function, +in case line offsets drift between TeX Live point releases.) + +--- a/texk/kpathsea/tex-file.c ++++ b/texk/kpathsea/tex-file.c +@@ -987,6 +987,12 @@ + thing for clients to call. */ + ++/* Calkit: defined in remote_fetch.c. On a local miss, fetch the file from the ++ remote texmf proxy into MEMFS and return its path (or NULL). */ ++extern string kpse_remote_fetch (const_string name, ++ kpse_file_format_type format); ++ + string + kpathsea_find_file (kpathsea kpse, const_string name, + kpse_file_format_type format, boolean must_exist) + { + string *ret_list = kpathsea_find_file_generic (kpse, name, format, + must_exist, false); + string ret = *ret_list; + free (ret_list); ++ if (ret == NULL) ++ ret = kpse_remote_fetch (name, format); + return ret; + } diff --git a/spikes/latex-package-proxy/README.md b/spikes/latex-package-proxy/README.md new file mode 100644 index 00000000..00bad19e --- /dev/null +++ b/spikes/latex-package-proxy/README.md @@ -0,0 +1,96 @@ +# Spike: LaTeX package proxy (on-demand TeX file fetching) + +**Question:** busytex only ships a *subset* of TeX Live, so real papers using +packages/classes it doesn't bundle (e.g. `sectsty`, and journal classes like +`aastex631`→`revtex4-1`) fail with no fix. Can we fetch missing files on demand +so they compile? + +**Answer: yes, the concept works — but the *delivery mechanism* matters.** + +## What's here + +- `proxy-server.py` — a **texmf file proxy** (the [Texlive-Ondemand](https://github.com/SwiftLaTeX/Texlive-Ondemand) + model). `GET /f/` resolves a file by name via `kpsewhich` inside a + full-TeX-Live container and returns its bytes; `GET /dir/` lists the + sibling files in that file's package directory. CORS-open. +- `run-spike.mjs` — drives the real busytex worker in a headless browser: + load `petebachant/boom-paper`, then loop **compile → scan the TeX log for + missing files → fetch them from the proxy → inject into the FS → recompile**. + +## Run it + +```sh +docker run -d --name tl-proxy --entrypoint /bin/sh texlive/texlive:latest-full -c "sleep infinity" +python3 proxy-server.py & # texmf proxy on :8771 +# dev frontend must be up (serves /tex busytex assets) +node run-spike.mjs # iterative compile against boom-paper +``` + +## Findings + +1. **The texmf proxy works perfectly.** A full TeX Live + `kpsewhich` resolves + any file by name (`revtex4-1.cls`, `sectsty.sty`, `pgfsys-pdftex.def`, …). + The public `texlive.swiftlatex.com` server is **dead** (DNS gone), so this + must be **self-hosted** — which is cheap (one container + a tiny endpoint). +2. **On-demand fetching resolves real, deep dependency trees.** Starting from + "aastex emulates revtex → no output", the iterative loop auto-fetched + **~94 files** — `revtex4-1.cls` + its 8 society `.rtx` files, then the entire + `tikz`/`pgf` core — driving boom-paper from zero output deep into a real + compile. So busytex *can* compile these papers once the files are present. +3. **The iterative log-parsing approach is the wrong delivery mechanism**, for + two reasons it surfaced concretely: + - **O(N) recompiles.** One recompile per missing file. boom-paper needs + 100+ files ⇒ 100+ recompiles ⇒ minutes. Unusable interactively. + - **Error-format whack-a-mole.** TeX announces missing files many ways: + `File \`x' not found`, `Driver file \`\`x''`, `\usepgflibrary{...}`, + interactive `Enter file name:` prompts, etc. Each needs bespoke parsing; + the loop got stuck on a pgf *library* file whose name isn't in the log. + +## Recommendation + +Adopt the proxy, but deliver files via an **in-engine kpathsea hook**, **not** +log-parsing: when kpathsea can't find a file, the WASM engine calls back to JS +with the **exact** filename + format; JS synchronously fetches it from the +**self-hosted texmf proxy** and returns it; the engine continues. **One compile, +exact names, no log parsing** — fixes both the O(N) recompiles and the +brittleness. (Sync-fetch-from-WASM is done via a synchronous XHR in the worker, +as SwiftLaTeX does, or Emscripten Asyncify.) + +Two ways to get that hook — assessed: + +### Path A — use SwiftLaTeX's engine (it already has the hook). ❌ Rejected. +- **AGPL-3.0 blocker.** SwiftLaTeX's LICENSE is GNU **AGPL-3.0**, and the + kpathsea-remote hook *is* their AGPL contribution. Using their engine in our + **MIT** editor triggers AGPL's network copyleft — a hosted SaaS built on it + must release its source under AGPL. Contradicts §0 (Path 1) and Calkit's MIT. +- Also older (**TeX Live 2020**), no npm package (WASM via GitHub releases, + last tagged 2022), uncertain maintenance. + +### Path B — patch busytex with our own MIT hook. ✅ Recommended. +- busytex is **MIT** (code) + permissive/LPPL TeX Live binaries, **TeX Live + 2023**, and already builds via a Makefile + Emscripten with source patches + (`fontconfig_emcc.patch`). We write a **clean-room** kpathsea patch (SwiftLaTeX + is a reference for the *technique*, not the code) → result stays MIT. +- **Work:** stand up the busytex build (Emscripten + cmake + p7zip, Docker, + multi-hour TeX Live compile) → add a focused kpathsea patch that, on a miss, + calls a JS callback that sync-fetches from our proxy into MEMFS and retries → + rebuild the ~30 MB wasm. The build infra is the main lift; the patch itself is + small and well-understood. +- **Pure-JS FS hooks are *not* enough:** kpathsea decides a file is absent + *before* `fopen` (it searches its db/dirs), so the search logic itself must be + patched — hence a rebuild, not just an Emscripten `FS` override. + +### Pragmatic fallback — server-side compile with full TeX Live. +For the heaviest papers, a **backend compile** (full TeX Live, no proxy needed — +the server has everything) is the simplest coverage. Cost: per-session server +compute (the thing WASM avoided), so keep it opt-in / for "export-quality" +builds. Dovetails with §3.1's "provenance-perfect builds with user compute". + +### The texmf proxy (either path / fallback) +`texlive/texlive` + a `kpsewhich`-backed endpoint, hosted by us (CORS-controlled, +cache aggressively, pin a TeX Live version). Cheap and ours. For Path B, expose +the protocol the patched engine expects (SwiftLaTeX uses `/pdftex//`). + +**Net:** coverage is solvable and the file source is trivial. The decision is +**patch busytex for an MIT in-engine hook** (Path B); SwiftLaTeX's engine is out +on AGPL. The remaining work is a one-time engine build — bounded, not open-ended. diff --git a/spikes/latex-package-proxy/proxy-server.py b/spikes/latex-package-proxy/proxy-server.py new file mode 100644 index 00000000..42421d51 --- /dev/null +++ b/spikes/latex-package-proxy/proxy-server.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Spike: a texmf file proxy backed by a full TeX Live (Texlive-Ondemand model). + +GET /f/ -> the file's bytes, resolved by name via `kpsewhich` inside a +long-lived `texlive/texlive` container (`tl-proxy`). 404 if not found. CORS open. + +This is the proof-of-concept for the package proxy: busytex (bundled, limited) +asks for a missing .cls/.sty/etc., we resolve it from a full TeX Live and hand +it back, then recompile. In production this would be a small hosted service. +""" +import subprocess +import urllib.parse +from http.server import BaseHTTPRequestHandler, HTTPServer + +CONTAINER = "tl-proxy" +PORT = 8771 + + +def kpse(name: str) -> bytes | None: + name = name.replace("/", "").replace("..", "") + try: + # -mktex=pk/tfm: generate bitmap fonts + metrics on demand. The WASM + # engine can't (mktexpk needs fork()), so it relies on us to produce + # them here (full TeX Live, fork works) and hand back the bytes. + path = subprocess.check_output( + ["docker", "exec", CONTAINER, "kpsewhich", + "-mktex=pk", "-mktex=tfm", name], + text=True, + timeout=60, + ).strip() + except subprocess.CalledProcessError: + return None + if not path: + return None + try: + return subprocess.check_output( + ["docker", "exec", CONTAINER, "cat", path], timeout=15 + ) + except subprocess.CalledProcessError: + return None + + +def pkg_dir_files(name: str) -> list[str] | None: + """All sibling filenames in the package directory that provides `name`.""" + name = name.replace("/", "").replace("..", "") + try: + path = subprocess.check_output( + ["docker", "exec", CONTAINER, "kpsewhich", name], + text=True, + timeout=15, + ).strip() + except subprocess.CalledProcessError: + return None + if not path: + return None + d = path.rsplit("/", 1)[0] + try: + out = subprocess.check_output( + ["docker", "exec", CONTAINER, "ls", d], text=True, timeout=15 + ) + except subprocess.CalledProcessError: + return None + return [f for f in out.split("\n") if f.strip()] + + +class Handler(BaseHTTPRequestHandler): + def do_GET(self): + # /dir/ -> JSON list of sibling files in 's package dir. + if self.path.startswith("/dir/"): + import json + + files = pkg_dir_files(urllib.parse.unquote(self.path[5:])) + self.send_response(200 if files is not None else 404) + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Content-Type", "application/json") + self.end_headers() + self.wfile.write(json.dumps(files or []).encode()) + return + if not self.path.startswith("/f/"): + self.send_response(404) + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + return + name = urllib.parse.unquote(self.path[3:]) + data = kpse(name) + if data is None: + self.send_response(404) + self.send_header("Access-Control-Allow-Origin", "*") + self.end_headers() + self.wfile.write(b"not found") + return + self.send_response(200) + self.send_header("Access-Control-Allow-Origin", "*") + self.send_header("Content-Type", "application/octet-stream") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def log_message(self, fmt, *args): + # Log every request so on-demand fetches are visible during testing. + print(f"{self.command} {self.path}", flush=True) + + +if __name__ == "__main__": + print(f"texmf proxy on http://127.0.0.1:{PORT}/f/") + HTTPServer(("127.0.0.1", PORT), Handler).serve_forever() diff --git a/spikes/latex-package-proxy/run-spike.mjs b/spikes/latex-package-proxy/run-spike.mjs new file mode 100644 index 00000000..4f27c1a4 --- /dev/null +++ b/spikes/latex-package-proxy/run-spike.mjs @@ -0,0 +1,206 @@ +// Spike: iterative package proxy. Load boom-paper, then loop: +// compile (busytex) -> detect missing .cls/.sty in the log -> fetch from the +// texmf proxy -> inject into the FS -> recompile, until it produces a PDF. +import pw from "/Users/pete/dev/calkit-cloud/frontend/node_modules/playwright-core/index.js" +const { chromium } = pw +const PROXY = "http://127.0.0.1:8771/f/" +const PROXY_DIR = "http://127.0.0.1:8771/dir/" +const ctx = await chromium.launchPersistentContext("/tmp/pp-prof", { + channel: "chrome", + headless: true, + args: ["--disable-web-security"], +}) +const page = await ctx.newPage() +await page.goto("http://localhost:5173/login", { waitUntil: "domcontentloaded" }) + +const result = await page.evaluate(async ({ PROXY, PROXY_DIR }) => { + const BASE = "/tex" + const DATA = [ + "texlive-basic.js", + "ubuntu-texlive-latex-base.js", + "ubuntu-texlive-latex-recommended.js", + "ubuntu-texlive-latex-extra.js", + "ubuntu-texlive-science.js", + "ubuntu-texlive-fonts-recommended.js", + ].map((p) => `${BASE}/${p}`) + const API = + "http://api.localhost/projects/petebachant/boom-paper/contents?path=" + const decode = (b) => + new TextDecoder().decode(Uint8Array.from(atob(b), (c) => c.charCodeAt(0))) + const toBytes = (b) => Uint8Array.from(atob(b), (c) => c.charCodeAt(0)) + const ext = (p) => { + const i = p.lastIndexOf(".") + return i < 0 ? "" : p.slice(i + 1).toLowerCase() + } + const TEXT = new Set(["tex", "bib", "cls", "sty", "bst", "def", "clo", "cfg"]) + const IMG = new Set(["png", "jpg", "jpeg", "pdf", "eps", "gif"]) + const get = async (p) => + (await fetch(API + encodeURIComponent(p))).json() + + const deps = [ + "paper/main.tex", + "paper/references.bib", + "paper/aasjournal.bst", + "paper/aastex631.cls", + "paper/results.tex", + "paper/diagrams", + "paper/figures", + ] + const files = [] + const seen = new Set() + async function addFile(p) { + if (seen.has(p)) return + seen.add(p) + const j = await get(p) + if (TEXT.has(ext(p))) { + const t = j.content + ? decode(j.content) + : j.url + ? await (await fetch(j.url)).text() + : "" + files.push({ path: p, contents: t }) + } else { + let bytes = null + if (j.content) bytes = toBytes(j.content) + else if (j.url) + bytes = new Uint8Array(await (await fetch(j.url)).arrayBuffer()) + if (bytes) files.push({ path: p, contents: bytes }) + } + } + for (const d of deps) { + if (d.startsWith(".")) continue + const j = await get(d) + if (j.type === "dir") { + for (const it of j.dir_items || []) + if (it.type !== "dir" && (TEXT.has(ext(it.name)) || IMG.has(ext(it.name)))) + await addFile(it.path) + } else if (TEXT.has(ext(d)) || IMG.has(ext(d))) await addFile(d) + } + const mainPath = "paper/main.tex" + const mainDir = "paper" + + const w = new Worker(`${BASE}/busytex_worker.js`) + await new Promise((res) => { + const h = ({ data }) => { + if (data.initialized !== undefined) { + w.removeEventListener("message", h) + res() + } + } + w.addEventListener("message", h) + w.postMessage({ + busytex_wasm: `${BASE}/busytex.wasm`, + busytex_js: `${BASE}/busytex.js`, + preload_data_packages_js: [`${BASE}/texlive-basic.js`], + data_packages_js: DATA, + texmf_local: [], + preload: true, + }) + }) + const compile = () => + new Promise((res) => { + const prints = [] + w.onmessage = ({ data }) => { + if (data.print !== undefined) { + prints.push(data.print) + return + } + if (data.initialized !== undefined) return + if (data.exception !== undefined) + res({ exception: String(data.exception).slice(0, 120), log: prints.join("\n") }) + else if (data.exit_code !== undefined) + res({ + exit: data.exit_code, + bytes: data.pdf ? data.pdf.byteLength : 0, + // The full TeX log is in the result (`data.log`); `print` messages + // are suppressed in silent mode. + log: data.log || prints.join("\n"), + }) + } + w.postMessage({ + files: [...files], + main_tex_path: mainPath, + bibtex: false, + verbose: "silent", + driver: "pdftex_bibtex8", + data_packages_js: DATA, + }) + }) + // Extract any TeX file token from lines that look like errors — TeX reports + // missing files in many formats (File `x' not found, Driver file ``x'', + // Enter file name, etc.), so be liberal rather than match each format. + const FILE_RE = + /[`'"( ]([\w.-]+\.(?:sty|cls|def|clo|cfg|rtx|fd|ldf|enc|tex))\b/g + function findMissing(log) { + const out = new Set() + // aastex-style "...include `revtex4-1.cls'" (a warning, no "not found") + const inc = /include `?([\w.-]+\.(?:cls|sty))'?/g + let m + while ((m = inc.exec(log)) !== null) out.add(m[1]) + // any file token on an error-ish line (covers "Driver file ``x''", etc.) + for (const line of log.split("\n")) { + if (!/not found|Error|Emergency|can't find|cannot/i.test(line)) continue + FILE_RE.lastIndex = 0 + let mm + while ((mm = FILE_RE.exec(line)) !== null) out.add(mm[1]) + } + // Never try to re-fetch the document's own files. + return [...out].filter( + (n) => !/^main\.(tex|log|aux|out|toc|bbl|blg|nav|snm)$/i.test(n), + ) + } + + const fetched = [] + const log = [] + let r + for (let iter = 1; iter <= 80; iter++) { + r = await compile() + if (r.bytes > 0) { + log.push(`iter ${iter}: SUCCESS, ${r.bytes} bytes`) + break + } + const missing = findMissing(r.log || "").filter((n) => !fetched.includes(n)) + if (missing.length === 0) { + log.push(`iter ${iter}: stuck (exit ${r.exit}, no new missing files)`) + break + } + const got = [] + const grab = async (n) => { + if (fetched.includes(n)) return + fetched.push(n) + try { + const resp = await fetch(PROXY + encodeURIComponent(n)) + if (resp.status === 200) { + files.push({ path: `${mainDir}/${n}`, contents: await resp.text() }) + got.push(n) + } + } catch {} + } + for (const name of missing) { + await grab(name) + // Deep deps: a class/style pulls in sibling files (e.g. revtex's .rtx + // society files) that TeX requests with formats my regex can't name. + if (/\.(cls|sty)$/i.test(name)) { + try { + const sibs = await ( + await fetch(`${PROXY_DIR}${encodeURIComponent(name)}`) + ).json() + for (const sib of sibs) + if (/\.(cls|sty|rtx|clo|def|cfg|fd|ltx)$/i.test(sib)) await grab(sib) + } catch {} + } + } + log.push(`iter ${iter}: missing [${missing.join(", ")}] -> fetched [${got.join(", ")}]`) + if (got.length === 0) break + } + return { + progress: log, + fetchedCount: fetched.length, + fetched, + final: { exit: r.exit, bytes: r.bytes, exception: r.exception }, + lastLogTail: (r.log || "").split("\n").filter((l) => l.trim()).slice(-12), + } +}, { PROXY, PROXY_DIR }) + +console.log(JSON.stringify(result, null, 2)) +await ctx.close() diff --git a/spikes/latex-wasm-busytex/.gitignore b/spikes/latex-wasm-busytex/.gitignore new file mode 100644 index 00000000..336fa8c3 --- /dev/null +++ b/spikes/latex-wasm-busytex/.gitignore @@ -0,0 +1,7 @@ +# Vendored busytex WASM engine + TeX Live data bundles — large binaries, not committed. +# Re-fetch with ./download-assets.sh (see README.md). +vendor/ + +# Generated by run-headless.mjs +out.pdf +out.png diff --git a/spikes/latex-wasm-busytex/README.md b/spikes/latex-wasm-busytex/README.md new file mode 100644 index 00000000..1c03c20c --- /dev/null +++ b/spikes/latex-wasm-busytex/README.md @@ -0,0 +1,57 @@ +# LaTeX → PDF in WASM — compile spike (§8.1) + +Throwaway spike for `LATEX_EDITOR_PLAN.md` Phase 0: prove that LaTeX compiles to PDF +entirely in the browser, and measure cold-start + compile time, before building the editor. + +## Engine & license (Path 1, MIT-clean) + +- Engine: **upstream `busytex/busytex`** WASM build — **TeX Live 2023**, emscripten 3.1.43. +- The busytex `.js` glue (`busytex_pipeline.js`, `busytex_worker.js`) is **MIT**; the + compiled `busytex.wasm` and `texlive-*.data` bundles carry TeX Live / LPPL (permissive) + licenses. This is clean to redistribute from an MIT project. +- We deliberately do **not** use TeXlyre's TeX Live 2026 build (`texlyre-busytex`), which is + **AGPL-3.0**. See `LATEX_EDITOR_PLAN.md` §0. +- `main.js` is **our own** thin loader around the MIT worker — no TeXlyre source is used. + +## Run it + +```sh +./download-assets.sh # ~135 MB from busytex GitHub releases (needs gh, authed) +node serve.mjs # http://localhost:8099 (sets COOP/COEP) +``` + +Open http://localhost:8099 and click **Compile sample**. The left pane streams the TeX log; +the right pane renders the produced PDF; the header shows cold-start / compile / total ms. + +## What it does + +- `vendor/busytex_worker.js` (MIT) runs the engine in a Web Worker. +- `main.js` initializes the pipeline, then compiles `sample/main.tex` with the + `pdftex_bibtex8` driver against the `texlive-basic` bundle. +- PDF bytes come back as a `Uint8Array` and are shown via a blob URL in an ` + + + + diff --git a/spikes/latex-wasm-busytex/main.js b/spikes/latex-wasm-busytex/main.js new file mode 100644 index 00000000..4c4dc075 --- /dev/null +++ b/spikes/latex-wasm-busytex/main.js @@ -0,0 +1,106 @@ +// Spike orchestrator: load the MIT busytex worker, compile sample/main.tex to PDF +// entirely in the browser, render it, and report timings. This is OUR OWN thin loader +// (Path 1) around the busytex MIT worker/pipeline glue — no TeXlyre code involved. + +const logEl = document.getElementById('log'); +const statusEl = document.getElementById('status'); +const metricsEl = document.getElementById('metrics'); +const frame = document.getElementById('pdf'); +const runBtn = document.getElementById('run'); + +const now = () => performance.now(); +const ms = (a, b) => `${Math.round(b - a)} ms`; + +function log(line) { + logEl.textContent += line + '\n'; + logEl.scrollTop = logEl.scrollHeight; +} +function setStatus(s) { statusEl.textContent = s; } + +// busytex driver options: pdftex_bibtex8 | xetex_bibtex8_dvipdfmx | luahbtex_bibtex8 | luatex_bibtex8 +const DRIVER = 'pdftex_bibtex8'; +const DATA_PACKAGES = ['texlive-basic.js']; // base TeX Live 2023 filesystem + +let worker; +let tWorkerStart, tInitialized, tCompileStart; + +async function main() { + runBtn.disabled = true; + logEl.textContent = ''; + metricsEl.textContent = ''; + frame.removeAttribute('src'); + setStatus('loading engine…'); + + const texSource = await (await fetch('sample/main.tex')).text(); + + tWorkerStart = now(); + // Worker lives in vendor/ so its importScripts('busytex_pipeline.js') and the + // bare asset filenames below all resolve relative to vendor/. + worker = new Worker('vendor/busytex_worker.js'); + + worker.onmessage = ({ data }) => { + if (data.print !== undefined) { log(data.print); return; } + + if (data.initialized !== undefined) { + tInitialized = now(); + setStatus('engine ready — compiling…'); + log(`\n=== engine initialized in ${ms(tWorkerStart, tInitialized)} ===`); + log('applet versions: ' + JSON.stringify(data.initialized) + '\n'); + tCompileStart = now(); + worker.postMessage({ + files: [{ path: 'main.tex', contents: texSource }], + main_tex_path: 'main.tex', + bibtex: null, // auto-detect; sample has no bibliography -> single pdflatex pass + verbose: 'silent', + driver: DRIVER, + data_packages_js: DATA_PACKAGES, + }); + return; + } + + if (data.exception !== undefined) { + setStatus('FAILED'); + log('\n!!! EXCEPTION:\n' + data.exception); + runBtn.disabled = false; + return; + } + + // Otherwise: the compile result {pdf, log, exit_code, logs} + const tDone = now(); + const ok = data.exit_code === 0 && data.pdf; + setStatus(ok ? 'compiled ✓' : `compile failed (exit ${data.exit_code})`); + metricsEl.innerHTML = [ + `engine cold-start: ${ms(tWorkerStart, tInitialized)}`, + `compile: ${ms(tCompileStart, tDone)}`, + `total: ${ms(tWorkerStart, tDone)}`, + `pdf size: ${data.pdf ? (data.pdf.byteLength / 1024).toFixed(1) + ' KB' : 'none'}`, + ].join('  |  '); + + if (ok) { + const blob = new Blob([data.pdf], { type: 'application/pdf' }); + frame.src = URL.createObjectURL(blob); + } else { + log('\n=== compile log ===\n' + (data.log || '(no log)')); + } + runBtn.disabled = false; + worker.terminate(); + }; + + worker.onerror = (e) => { + setStatus('worker error'); + log(`\n!!! worker error: ${e.message} @ ${e.filename}:${e.lineno}`); + runBtn.disabled = false; + }; + + // Initialize the pipeline (paths are relative to the worker's vendor/ dir). + worker.postMessage({ + busytex_wasm: 'busytex.wasm', + busytex_js: 'busytex.js', + preload_data_packages_js: DATA_PACKAGES, + data_packages_js: DATA_PACKAGES, + texmf_local: [], + preload: true, + }); +} + +runBtn.addEventListener('click', main); diff --git a/spikes/latex-wasm-busytex/run-headless.mjs b/spikes/latex-wasm-busytex/run-headless.mjs new file mode 100644 index 00000000..0769c6a0 --- /dev/null +++ b/spikes/latex-wasm-busytex/run-headless.mjs @@ -0,0 +1,51 @@ +// Headless driver for the spike: loads the page, clicks Compile, waits for the +// result, prints metrics + a PDF artifact. Uses the frontend's playwright + system Chrome. +import pw from '/Users/pete/dev/calkit-cloud/frontend/node_modules/playwright-core/index.js'; +import { writeFileSync } from 'node:fs'; +const { chromium } = pw; + +const PAGE_URL = process.env.URL || 'http://localhost:8099/'; +const browser = await chromium.launch({ channel: 'chrome', headless: true }); +const page = await browser.newPage(); +page.on('console', (m) => console.log(' [page]', m.text())); +page.on('pageerror', (e) => console.log(' [pageerror]', e.message)); + +await page.goto(PAGE_URL, { waitUntil: 'load' }); + +// Capture the compiled PDF bytes by hooking Blob URL creation isn't trivial; instead +// re-expose the last result from the worker via a global the page sets. +await page.evaluate(() => { window.__lastPdfLen = 0; }); + +await page.click('#run'); + +const deadline = Date.now() + 240_000; +let status = ''; +while (Date.now() < deadline) { + status = await page.textContent('#status'); + if (/compiled|failed|FAILED|error/.test(status)) break; + await page.waitForTimeout(500); +} + +const metrics = (await page.textContent('#metrics'))?.replace(/ /g, ' ').replace(/\s+/g, ' ').trim(); +console.log('\nSTATUS :', status); +console.log('METRICS:', metrics || '(none)'); + +// Pull the PDF from the iframe blob URL into the page and out to Node. +const pdfB64 = await page.evaluate(async () => { + const src = document.getElementById('pdf').getAttribute('src'); + if (!src) return null; + const buf = await (await fetch(src)).arrayBuffer(); + let bin = ''; const bytes = new Uint8Array(buf); + for (let i = 0; i < bytes.length; i++) bin += String.fromCharCode(bytes[i]); + return btoa(bin); +}); +if (pdfB64) { + writeFileSync('out.pdf', Buffer.from(pdfB64, 'base64')); + console.log('PDF : wrote out.pdf (' + (pdfB64.length * 0.75 / 1024).toFixed(1) + ' KB)'); +} +await page.screenshot({ path: 'out.png', fullPage: false }); + +await browser.close(); +const ok = /compiled/.test(status) && !!pdfB64; +console.log('\nRESULT :', ok ? 'PASS ✓' : 'FAIL ✗'); +process.exit(ok ? 0 : 1); diff --git a/spikes/latex-wasm-busytex/sample/main.tex b/spikes/latex-wasm-busytex/sample/main.tex new file mode 100644 index 00000000..e8f0bce9 --- /dev/null +++ b/spikes/latex-wasm-busytex/sample/main.tex @@ -0,0 +1,25 @@ +\documentclass{article} +\usepackage{amsmath} +\usepackage{graphicx} +\usepackage{hyperref} + +\title{Calkit LaTeX Editor --- WASM Compile Spike} +\author{busytex (TeX Live 2023) in the browser} +\date{\today} + +\begin{document} +\maketitle + +\section{It compiles} +This PDF was produced entirely client-side by the \texttt{busytex} WebAssembly +build of pdf\LaTeX{}, with no compile server. A representative equation: +\begin{equation} + \int_{0}^{\infty} e^{-x^2}\,\mathrm{d}x = \frac{\sqrt{\pi}}{2}. +\end{equation} + +\section{Why this spike exists} +To confirm in-browser LaTeX compilation is viable for the editor preview, and to +measure cold-start and compile time before building the editor UI. See +\href{https://example.invalid}{the plan} for context. + +\end{document} diff --git a/spikes/latex-wasm-busytex/serve.mjs b/spikes/latex-wasm-busytex/serve.mjs new file mode 100644 index 00000000..53f17b75 --- /dev/null +++ b/spikes/latex-wasm-busytex/serve.mjs @@ -0,0 +1,47 @@ +// Minimal static server for the spike. +// Sets COOP/COEP (cross-origin isolation) in case the emscripten build wants +// SharedArrayBuffer, and serves .wasm/.data with sane types. Node >= 18, no deps. +import { createServer } from 'node:http'; +import { readFile, stat } from 'node:fs/promises'; +import { extname, join, normalize } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = fileURLToPath(new URL('.', import.meta.url)); +const PORT = process.env.PORT ? Number(process.env.PORT) : 8099; + +const TYPES = { + '.html': 'text/html; charset=utf-8', + '.js': 'text/javascript; charset=utf-8', + '.mjs': 'text/javascript; charset=utf-8', + '.json': 'application/json', + '.wasm': 'application/wasm', + '.data': 'application/octet-stream', + '.tex': 'text/plain; charset=utf-8', + '.pdf': 'application/pdf', +}; + +createServer(async (req, res) => { + // Cross-origin isolation — harmless when unused, required if the engine uses threads. + res.setHeader('Cross-Origin-Opener-Policy', 'same-origin'); + res.setHeader('Cross-Origin-Embedder-Policy', 'require-corp'); + res.setHeader('Cross-Origin-Resource-Policy', 'same-origin'); + + try { + const urlPath = decodeURIComponent((req.url || '/').split('?')[0]); + const rel = normalize(urlPath === '/' ? '/index.html' : urlPath).replace(/^(\.\.[/\\])+/, ''); + const filePath = join(ROOT, rel); + if (!filePath.startsWith(ROOT)) { res.writeHead(403).end('forbidden'); return; } + + const info = await stat(filePath); + if (info.isDirectory()) { res.writeHead(403).end('forbidden'); return; } + + const body = await readFile(filePath); + res.setHeader('Content-Type', TYPES[extname(filePath)] || 'application/octet-stream'); + res.setHeader('Content-Length', info.size); + res.writeHead(200).end(body); + } catch (err) { + res.writeHead(err.code === 'ENOENT' ? 404 : 500).end(String(err)); + } +}).listen(PORT, () => { + console.log(`Spike server on http://localhost:${PORT} (Ctrl-C to stop)`); +});