Skip to content

Commit 218e248

Browse files
richardr1126richardr1126
authored
Merge pull request #90: Refactor compute architecture with workerized ONNX PDF/Whisper pipeline
- Monorepo Architecture: Separates processing into compute/core and compute/worker Fastify service. - NATS JetStream: Migrates processing queues to NATS JetStream with lazy connection and 120s idle sleep for Railway. - ONNX PDF Layout: Upgrades to PP-DocLayout-V3 with block overlay UI, force reparse, and block-level skip playback rules. - ONNX Whisper Alignment: Replaces whisper.cpp with a pure JS ONNX alignment decoder. - Onboarding Coordinator: Integrates OnboardingFlowContext to manage sequential modals (Privacy, Claim Guest Data, Dexie Cloud Migration). - Database Schema: Adds documentSettings table and tracks parsing stages (parseState, parsedJsonKey) in the documents table. - Error Contracts: Standardizes API endpoints on ServerAppError responses.
2 parents d90e48a + 3db193e commit 218e248

234 files changed

Lines changed: 27756 additions & 2119 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 48 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,10 @@
11

2+
# Logging
3+
# pretty (default): human-readable logs
4+
# json: structured logs for log platforms
5+
# LOG_FORMAT=pretty
6+
# LOG_LEVEL=info
7+
28
# Local / OpenAI TTS API Configuration (default)
39
# Suggest using https://github.com/remsky/Kokoro-FastAPI
410
#
@@ -77,22 +83,52 @@ RUN_FS_MIGRATIONS=
7783
IMPORT_LIBRARY_DIR=
7884
IMPORT_LIBRARY_DIRS=
7985

80-
# (Required without Docker) Path to your local whisper.cpp CLI binary for STT timestamp generation
81-
WHISPER_CPP_BIN=/whisper.cpp/build/bin/whisper-cli
86+
# Compute
87+
# Embedded/local (default): leave COMPUTE_WORKER_URL empty.
88+
# External worker: set COMPUTE_WORKER_URL + COMPUTE_WORKER_TOKEN.
89+
# External worker split:
90+
# - App side (this root `.env`): set only routing/auth + shared timeout/stale overrides.
91+
# - Worker side (`compute/worker/.env*` or worker platform env): set NATS_*, S3_*, model base URLs, and worker tuning.
92+
# Details: docs/deploy/compute-worker
93+
# COMPUTE_WORKER_URL=http://localhost:8081
94+
# COMPUTE_WORKER_TOKEN=local-compute-token
95+
# Optional embedded startup controls:
96+
# EMBEDDED_COMPUTE_WORKER_PORT=8081
97+
# EMBEDDED_NATS_PORT=4222
98+
# EMBEDDED_NATS_MONITOR_PORT=8222
99+
# EMBEDDED_NATS_STORE_DIR=docstore/nats/jetstream
100+
# `NATS_URL` here is for embedded startup only. In external worker mode, set `NATS_URL` on the worker service env.
101+
# NATS_URL=nats://127.0.0.1:4222
102+
# Optional worker log level:
103+
# COMPUTE_LOG_LEVEL=info
104+
# Optional shared compute tuning:
105+
# COMPUTE_JOB_CONCURRENCY=1
106+
# COMPUTE_WHISPER_TIMEOUT_MS=30000
107+
# COMPUTE_PDF_TIMEOUT_MS=300000
108+
# COMPUTE_OP_STALE_MS=1800000
109+
110+
# Optional Whisper ONNX base URL override (must contain all expected files)
111+
# Default expected Whisper variant is q4:
112+
# - onnx/encoder_model_q4.onnx
113+
# - onnx/decoder_model_merged_q4.onnx
114+
# - onnx/decoder_with_past_model_q4.onnx
115+
# WHISPER_MODEL_BASE_URL=https://huggingface.co/onnx-community/whisper-base_timestamped/resolve/main
116+
117+
# Optional PDF layout ONNX base URL override (must contain all expected files)
118+
# PDF_LAYOUT_MODEL_BASE_URL=https://huggingface.co/Bei0001/PP-DocLayoutV3-ONNX/resolve/main
82119

83120
# (Optional) Override ffmpeg binary path used for audiobook processing
84121
FFMPEG_BIN=
85122

86123
# (Optional) Client feature flags — seeded into the admin-managed runtime
87124
# config on first boot, then ignored. Edit values from Settings → Admin →
88125
# Site features instead of redeploying. SSR-injected so they take effect
89-
# without rebuilding (unlike the old NEXT_PUBLIC_* build-time pattern).
90-
# NEXT_PUBLIC_ENABLE_DOCX_CONVERSION=true
91-
# NEXT_PUBLIC_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=true
92-
# NEXT_PUBLIC_ENABLE_TTS_PROVIDERS_TAB=true
93-
# NEXT_PUBLIC_ENABLE_USER_SIGNUPS=true
94-
# NEXT_PUBLIC_RESTRICT_USER_API_KEYS=true
95-
# NEXT_PUBLIC_DEFAULT_TTS_PROVIDER=custom-openai
96-
# NEXT_PUBLIC_CHANGELOG_FEED_URL=https://docs.openreader.richardr.dev/changelog/manifest.json
97-
# NEXT_PUBLIC_ENABLE_AUDIOBOOK_EXPORT=true
98-
# NEXT_PUBLIC_ENABLE_WORD_HIGHLIGHT=true
126+
# without rebuilding (unlike the old build-time public-env pattern).
127+
# RUNTIME_SEED_ENABLE_DOCX_CONVERSION=true
128+
# RUNTIME_SEED_ENABLE_DESTRUCTIVE_DELETE_ACTIONS=true
129+
# RUNTIME_SEED_ENABLE_TTS_PROVIDERS_TAB=true
130+
# RUNTIME_SEED_ENABLE_USER_SIGNUPS=true
131+
# RUNTIME_SEED_RESTRICT_USER_API_KEYS=true
132+
# RUNTIME_SEED_DEFAULT_TTS_PROVIDER=custom-openai
133+
# RUNTIME_SEED_CHANGELOG_FEED_URL=https://docs.openreader.richardr.dev/changelog/manifest.json
134+
# RUNTIME_SEED_ENABLE_AUDIOBOOK_EXPORT=true

‎.github/workflows/docker-publish.yml‎

Lines changed: 87 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -22,30 +22,17 @@ jobs:
2222
prepare:
2323
runs-on: ubuntu-24.04
2424
outputs:
25-
image_name: ${{ steps.image-name.outputs.image_name }}
26-
legacy_image_name: ${{ steps.image-name.outputs.legacy_image_name }}
27-
tags: ${{ steps.meta.outputs.tags }}
28-
labels: ${{ steps.meta.outputs.labels }}
25+
web_image_name: ${{ steps.image-name.outputs.web_image_name }}
26+
web_legacy_image_name: ${{ steps.image-name.outputs.web_legacy_image_name }}
27+
compute_worker_image_name: ${{ steps.image-name.outputs.compute_worker_image_name }}
2928
steps:
3029
- name: Compute Docker image names
3130
id: image-name
3231
run: |
3332
owner="${GITHUB_REPOSITORY_OWNER,,}"
34-
echo "image_name=${owner}/openreader" >> "$GITHUB_OUTPUT"
35-
echo "legacy_image_name=${owner}/openreader-webui" >> "$GITHUB_OUTPUT"
36-
37-
- name: Extract metadata (tags, labels) for Docker
38-
id: meta
39-
uses: docker/metadata-action@v6
40-
with:
41-
images: |
42-
${{ env.REGISTRY }}/${{ steps.image-name.outputs.image_name }}
43-
${{ env.REGISTRY }}/${{ steps.image-name.outputs.legacy_image_name }}
44-
tags: |
45-
type=raw,value=latest,enable=${{ (github.event_name == 'push' && !contains(github.ref, '-pre')) || (github.event_name == 'workflow_dispatch' && inputs.use_latest_tag == true) }},priority=1000
46-
type=semver,pattern={{version}}
47-
type=ref,event=tag
48-
type=ref,event=branch,enable=${{ github.event_name == 'workflow_dispatch' && inputs.use_latest_tag != true }}
33+
echo "web_image_name=${owner}/openreader" >> "$GITHUB_OUTPUT"
34+
echo "web_legacy_image_name=${owner}/openreader-webui" >> "$GITHUB_OUTPUT"
35+
echo "compute_worker_image_name=${owner}/openreader-compute-worker" >> "$GITHUB_OUTPUT"
4936
5037
build:
5138
needs: prepare
@@ -58,12 +45,30 @@ jobs:
5845
fail-fast: false
5946
matrix:
6047
include:
61-
- arch: amd64
48+
- image_target: web
49+
arch: amd64
6250
platform: linux/amd64
6351
runner: ubuntu-24.04
64-
- arch: arm64
52+
context: .
53+
dockerfile: ./Dockerfile
54+
- image_target: web
55+
arch: arm64
6556
platform: linux/arm64
6657
runner: ubuntu-24.04-arm
58+
context: .
59+
dockerfile: ./Dockerfile
60+
- image_target: compute-worker
61+
arch: amd64
62+
platform: linux/amd64
63+
runner: ubuntu-24.04
64+
context: .
65+
dockerfile: ./compute/worker/Dockerfile
66+
- image_target: compute-worker
67+
arch: arm64
68+
platform: linux/arm64
69+
runner: ubuntu-24.04-arm
70+
context: .
71+
dockerfile: ./compute/worker/Dockerfile
6772

6873
steps:
6974
- name: Checkout repository
@@ -82,46 +87,63 @@ jobs:
8287
username: ${{ github.actor }}
8388
password: ${{ secrets.GITHUB_TOKEN }}
8489

85-
- name: Build and push Docker image (${{ matrix.arch }})
90+
- name: Select image name
91+
id: image
92+
run: |
93+
if [ "${{ matrix.image_target }}" = "web" ]; then
94+
echo "image_name=${{ needs.prepare.outputs.web_image_name }}" >> "$GITHUB_OUTPUT"
95+
else
96+
echo "image_name=${{ needs.prepare.outputs.compute_worker_image_name }}" >> "$GITHUB_OUTPUT"
97+
fi
98+
99+
- name: Build and push Docker image (${{ matrix.image_target }} / ${{ matrix.arch }})
86100
id: build
87101
uses: docker/build-push-action@v7
88102
with:
89-
context: .
103+
context: ${{ matrix.context }}
104+
file: ${{ matrix.dockerfile }}
90105
platforms: ${{ matrix.platform }}
91-
labels: ${{ needs.prepare.outputs.labels }}
92-
cache-from: type=gha,scope=${{ matrix.arch }}
93-
cache-to: type=gha,mode=max,scope=${{ matrix.arch }}
106+
cache-from: type=gha,scope=${{ matrix.image_target }}-${{ matrix.arch }}
107+
cache-to: type=gha,mode=max,scope=${{ matrix.image_target }}-${{ matrix.arch }}
94108
provenance: false
95-
outputs: type=image,name=${{ env.REGISTRY }}/${{ needs.prepare.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
109+
outputs: type=image,name=${{ env.REGISTRY }}/${{ steps.image.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
96110

97111
- name: Export digest
98112
run: |
99-
mkdir -p /tmp/digests
113+
mkdir -p /tmp/digests/${{ matrix.image_target }}
100114
digest="${{ steps.build.outputs.digest }}"
101-
touch "/tmp/digests/${digest#sha256:}"
115+
touch "/tmp/digests/${{ matrix.image_target }}/${digest#sha256:}"
102116
103117
- name: Upload digest
104118
uses: actions/upload-artifact@v7
105119
with:
106-
name: digests-${{ matrix.arch }}
107-
path: /tmp/digests
120+
name: digests-${{ matrix.image_target }}-${{ matrix.arch }}
121+
path: /tmp/digests/${{ matrix.image_target }}
108122
if-no-files-found: error
109123
retention-days: 1
110124

111125
merge:
112126
needs: [prepare, build]
113-
runs-on: ubuntu-24.04
127+
runs-on: ${{ matrix.runner }}
114128
permissions:
115129
actions: read
116130
contents: read
117131
packages: write
132+
strategy:
133+
fail-fast: false
134+
matrix:
135+
include:
136+
- image_target: web
137+
runner: ubuntu-24.04
138+
- image_target: compute-worker
139+
runner: ubuntu-24.04
118140

119141
steps:
120142
- name: Download digests
121143
uses: actions/download-artifact@v8
122144
with:
123-
path: /tmp/digests
124-
pattern: digests-*
145+
path: /tmp/digests/${{ matrix.image_target }}
146+
pattern: digests-${{ matrix.image_target }}-*
125147
merge-multiple: true
126148

127149
- name: Set up Docker Buildx
@@ -134,20 +156,46 @@ jobs:
134156
username: ${{ github.actor }}
135157
password: ${{ secrets.GITHUB_TOKEN }}
136158

159+
- name: Select image names
160+
id: image
161+
run: |
162+
if [ "${{ matrix.image_target }}" = "web" ]; then
163+
echo "image_name=${{ needs.prepare.outputs.web_image_name }}" >> "$GITHUB_OUTPUT"
164+
echo "metadata_images<<EOF" >> "$GITHUB_OUTPUT"
165+
echo "${{ env.REGISTRY }}/${{ needs.prepare.outputs.web_image_name }}" >> "$GITHUB_OUTPUT"
166+
echo "${{ env.REGISTRY }}/${{ needs.prepare.outputs.web_legacy_image_name }}" >> "$GITHUB_OUTPUT"
167+
echo "EOF" >> "$GITHUB_OUTPUT"
168+
else
169+
echo "image_name=${{ needs.prepare.outputs.compute_worker_image_name }}" >> "$GITHUB_OUTPUT"
170+
echo "metadata_images=${{ env.REGISTRY }}/${{ needs.prepare.outputs.compute_worker_image_name }}" >> "$GITHUB_OUTPUT"
171+
fi
172+
173+
- name: Extract metadata (tags, labels) for Docker
174+
id: meta
175+
uses: docker/metadata-action@v6
176+
with:
177+
images: ${{ steps.image.outputs.metadata_images }}
178+
tags: |
179+
type=raw,value=latest,enable=${{ (github.event_name == 'push' && !contains(github.ref, '-pre')) || (github.event_name == 'workflow_dispatch' && inputs.use_latest_tag == true) }},priority=1000
180+
type=semver,pattern={{version}}
181+
type=ref,event=tag
182+
type=ref,event=branch,enable=${{ github.event_name == 'workflow_dispatch' && inputs.use_latest_tag != true }}
183+
137184
- name: Create manifest list and push
138-
working-directory: /tmp/digests
185+
working-directory: /tmp/digests/${{ matrix.image_target }}
139186
run: |
140187
docker buildx imagetools create \
141188
$(echo "$TAGS" | xargs -I {} echo -t {}) \
142-
$(printf '${{ env.REGISTRY }}/${{ needs.prepare.outputs.image_name }}@sha256:%s ' *)
189+
$(printf '${{ env.REGISTRY }}/${{ steps.image.outputs.image_name }}@sha256:%s ' *)
143190
env:
144-
TAGS: ${{ needs.prepare.outputs.tags }}
191+
TAGS: ${{ steps.meta.outputs.tags }}
145192

146193
- name: Output build information
147194
run: |
148-
echo "✅ Docker images built and pushed successfully!"
195+
echo "✅ Docker image built and pushed successfully!"
196+
echo "🐋 Target: ${{ matrix.image_target }}"
149197
echo "🐋 Images:"
150-
echo '${{ needs.prepare.outputs.tags }}' | sed 's/^/ - /'
198+
echo '${{ steps.meta.outputs.tags }}' | sed 's/^/ - /'
151199
echo "📝 Event: ${{ github.event_name }}"
152200
if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
153201
echo "📝 Triggered by manual workflow dispatch on branch: ${{ github.ref_name }}"

‎.github/workflows/playwright.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,19 @@ jobs:
3434
sudo install -m 0755 ./weed /usr/local/bin/weed
3535
rm -f ./weed
3636
weed version
37+
- name: Install NATS server binary
38+
run: |
39+
curl -fsSL -o /tmp/nats-server.tar.gz \
40+
https://github.com/nats-io/nats-server/releases/download/v2.12.1/nats-server-v2.12.1-linux-amd64.tar.gz
41+
tar -xzf /tmp/nats-server.tar.gz -C /tmp
42+
sudo install -m 0755 /tmp/nats-server-v2.12.1-linux-amd64/nats-server /usr/local/bin/nats-server
43+
nats-server -v
3744
- name: Install dependencies
3845
run: pnpm install --frozen-lockfile
46+
- name: Build app and enforce bundle guard
47+
run: |
48+
pnpm build
49+
pnpm build:bundle-guard
3950
- name: Verify ffprobe
4051
run: ffprobe -version
4152
- name: Install Playwright Browsers

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,9 @@ yarn-error.log*
3333

3434
# env files (can opt-in for committing if needed)
3535
.env
36+
.env.prod
3637
.dockerignore
38+
*.creds
3739

3840
# vercel
3941
.vercel

‎Dockerfile‎

Lines changed: 18 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,14 @@
1-
# Stage 1: build whisper.cpp (no model download – the app handles that)
2-
FROM alpine:3.23 AS whisper-builder
3-
4-
RUN apk add --no-cache git cmake build-base
5-
6-
WORKDIR /opt
7-
8-
ARG TARGETARCH
9-
10-
RUN git clone --depth 1 https://github.com/ggml-org/whisper.cpp.git && \
11-
cd whisper.cpp && \
12-
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=OFF $( [ "$TARGETARCH" = "arm64" ] && echo "-DGGML_CPU_ARM_ARCH=armv8-a" || true ) && \
13-
cmake --build build -j
14-
15-
# Stage 1b: extract seaweedfs weed binary (for optional embedded weed mini)
1+
# Stage 1: extract seaweedfs weed binary (for optional embedded weed mini)
162
# Pin to 4.18 because CI observed upload regressions on 4.19.
173
FROM chrislusf/seaweedfs:4.18 AS seaweedfs-builder
184
RUN cp "$(command -v weed)" /tmp/weed && \
195
(wget -qO /tmp/SeaweedFS-LICENSE.txt "https://raw.githubusercontent.com/seaweedfs/seaweedfs/master/LICENSE" || \
206
wget -qO /tmp/SeaweedFS-LICENSE.txt "https://raw.githubusercontent.com/seaweedfs/seaweedfs/main/LICENSE")
217

8+
# Stage 1b: extract nats-server binary for embedded single-container worker mode.
9+
FROM nats:2.11-alpine AS nats-builder
10+
RUN cp "$(command -v nats-server)" /tmp/nats-server
11+
2212

2313
# Stage 2: build the Next.js app
2414
FROM node:lts-alpine AS app-builder
@@ -29,8 +19,10 @@ RUN npm install -g pnpm@11.1.2
2919
# Create app directory
3020
WORKDIR /app
3121

32-
# Copy package files
22+
# Copy workspace manifests needed for dependency installation
3323
COPY package.json pnpm-lock.yaml pnpm-workspace.yaml ./
24+
COPY compute/core/package.json ./compute/core/package.json
25+
COPY compute/worker/package.json ./compute/worker/package.json
3426

3527
# Install dependencies
3628
RUN pnpm install --frozen-lockfile
@@ -59,29 +51,30 @@ FROM node:lts-alpine AS runner
5951
# ffmpeg is provided by ffmpeg-static from node_modules.
6052
RUN apk add --no-cache ca-certificates libreoffice-writer
6153

62-
# Install pnpm globally for running the app
63-
RUN npm install -g pnpm@11.1.2
54+
# Install pnpm for runtime process commands.
55+
RUN npm install -g pnpm@10.33.4
6456

6557
# App runtime directory
6658
WORKDIR /app
6759

68-
# Copy built app and dependencies from the builder stage
60+
# Copy built app and runtime files from the builder stage (non-standalone runtime).
6961
COPY --from=app-builder /app ./
7062
# Include third-party license report and copied license texts at a stable path in the image.
7163
COPY --from=app-builder /app/THIRD_PARTY_LICENSES /licenses
7264
# Include SeaweedFS license text for the copied weed binary.
7365
COPY --from=seaweedfs-builder /tmp/SeaweedFS-LICENSE.txt /licenses/SeaweedFS-LICENSE.txt
66+
# Include static model notices for runtime-downloaded assets.
67+
COPY --from=app-builder /app/compute/core/src/pdf/assets/LICENSE.txt /licenses/pp-doclayoutv3-LICENSE.txt
7468

75-
# Copy the compiled whisper.cpp build output into the runtime image
76-
# (includes whisper-cli and its shared libraries, e.g. libwhisper.so, libggml.so)
77-
COPY --from=whisper-builder /opt/whisper.cpp/build /opt/whisper.cpp/build
7869
# Copy seaweedfs weed binary for optional embedded local S3.
7970
COPY --from=seaweedfs-builder /tmp/weed /usr/local/bin/weed
8071
RUN chmod +x /usr/local/bin/weed
72+
# Copy nats-server binary for embedded local JetStream.
73+
COPY --from=nats-builder /tmp/nats-server /usr/local/bin/nats-server
74+
RUN chmod +x /usr/local/bin/nats-server
8175

82-
# Point the app at the compiled whisper-cli binary and ensure its libs are discoverable
83-
ENV WHISPER_CPP_BIN=/opt/whisper.cpp/build/bin/whisper-cli
84-
ENV LD_LIBRARY_PATH=/opt/whisper.cpp/build
76+
# Include OpenAI Whisper license text for runtime-downloaded ONNX artifacts.
77+
COPY --from=app-builder /app/compute/core/src/whisper/assets/LICENSE.txt /licenses/openai-whisper-LICENSE.txt
8578

8679
# Expose the port the app runs on
8780
EXPOSE 3003

0 commit comments

Comments
 (0)