Skip to content

2026-05-13_v3.0_release.md

Latest commit

 

History

History
184 lines (124 loc) · 11.8 KB

2026-05-13_v3.0_release.md

File metadata and controls

184 lines (124 loc) · 11.8 KB
Raw

v3.0_initial Release — Major Architectural Refactor

Released: 2026-05-13

A major refactor of the Versatile-OCR-Program from a flat-script structure (v1.0_initial / v2.0_initial) to a modular, config-driven, automation-ready architecture. The pipeline behavior and OCR output remain unchanged — this release is a pure refactor + cleanup. All previously hardcoded user-tunable values are centralized in config.yaml and prompts/.

What's New (v2 → v3 Architectural Changes)

[1] Modular Structure + Auto-Run Wrappers

The flat 5-file structure (v2.0_initial/ containing Dockerfile, advanced_ocr.py, custom_doclayout_yolo.py, ocr_stage1.py, ocr_stage2.py) has been reorganized into:

  • src/ocr/ — OCR engine modules (advanced_ocr.py, custom_doclayout_yolo.py, docker_build.py, Dockerfile)
  • src/stages/ — Stage execution wrappers (ocr_stage_1.py, ocr_stage_2.py — note the underscore in the new filenames)

Two new auto-run wrappers were added to the root for batch execution:

  • auto_run_stage1.py — Recursive PDF directory scanning. Walks an input tree, finds every directory containing PDFs, and invokes ocr_stage_1.py per directory. Skips hidden directories automatically.
  • auto_run_stage2.py — Parallel processing of multiple GCS paths using ThreadPoolExecutor (default 70 workers). Real-time per-worker logging with worker IDs.

[2] Config-Driven Architecture

All hardcoded user-tunable values have been moved to config.yaml. Default values match the prior hardcoded values exactly — no behavior change. The new keys are:

  • ocr.language_hints (default ["ja", "en", "ko"]) — Google Vision OCR language hints
  • ocr.pdf_dpi (default 200) — DPI used by pdf2image.convert_from_path
  • ocr.iou_threshold (default 0.5) — Region duplicate-merge IoU threshold
  • ocr.image_processing.vision_max_dim (default 1600) — Image resize cap for Vision API
  • ocr.image_processing.gemini_max_dim (default 1024) — Image resize cap for Gemini API
  • ocr.image_processing.jpeg_quality (default 85) — JPEG re-encoding quality
  • gcs.stage1_prefix (default "stage_1") — GCS path prefix for Stage 1 output
  • gcs.stage2_prefix (default "stage_2") — GCS path prefix for Stage 2 output
  • gcs.validate_max_results (default 5) — Sample size for GCS path validation
  • gemini.model (default "gemini-2.0-flash") — Gemini model for figure/table analysis
  • gemini.figure_prompt_path (default "prompts/gemini_figure.txt")
  • gemini.table_prompt_path (default "prompts/gemini_table.txt")
  • stage2.system_prompt_path (default "prompts/chatgpt_stage2.md")
  • docker.source_files.prompts (new entry: "prompts/") — Tells docker_build.py to copy the prompts directory into the build context

auto_run.yaml separates execution parameters (input directory, mode, GCS paths, worker count) from the global config.yaml.

[3] Prompt Externalization (prompts/)

The long inline prompt strings previously hardcoded inside Python source have been extracted to dedicated files:

  • prompts/gemini_figure.txt — Gemini API prompt for figure region analysis (was advanced_ocr.py:714 inline)
  • prompts/gemini_table.txt — Gemini API prompt for table region analysis (was advanced_ocr.py:892 inline)
  • prompts/chatgpt_stage2.md — ChatGPT Stage 2 region-preserving system prompt (was ocr_stage_2.py:251 inline)

A small _load_prompt(path) helper resolves relative paths against the config.yaml location (recorded in _config_dir when the config is loaded), so prompts load correctly from any working directory or inside Docker.

[4] GCS-Only Output Architecture

The local output directory pattern (/tmp/ocr_output) used in v2 is removed. Stage 1 uploads results directly to GCS, and Stage 2 reads from and writes to GCS exclusively. This decouples the two stages cleanly and enables parallel re-runs.

process_folder_stage2() is idempotent: it counts JSON files in Stage 1 vs Stage 2 directories and skips already-complete folders.

[5] Stage 2 Simplification

The [Formula Start]/[Formula End], [Image Start]/[Image End], [Table Start]/[Table End] placeholder system used in v2 to mark special content for ChatGPT preservation is removed. Stage 1 now emits a simplified output (regions array only — no flattened text, no metadata), and Stage 2's ChatGPT prompt enforces strict region-structure preservation (same count, same type/coords/id fields, only text modified).

Three-tier fallback in chatgpt_correct_text:

  1. JSON parse failure → use original regions
  2. Region-structure validation failure (validate_region_structure) → use original regions
  3. Exception → empty result

Region IDs are forcibly re-assigned by the code (page_{N}_region_{i}) regardless of what ChatGPT returns.

[6] Unicode Filename Compatibility

validate_direct_path() and related GCS path resolution now tries NFC normalization first, then falls back to NFD. This handles Korean filenames uploaded from macOS environments (a previously fragile area).

[7] Operational Tools

  • docker_build.py — Safe Docker build automation using a temporary build context. Smart config.yaml discovery (walks up directory tree), per-source-file resolution, file existence check (--check-files), and optional context preservation for debugging (--keep-context). Now supports directory copy via shutil.copytree (required for the new prompts/ directory).
  • temp_controller.ipynb — Manual debug notebook. Not committed to the repository (see .gitignore); kept locally for ad-hoc single-directory runs during development.

Pre-commit Cleanup

The following non-functional changes were applied alongside the architectural refactor. All preserve external behavior.

Security

  • Removed leaked GEMINI_API_KEY debug print at src/stages/ocr_stage_1.py:37. The line print("DEBUG GEMINI:", os.getenv("GEMINI_API_KEY")) exposed the API key in plain text on every stage 1 run.

Hygiene

  • Promoted import re to the top-level of src/ocr/advanced_ocr.py (was duplicated as in-function imports inside _process_figure_region and _process_table_region).
  • Removed duplicate method definitions of _process_title_region and _process_list_region in src/ocr/advanced_ocr.py. The second definitions (at the bottom of the class) were __doc__/comment variants of the earlier ones with identical functional code; they have been removed and the earlier (more descriptive) definitions retained.
  • Fixed src/ocr/Dockerfile:6 comment: NumPy <2.0 (1.24.3) → NumPy 1.26.4 (matches the actual numpy==1.26.4 install).
  • Pinned google-genai==1.68.0 and python-dotenv==1.2.2 in src/ocr/Dockerfile (previously unpinned).

Centralization (Hash Algorithm Constants)

Two cache-key algorithm constants were promoted to module-level constants but not externalized to config, since changing them invalidates all cached results:

HASH_THUMBNAIL_SIZE = 32
HASH_JPEG_QUALITY = 50

These are not user-tunable in the practical sense.

.gitignore Additions

The following project-specific patterns were added:

src/input/          # Personal learning materials (Korean math textbook PDFs)
cache/              # OCR image hash cache (contains personal data results)
*.swp               # Vim swap files
temp_controller.ipynb  # Manual debug notebook

Migration Guide (v2.0_initial → v3.0_initial)

If you were running v2.0_initial/ocr_stage1.py directly:

  1. Required env vars (unchanged): OPENAI_API_KEY, MATHPIX_APP_ID, MATHPIX_APP_KEY, GOOGLE_SHEETS_SPREADSHEET_ID, GOOGLE_APPLICATION_CREDENTIALS, GEMINI_API_KEY.

  2. New required files at repo root: config.yaml and auto_run.yaml. The example values in the committed config.yaml reflect a specific local environment (/home/jupyter/Program/.env, /home/jupyter/Program/credentials, etc.) — adjust paths to your environment before running.

  3. Output destination: v2 produced both local output files and GCS uploads; v3 is GCS-only. Update any downstream consumers that read local output files.

  4. Stage 1 changes:

    • Entry script renamed: ocr_stage1.py → src/stages/ocr_stage_1.py
    • For recursive batch runs, use the new auto_run_stage1.py wrapper.
    • Output JSON now contains regions only (the flattened text field, image_path metadata, and processing_timestamp have been removed).
    • GCS path structure: stage_1/{relative_path}/{pdf_name}/page_{NNN}.json where stage_1 is configurable via gcs.stage1_prefix.

  5. Stage 2 changes:

    • Entry script renamed: ocr_stage2.py → src/stages/ocr_stage_2.py
    • For parallel multi-path runs, use the new auto_run_stage2.py wrapper.
    • System prompt now externalized to prompts/chatgpt_stage2.md (path configurable in stage2.system_prompt_path).
    • Model is configurable via stage2.model (default gpt-5-nano).
    • Special content placeholders ([Formula Start]/[Formula End] etc.) are no longer recognized — Stage 1 output is already simplified to a regions-only array.

  6. Docker:

    • Dockerfile relocated to src/ocr/Dockerfile.
    • Prefer docker_build.py over direct docker build — it manages a temporary build context driven by docker.source_files in config.yaml.
    • The build context now includes the prompts/ directory (configured via docker.source_files.prompts and copied by docker_build.py's create_build_context).

  7. Tunable values: Anything you might want to vary per user/environment (OCR language hints, PDF DPI, image max dimensions, JPEG quality, Gemini model name, GCS path prefixes, prompt texts) is now in config.yaml or prompts/. Edit there, not in source code.

  8. Jupyter / Vertex AI users: Docker group permission setup is still required — see setup_guide.md §3 (sudo usermod -aG docker jupyter && sudo reboot).

Affected Files

File Change
auto_run.yaml NEW
auto_run_stage1.py NEW
auto_run_stage2.py NEW
config.yaml NEW (with v3 centralization items)
src/ocr/advanced_ocr.py Relocated from v2.0_initial/, refactored for config-driven settings, prompt externalization, hash constants, method dedup, top-level import re
src/ocr/custom_doclayout_yolo.py Relocated from v2.0_initial/, already config-driven in upstream
src/ocr/docker_build.py NEW (safe build context automation, supports directory copy)
src/ocr/Dockerfile Refactored from v2.0_initial/, COPY prompts/, pinned google-genai and python-dotenv, NumPy comment fix
src/stages/ocr_stage_1.py Renamed from ocr_stage1.py, container path mapping, debug print removed, config-driven Docker execution
src/stages/ocr_stage_2.py Renamed from ocr_stage2.py, region-preserving prompt externalized, idempotent processing, Unicode NFC/NFD fallback, GCS prefix from config
prompts/ NEW directory (3 prompt files)
changes/ NEW directory (2 documents — v2.0_initial.md migrated from patch_notes/, plus this release doc)
legacy/ NEW directory — v1.0_initial/ and v2.0_initial/ moved here, plus a README.md explaining the move
setup_guide.md Rewritten for v3 (clone → API keys → Docker → config → run → verification → troubleshooting)
usage.md NEW — operational manual (CLI flags, scenarios, config reference, prompt editing, results, idempotency, FAQ)
.gitignore Extended with 4 project-specific patterns
sample_images/, README.md, LICENSE, planned_features.md Unchanged
patch_notes/ Removed (content moved into changes/)

Recommendation

Use this version (main after this release) for any new deployment of Versatile-OCR-Program. The legacy/v1.0_initial/ and legacy/v2.0_initial/ folders remain available for reference but should not be used for new work.

For questions, contact: raphael.es.seo@gmail.com