refactor: consistent cloning & pattern-handling by filipchristiansen · Pull Request #388 · coderamp-labs/gitingest

filipchristiansen · 2025-07-06T12:19:16Z

✨ Refactor: consistent cloning & pattern-handling

Closes #196
Closes #344
Prepares for #342 (digest prefix/suffix) and #343 (caching)

Why

Cloning behaved differently depending on whether they supplied a branch, tag or commit.
→ Caching and reproducibility suffered.
Pattern handling lived in query_parser and duplicated ignore/include logic.
→ Hard to unit-test and reuse.
Windows CI occasionally failed when temp clones contained read-only
packed-object files.
→ Shallow-clone cleanup broke with WinError 5.

What’s new

Unified commit resolution
utils.git_utils.resolve_commit() guarantees we always fetch the exact SHA
(HEAD / branch / tag) before checkout.
Deterministic → enables caching (feat: Implement caching on a per-commit basis #343).
New pattern helpers
utils.pattern_utils.process_patterns() centralises include/exclude parsing
(moved out of query_parser). Adds thorough tests.
Windows-safe repo cleanup
_handle_remove_readonly on-error callback makes read-only Git objects
writable and retries shutil.rmtree(), preventing WinError 5 in CI.
Public API cleanup (BREAKING)
- parse_query() removed in favour of
  - parse_remote_repo() (URLs/slugs)
  - parse_local_dir_path() (local paths)
- clone_repo, ingest_query, parse_query are no longer re-exported from
  gitingest.__init__.
Clone rewrite
- Single entry point that always does: shallow clone → sparse-checkout
  (optional) → fetch commit → checkout → submodule update (optional).
- _checkout_partial_clone renamed & moved → git_utils.checkout_partial_clone.
Server updates
- query_processor uses new pattern utilities and passes a typed enum.
- IngestRequest.validate_input_text() now removes any .git suffix.
Delete _is_safe_symlink
Remove pattern validation (delete _is_valid_pattern, InvalidPatternError, and test_parse_patterns_invalid_characters)
Tests added/updated

clone, pattern, parser, summary

File changes

`gitingest`

File	Changes
`__init__.py`	Stop exposing internal helpers (`clone_repo`, `ingest_query`, `parse_query`)
`clone.py`	Rewrite `clone_repo` to call `resolve_commit`, sparse checkout, and uniform fetch/checkout steps; move helper & rename
`entrypoint.py`	Add `_handle_remove_readonly` callback for Windows temp-dir cleanup; `ingest_async` now uses `parse_remote_repo` / `parse_local_dir_path`
`output_formatter.py`	Add missing `Tag` prefix in `_create_summary_prefix`
`query_parser.py`	Remove `parse_query`; move pattern helpers to `pattern_utils.py`
`utils/exceptions.py`	Delete `InvalidPatternError`
`utils/git_utils.py`	New `resolve_commit` helper
`utils/os_utils.py`	Rename `ensure_directory` → `ensure_directory_exists_or_create`
`utils/pattern_utils.py`	Consolidated helpers + `process_patterns`
`utils/query_parser_utils.py`	Delete `_is_valid_pattern`
`path_utils.py` (removed)	Delete obsolete helper `_is_safe_symlink`

`server`

File	Changes
`models.py`	Strip `.git` suffix in `validate_input_text`
`query_processor.py`	Replace `parse_query` with `parse_remote_repo`, integrate `PatternType`, use `process_patterns`
`routers_utils.py`	Coerce `pattern_type` into `PatternType` enum

`tests`

File	Changes
`conftest.py`	Adjust fixtures
`query_parser/test_git_host_agnostic.py`	Switch to `parse_remote_repo`
`query_parser/test_query_parser.py`	Switch to `parse_remote_repo`; add `parse_local_dir_path` coverage
`test_clone.py`	Update expectations and fixtures
`test_pattern_utils.py`	Add tests for `_parse_patterns` and `process_patterns`; remove `test_parse_patterns_invalid_characters` (pattern validation no longer enforced)
`test_summary.py`	Verify that `gitingest.ingest()` emits correct summaries

ix-56h

LGTM

filipchristiansen · 2025-07-14T13:05:16Z

TODO:

delete _is_valid_pattern, InvalidPatternError, _is_safe_symlink
call resolve_commit in parse_remote_repo instead of in clone_repo

Add `_handle_remove_readonly` on-error callback for `shutil.rmtree` that removes the read-only attribute and retries, preventing WinError 5 during temp-repo cleanup in tests.

fix blob/tag added test

…mlink

NicolasIRAGNE

LGTM
Thanks Filip!

github-actions · 2025-07-23T14:35:42Z

⚙️ Preview environment was undeployed.

filipchristiansen mentioned this pull request Jul 6, 2025

feat: avoid using branch references #344

Closed

filipchristiansen changed the title ~~resolve commit~~ feat: resolve commit Jul 9, 2025

github-advanced-security AI found potential problems Jul 9, 2025

View reviewed changes

Comment thread .github/workflows/pr-title-check.yml Fixed

Comment thread src/server/routers/ingest.py Fixed

Comment thread src/server/routers/ingest.py Fixed

github-advanced-security AI found potential problems Jul 9, 2025

View reviewed changes

Comment thread src/static/js/utils.js Fixed

filipchristiansen force-pushed the resolve-commit branch 3 times, most recently from 7ab4af2 to 831a36d Compare July 9, 2025 20:32

filipchristiansen requested a review from Copilot July 9, 2025 20:33

This comment was marked as resolved.

Sign in to view

filipchristiansen force-pushed the resolve-commit branch 6 times, most recently from cf1aa6f to d1224a6 Compare July 9, 2025 23:45

filipchristiansen added the enhancement label Jul 9, 2025

filipchristiansen marked this pull request as ready for review July 9, 2025 23:47

filipchristiansen requested review from NicolasIRAGNE and Copilot July 9, 2025 23:48

This comment was marked as resolved.

Sign in to view

filipchristiansen force-pushed the resolve-commit branch 2 times, most recently from 979c88a to 3031e7c Compare July 10, 2025 00:07

filipchristiansen changed the title ~~feat: resolve commit~~ refactor: consistent cloning & pattern-handling Jul 10, 2025

filipchristiansen requested a review from ix-56h July 12, 2025 19:39

ix-56h reviewed Jul 12, 2025

View reviewed changes

Comment thread src/gitingest/entrypoint.py

ix-56h reviewed Jul 12, 2025

View reviewed changes

Comment thread .github/workflows/publish_to_pypi.yml Outdated

ix-56h reviewed Jul 13, 2025

View reviewed changes

Comment thread src/gitingest/entrypoint.py

This comment was marked as resolved.

Sign in to view

ix-56h approved these changes Jul 13, 2025

View reviewed changes

This comment was marked as off-topic.

Sign in to view

NicolasIRAGNE reviewed Jul 14, 2025

View reviewed changes

Comment thread src/gitingest/clone.py

NicolasIRAGNE reviewed Jul 14, 2025

View reviewed changes

Comment thread src/gitingest/utils/pattern_utils.py Outdated

This comment was marked as duplicate.

Sign in to view

filipchristiansen force-pushed the resolve-commit branch from f98250f to 8438492 Compare July 16, 2025 19:16

filipchristiansen and others added 10 commits July 23, 2025 11:47

resolve commit

48fc086

add test_summary.py

259044d

replace non-ASCII chars

293b4ae

fix(entrypoint): handle read-only Git objects on Windows

b8ac225

Add `_handle_remove_readonly` on-error callback for `shutil.rmtree` that removes the read-only attribute and retries, preventing WinError 5 during temp-repo cleanup in tests.

python versions compat: replace onerror with onexc for Python 3.12+

28cb287

fix typo

8c0503f

add more blob/tag tests

fe81708

fix blob/tag added test

chore: delete _is_valid_pattern, InvalidPatternError, and _is_safe_sy…

01ca7f2

…mlink

chore: delete test_parse_patterns_invalid_characters

8a2c6cb

call resolve_commit in parse_remote_repo instead of in clone_repo

c5c9d5d

filipchristiansen force-pushed the resolve-commit branch from 2d6d8c0 to c5c9d5d Compare July 23, 2025 09:47

filipchristiansen requested review from NicolasIRAGNE, Copilot and ix-56h July 23, 2025 09:48

This comment was marked as outdated.

Sign in to view

filipchristiansen added 4 commits July 23, 2025 12:05

fix ci

30eb2aa

factor out _calculate_expected_lines helper function

61803df

make CloneConfig a pydantic BaseModel

e79a9a4

move CloneConfig to its own file

3015487

NicolasIRAGNE approved these changes Jul 23, 2025

View reviewed changes

filipchristiansen removed the request for review from ix-56h July 23, 2025 14:34

filipchristiansen merged commit a99089a into main Jul 23, 2025
25 checks passed

filipchristiansen deleted the resolve-commit branch July 23, 2025 14:35

Uh oh!

Conversation

filipchristiansen commented Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ Refactor: consistent cloning & pattern-handling

Why

What’s new

File changes

gitingest

server

tests

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

This comment was marked as resolved.

ix-56h left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

Uh oh!

This comment was marked as duplicate.

filipchristiansen commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO:

Uh oh!

This comment was marked as outdated.

Uh oh!

NicolasIRAGNE left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

filipchristiansen commented Jul 6, 2025 •

edited

Loading

`gitingest`

`server`

`tests`

filipchristiansen commented Jul 14, 2025 •

edited

Loading