feat(PLU-441): add OAuth support to GitHub source connector (#759)

## Summary

Prepares the GitHub source connector for Foundation by adding OAuth
auth, blob-SHA-based incremental change detection, and hardening
indexing/downloading (API-call economy, tree-truncation handling, a
download-path bug fix, consistent error classification, and
rate-limit-aware retries). PAT configs continue to work unchanged.

## What changed

- **OAuth support.** `GithubAccessConfig` accepts `oauth_token` /
`refresh_token` alongside the existing `access_token` (PAT), matching
the field contract the platform's `init-oauth-refresh` container
expects. `oauth_token` takes precedence; token refresh is handled
upstream (PLU-381), not by the connector. PAT and OAuth are mutually
exclusive.
- **Blob SHA as `metadata.version`.** The indexer emits each file's git
blob SHA (a content hash already in the tree listing — no extra call) so
the platform's per-record incremental skip logic works correctly. The
old tree ETag was shared across every file and couldn't drive per-file
skip. `metadata.date_modified` is no longer set (git trees carry no
reliable per-file timestamp; `version` is the sole skip signal).
- **API-call economy + tree truncation.** Fetches the
repo/default-branch once per run instead of twice per file (`2N+3` → `2`
calls). When GitHub truncates the recursive tree (~100k-entry / 7 MB
cap, non-paginable), falls back to a per-directory walk so large repos
index completely. Skips empty files, directories, and submodules. The
downloader caches the repo object per instance (i.e. per process/pod),
so repeated downloads in one worker skip the redundant GET
/repos/{owner}/{repo} — safe under pod fan-out since nothing is shared
across processes.


- **Download-path fix.** `GithubDownloader.run` now creates the parent
directory chain before writing — previously every download of a nested
(or root) path failed with `FileNotFoundError`.
- **Error classification + retries.** GitHub errors map consistently
(auth 401/403 → `UserAuthError`, throttle → `RateLimitError`, 5xx →
`ProviderError`; network resets/timeouts →
`SourceConnectionNetworkError`). Transient throttles / 5xx / network
errors are retried with exponential backoff that honors GitHub's
`Retry-After` (secondary limit) and `x-ratelimit-reset` (primary limit),
capped at 300s. A 403 with an exhausted window is treated as a retriable
throttle, not a permanent auth failure. Attempt budget via `max_retries`
(default 10) on both indexer and downloader configs. The raw-blob fetch
now uses a bounded request timeout. Adds `tenacity` to the `github`
extra.

## Testing

- **Unit:** new `test/unit/connectors/test_github.py` (61 cases)
covering auth config, error classification, blob-SHA versioning,
recursive listing + truncation walk, downloader caching/fetch, backoff
parsing, and the retry loop. All pass in ~0.2s (retries mocked to not
sleep).
- **Integration:** `test/integration/connectors/test_github.py`
refactored to share a validation helper; added
`test_github_source_oauth` exercising the OAuth token path (requires
`GH_OAUTH_ACCESS_TOKEN`). Existing PAT test preserved.
- `ruff check` clean.

## Notes

- Version bumped to `1.7.1`; `tenacity` added to the `github` extra with
a matching `uv.lock` edit (`uv lock` couldn't be run in-sandbox due to
the Azure-CLI auth requirement, so the lock was updated by hand to match
uv's format — worth a quick `uv lock` confirmation in CI).
- Companion changes for incremental support + OAuth refresh live in
`platform-etl-orchestration` (`etl_job_api` incremental registry,
`etl-operator` `GithubOAuthRefresher`).

---------

Co-authored-by: Sean Mullen <sean@unstructured.io>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

Author: 4 people

CHANGELOG.md

@@ -1,3 +1,9 @@
+## [1.7.4]
+
+### Enhancements
+
+- **feat(PLU-441): ready the GitHub source connector for Foundation.** Adds OAuth support (`oauth_token`/`refresh_token` alongside the existing PAT, refresh handled upstream per PLU-381) and emits the git blob SHA as `metadata.version` for incremental change detection. Indexing fetches the repo once per run instead of twice per file, falls back to a per-directory walk when GitHub truncates the recursive tree, and skips empty files, directories, and submodules. Fixes downloads failing with `FileNotFoundError` on nested paths. GitHub API errors are classified consistently (auth → `UserAuthError`, throttle → `RateLimitError`, 5xx → `ProviderError`), and transient throttles / 5xx / network errors are retried with exponential backoff that honors GitHub's `Retry-After` (`max_retries`, default 10).
+
 ## [1.7.3]
 
 ### Fixes

pyproject.toml

@@ -52,7 +52,7 @@ dropbox = ["dropboxdrivefs", "fsspec"]
 duckdb = ["pandas", "duckdb"]
 elasticsearch = ["elasticsearch[async]<9.0.0"]
 gcs = ["gcsfs", "fsspec", "beautifulsoup4"]
-github = ["pygithub>1.58.0", "requests"]
+github = ["pygithub>1.58.0", "requests", "tenacity"]
 gitlab = ["python-gitlab"]
 google-drive = ["google-api-python-client", "tenacity"]
 hubspot = ["hubspot-api-client", "urllib3"]

test/integration/connectors/test_github.py

@@ -20,16 +20,7 @@
 )
 
 
-@pytest.mark.tags(CONNECTOR_TYPE, SOURCE_TAG, UNCATEGORIZED_TAG)
-@pytest.mark.asyncio
-@requires_env("GH_READ_ONLY_ACCESS_TOKEN")
-async def test_github_source(temp_dir):
-    access_token = os.environ["GH_READ_ONLY_ACCESS_TOKEN"]
-    connection_config = GithubConnectionConfig(
-        access_config=GithubAccessConfig(access_token=access_token),
-        url="dcneiner/Downloadify",
-    )
-
+async def _validate_github_source(connection_config: GithubConnectionConfig, temp_dir):
     indexer = GithubIndexer(
         connection_config=connection_config,
         index_config=GithubIndexerConfig(file_glob=["*.txt", "*.html"]),
@@ -52,3 +43,29 @@ async def test_github_source(temp_dir):
             postdownload_file_data_check=source_filedata_display_name_set_check,
         ),
     )
+
+
+@pytest.mark.tags(CONNECTOR_TYPE, SOURCE_TAG, UNCATEGORIZED_TAG)
+@pytest.mark.asyncio
+@requires_env("GH_READ_ONLY_ACCESS_TOKEN")
+async def test_github_source(temp_dir):
+    access_token = os.environ["GH_READ_ONLY_ACCESS_TOKEN"]
+    connection_config = GithubConnectionConfig(
+        access_config=GithubAccessConfig(access_token=access_token),
+        url="dcneiner/Downloadify",
+    )
+    await _validate_github_source(connection_config, temp_dir)
+
+
+@pytest.mark.tags(CONNECTOR_TYPE, SOURCE_TAG, UNCATEGORIZED_TAG)
+@pytest.mark.asyncio
+@requires_env("GH_OAUTH_ACCESS_TOKEN")
+async def test_github_source_oauth(temp_dir):
+    # The platform's init-oauth-refresh container populates oauth_token with a fresh access
+    # token before each run; here we supply an already-valid OAuth token to exercise the path.
+    oauth_token = os.environ["GH_OAUTH_ACCESS_TOKEN"]
+    connection_config = GithubConnectionConfig(
+        access_config=GithubAccessConfig(oauth_token=oauth_token),
+        url="dcneiner/Downloadify",
+    )
+    await _validate_github_source(connection_config, temp_dir)