BATCH-3679: gate S3 default client on IS_LUIGI1_DEPRECATED flag

[andresvelasquez2-affirm]

Description

Adds an environment-driven switch that picks the default S3 client implementation based on whether luigi1 is still installed alongside this fork.

• New private module luigi/contrib/_luigi1_compat.py exposing one symbol — IS_LUIGI1_DEPRECATED:
  • False when import luigi1 succeeds (legacy stack still around)
  • True when import luigi1 raises any Exception (legacy stack gone)
• luigi/contrib/s3.py now binds the public aliases through this flag:
  • IS_LUIGI1_DEPRECATED is True → S3Client = S3ClientBoto3, ReadableS3File = ReadableS3FileBoto3
  • IS_LUIGI1_DEPRECATED is False → S3Client = S3ClientBoto1, ReadableS3File = ReadableS3FileBoto1
• S3ClientBoto1 / S3ClientBoto3 (and the matching ReadableS3File* classes) remain public, so callers that need to pin a stack regardless of environment continue to do so via client=.
• client= kwargs on S3Target, S3PathTask, S3EmrTask, S3FlagTask are unchanged. The flag covers the default dispatch; the kwarg covers explicit injection (custom credentials, assumed roles, moto-mocked clients, RedshiftManifestTask's self._client forwarding).
• Version bump in setup.py.
• CLAUDE.md and PLAN_Py39_P312_DUAL_COMPATIBILITY.md updated with the rationale, the surface controlled by the flag, and the test infra changes.

Motivation and Context

BATCH-3679. The base branch (rafay/BATCH-3679-luigi-py-312-upgrade) ships both boto1 and boto3 stacks side-by-side and pins the default to boto1 for backwards compatibility. That forces every caller in a modern environment (where luigi1 has already been removed) to migrate explicitly to client=S3ClientBoto3() before they can drop luigi1.

This PR makes the cutover environmental instead of per-call-site: while luigi1 is installed the legacy boto1 default is preserved verbatim, and the day luigi1 is uninstalled the same call sites land on boto3 with no code change. Existing client= overrides continue to win when callers need to pin a specific stack.

Have you tested this? If so, how?

I have included unit tests, and I have run the s3 contrib tests across 7 environment combinations.

Unit tests:

• test/contrib/luigi1_compat_test.py — new file. Covers all three branches of IS_LUIGI1_DEPRECATED: importable (False), unimportable (True via ImportError), and importable-but-raises-non-ImportError (True). Uses a sys.meta_path finder to actively block / inject luigi1 so the test is deterministic regardless of whether luigi1 is genuinely installed in the env.
• test/contrib/s3_test.py::TestFlagDrivenS3Dispatch — new test class. Pins the dispatch contract: alias resolution per flag, _readable_file_cls wiring, default client injection in S3Target / S3PathTask / S3EmrTask / S3FlagTask, same-stack and cross-stack client= override, and (for the harder reload case) the alias flip in both directions. Two simple, hermetic tests pin the literal class identity of S3Target('s3://...').fs per flag value, gated on the natural flag state — coverage of both directions comes from running under different uv scenarios:
  • test_s3_target_default_fs_is_boto3_when_flag_true — @unittest.skipUnless(IS_LUIGI1_DEPRECATED, ...). Asserts target.fs is an S3ClientBoto3 (and not S3ClientBoto1). Exercised in scenarios 2/3/4/5.
  • test_s3_target_default_fs_is_boto1_when_flag_false — @unittest.skipUnless(not IS_LUIGI1_DEPRECATED and HAS_BOTO_PKG, ...). Asserts target.fs is an S3ClientBoto1 (and not S3ClientBoto3). Exercised in scenario 6.

Test-infra refinements made along the way:

• The legacy try: import boto / HAS_BOTO setup was replaced with flag-aware predicates: USING_BOTO1, HAS_BOTO_PKG, BOTO1_RUNNABLE = USING_BOTO1 and HAS_BOTO_PKG, S3CLIENT_INSTANTIABLE = HAS_BOTO_PKG if USING_BOTO1 else True. Class-level skips on TestS3Target / TestS3Client (and per-test on the dispatch tests that construct S3Client()) ensure degenerate configs (flag selects boto1 but boto missing) skip cleanly instead of failing at import or instantiation.
• S3ResponseError is now bound to the active stack, not just package availability — boto can be on path while the flag selects boto3, and in that case errors come from botocore.
• mock_s3 / mock_sts selection is now stack-aware. Under the boto1 stack we prefer moto 1.x's HTTPretty-backed mock_s3_deprecated / mock_sts_deprecated, which patch at the socket level and therefore catch boto1's raw http.client traffic. The default mock_s3 only patches requests / urllib3 and silently lets boto1 traffic reach real AWS; that's why the boto1 path looked "broken" before this fix even though the test code was fine. moto 4+ dropped support for boto1 mocking entirely.
• test_init_with_config_and_roles had a stale assertion expecting s3_client.s3.access_key == 'AKIAIOSFODNN7EXAMPLE' — moto's STS mock now (correctly) returns AWS-canonical ASIA…-prefixed temporary credentials. Loosened to assert the ASIA STS prefix and a non-empty secret key, which is what the test was actually meant to verify.

Scenario matrix (uv run --no-project --index-url https://pypi.affirm-build.com/artifactory/api/pypi/pypi/simple/ --with-editable . --with pytest --with mock --with requests <stack-deps> python -m pytest -q test/contrib/s3_test.py test/contrib/luigi1_compat_test.py):

#	Python	Stack deps	luigi1	Result
1	3.12	boto + boto3==1.33.13 + moto==1.3.14	—	collection error — boto + moto 1.x are not py3.12 compatible (expected; can't run the boto1 stack on py3.12)
2	3.12	boto3 + moto>=5,<6	—	59 pass / 14 skip / 0 fail
3	3.9	boto + boto3==1.33.13 + moto==1.3.14	—	60 pass / 13 skip / 0 fail
4	3.9	boto3==1.33.13 + moto==1.3.14	—	60 pass / 13 skip / 0 fail (moto 1.x transitively requires boto, so this collapses to scenario 3)
5	3.9	boto3 + moto>=5,<6	—	59 pass / 14 skip / 0 fail
6	3.9	boto + boto3==1.33.13 + moto==1.3.14 + luigi1	yes	72 pass / 1 skip / 0 fail
7	3.9	boto3 + moto>=5,<6 + luigi1	yes	8 pass / 65 skip / 0 fail — degenerate config (luigi1 says use boto1 but boto is missing); flag dispatch contract still verified by the 8 non-instantiating tests

Every viable scenario is green. Scenario 1 is the only failure and it's the predicted "boto1 isn't py3.12 compatible" outcome — there's no way to run that combination because the legacy stack itself doesn't load on py3.12. Both flag states are exercised: scenarios 2/3/4/5 cover flag-True (boto3 default), scenario 6 covers flag-False (boto1 default with the proper HTTPretty-backed moto mock).