feat(scripts): Add dependency version scanner tool (#16867)

This adds a utility with the ability to scan for common references to
dependencies (Python runtimes and package dependencies) to facilitate
updating code when runtimes and dependencies change.

* It can be run against an entire repo OR against specific packages
within a monorepo
* It is customizable with [regex patterns and examples
here](https://github.com/googleapis/google-cloud-python/pull/16867/changes#diff-d17423afcf0604a287af7c6a590da37df7105674bbb715248d4f0c53545f3ea9)
* The test suite checks each regex against the examples to ensure the
efficacy of the patterns
* The current patterns account for edge cases such as finding `< 3.8`
when searching for references to `3.7` since they are semantically
equivalent even if syntactically different.
* The scanner produces a CSV report with:
```
path/filename, package name, line number, matching pattern, full line for context, etc.
```

Author: chalmerlowe

scripts/version_scanner/.gitignore

@@ -0,0 +1,2 @@
+.conductor/
+scanner_report.csv

scripts/version_scanner/.scannerignore

@@ -0,0 +1,22 @@
+# Directories and files to ignore by the version scanner
+.git
+__pycache__
+.tox
+.nox
+venv
+.venv
+.conductor
+version_scanner
+docs
+samples
+changelog.md
+.librarian
+goldens
+# Ignore pandoc references in repositories.bzl
+repositories.bzl
+
+# Ignore binary media files
+*.jpg
+*.png
+*.gif
+*.ico

scripts/version_scanner/README.md

@@ -0,0 +1,179 @@
+# Automated Dependency Version Scanner
+
+This tool scans the repository for hardcoded references to specific dependency versions (like Python 3.7) that need to be upgraded or removed.
+
+## Usage
+
+Run the script from the repository root:
+
+```bash
+python3 scripts/version_scanner/version_scanner.py -d <dependency> -v <version> [options]
+```
+
+### Options
+
+*   `-d`, `--dependency`: Name of the dependency (e.g., python, protobuf)
+*   `-v`, `--version`: Specific version to search for (e.g., 3.7, 4.25.8)
+*   `-p`, `--path`: Root directory to scan (defaults to current directory)
+*   `--package`: Specific subdirectory filter (useful for monorepos)
+*   `--package-file`: Path to a file containing a list of package directories to scan (e.g., `scripts/version_scanner/small_package_list.txt`)
+*   `--config`: Path to the regex configuration file (defaults to scripts/version_scanner/regex_config.yaml)
+*   `-o`, `--output`: Path to the output CSV file (defaults to <dependency>-<version>-<timestamp>.csv)
+*   `--github-repo`: GitHub repository URL base (defaults to https://github.com/googleapis/google-cloud-python)
+*   `--branch`: GitHub branch for links (defaults to main)
+
+## Installation & Setup
+
+By default, the core scanner only depends on Python's standard library and **`pyyaml`** to read the configuration file.
+
+If you want to use the Google Drive upload feature (`--upload`), you must install the optional Google API client dependencies:
+```bash
+pip install -r scripts/version_scanner/requirements.txt
+```
+
+## Scope: Handwritten vs. Generated Code
+
+> [!NOTE]
+> **This scanner is primarily intended for auditing handwritten code, configuration files, CI scripts, and documentation.**
+> You do **not** need to scan or manually edit auto-generated GAPIC libraries. Any dependency updates for generated code are handled upstream by editing the generator templates in the `gapic-generator-python` repository. When the templates are updated, the changes naturally trickle downstream to correct all generated client libraries upon the next regeneration.
+
+## Limitations
+
+*   **Single-Line Matching Only**: The scanner processes files line-by-line to ensure high performance and simplicity. Consequently, version declarations or dependency lists that span across multiple lines (such as multiline lists in a `setup.py` file) will not be caught by the regex patterns.
+
+## Configuration
+
+The scanner uses a YAML configuration file (`regex_config.yaml`) to define rules and regex patterns.
+
+## Ignoring Directories
+
+You can create a `.scannerignore` file in the directory you are scanning (usually the repo root) to list directories to skip, one per line.
+
+## Known Issues & Future Investigations
+- **Binary Ignores in `.scannerignore`**: Recursive wildcard ignores (e.g., `*.jpg`) currently do not effectively ignore deeply nested binary files. The scanner logic should be investigated to support robust globbing or full-path suffix matching.
+
+---
+
+## Universal Prompt for EOL Runtime & Dependency Migration
+
+### Context & Overview
+
+#### Overview
+This prompt is provided as an example and outlines the approach to update Python packages to drop support for end-of-life Python runtimes (3.7, 3.8, 3.9) OR for deprecated dependencies, and ensure the packages are configured for modern Python. This may help speed up your ability to resolve version mismatches. This prompt is provided with no guarantees, your mileage may vary. LLMs may make mistakes, always double check the LLM's work and test thoroughly.
+
+#### High-Level Strategy
+- **One Branch Per Package**: To keep PRs manageable and isolated, we suggest a dedicated worktree and branch for each package (e.g., `feat/drop-<dependency>-<version>-<package-name>` i.e. `feat/drop-protobuf-4.25.8-google-cloud-bigquery`).
+- **Small & Reversible Commits**: Group changes into logical commits (Metadata, Nox, Docs, Cleanup, Tests) following Conventional Commits.
+
+---
+
+### Per-Package Workflow
+
+Follow these steps for each package in the target list. Context and warnings are provided inline before the steps where they apply.
+
+#### Step 1: Sync & Branch
+1. Ensure `main` branch is up to date.
+2. Create the feature branch: `git checkout -b feat/drop-<dependency>-<version>-<package-name>`.
+
+#### Step 2: Scan (Baseline)
+1. Run the `version_scanner` for the package to get a list of all occurrences of the dependency and version.
+   > [!TIP]
+   > Use `# version-scanner: ignore` or `ignore-next-line` in code to silence true false-positives and maintain clean reports.
+
+---
+
+#### 💡 Context for Step 3: Standards & Cleanup
+*Before applying changes, review these standards to ensure consistency:*
+
+##### Runtime Version Checks
+- **Standard**: Use `sys.version_info < (X, Y)`.
+- **Rationale**: Python compares tuples lexicographically, making this robust.
+- **Avoid**: `sys.version_info.minor < Y` or string conversions.
+
+##### Pytest Skips
+- **Standard**: `@pytest.mark.skipif(sys.version_info < (X, Y), reason="Requires Python X.Y+")`.
+- **Avoid**: String-based conditions like `@pytest.mark.skipif("sys.version_info < ...")`.
+
+##### Noxfile Version Matches
+- **Standard**: `session.python == "X.Y"` (Nox uses strings).
+- **Avoid**: `float(session.python) < X.Y` (fails for `3.10`).
+
+##### Cleanup Rules
+- **Polyfills**: Remove dead `try/except` blocks guarding polyfills for features now standard in 3.10+.
+- **Obsolete Skips**: Remove pytest skips for features now universally available.
+
+##### Dependency Specific rules
+- Use idiomatic python references to detect dependency versions and to compare against the target version.
+
+---
+
+#### 💡 Context for Step 3: Disposition Rules
+*Every reference to the dependency version found by the scanner must be dispositioned in one of these ways:*
+
+1. **Update**: Update the reference if still necessary (e.g., changing `3.9` to `3.10` in support files).
+2. **Delete**: Delete if no longer relevant (dead code, obsolete comments).
+3. **Pragma Ignore**: Use `# version-scanner: ignore` or `# version-scanner: ignore-next-line` but ONLY for immutable historical facts or true false positives. Do NOT use for things that might change in future upgrades.
+
+#### Step 3: Apply Changes
+1. Update `setup.py` or `pyproject.toml` metadata and `requires-python`.
+2. Update `noxfile.py` to remove old versions from sessions.
+3. Update `README.rst` and `CONTRIBUTING.rst` documentation.
+4. Remove compatibility code and skips based on the standards above.
+5. **Sync Documentation**: If the package has a `docs` folder containing a `README.rst`, copy the updated top-level `README.rst` to overwrite it (unless it is a symlink).
+6. Continue with the update process until all rows from the scan have been properly dispositioned.
+
+---
+
+#### Step 4: Verify (Post-Scan)
+1. Run the `version_scanner` again. The result should be 0 matches (or only valid ignores).
+
+---
+
+#### 💡 Context for Step 5: Constraints & Conflicts
+*Review these lessons learned when dealing with constraints:*
+
+- **Lowest Runtime Constraints**: The file for the lowest accepted runtime (e.g., `constraints-3.10.txt`) must have pins matching the lowest acceptable versions in `setup.py` or `pyproject.toml`.
+- **Philosophy on Warnings**: Do not simply block warnings (like `six` or `pkg_resources`) to make tests pass. **Bump the lower bounds** of dependencies to versions that don't trigger warnings on the current lowest acceptable runtime. This protects customers who use strict warning filters.
+- **SQLAlchemy Transition**: For libraries supporting both 1.4 and 2.0, use `SQLALCHEMY_SILENCE_UBER_WARNING=1` in specific legacy Nox sessions rather than silencing globally.
+
+---
+
+#### Step 5: Local Test
+1. Run unit tests using Nox (e.g., `nox -s unit`).
+   > [!TIP]
+   > Use `nox -s unit-3.10` to save time when debugging specific runtime failures.
+2. Run `blacken` and `lint` sessions.
+
+#### Step 6: Push & PR
+1. Push the branch and create the PR using the template in the Appendix.
+
+---
+
+## Appendix
+
+### PR Template [^1]
+```text
+This PR updates `<dependency>` to establish version x.y.z as the minimum supported version.
+
+### Changes
+* Configuration: Updated `setup.py` and `noxfile.py` to require <dependency> <version> and remove references to older versions.
+* Cleanup: Removed dead code and polyfills no longer needed.
+
+Fixes internal issue: http://b/482126936 🦕
+```
+
+---
+
+## Candidates for `.conductor` or `gemini.md`
+
+*The following guidelines are universal for AI assistants workin' in this repo and should be moved to `.conductor` files or Gemini memories:*
+
+1. **AI & LLM Guidelines for Verification**:
+   - Use Git Worktrees to scan branches without switching.
+   - Run scanner from main branch pointing to worktree.
+   - Bypass env artifacts by worktree only checking out tracked files.
+2. **Automated Bisection**:
+   - Use `version_bisector.py` to find lowest workable versions.
+   - Abort tests early as soon as collection succeeds to save time.
+
+[^1]: Adapted from the standard PR template used in this repository.

scripts/version_scanner/benchmark.py

@@ -0,0 +1,180 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import os
+import random
+import subprocess
+import sys
+import tempfile
+import time
+from typing import List, Dict
+
+def get_package_subset(packages_dir: str, count: int) -> List[str]:
+    """
+    Get a randomized subset of package names from the specified directory.
+    
+    Args:
+        packages_dir: Path to the directory containing packages.
+        count: Number of packages to return.
+        
+    Returns:
+        A list of package directory names.
+    """
+    all_packages = [d for d in os.listdir(packages_dir) if os.path.isdir(os.path.join(packages_dir, d))]
+        
+    if count >= len(all_packages):
+        return all_packages
+        
+    return random.sample(all_packages, count)
+
+def run_benchmark(
+    scanner_path: str,
+    root_path: str,
+    package_file: str,
+    dependency: str,
+    version: str
+) -> float:
+    """
+    Run the scanner and return the duration in seconds.
+    """
+    cmd = [
+        "python3", scanner_path,
+        "-d", dependency,
+        "-v", version,
+        "-p", root_path,
+        "--package-file", package_file
+    ]
+    
+    start_time = time.perf_counter()
+    
+    try:
+        result = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    except subprocess.CalledProcessError as e:
+        print(f"Error running benchmark: {e}")
+        return -1.0
+        
+    duration = time.perf_counter() - start_time
+    return duration
+
+def run_benchmarks(
+    scanner_path: str,
+    root_path: str,
+    packages_dir: str,
+    counts: List[int],
+    dependency: str,
+    version: str
+) -> Dict[int, float]:
+    """Runs benchmarks for specified counts and returns a dict of results."""
+    results = {}
+    
+    for count in counts:
+        subset = get_package_subset(packages_dir, count)
+        print(f"  Testing {len(subset)} packages (e.g., {subset[:3]}...)")
+        
+        # Create temp package file
+        with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
+            for pkg in subset:
+                f.write(f"packages/{pkg}\n")
+            pkg_file = f.name
+                
+        try:
+            duration = run_benchmark(scanner_path, root_path, pkg_file, dependency, version)
+            results[count] = duration
+        finally:
+            # Clean up
+            if os.path.exists(pkg_file):
+                os.remove(pkg_file)
+            
+    return results
+
+def main():
+    parser = argparse.ArgumentParser(description="Benchmark the version scanner.")
+    
+    parser.add_argument(
+        "-s", "--scanner-path",
+        default="version_scanner.py",
+        help="Path to version_scanner.py"
+    )
+    
+    parser.add_argument(
+        "-r", "--root-path",
+        required=True,
+        help="Path to the monorepo root directory"
+    )
+    
+    parser.add_argument(
+        "-p", "--packages-dir",
+        help="Path to packages directory (defaults to <root-path>/packages)"
+    )
+    
+    parser.add_argument(
+        "-d", "--dependency",
+        default="python",
+        help="Dependency to search for"
+    )
+    
+    parser.add_argument(
+        "-v", "--version",
+        default="3.7",
+        help="Version to search for"
+    )
+    
+    parser.add_argument(
+        "-c", "--counts",
+        default="1,10,50",
+        help="Comma-separated list of package counts to test"
+    )
+    
+    args = parser.parse_args()
+    
+    packages_dir = args.packages_dir or os.path.join(args.root_path, "packages")
+    
+    if not os.path.exists(packages_dir):
+        print(f"Error: Packages directory not found: {packages_dir}", file=sys.stderr)
+        sys.exit(1)
+        
+    counts = [int(c) for c in args.counts.split(',')]
+    
+    all_packages = [d for d in os.listdir(packages_dir) if os.path.isdir(os.path.join(packages_dir, d))]
+        
+    total_packages = len(all_packages)
+    
+    print(f"Found {total_packages} packages in {packages_dir}")
+    
+    # Filter counts that are greater than total packages
+    counts = [c for c in counts if c <= total_packages]
+    # Add total if not already there
+    if total_packages not in counts:
+        counts.append(total_packages)
+        
+    print(f"Running benchmarks for counts: {counts}")
+    
+    results = run_benchmarks(
+        scanner_path=args.scanner_path,
+        root_path=args.root_path,
+        packages_dir=packages_dir,
+        counts=counts,
+        dependency=args.dependency,
+        version=args.version
+    )
+    
+    print("\nBenchmark Results:")
+    print(f"{'Packages':<10} | {'Time (seconds)':<15}")
+    print("-" * 30)
+    for count, duration in results.items():
+        print(f"{count:<10} | {duration:<15.4f}")
+
+if __name__ == "__main__":
+    main()