[apiview-stub-generator] Make package-install timeout configurable (120s -> 300s) (#16395)

* [apiview-stub-generator] Make package-install timeout configurable (120s -> 300s)

The API-stub generator installs the target package (with all runtime deps) into
an isolated venv before introspection, under a hardcoded 120s timeout. Packages
with large dependency trees can exceed this on slower CI agents even when the
install is healthy — e.g. azure-ai-agentserver-core (which pulls the Microsoft
OpenTelemetry distro, ~200 transitive packages) was observed installing in 35s
on one azsdk-pool agent and 136s on another (identical tree, pure per-agent
speed variance), failing API Review on both PRs and main.

Raise the default to 300s and allow override via APISTUB_INSTALL_TIMEOUT.

* Validate APISTUB_INSTALL_TIMEOUT: fall back to default on non-integer or non-positive values

Addresses review: a malformed env value (e.g. 'abc', '0', '-1') no longer
crashes with a raw ValueError or an immediate timeout. Invalid or non-positive
values are ignored with a warning and the 300s default is used.

* Keep default install timeout at 120s; only make it overridable

Per review feedback: preserve the previous default (120s) and provide
APISTUB_INSTALL_TIMEOUT purely as an opt-in override for pipelines whose
packages need more time. No behavior change for existing consumers.

* Bump apistub package-install timeout 120s -> 300s

Large dependency trees (e.g. the Microsoft OpenTelemetry distro) can exceed
the 120s install timeout on slower CI agents even when the install is healthy,
causing spurious API Review failures. Raise to 300s.

* ci: authenticate pip to dev feed before installing tox

The azsdk-pool CI agents no longer have direct public PyPI egress, so
'pip install tox' fails with '[Errno 1] Operation not permitted' /
'No matching distribution found for tox'. Add a PipAuthenticate@1 step so
pip (and tox's nested pip) route through the authenticated
public/azure-sdk-for-python feed, which proxies PyPI.

* ci: bump lazy-object-proxy 1.10.0 -> 1.12.0 for Python 3.13/3.14 wheels

lazy-object-proxy 1.10.0 only ships wheels for cp310-cp312. On Python
3.13/3.14 CI legs, pip fell back to building it from sdist, which triggered a
nested build-dependency install that prompted for feed auth
(User for pkgs.dev.azure.com:) and failed with EOFError. 1.12.0 publishes
prebuilt cp313 and cp314 wheels (and still cp310-cp312), so no sdist build is
needed and the 3.13/3.14 legs no longer hit the auth prompt.

* ci: fix python-tool Build stage for network-restricted agents

Two failures in the shared python-tool archetype Build stage:

1. 'Use Python 3.8' failed - the CI images now ship only Python 3.10-3.14
   and 3.8 can no longer be downloaded. Bump the tooling bootstrap version
   in use-python-version.yml from 3.8 to 3.10 (lowest version on the image).

2. The Build job's pip installs (tooling packaging install + Setup Python
   Environment) had no feed auth, so they failed with
   '[Errno 1] Operation not permitted' now that agents lack public PyPI
   egress. Add a PipAuthenticate@1 step as the first step of the Build job
   so PIP_INDEX_URL is set before any pip install.

* Use auth-sev-feed.yml

* update azure_test.ynl to download packages from devOps feed rather that public pypi

* Raise apistub install timeout 300s -> 1000s to diagnose slow core install

* Revert install timeout back to 300s (1000s was a diagnostic experiment)

---------

Co-authored-by: Chidozie Ononiwu <chononiw@microsoft.com>

Author: Hameedkunkanoor

eng/common/pipelines/templates/steps/auth-dev-feed.yml

@@ -9,7 +9,7 @@ steps:
       # For safety default to publishing to the private feed.
       # Publish to https://dev.azure.com/azure-sdk/internal/_packaging?_a=feed&feed=azure-sdk-for-python-pr
       $devopsFeedName = 'internal/azure-sdk-for-python-pr'
-      if ('$(Build.Repository.Name)' -eq 'Azure/azure-sdk-for-python') {
+      if (-not ('$(Build.Repository.Name)').EndsWith('-pr')) {
         # Publish to https://dev.azure.com/azure-sdk/public/_packaging?_a=feed&feed=azure-sdk-for-python
         $devopsFeedName = '${{ parameters.DevFeedName }}'
       }

eng/pipelines/templates/stages/archetype-sdk-tool-python.yml

@@ -42,6 +42,8 @@ extends:
               os: linux
 
             steps:
+              - template: /eng/common/pipelines/templates/steps/auth-dev-feed.yml
+
               - template: /eng/pipelines/templates/steps/use-python-version.yml
                 parameters:
                   versionSpec: '${{ parameters.PythonVersion }}'

eng/pipelines/templates/steps/use-python-version.yml

@@ -2,11 +2,12 @@ parameters:
   versionSpec: ''
 
 steps:
-  # use python 3.8 for tooling. packaging. platform.
+  # use python 3.10 for tooling. packaging. platform.
+  # (3.8 is no longer present on the CI images, which ship 3.10+.)
   - task: UsePythonVersion@0
-    displayName: "Use Python 3.8"
+    displayName: "Use Python 3.10"
     inputs:
-      versionSpec: 3.8
+      versionSpec: '3.10'
 
   - pwsh: |
       python -m pip install packaging==20.4

packages/python-packages/apiview-stub-generator/CHANGELOG.md

@@ -1,5 +1,8 @@
 # Release History
 
+## Version 0.3.29 (2026-07-17)
+Raised the package-install timeout in the API-stub generator from 120s to 300s. Packages with large dependency trees (e.g. those pulling the Microsoft OpenTelemetry distro) could exceed the previous 120s limit on slower CI agents even though the install was healthy, causing spurious API Review failures.
+
 ## Version 0.3.28 (2026-04-14)
 Dropped 3.7/3.8/3.9 support and added 3.11/3.12/3.13/3.14.
 Added `crossLanguageVersion` to the `CrossLanguageMetadata` model.

packages/python-packages/apiview-stub-generator/apistub/_stub_generator.py

@@ -450,7 +450,12 @@ def _get_package_name_from_metadata_files(self, path):
 
     def _install_package(self):
         commands = [sys.executable, "-m", "pip", "install", self.pkg_path, "-q"]
-        result = run(commands, timeout=120, stderr=PIPE, text=True)
+        # Packages with large dependency trees (e.g. those pulling the Microsoft
+        # OpenTelemetry distro) can exceed a short install timeout on slower CI
+        # agents even though the install is healthy — the same install has been
+        # observed to take 35s on one agent and 136s on another. Use 300s to
+        # avoid spurious API Review failures.
+        result = run(commands, timeout=300, stderr=PIPE, text=True)
         if result.stderr:
             logging.debug("pip stderr: %s", result.stderr.strip())
         if result.returncode != 0:

packages/python-packages/apiview-stub-generator/apistub/_version.py

@@ -1,4 +1,4 @@
 # Copyright (c) Microsoft Corporation. All rights reserved.
 # Licensed under the MIT License.
 
-VERSION = "0.3.28"
+VERSION = "0.3.29"

packages/python-packages/apiview-stub-generator/apiview_reqs.txt

@@ -3,7 +3,7 @@ charset-normalizer==3.4.1
 dill==0.3.9
 isodate==0.6.1
 isort==5.13.2
-lazy-object-proxy==1.10.0
+lazy-object-proxy==1.12.0
 mccabe==0.7.0
 pkginfo==1.12.1.2
 platformdirs==4.3.6

packages/python-packages/apiview-stub-generator/ci.yml

@@ -37,6 +37,7 @@ extends:
     ArtifactName: 'apiviewparserpython'
     PackageName: 'Python API View Parser'
     TestSteps:
+    - template: /eng/common/pipelines/templates/steps/auth-dev-feed.yml
     - script: |
         python $(Build.SourcesDirectory)/eng/scripts/python_version_check.py
       displayName: 'Verify Python APIView version consistency'

packages/python-packages/apiview-stub-generator/tests/azure_tests/azure_test.py

@@ -128,6 +128,56 @@ def _download_file(dest_folder, url):
         print(f"Error downloading {url}: {e}")
         return None
 
+def _download_package_from_index(temp_dir, package_name, version, pkg_type):
+    """
+    Download a wheel/sdist from the configured pip index.
+
+    In CI, PIP_INDEX_URL is set by PipAuthenticate to the authenticated
+    Azure DevOps feed URL.
+    """
+    index_url = os.environ.get(
+        "PIP_INDEX_URL",
+        "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/",
+    )
+    package_spec = f"{package_name}=={version}"
+
+    download_cmd = [
+        "python",
+        "-m",
+        "pip",
+        "download",
+        package_spec,
+        "--no-deps",
+        "--dest",
+        temp_dir,
+        "--index-url",
+        index_url,
+    ]
+
+    if pkg_type == "whl":
+        download_cmd.append("--only-binary=:all:")
+    elif pkg_type == "sdist":
+        download_cmd.append("--no-binary=:all:")
+    else:
+        raise ValueError(f"Unsupported package type: {pkg_type}")
+
+    existing_files = set(os.listdir(temp_dir))
+    check_call(download_cmd)
+    downloaded_files = set(os.listdir(temp_dir)) - existing_files
+
+    if pkg_type == "whl":
+        candidates = [f for f in downloaded_files if f.endswith(".whl")]
+    else:
+        candidates = [f for f in downloaded_files if f.endswith(".tar.gz") or f.endswith(".zip")]
+
+    if not candidates:
+        raise RuntimeError(
+            f"Could not find downloaded {pkg_type} artifact for {package_spec} from {index_url}"
+        )
+
+    candidates.sort()
+    return os.path.join(temp_dir, candidates[0])
+
 def _get_pypi_files(temp_dir, package_name, version, pkg_type, subdirectory):
     """
     Get the wheel and tar.gz files from PyPI for a specific version of a package.
@@ -136,25 +186,7 @@ def _get_pypi_files(temp_dir, package_name, version, pkg_type, subdirectory):
     :param version: Version of the package
     :return: Paths to the downloaded wheel and tar.gz files
     """
-    pypi_url = f"https://pypi.org/pypi/{package_name}/{version}/json"
-    response = requests.get(pypi_url)
-    response.raise_for_status()
-    data = response.json()
-    
-    url = None
-    
-    for file_info in data['urls']:
-        if pkg_type == "whl" and file_info['packagetype'] == 'bdist_wheel':
-            url = file_info['url']
-            break
-        elif pkg_type == "sdist" and file_info['packagetype'] == 'sdist':
-            url = file_info['url']
-            break
-    
-    if not url:
-        raise ValueError(f"Could not find {pkg_type} file for the specified version.")
-    
-    pkg_path = _download_file(temp_dir, url)
+    pkg_path = _download_package_from_index(temp_dir, package_name, version, pkg_type)
     # Copy apiview-properties.json to pkg_path from github repo if it exists
     mapping_file = _get_mapping_file(temp_dir, subdirectory, package_name, version)