fix(dynamic-instrumentation): rebind FastAPI Depends() sub-dependencies (#785)

## Description

A **function-level** Dynamic Instrumentation breakpoint set on a
function that is used as a FastAPI `Depends()` **sub-dependency**
silently never fires. The instrumentation status stays `READY` (no
`ERROR`) and **zero snapshots** are produced, even though the dependency
callable executes on every request.

## Root cause

When DI instruments a module-level function it replaces it with a
wrapper and then calls `_patch_framework_references` →
`_patch_single_fastapi_app` to fix up framework-held references. That
patcher rebinds:

- `route.endpoint`, and
- the top-level `route.dependant.call`

…but the per-request dependency solver invokes sub-dependency callables
via `route.dependant.dependencies[*].call` (recursively). Those
references were **never** rebound, so FastAPI kept calling the original,
unwrapped function object and the DI wrapper never ran.

This is distinct from the inherited-method / nested-class silent-failure
cases: here the code-object identity is exactly correct
(`route.dependant.dependencies[0].call is original_func`); the only
problem is that the patcher didn't reach the sub-dependency reference.

```python
async def square_dependency(value: int = 7) -> int:   # used only via Depends()
    return value * value

@app.get("/dep")
async def decorators_dependency(dep: int = Depends(square_dependency)):
    return {"result": dep}
```

A breakpoint on `square_dependency` (function-level) produced 0
snapshots; the same function with a **line-level** breakpoint fired
correctly (the line engine binds the code object directly), which
isolates the defect to the function-level FastAPI-patch path.

## Fix

Walk the whole `Dependant` tree (`endpoint` + `dependant` + nested
`dependencies`) and rebind only the references that match the target
function. Restricting rebinding to matches — rather than the previous
unconditional `route.endpoint = new_func` — prevents clobbering an
unrelated endpoint on a route that matched solely via one of its
sub-dependencies. The same recursive walk is applied in both the
identity-match pass and the name+module fallback, so the fallback now
also reaches sub-dependencies.

## Testing

- Added two regression tests in `test_function_wrapper_fastapi.py`:
- `test_patch_fastapi_routes_sub_dependency` — a `Depends()`
sub-dependency is rebound on `dependant.dependencies[*].call`, the
original is no longer referenced, and the route's own endpoint is not
clobbered.
- `test_patch_fastapi_routes_only_target_sub_dependency_rebound` — with
two `Depends()` on one route, only the targeted sub-dependency is
rebound; the other is left intact.
- Full debugger test suite passes locally: **547 passed, 43 skipped**
(the FastAPI file: 11 passed, 9 pre-existing + 2 new).
- Verified end-to-end on the live Application Signals pipeline (FastAPI
app → CloudWatch Agent → CloudWatch Logs): the function-level breakpoint
on the `Depends()` sub-dependency now reaches status `ACTIVE`
("breakpoint is being hit") and produces a snapshot capturing the
argument (`value=7`) and return (`49`); a sibling route-handler
breakpoint passed as a same-app control, and the endpoint response was
unchanged.

## By submitting this pull request, I confirm that my contribution is
made under the terms of the Apache 2.0 license.

Author: syed-ahsan-ishtiaque

aws-opentelemetry-distro/src/amazon/opentelemetry/distro/debugger/_function_wrapper.py

@@ -1170,29 +1170,58 @@ def _patch_single_fastapi_app(  # pylint: disable=too-many-locals
     ):
         """Patch route handler references in a single FastAPI app instance.
 
-        Rebinds both ``route.endpoint`` and ``route.dependant.call`` (the attribute the
-        dispatcher actually invokes). The pre-built Dependant remains valid because the
-        DI wrapper preserves the handler signature via functools.wraps, so no rebuild is
-        needed. ``app.include_router`` flattens sub-routers into ``app.router.routes``,
-        so iterating that list covers all routes.
+        Rebinds ``route.endpoint``, ``route.dependant.call``, and — recursively —
+        every matching ``route.dependant.dependencies[*].call`` (the attributes the
+        dispatcher actually invokes). The last one matters for functions used as
+        ``Depends(...)`` sub-dependencies: the per-request solver calls those
+        directly, so rebinding only the top-level handler would leave a
+        sub-dependency breakpoint silently never firing. The pre-built Dependant
+        remains valid because the DI wrapper preserves the handler signature via
+        functools.wraps, so no rebuild is needed. ``app.include_router`` flattens
+        sub-routers into ``app.router.routes``, so iterating that list covers all
+        routes.
         """
         router = getattr(fastapi_app, "router", None)
         routes = getattr(router, "routes", None)
         if not isinstance(routes, list):
             logger.debug("FastAPI app '%s' has no router.routes list", app_name)
             return
 
-        def _rebind(route) -> None:
-            route.endpoint = new_func
-            dependant = getattr(route, "dependant", None)
-            if dependant is not None and hasattr(dependant, "call"):
+        def _rebind_dependant(dependant, matches) -> int:
+            """Replace every matching ``call`` in a Dependant tree with new_func.
+
+            FastAPI stores ``Depends(...)`` sub-dependency callables on
+            ``dependant.dependencies[*].call`` (recursively). The per-request
+            solver invokes those directly, so a function used only as a
+            sub-dependency is never reached by rebinding the top-level
+            ``dependant.call`` alone. Walks the whole tree and rebinds only the
+            references that match the target, so a route matched via one
+            sub-dependency does not have its unrelated calls clobbered.
+            Returns the number of references rebound.
+            """
+            if dependant is None:
+                return 0
+            count = 0
+            if matches(getattr(dependant, "call", None)):
                 dependant.call = new_func
-
-        def _is_identity_match(route) -> bool:
-            if getattr(route, "endpoint", None) is original_func:
-                return True
-            dependant = getattr(route, "dependant", None)
-            return dependant is not None and getattr(dependant, "call", None) is original_func
+                count += 1
+            for sub in getattr(dependant, "dependencies", None) or []:
+                count += _rebind_dependant(sub, matches)
+            return count
+
+        def _rebind_route(route, matches) -> int:
+            """Rebind matching references on a route: its ``endpoint`` and every
+            matching ``call`` in its Dependant tree. Only references that match
+            are touched. Returns the number of references rebound."""
+            count = 0
+            if matches(getattr(route, "endpoint", None)):
+                route.endpoint = new_func
+                count += 1
+            count += _rebind_dependant(getattr(route, "dependant", None), matches)
+            return count
+
+        def _is_identity(func) -> bool:
+            return func is original_func
 
         # Identity match first. Guard each route independently so one bad route
         # cannot abort patching of the others.
@@ -1201,17 +1230,17 @@ def _is_identity_match(route) -> bool:
             try:
                 if getattr(route, "endpoint", None) is None:
                     continue  # not an APIRoute (e.g. Mount/WebSocketRoute)
-                if _is_identity_match(route):
-                    _rebind(route)
-                    patched_count += 1
+                rebound = _rebind_route(route, _is_identity)
+                if rebound:
+                    patched_count += rebound
                     logger.debug("Patched FastAPI route '%s' to use DI wrapper", getattr(route, "path", "?"))
             except Exception as exc:
                 logger.warning("Error patching FastAPI route '%s': %s", getattr(route, "path", "?"), exc)
                 continue
 
         if patched_count > 0:
             logger.debug(
-                "Patched %d FastAPI route(s) for function '%s' on app '%s'",
+                "Patched %d FastAPI reference(s) for function '%s' on app '%s'",
                 patched_count,
                 func_name,
                 app_name,
@@ -1223,29 +1252,36 @@ def _is_identity_match(route) -> bool:
         # so requiring both name and module avoids patching a same-named handler from a
         # different module. If the original has no __module__, fall back to name-only.
         def _matches(endpoint):
+            if endpoint is None:
+                return False
             if getattr(endpoint, "__name__", None) != func_name:
                 return False
             if func_module is None:
                 return True
             return getattr(endpoint, "__module__", None) == func_module
 
-        matching_routes = [r for r in routes if getattr(r, "endpoint", None) is not None and _matches(r.endpoint)]
-        if matching_routes:
+        fallback_count = 0
+        for route in routes:
+            try:
+                if getattr(route, "endpoint", None) is None:
+                    continue
+                rebound = _rebind_route(route, _matches)
+                if rebound:
+                    fallback_count += rebound
+                    logger.debug("Patched FastAPI route '%s' by name+module match", getattr(route, "path", "?"))
+            except Exception as exc:
+                logger.warning("Error patching FastAPI route '%s': %s", getattr(route, "path", "?"), exc)
+                continue
+
+        if fallback_count > 0:
             logger.debug(
-                "FastAPI app '%s' has %d route(s) with name '%s' (module '%s') but identity "
-                "mismatch. Patching by name+module.",
+                "FastAPI app '%s' had identity mismatch; patched %d reference(s) for '%s' (module '%s') "
+                "by name+module match.",
                 app_name,
-                len(matching_routes),
+                fallback_count,
                 func_name,
                 func_module,
             )
-            for route in matching_routes:
-                try:
-                    _rebind(route)
-                    logger.debug("Patched FastAPI route '%s' by name+module match", getattr(route, "path", "?"))
-                except Exception as exc:
-                    logger.warning("Error patching FastAPI route '%s': %s", getattr(route, "path", "?"), exc)
-                    continue
 
     @staticmethod
     def _get_qualified_name(original_func: Callable) -> str:

aws-opentelemetry-distro/tests/amazon/opentelemetry/distro/debugger/test_function_wrapper_fastapi.py

@@ -164,6 +164,86 @@ def other_view():
         self.assertIs(route.endpoint, other_endpoint)
         self.assertIs(route.dependant.call, other_call)
 
+    # ------------------------------------------------------------------
+    # Depends() sub-dependency patching (regression: sub-dependency
+    # breakpoints silently never fired because only the top-level
+    # dependant.call was rebound, never dependant.dependencies[*].call)
+    # ------------------------------------------------------------------
+
+    def test_patch_fastapi_routes_sub_dependency(self):
+        """A function used as a Depends() sub-dependency is rebound on dependant.dependencies[*].call.
+
+        The per-request solver invokes the sub-dependency via
+        route.dependant.dependencies[i].call, not the top-level endpoint, so
+        rebinding only endpoint/dependant.call would leave it pointing at the
+        original (the breakpoint would silently never fire).
+        """
+        try:
+            from fastapi import Depends, FastAPI
+        except ImportError:
+            self.skipTest("FastAPI not installed")
+
+        def square_dependency(value: int = 7) -> int:
+            return value * value
+
+        app = FastAPI()
+
+        @app.get("/dep")
+        def decorators_dependency(dep: int = Depends(square_dependency)):
+            return {"result": dep}
+
+        route = _route_for(app, "/dep")
+        sub_calls = [d.call for d in route.dependant.dependencies]
+        # Precondition: the solver currently points at the ORIGINAL sub-dependency.
+        self.assertIn(square_dependency, sub_calls)
+
+        wrapper = lambda value=7: value * value  # noqa: E731
+        wrapper.__name__ = square_dependency.__name__
+        wrapper.__module__ = square_dependency.__module__
+
+        mod = _make_module(self.module_name, app=app, square_dependency=square_dependency)
+
+        FunctionWrapper._patch_fastapi_routes(mod, square_dependency, wrapper)
+
+        rebound_calls = [d.call for d in route.dependant.dependencies]
+        self.assertIn(wrapper, rebound_calls)
+        self.assertNotIn(square_dependency, rebound_calls)
+        # The route's own endpoint (a different function) must not be clobbered.
+        self.assertIsNot(route.endpoint, wrapper)
+
+    def test_patch_fastapi_routes_only_target_sub_dependency_rebound(self):
+        """With two Depends() on a route, only the targeted sub-dependency is rebound."""
+        try:
+            from fastapi import Depends, FastAPI
+        except ImportError:
+            self.skipTest("FastAPI not installed")
+
+        def square_dependency(value: int = 7) -> int:
+            return value * value
+
+        def inner_dep() -> int:
+            return 1
+
+        app = FastAPI()
+
+        @app.get("/nested")
+        def nested_dependency(dep: int = Depends(square_dependency), x: int = Depends(inner_dep)):
+            return {"result": dep + x}
+
+        route = _route_for(app, "/nested")
+        wrapper = lambda value=7: value * value  # noqa: E731
+        wrapper.__name__ = square_dependency.__name__
+        wrapper.__module__ = square_dependency.__module__
+
+        mod = _make_module(self.module_name, app=app, square_dependency=square_dependency)
+
+        FunctionWrapper._patch_fastapi_routes(mod, square_dependency, wrapper)
+
+        rebound_calls = [d.call for d in route.dependant.dependencies]
+        self.assertIn(wrapper, rebound_calls)  # target rebound
+        self.assertIn(inner_dep, rebound_calls)  # the other dep left intact
+        self.assertNotIn(square_dependency, rebound_calls)
+
     # ------------------------------------------------------------------
     # Integration: _replace_function_in_module with FastAPI patching
     # ------------------------------------------------------------------