DatabaseSessionService.append_event raises MissingGreenlet with asyncpg after client disconnect (regression from da73e718ef)

[KrishnaVemu]

Summary

With postgresql+asyncpg:// and DatabaseSessionService, every SSE/stream
client disconnect mid-run produces:

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called;
can't call await_only() here.

Reproduces 100% of the time when the client closes the stream while
append_event is committing.

Version

• google-adk: 1.32.0 (verified identical code on main / 1.33 / 1.34 / 2.0)
• SQLAlchemy: 2.x
• asyncpg: 0.31.0
• Python: 3.12
• Driver recommended by ADK docs: asyncpg

Root cause

1. database_session_service.py:208 forces pool_pre_ping=True for any
   non-SQLite URL (added in da73e71 on 2026-02-03 — this is a regression
   for asyncpg users).
2. append_event at L740 does await sql_session.commit(), then at L743
   accesses storage_session.update_time via
   storage_session.get_update_timestamp(is_sqlite).
3. With async engine + commit, that attribute access lazy-loads → pool
   checkout → _do_ping_w_event → await_only() outside the greenlet
   bridge → MissingGreenlet.
4. Normally the request completes before this triggers. A client disconnect
   raises GeneratorExit through parallel_agent._merge_agent_run, which
   surfaces the bad code path.

Repro

• Run any agent under an SSE endpoint with DatabaseSessionService(db_url="postgresql+asyncpg://...") (no extra kwargs — defaults only).
• Disconnect the client mid-stream (close tab / abort fetch).
• Server log shows the MissingGreenlet stack from append_event.

Stack trace (abridged)

File ".../google/adk/sessions/database_session_service.py", line 743, in append_event
    session.last_update_time = storage_session.get_update_timestamp(is_sqlite)
File ".../google/adk/sessions/schemas/v1.py", line 131, in get_update_timestamp
    return self.update_time.timestamp()
File ".../sqlalchemy/orm/attributes.py", line 569, in __get__
    return self.impl.get(state, dict_)
...
File ".../sqlalchemy/dialects/postgresql/asyncpg.py", line 816, in ping
    _ = self.await_(self._async_ping())
File ".../sqlalchemy/util/_concurrency_py3k.py", line 123, in await_only
    raise exc.MissingGreenlet(...)
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called;
can't call await_only() here.

Workaround (confirmed)

Pass pool_pre_ping=False — kwargs are forwarded to create_async_engine:

DatabaseSessionService(db_url=url, pool_pre_ping=False)

But pool_pre_ping is the recommended setting for production, so
disabling it is not a real fix.

Prior reports (closed stale, not fixed)

• DatabaseSessionService with Postgres: sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called #548
• greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s) #1276
• DatabaseSessionService uses sync DB operations - needs async variant #1005, synchronous psycopg2 dependency in postgres database sessionservice , while adk is async all the way #1750 (related)
• Add support for reusing an existing SQLAlchemy AsyncEngine in DatabaseSessionService #3565 (related pool/asyncpg issue, still open)

Suggested fixes

1. Refresh update_time explicitly before commit, so the post-commit
   access doesn't lazy-load:

   await sql_session.refresh(storage_session, ["update_time"])
   await sql_session.commit()

2. Read update_time into a local before commit (cheapest fix):

   updated_at = storage_session.update_time
   await sql_session.commit()
   session.last_update_time = updated_at.timestamp()

3. Use expire_on_commit=False consistently and guard the lazy-load path.

Happy to send a PR if a maintainer can confirm preferred approach.

[Ilya0527]

The MissingGreenlet on client-disconnect mid-commit is the signature of asyncio cancelling the task between the SQLAlchemy session context and the actual await. The greenlet bridge SQLAlchemy uses for sync-style API over asyncio drivers only exists inside AsyncSession's context — if the surrounding task gets cancelled (which client-disconnect triggers via the SSE stream's cancellation propagation), the next await_only() call lands without its greenlet.

The regression at da73e718ef likely moved either (a) the await session.commit() outside the async with AsyncSession(...) as session: block, or (b) replaced an explicit session.execute(...) await with an implicit one that runs through the greenlet bridge.

Two structural fixes worth considering in the same PR:

1. Shield the commit from cancellation with asyncio.shield(session.commit()) — the commit completes even if the SSE task is cancelled. Trade-off: the client disconnect doesn't actually abort the write, so transient client churn can still incur DB cost. Usually correct for audit/append-only writes; wrong for ephemeral session-state writes.

2. Detect-and-skip on cancellation — wrap append_event in a try/except asyncio.CancelledError that checks if the session is still attached before re-raising. Cleaner semantics but masks the original cancellation chain from upstream observers.

The cleanest test fixture I'd want before either fix lands: a parametrized test that fires client.cancel() at every await-point in append_event. Currently the bug is "100% repro on disconnect mid-commit" — the fixture surface is wider than that single point.

Which commit shape is closer to your reproducer: option 1's symptom (commit outside session context) or option 2's (implicit await through bridge)?

[KrishnaVemu]

Thanks for digging in — but I think the cancellation framing is leading us away from the actual mechanism. Let me lay out what I've verified, because the specifics matter for the fix.

1. The greenlet bridge is per-await, not per-AsyncSession context.

Each await async_session.something() spawns a fresh greenlet via greenlet_spawn, runs the sync ORM code inside it, and the greenlet exits when that await returns. It does not persist across awaits, and async with AsyncSession(...) is not what creates it. So "the surrounding task gets cancelled and the next await_only() lands without its greenlet" mischaracterizes the bridge. Cancellation between awaits would just give a normal CancelledError, not MissingGreenlet.

2. da73e718ef did not move the commit or swap explicit awaits for implicit ones.

The diff is one line: engine_kwargs.setdefault("pool_pre_ping", True) for non-SQLite engines. That's it. Verifiable in database_session_service.py:208. The commit shape is neither (a) nor (b) you proposed.

3. The actual mechanism is pool_pre_ping + onupdate=func.now() + sync attribute access.

schemas/v1.py:96 declares update_time with onupdate=func.now(). After the UPDATE flush, SQLAlchemy doesn't know what timestamp Postgres wrote, so it marks the attribute as expired — independent of expire_on_commit. Then database_session_service.py:743 reads it:

await sql_session.commit()                                                   # L740
session.last_update_time = storage_session.get_update_timestamp(is_sqlite)   # L743

The read on L743 happens via __get__ on a Mapped attribute — plain sync attribute access, no await — which triggers _load_expired → load_on_pk_identity → sync pool checkout → pool_pre_ping=True → asyncpg dialect's do_ping → await_only(self._async_ping()) → no surrounding greenlet → MissingGreenlet. The full chain is in the trace at the top of this issue.

Client disconnect is not required — it just changes the pool's checkout timing so the bug surfaces reliably. Without disconnect, the same connection used for the commit is sometimes still hot in the fairy and pre-ping is skipped; with disconnect, the teardown returns it and the next checkout pings.

4. Why the proposed fixes don't address it:

• asyncio.shield(session.commit()) — the commit is not what fails. L743 fires after a successful commit. Shielding the commit changes nothing.
• except asyncio.CancelledError around append_event — there's no CancelledError in this trace, only MissingGreenlet. The bug reproduces without any cancellation.

5. Cleanest structural fix is at L737–L743, not at cancellation boundaries.

Either:

(a) Read into a local before commit so no lazy-load is needed:

storage_session.update_time = update_time
await sql_session.flush()
await sql_session.refresh(storage_session, ["update_time"])  # one await, inside the bridge
updated_at = storage_session.update_time
await sql_session.commit()
session.last_update_time = (
    updated_at.replace(tzinfo=timezone.utc).timestamp() if is_sqlite
    else updated_at.timestamp()
)

(b) Drop onupdate=func.now() and trust the Python-assigned value (it's already set on L737), which removes the post-commit refresh entirely.

Both make the bug structurally impossible regardless of pool_pre_ping or cancellation timing. A regression test should just hit the endpoint normally — no client cancellation choreography needed.

[surajksharma07]

@KrishnaVemu the pool_pre_ping default introduced in da73e71 combined with the expired update_time on the post-commit attribute access is a precise root cause.

Worth trying: move both get_update_timestamp(is_sqlite) and get_update_marker() into locals before the await sql_session.commit() at L747, then assign them to session after.
That removes any post-commit ORM attribute access entirely and sidesteps the lazy-load/greenlet bridge issue.

Could you test this thoroughly in your environment and confirm it consistently clears the MissingGreenlet across your use cases.

[Ilya0527]

The pool_pre_ping=True + onupdate=func.now() + sync attribute access chain is a precise diagnosis. The _load_expired → sync pool checkout → asyncpg do_ping → await_only path is exactly the kind of bug that doesn't appear in the cancellation chain because it isn't cancellation-driven — the SSE disconnect just changes pool-checkout timing enough to make the synchronous re-attach attempt land outside the active greenlet. Without disconnect, the same code path either reuses the hot connection (no ping) or hits a fresh checkout that happens to land inside a different await-bridge.

surajksharma07's suggestion (read get_update_timestamp / get_update_marker into locals before await sql_session.commit()) sidesteps the post-commit attribute access, which removes the trigger for _load_expired. The mechanism is sound. One adjacent hazard worth flagging before the fix lands as the canonical answer:

storage_session.get_update_timestamp(is_sqlite) returns either the application-supplied timestamp or func.now() (server-side). The pre-commit local capture works for the application-supplied branch — the value is already known before the commit. For the server-side branch (func.now()), the pre-commit read is reading the SQL expression object, not the value Postgres will write at commit time. Whatever the local stores is the SQLAlchemy func.now() clause, and downstream code that assigns it to session.last_update_time will either coerce to a string repr or store a ClauseElement that breaks any consumer expecting a datetime.

The safer shape is one of:

1. Force default= (client-side) instead of onupdate=server-now for the update_time column. The _load_expired path doesn't fire because the attribute is no longer marked expired post-flush. Costs a small precision/skew gap (Python clock vs. DB clock), which for session-update timestamps is usually acceptable. Two-line change in schemas/v1.py.

2. Keep the server-side default, but capture the value via RETURNING update_time in the same commit. SQLAlchemy 2.x supports update().returning(...) and asyncpg honors it; the value lands as a result row before the commit closes, avoiding the post-commit re-attach.

3. Disable pool_pre_ping for asyncpg specifically. The da73e718ef change defaulted it for all non-SQLite engines, but pool_pre_ping was originally designed for sync drivers — asyncpg's reconnect handling is already robust enough that the ping is more cost than safety. engine_kwargs.setdefault("pool_pre_ping", not is_asyncpg(url)) resolves the immediate regression without touching the schema or commit path.

Each has different blast-radius. Option 3 is the smallest reversal of the regression; option 1 is the cleanest schema-level fix; option 2 keeps server-side timestamping intact. Option 2's main risk is that it changes the SQL shape of every session update, which may affect Postgres query plan caching in long-running deployments.

Falsifiable check that distinguishes 3 from 1+2: revert da73e718ef locally, re-run the SSE-disconnect repro. If MissingGreenlet no longer appears, option 3 is the minimum fix and the post-commit attribute access is merely the trigger, not the root issue.