ent-prod env reports Invalid database url

[data-sync-user]

Sentry Issue: SYNCSTORAGE-RS-HG

panic: called `Result::unwrap()` on an `Err` value: ApiError { kind: Db(DbError { kind: Common(SyncstorageDbError { kind: Internal("Invalid database url: mysql://root@127.0.0.1/syncstorage"), status: 500, backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
 }), status: 500, backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
 }), backtrace:    0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: <unknown>
  15: <unknown>
  16: <unknown>
  17: <unknown>
, status: 500 }
  File "main.rs", line 57, in syncserver::main::{{closure}}
  File "runtime.rs", line 138, in actix_rt::runtime::Runtime::block_on
  File "system.rs", line 240, in actix_rt::system::SystemRunner::block_on
  File "main.rs", line 27, in syncserver::main
  ?, in main
...
(66 additional frame(s) were not displayed)

A recent new release seems to have auto deployed this environment, which then began producing this error. ent-prod was probably originally configured to point to Spanner (never migrated to postgres). I'm not sure why this error started occurring just now.

The ent-stage/prod envs are likely to be decommissioned but in the meantime, we can probably apply a quick fix to stop this event.

┆Issue is synchronized with this Jira Task

[data-sync-user]

Taddes Korris commented from Jira:

Hi[~accountid:60ec31a9081fcd0070a1db99]! [~accountid:5d264129831d7b0bcfbe3cfb] had told me you might have some context on this as I know the original plan was to decommission this env, but we are keeping it in the end. Just checking in because if it stays, we’ll want to make sure the env is properly configured and we address this Sentry invalid db url.

[data-sync-user]

Taddes Korris commented from Jira:

Think I sussed out the origin of this (probably). ArgoCD Image Updater auto-deployed a new latest tag (both ent images are pinned to tag: latest, pullPolicy: Always), which restarted the pods at some point. The fresh pod re-read the current (incomplete) config which was maybe migrated to the new mozlcoud charts and panicked. The old pod was likely running from before the config drifted, so nothing failed until the restart.

[data-sync-user]

Taddes Korris commented from Jira:

After external discussion, we will do a full decommission of the environments, so a subsequent task will remove the configs from webservices-infra. Src in Slack https://mozilla.slack.com/archives/C09T69S05C4/p1783439254561659?thread_ts=1781040118.256139&cid=C09T69S05C4