add: load schema cache dump from URI at startup

[mkleczek]

Resolves #4999
by making it unnecessary.

[steve-chavez]

Helping to clarify, the motivation is to solve the same problems on #4999. This by loading the schema cache from a file.

[steve-chavez]

Q: If the schema cache file is stale, we still pick it up and serve requests as "best effort"?

[mkleczek]

Helping to clarify, the motivation is to solve the same problems on #4999. This by loading the schema cache from a file.

@steve-chavez I am still working on the right design and will provide more details soon.

I believe now that we can solve in this PR all pain points we want to solve with materialized views.

[mkleczek]

Q: If the schema cache file is stale, we still pick it up and serve requests as "best effort"?

Yes the idea is to treat schema cache preloaded from the dump as stale and trigger asynchronous load from the db.

[steve-chavez]

So if the URI is slow then we will prefer querying the schema cache.

Won't this will always be the case when doing HTTP requests? It seems like this feature will only make sense when it's obtained from a file since that'd be faster.

[mkleczek]

So if the URI is slow then we will prefer querying the schema cache.

Won't this will always be the case when doing HTTP requests? It seems like this feature will only make sense when it's obtained from a file since that'd be faster.

Well... given performance issues with schema cache queries we've seen reported, I wouldn't be so sure.

HTTP is needed because:

1. In k8s environment it is rare to have a shared filesystem to read the file from.
2. Since we provide schema cache dump from admin server it is very convenient for blue/green deployments - just configure each to point to the other one's admin endpoint and you have schema cache pre-loading for free.

Besides: why would HTTP be slow? The most important thing here is not the access protocol but the fact that we serve the dump from memory. I am sure establishing database session and then querying the catalog is going to be much slower than HTTP GET from localhost.
I am pretty sure HTTP is going to be faster than reading from a materialized view as well.

But really we don't want to decide up front which method is preferred. We need to query the database anyway because pre-loaded dump might be stale. So why wait if we are going to do it anyway? It is better to race - that way we are going to have startup as fast as possible - we really don't care which method wins. We only need to make sure the database result is not overwritten.

There is also another angle: we want to deal with thundering herd issue when querying the db. In clustered environments HTTP gives us a natural way to load balance initial schema cache load among existing PostgREST instances. Once we have #4643 or similar it will become even more important as it might affect startup time.

[steve-chavez]

Good points. So we're always going to race here to get the fastest possible startup, got it. We should make this clear on the docs.

[steve-chavez]

Since this is about fast startups, I was wondering if you have considered the in-db config + role settings queries, which if used are also required for startup (done prior to the schema cache):

postgrest/src/PostgREST/Config/Database.hs

Lines 96 to 134 in 6191243

	queryDbSettings :: Maybe Text -> Session [(Text, Text)]
	queryDbSettings preConfFunc =
	SQL.transactionNoRetry SQL.ReadCommitted SQL.Read $ SQL.statement dbSettingsNames $ SQL.Statement sql (arrayParam HE.text) decodeSettings True
	where
	sql = encodeUtf8 [trimming|
	WITH
	role_setting AS (
	SELECT setdatabase as database,
	unnest(setconfig) as setting
	FROM pg_catalog.pg_db_role_setting
	WHERE setrole = quote_ident(CURRENT_USER)::regrole::oid
	AND setdatabase IN (0, (SELECT oid FROM pg_catalog.pg_database WHERE datname = CURRENT_CATALOG))
	),
	kv_settings AS (
	SELECT database,
	substr(setting, 1, strpos(setting, '=') - 1) as k,
	substr(setting, strpos(setting, '=') + 1) as v
	FROM role_setting
	${preConfigF}
	)
	SELECT DISTINCT ON (key)
	replace(k, '${prefix}', '') AS key,
	v AS value
	FROM kv_settings
	WHERE k = ANY($$1) AND v IS NOT NULL
	ORDER BY key, database DESC NULLS LAST;
	|]
	preConfigF = case preConfFunc of
	Nothing -> mempty
	Just func -> [trimming|
	UNION
	SELECT
	null as database,
	x as k,
	current_setting(x, true) as v
	FROM unnest($$1) x
	JOIN ${func}() _ ON TRUE
	|]::Text
	decodeSettings = HD.rowList $ (,) <$> column HD.text <*> column HD.text

postgrest/src/PostgREST/Config/Database.hs

Lines 136 to 185 in 6191243

	queryRoleSettings :: PgVersion -> Session (RoleSettings, RoleIsolationLvl)
	queryRoleSettings pgVer =
	SQL.transactionNoRetry SQL.ReadCommitted SQL.Read $ SQL.statement mempty $ SQL.Statement sql HE.noParams (processRows <$> rows) True
	where
	sql = encodeUtf8 [trimming|
	with
	role_setting as (
	select r.rolname, unnest(r.rolconfig) as setting
	from pg_auth_members m
	join pg_roles r on r.oid = m.roleid
	where member = quote_ident(current_user)::regrole::oid
	),
	kv_settings AS (
	SELECT
	rolname,
	substr(setting, 1, strpos(setting, '=') - 1) as key,
	substr(setting, strpos(setting, '=') + 1) as value
	FROM role_setting
	),
	iso_setting AS (
	SELECT rolname, value
	FROM kv_settings
	WHERE key = 'default_transaction_isolation'
	)
	select
	kv.rolname,
	i.value as iso_lvl,
	coalesce(array_agg(row(kv.key, kv.value)) filter (where key <> 'default_transaction_isolation'), '{}') as role_settings
	from kv_settings kv
	join pg_settings ps on ps.name = kv.key and (ps.context = 'user' ${hasParameterPrivilege})
	left join iso_setting i on i.rolname = kv.rolname
	group by kv.rolname, i.value;
	|]
	hasParameterPrivilege
	| pgVer >= pgVersion150 = "or has_parameter_privilege(quote_ident(current_user)::regrole::oid, ps.name, 'set')"
	| otherwise = ""
	processRows :: [(Text, Maybe Text, [(Text, Text)])] -> (RoleSettings, RoleIsolationLvl)
	processRows rs =
	let
	rowsWRoleSettings = [ (x, z) | (x, _, z) <- rs ]
	rowsWIsolation = [ (x, y) | (x, Just y, _) <- rs ]
	in
	( HM.fromList $ bimap encodeUtf8 (HM.fromList . ((encodeUtf8 *** encodeUtf8) <$>)) <$> rowsWRoleSettings
	, HM.fromList $ (encodeUtf8 *** toIsolationLevel) <$> rowsWIsolation
	)
	rows :: HD.Result [(Text, Maybe Text, [(Text, Text)])]
	rows = HD.rowList $ (,,) <$> column HD.text <*> nullableColumn HD.text <*> compositeArrayColumn ((,) <$> compositeField HD.text <*> compositeField HD.text)

The in-db configs are not cached but the role settings are cached here:

postgrest/src/PostgREST/Config.hs

Lines 128 to 129 in 6191243

	, configRoleSettings :: RoleSettings
	, configRoleIsoLvl :: RoleIsolationLvl

When adding those I wondered if they should go into schema cache or not, I decided to make them separate to save some work on filtering them for the openAPI output. But with this feature, it looks we should make them part of the schema cache to really get the faster startup?

[steve-chavez]

in-db config + role settings queries

Actually I don't think the "in-db config" can be put inside the schema cache because it depends on it (db-schemas, db-extra-search-path) but the role settings could be put inside the schema cache to save some time.

[steve-chavez]

We should be clear here that even if we load the schema cache from URL, this will be overridden later when the schema cache comes from the db.

[steve-chavez]

This section is more like a part of a how-to. I think we have two options:

1. Remove it. Simplest option.
   • The reference is already pretty clear IMO.
2. Or make it part of a new how-to, like "Fast cold starts for PostgREST".
   • Maybe it could fit into an explanation too.

[steve-chavez]

@mkleczek Does this require #5083? What happens right now since we don't have the pdIsoLvl?

[taimoorzaeem]

Once #5083 gets merged, this can be remove by just adding:

deriving instance JSON.FromJSON SQL.IsolationLevel

and deriving the instance for Routine type like:

data Routine = ...
  ...
  deriving (Eq, Show, Generic, JSON.ToJSON, JSON.FromJSON)

[steve-chavez]

@mkleczek Please address the feedback here 🙏