cli: detect sql-server via creds file before probing DBs#11115
Open
zook-bot wants to merge 1 commit into
Open
Conversation
buildLateBinder previously gated FindAndLoadLocalCreds on an access-mode probe that loaded the target env's DoltDB and, if that lookup didn't resolve, iterated mrEnv and loaded every other database in the data dir. Each load attempts the journal manifest's exclusive file lock with a 100 ms timeout (file_manifest.go) plus the usual manifest/journal I/O, so when a sql-server holds the locks the probe costs roughly 100 ms × N databases just to learn something a single file read would have told us. Since FindAndLoadLocalCreds is the only way the CLI auto-connects to a local server anyway, hoist it to be the first signal. If a sql-server.info exists up-tree, we connect; otherwise we fall through to local mode. The access-mode probe is removed: it never influenced the local-mode fallback (no creds file -> local regardless) and was only gating the creds lookup. Behavior change: a stale sql-server.info left behind by a crashed server will now surface a connection error instead of silently falling back to local mode. Deleting the stale file restores prior behavior. Benchmark (100 databases, sql-server running, dolt remote -v from the top-level data dir): before: 10.6 s loads every DB, then fails with "not a valid dolt repo" after: 0.18 s reads sql-server.info, then fails with the same error The dolt commands branch -a / status see a similar ~2-3x speedup when invoked with --use-db against a server.
6 tasks
Contributor
|
Hi @zook-bot. Thank's for your interest in Dolt and for the contribution. The semantics and cleanliness around database loading and locking aren't great currently and they reflect quite a few warts accrued on the path-dependent walk they took from purely-CLI-based-tool to SQL-server to CLI-operations-invoked over SQL, etc. I have taken a slightly different approach to try to fix the pain points brought up here and in #11116. The two relevant PRs have landed in
Would you be able to checkout |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
buildLateBinderingo/cmd/dolt/dolt.gopreviously gatedFindAndLoadLocalCredson an access-mode probe that loaded the target env's DoltDB and, if that didn't resolve, iteratedmrEnvand loaded every other database in the data dir. Each load attempts the journal manifest's exclusive file lock with a 100ms timeout (go/store/nbs/file_manifest.go:46) plus the usual manifest/journal I/O, so when a sql-server holds the locks the probe costs ~100ms × N databases just to infer what a single file read would have told us.Since
FindAndLoadLocalCredsis the only way the CLI auto-connects to a local server, hoist it to be the first signal. Ifsql-server.infoexists up-tree, connect; otherwise fall through to local mode. The access-mode probe is removed — it never influenced the local-mode fallback (no creds file → local regardless) and was only gating the creds lookup.Behavior change
A stale
sql-server.infoleft behind by a crashed server will now surface a connection error instead of silently falling back to local mode. Deleting the stale file restores prior behavior. The trade-off felt fair given the perf win — happy to revisit if reviewers disagree.Benchmark
100 databases, sql-server running,
dolt remote -vfrom the top-level data dir:dolt remote -v(no--use-db, top-level data dir)dolt --use-db db_1 remote -vdolt --use-db db_1 branch -adolt --use-db db_1 statusIn the worst case both binaries still print "not a valid dolt repository" (
CheckEnvIsValidis a separate gate). The new one just fails fast.Test plan
go test ./cmd/dolt/commands/...— failures present with fix are identical to those onmain(pre-existing flakes; verified withgit stash)bats remote-cmd.bats— 8/8bats remotes.bats— 70/70bats remotes-push-pull.bats— 33/33bats sql-server-remotesrv.bats— 25/25Follow-up
A companion PR will address the related cost in
MultiEnvForDirectory(an eagerLoadDoltDBon the rootEnv that fires before this code runs). It accounts for the remaining ~1s when invoking from inside a single DB while a server holds its lock.