-
Notifications
You must be signed in to change notification settings - Fork 0
Changestream parallel #40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
FGasper
wants to merge
32
commits into
main
Choose a base branch
from
changestream_parallel
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
32 commits
Select commit
Hold shift + click to select a range
423a12b
save
FGasper f3e8e18
save revert
FGasper e967650
save
FGasper cb14fad
save
FGasper cd1120d
fetch driver
FGasper c22c845
fix ctx
FGasper 6471ef7
fix
FGasper 372ae82
vendor
FGasper faf6f2d
error
FGasper 9580530
nil
FGasper c822c27
tryNext
FGasper 44f8481
set session earlier
FGasper ec23313
work with all change events
FGasper 42f032e
allow custom
FGasper 028a3e3
no sync
FGasper d6cde6a
add test
FGasper 776ccbd
pedantry
FGasper 7535033
csopts
FGasper 2c114dd
rewrwite test
FGasper 192f146
more interesting test
FGasper b4497f2
tidy
FGasper d314384
lint
FGasper 35dec18
test version
FGasper 3be5b68
lines
FGasper 50e48a5
lint
FGasper aacb984
moar
FGasper c130b2a
maybve better
FGasper d44deb5
notices
FGasper 2d25481
simplify
FGasper b832e66
preliminary resume token
FGasper f5f93bb
tweaked/fixed
FGasper 6f5e257
token
FGasper File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| name: Integration | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| integration: | ||
| name: ${{ matrix.mongo_version }} ${{ matrix.topology.name }} | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| mongo_version: ["4.0", "4.2", "4.4", "5.0", "6.0", "7.0", "8.0"] | ||
| topology: | ||
| - name: replset | ||
| args: "--replicaset" | ||
| - name: sharded | ||
| args: "--sharded 2 --replicaset" | ||
| steps: | ||
| - name: Check out repo | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Install tools with `mise` | ||
| uses: jdx/mise-action@v3 | ||
| with: | ||
| # This matches what we use in the mongosync and mongo-tools repos. | ||
| version: 2026.2.19 | ||
|
|
||
| - name: Start MongoDB cluster | ||
| run: | | ||
| docker run \ | ||
| --network host \ | ||
| --name mongo \ | ||
| --detach \ | ||
| --rm \ | ||
| felipegasper298/mongo-provision \ | ||
| ${{ matrix.mongo_version }} \ | ||
| ${{ matrix.topology.args }} | ||
|
|
||
| - name: Await cluster readiness | ||
| run: | | ||
| for i in $(seq 1 120); do | ||
| if docker exec mongo test -f /ready; then | ||
| echo "Cluster ready after ${i} attempts" | ||
| exit 0 | ||
| fi | ||
| echo "Waiting for cluster... (${i}/120)" | ||
| sleep 5 | ||
| done | ||
| echo "Cluster did not become ready in time" | ||
| docker logs mongo | ||
| exit 1 | ||
|
|
||
| - name: Extract connection string | ||
| id: conn | ||
| run: | | ||
| uri=$(docker exec mongo cat /ready | jq -r .connection_string) | ||
| echo "uri=${uri}" >> "$GITHUB_OUTPUT" | ||
| echo "Connection string: ${uri}" | ||
|
|
||
| - name: Run integration tests | ||
| env: | ||
| MIGRATION_TOOLS_MONGODB_URI: ${{ steps.conn.outputs.uri }} | ||
| MIGRATION_TOOLS_MONGODB_TOPOLOGY: ${{ matrix.topology.name }} | ||
| MIGRATION_TOOLS_MONGODB_VERSION: ${{ matrix.mongo_version }} | ||
| run: go test -v -run Integration ./... | ||
|
|
||
| - name: Show MongoDB logs on failure | ||
| if: failure() | ||
| run: docker logs mongo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| package internal | ||
|
|
||
| import ( | ||
| "os" | ||
| "testing" | ||
| ) | ||
|
|
||
| const ( | ||
| // The environment variable to use for the connection string in tests. | ||
| connStrEnv = "MIGRATION_TOOLS_MONGODB_URI" | ||
| topologyEnv = "MIGRATION_TOOLS_MONGODB_TOPOLOGY" | ||
| dbVersionEnv = "MIGRATION_TOOLS_MONGODB_VERSION" | ||
| ) | ||
|
|
||
| // GetTopology returns the provisioned cluster’s topology. | ||
| func GetTopology(t *testing.T) string { | ||
| return getEnvOrSkip(t, topologyEnv) | ||
| } | ||
|
|
||
| func GetDBVersion(t *testing.T) string { | ||
| return getEnvOrSkip(t, dbVersionEnv) | ||
| } | ||
|
|
||
| // GetConnStr returns the provisioned cluster’s connection string. | ||
| func GetConnStr(t *testing.T) string { | ||
| return getEnvOrSkip(t, connStrEnv) | ||
| } | ||
|
|
||
| func getEnvOrSkip(t *testing.T, envName string) string { | ||
| val := os.Getenv(envName) | ||
| if val == "" { | ||
| t.Skipf("%#q not set", envName) | ||
| } | ||
|
|
||
| return val | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| package legacytools | ||
|
|
||
| import ( | ||
| "github.com/samber/lo" | ||
| "go.mongodb.org/mongo-driver/v2/x/mongo/driver/topology" | ||
| ) | ||
|
|
||
| // cf. https://github.com/mongodb/specifications/blob/b519824da64005cdf99ca680fc49c4e278af0ef3/source/wireversion-featurelist/wireversion-featurelist.md | ||
| var dbWireVersion = map[string]int32{ | ||
| "4.0": 7, | ||
| "4.2": 8, | ||
| } | ||
|
|
||
| // SetDriverCompatibility alters the driver’s global state so that it will | ||
| // try to support an older DB version than it otherwise would. | ||
| // | ||
| // It goes without saying: HANDLE WITH CARE! | ||
| func SetDriverCompatibility(dbVersion string) { | ||
| wireVersion, ok := dbWireVersion[dbVersion] | ||
| lo.Assert( | ||
| ok, | ||
| "need wire version for db version %#q", | ||
| dbVersion, | ||
| ) | ||
|
|
||
| topology.MinSupportedMongoDBVersion = dbVersion | ||
| topology.SupportedWireVersions.Min = wireVersion | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
README.md:44The README says to callinternal.GetConnStr(), but the helper ininternal/test.gorequires a*testing.Targument (internal.GetConnStr(t)).Severity: low
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.