MongoDBContainer consumes excessive CPU after #1070

[CodingCanuck]

Expected Behaviour
MongoDBContainer should not consume excessive amounts of CPU

Actual Behaviour
After starting MongoDBContainer, it consumes 50-100% of 1 CPU core, due to repeated replicaset re-initialization

Testcontainer Logs

2025-09-25T09:08:34.162Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.161+00:00"},"s":"I",  "c":"NETWORK",  "id":6788700, "ctx":"conn80","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":2}}
2025-09-25T09:08:34.162Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.162+00:00"},"s":"I",  "c":"COMMAND",  "id":21577,   "ctx":"conn80","msg":"Initiate: no configuration specified. Using a default configuration for the set"}
2025-09-25T09:08:34.162Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.162+00:00"},"s":"I",  "c":"COMMAND",  "id":21578,   "ctx":"conn80","msg":"Created configuration for initiation","attr":{"config":{"_id":"rs0","version":1,"members":[{"_id":0,"host":"666818ea5142:27017"}]}}}
2025-09-25T09:08:34.162Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.162+00:00"},"s":"I",  "c":"REPL",     "id":21356,   "ctx":"conn80","msg":"replSetInitiate admin command received from client"}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.167Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn79","msg":"Connection ended","attr":{"remote":"127.0.0.1:43720","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"a9480a47-939b-4d5a-978f-fc8fe7a08165"}},"connectionId":79,"connectionCount":4}}
2025-09-25T09:08:34.167Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn76","msg":"Connection ended","attr":{"remote":"127.0.0.1:43686","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"35b13d63-89b3-4bab-b36d-f6e961a21092"}},"connectionId":76,"connectionCount":3}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.167Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn78","msg":"Connection ended","attr":{"remote":"127.0.0.1:43714","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"5dfa454c-6f50-44f0-bdb1-b168ae7b0229"}},"connectionId":78,"connectionCount":2}}
2025-09-25T09:08:34.167Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn80","msg":"Connection ended","attr":{"remote":"127.0.0.1:43732","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"f5463634-b201-4a16-9013-264168499a7c"}},"connectionId":80,"connectionCount":1}}
2025-09-25T09:08:34.167Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.166+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn77","msg":"Connection ended","attr":{"remote":"127.0.0.1:43698","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"873968be-b359-4e74-b8bb-7e00dddc898d"}},"connectionId":77,"connectionCount":0}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.677Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.676+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:43738","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"5396470b-fb96-4f02-87fa-cbdbd35c005c"}},"connectionId":81,"connectionCount":1}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.679Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.678+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn81","msg":"client metadata","attr":{"remote":"127.0.0.1:43738","client":"conn81","negotiatedCompressors":[],"doc":{"application":{"name":"mongosh 2.5.6"},"driver":{"name":"nodejs|mongosh","version":"6.16.0|2.5.6"},"platform":"Node.js v20.19.4, LE","os":{"name":"linux","architecture":"arm64","version":"4.14.209-160.339.amzn2.aarch64","type":"Linux"},"env":{"container":{"runtime":"docker"}}}}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.720Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.719+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:43754","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"b491bb8a-927b-4dfc-9aaf-f8b2d06b72ee"}},"connectionId":82,"connectionCount":2}}
2025-09-25T09:08:34.720Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.719+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:43760","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"a179462c-6131-42e1-a5da-cafd8d6c4f22"}},"connectionId":83,"connectionCount":3}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.720Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.719+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn82","msg":"client metadata","attr":{"remote":"127.0.0.1:43754","client":"conn82","negotiatedCompressors":[],"doc":{"application":{"name":"mongosh 2.5.6"},"driver":{"name":"nodejs|mongosh","version":"6.16.0|2.5.6"},"platform":"Node.js v20.19.4, LE","os":{"name":"linux","architecture":"arm64","version":"4.14.209-160.339.amzn2.aarch64","type":"Linux"},"env":{"container":{"runtime":"docker"}}}}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.721Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.720+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn83","msg":"client metadata","attr":{"remote":"127.0.0.1:43760","client":"conn83","negotiatedCompressors":[],"doc":{"application":{"name":"mongosh 2.5.6"},"driver":{"name":"nodejs|mongosh","version":"6.16.0|2.5.6"},"platform":"Node.js v20.19.4, LE","os":{"name":"linux","architecture":"arm64","version":"4.14.209-160.339.amzn2.aarch64","type":"Linux"},"env":{"container":{"runtime":"docker"}}}}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.722Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.722+00:00"},"s":"I",  "c":"NETWORK",  "id":6788700, "ctx":"conn82","msg":"Received first command on ingress connection since session start or auth handshake","attr":{"elapsedMillis":2}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.723Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.722+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:43762","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"0a3db8a0-36fa-492b-ae7e-ead0ced31d85"}},"connectionId":84,"connectionCount":4}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.723Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.723+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:43770","isLoadBalanced":false,"uuid":{"uuid":{"$uuid":"08528504-3252-40c7-89fa-7439b1bbe6b9"}},"connectionId":85,"connectionCount":5}}

stdout | packages/modules/mongodb/src/mongodb-container.test.ts > MongoDBContainer > should have low CPU usage
2025-09-25T09:08:34.724Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.723+00:00"},"s":"I",  "c":"COMMAND",  "id":21577,   "ctx":"conn82","msg":"Initiate: no configuration specified. Using a default configuration for the set"}
2025-09-25T09:08:34.724Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.723+00:00"},"s":"I",  "c":"COMMAND",  "id":21578,   "ctx":"conn82","msg":"Created configuration for initiation","attr":{"config":{"_id":"rs0","version":1,"members":[{"_id":0,"host":"666818ea5142:27017"}]}}}
2025-09-25T09:08:34.724Z testcontainers:containers [666818ea5142] {"t":{"$date":"2025-09-25T09:08:34.723+00:00"},"s":"I",  "c":"REPL",     "id":21356,   "ctx":"conn82","msg":"replSetInitiate admin command received from client"}

Steps to Reproduce

1. Add a test to mongodb-container.test.ts like the following, which waits for the mongo container to start, then repeatedly prints out docker stats to show container CPU usage (but isn't actually asserting correct CPU usage):

// New imports and methods
import { exec } from "node:child_process";
import { promisify } from "node:util";
const execAsync = promisify(exec);

describe("MongoDBContainer", { timeout: 240_000 }, () => {
  // Add this test after existing tests

  it("should have low CPU usage", async () => {
    const image = "mongo:8.0.12";
    await using container = await new MongoDBContainer(image).start();

    const seconds = 10;
    console.log(`Sleeping for ${seconds} seconds...`);
    await new Promise((resolve) => setTimeout(resolve, seconds * 1000));
    console.log("Done sleeping");

    for (let i = 0; i < 5; i++) {
      const { stdout } = await execAsync("docker stats --no-stream");
      let output = `Docker stats ${i}: \n${stdout}`;
      // Prepend a unique string to each line of stdout so it can be grepped
      output = output.replaceAll("\n", "\n======");

      console.log(output);
    }
  });
});

2. Run this test, and grep for relevant console output to just see the Mongo docker container's CPU usage:

npx vitest run packages/modules/mongodb/src/mongodb-container.test.ts -t "should have low CPU usage" --silent=false |& grep "======" | grep -v "testcontainers-"

3. Run this test before and after Add support for MongoDBContainer credentials #1070 , which introduced this CPU-usage regression.

Sample CPU usage stats, after that change (run git checkout c0571da):

======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======ef5b48a8e8fe   focused_greider                    51.46%    123.5MiB / 7.654GiB   1.58%     1.11kB / 126B   0B / 1.03MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======ef5b48a8e8fe   focused_greider                    53.41%    266.1MiB / 7.654GiB   3.40%     1.11kB / 126B   0B / 1.03MB   89
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======ef5b48a8e8fe   focused_greider                    46.68%    181.4MiB / 7.654GiB   2.31%     1.11kB / 126B   0B / 1.33MB   83
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======ef5b48a8e8fe   focused_greider                    52.25%    123.3MiB / 7.654GiB   1.57%     1.11kB / 126B   0B / 1.33MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======ef5b48a8e8fe   focused_greider                    48.96%    262.9MiB / 7.654GiB   3.36%     1.11kB / 126B   0B / 1.34MB   88
======

Sample CPU usage stats, before that change (git checkout c0571da^):

======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d3d4c9762170   blissful_satoshi                   2.47%     120.6MiB / 7.654GiB   1.54%     1.11kB / 126B   0B / 1.04MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d3d4c9762170   blissful_satoshi                   1.44%     120.5MiB / 7.654GiB   1.54%     1.11kB / 126B   0B / 1.04MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d3d4c9762170   blissful_satoshi                   1.82%     120.8MiB / 7.654GiB   1.54%     1.11kB / 126B   0B / 1.13MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d3d4c9762170   blissful_satoshi                   1.84%     120.5MiB / 7.654GiB   1.54%     1.11kB / 126B   0B / 1.13MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d3d4c9762170   blissful_satoshi                   1.37%     120.3MiB / 7.654GiB   1.53%     1.11kB / 126B   0B / 1.13MB   76
======

Environment Information

• Operating System: Mac OS X Sequoia 15.6.1
• Docker Version: Docker Desktop 4.46.0,204649, Docker version 28.4.0, build d8eb465
• Node version: v24.8.0 or v20.18.3
• Testcontainers version: v11.4.0 or higher (regression occurred in Add support for MongoDBContainer credentials #1070 )

(Note, environment / architecture likely doesn't matter to reproduce this issue)

[CodingCanuck]

One possible fix for this CPU usage issue is to change this healthcheck retry code so it's not trying to run a command every 250ms that re-initializes the replicaset.

So replace this:

      interval: 250,
      timeout: 60000,
      retries: 1000,

With this:

      timeout: 60000,
      retries: 0,

Which fixes the high-CPU regression. Sample CPU usage printed by the above test after that fix:

======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======1fd86ab39c17   elegant_kepler                     1.66%     129.7MiB / 7.654GiB   1.66%     1.11kB / 126B   0B / 1.12MB   74
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======1fd86ab39c17   elegant_kepler                     1.39%     129.5MiB / 7.654GiB   1.65%     1.11kB / 126B   0B / 1.12MB   74
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======1fd86ab39c17   elegant_kepler                     1.43%     129.5MiB / 7.654GiB   1.65%     1.11kB / 126B   0B / 1.12MB   74
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======1fd86ab39c17   elegant_kepler                     1.49%     129.5MiB / 7.654GiB   1.65%     1.11kB / 126B   0B / 1.12MB   74
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======1fd86ab39c17   elegant_kepler                     1.56%     129.2MiB / 7.654GiB   1.65%     1.11kB / 126B   0B / 1.12MB   74
======

@digital88 and @cristianrgreco I see you were involved in the PR review that increased CPU usage. What do you think of a fix like this? (Want me to send out a PR?)

I'll admit I'm not familiar with Mongo (I'm just writing integration tests in a project that uses Mongo), so let me know if there's an option for a cleaner fix here.

I also don't have good suggestions for how to add a regression test to catch this sort of excessive CPU consumption: suggestions also welcome for ways to test this behavior.

[digital88]

@digital88 and @cristianrgreco I see you were involved in the PR review that increased CPU usage. What do you think of a fix like this? (Want me to send out a PR?)

Hi, I did write this code. We actually need infinite retries because mongo forks and relaunches daemon when initializing with username and password. So without retries we may fail this healthcheck and wait strategy never completes.

I think we need to check rs.status() before rs.initiate();.

[CodingCanuck]

@digital88 thanks for the advice and your quick response!

@digital88 and @cristianrgreco I see you were involved in the PR review that increased CPU usage. What do you think of a fix like this? (Want me to send out a PR?)

Hi, I did write this code. We actually need infinite retries because mongo forks and relaunches daemon when initializing with username and password. So without retries we may fail this healthcheck and wait strategy never completes.

Darn :/

Though I'm a bit confused: if forking was an issue, wouldn't the "should connect with credentials" test you added fail if the health check failed in this case? (Or would this be a racy failure that wouldn't be deterministically caught by that test?)

I think we need to check rs.status() before rs.initiate();.

(Note: as mentioned above, I'm not familiar with Mongo, so this might quickly get out of my depth!)

I tried changing the health check to this (inspired by random github code like this):

`'try { rs.status(); } catch (e){ rs.initiate(); } while (db.runCommand({isMaster: 1}).ismaster==false) { sleep(100); }'`

It unfortunately doesn't fix the high CPU use, and it also causes the older mongo 4.4 test to fail with a timeout (not sure why: presumably status() throws on newer releases, but not older ones?)

testcontainers-node % npx vitest run packages/modules/mongodb/src/mongodb-container.test.ts 
...
❯ packages/modules/mongodb/src/mongodb-container.test.ts (5 tests | 1 failed) 120395ms
   ✓ MongoDBContainer > should work with mongo:8.0.12  2310ms
   ✓ MongoDBContainer > should work with mongo:6.0.25  2116ms
   × MongoDBContainer > should work with mongo:4.4.29 120394ms
     → Health check not healthy after 120000ms
   ✓ MongoDBContainer > should connect with credentials  4031ms
   ✓ MongoDBContainer > should have low CPU usage  23431ms

npx vitest run packages/modules/mongodb/src/mongodb-container.test.ts -t "should have low CPU usage" --silent=false |& grep "======" | grep -v "testcontainers-"
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d6c05951ba14   great_meitner                      48.88%    191.6MiB / 7.654GiB   2.45%     1.11kB / 126B   0B / 1.02MB   83
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d6c05951ba14   great_meitner                      55.32%    133.1MiB / 7.654GiB   1.70%     1.11kB / 126B   0B / 1.02MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d6c05951ba14   great_meitner                      52.91%    132.8MiB / 7.654GiB   1.69%     1.11kB / 126B   0B / 1.36MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d6c05951ba14   great_meitner                      51.86%    133.2MiB / 7.654GiB   1.70%     1.11kB / 126B   0B / 1.36MB   76
======
======CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O     PIDS
======d6c05951ba14   great_meitner                      46.21%    271.8MiB / 7.654GiB   3.47%     1.11kB / 126B   0B / 1.37MB   88
======

[digital88]

Though I'm a bit confused: if forking was an issue, wouldn't the "should connect with credentials" test you added fail if the health check failed in this case?

The first couple of healthcheck tries will fail, but it will eventually succeed. The wait strategy will wait for successfull healthcheck. Thats why 1000 retries needed.

and it also causes the older mongo 4.4 test to fail with a timeout

Yes, I encountered this too, I guess thats why I settled for current implementation. But I agree, CPU usage should be mitigated. Maybe use the old rs init code for older mongo and yours version for >= 5.0 ?

[CodingCanuck]

Though I'm a bit confused: if forking was an issue, wouldn't the "should connect with credentials" test you added fail if the health check failed in this case?

The first couple of healthcheck tries will fail, but it will eventually succeed. The wait strategy will wait for successfull healthcheck. Thats why 1000 retries needed.

What I meant to say: if I disable health check retries entirely, the "should connect with credentials" test passes (with what I assume is a single health check run / single initialization).

Is that expected? Based on your description of the daemon issue, I'd expect that test to fail if daemon-forking caused a deterministic failure. (But I'd expect it to only sometimes-fail if this is a rare race condition)

and it also causes the older mongo 4.4 test to fail with a timeout

Yes, I encountered this too, I guess thats why I settled for current implementation. But I agree, CPU usage should be mitigated. Maybe use the old rs init code for older mongo and yours version for >= 5.0 ?

Different logic for older versions seems totally reasonable.

I couldn't get a status-conditional initialization to work, though: my first attempt didn't fix high CPU usage on newer mongos.

Could you help me write the correct if-status-then-initialize logic that you have in mind? (From your description, it sounds like the one-initialize-only logic isn't safe on newer mongos either, though I'd love a test that would fail if I introduced that sort of error)

[digital88]

status() throws on newer releases

Also, rs.status() returns JSON we need to check some fields inside this JSON. Like if (rs.status().ok==1) {} else {sleep(100)}

[digital88]

But I'd expect it to only sometimes-fail if this is a rare race condition

Yes, exactly. It fails non-deterministically, almost always in CI :)

[digital88]

Could you help me write the correct if-status-then-initialize logic that you have in mind?

I will try to open a PR later

[CodingCanuck]

Could you help me write the correct if-status-then-initialize logic that you have in mind?

I will try to open a PR later

That would be great, thank you!