Support for waiting sequentically

[nidomiro]

Right now I can use the Wait.forAll-Stategy to wait for multiple conditions to be ready.
This does however check for all conditions in parallel.
For my usecase I need a sequence of things to be checked.

My usecase is a postgres DB that is up for init, then shuts down for the actual DB to come up.
So my Wait strategy is:

1. Wait for the log message: PostgreSQL init process complete; ready for start up.
2. Wait for the log message: database system is ready to accept connections
3. Wait.forListeningPorts()

I can implement this change, but there are multiple variants on how to do that.

1. Alter Wait.forAll() to accept an optional parameter: Wait.forAll([], "ExecuteInSequence") or Wait.forAll([], "ExecuteInParallel" ) where "ExecuteInParallel" is used by default, to keep compatibility
2. Create a new Wait method, like Wait.forAllInSequence()

What would be your preferred solution?

[cristianrgreco]

Hi @nidomiro,

For complex custom wait strategies, I'd suggest just defining your own. It's the first I've heard of the use case where the strategies must be executed sequentially. Docs on how to do that here: https://node.testcontainers.org/features/wait-strategies/#other-startup-strategies

If you simply wish to have a working wait strategy for Postgres, I'd suggest you use the PostgresqlContainer (https://node.testcontainers.org/modules/postgresql/), which implements a health check wait strategy, instead of trying to have a workaround with logs and ports: https://github.com/testcontainers/testcontainers-node/blob/main/packages/modules/postgresql/src/postgresql-container.ts#L39-L47

[nidomiro]

Hi @cristianrgreco ,

thanks for your reply.

I was using the said PostgresqlContainer and ran into the issue that my tests failed because I could not connect to the db. After some investigation I found out that the container needs another ~2 sec to come up, after started() returns. (tested with creating a (node-) psql client in a loop and stop the timer when it worked).

This lead me down the path of discovering that Wait.forAll() does not support waiting for the single checks to happen in sequence, but only in parallel.

In my opinion, this is also needed for your PostgresqlContainer since the healthcheck command seems to also give green light, when the init runs.

Also the implementation of this feature is not difficult. My current workaround implementation is a copy of

testcontainers-node/packages/testcontainers/src/wait-strategies/composite-wait-strategy.ts

Line 26 in 4471f40

Promise.all(

where I replaced the Promise.all with a loop, waiting in order.

[cristianrgreco]

I was using the said PostgresqlContainer and ran into the issue that my tests failed because I could not connect to the db. After some investigation I found out that the container needs another ~2 sec to come up, after started() returns. (tested with creating a (node-) psql client in a loop and stop the timer when it worked).

I'm surprised at this. In this project's pipelines we frequently run tests against the Postgres container, as well as variants like PGVector, Postgis and Timescale, and never seen an issue.

Also the implementation of this feature is not difficult

It's not about the implementation being easy/difficult, it's about keeping the API of this library concise.

TBH I don't see harm in adding this feature. A new method, Wait.forAllInSequence() sounds good to me. If you're going to raise a PR please also update the docs.

[nidomiro]

I just cloned the repo and ran the tests and there are a lot of test-failures on my machine (Apple M2 Pro, 32 GB Ram, running colima).
Is there more todo than npm install and then npm test?

$ colima version
colima version 0.8.1
git commit: 96598cc5b64e5e9e1e64891642b91edc8ac49d16

runtime: docker
arch: aarch64
client: v28.0.4
server: v27.4.0

My colima startup command: colima start --arch aarch64 --vm-type=vz --vz-rosetta --cpu 6 --memory 12 --disk 150

Part of the test output:

packages/testcontainers/src/wait-strategies/health-check-wait-strategy.test.ts > HealthCheckWaitStrategy > should wait for health check
TypeError: fetch failed
 ❯ checkContainerIsHealthy packages/testcontainers/src/utils/test-helper.ts:13:20
     11| export const checkContainerIsHealthy = async (container: StartedTestContainer): Promise<void> => {
     12|   const url = `http://${container.getHost()}:${container.getMappedPort(8080)}`;
     13|   const response = await fetch(`${url}/hello-world`);
       |                    ^
     14|   expect(response.status).toBe(200);
     15| };
 ❯ packages/testcontainers/src/wait-strategies/health-check-wait-strategy.test.ts:18:5

{
  stack: 'AggregateError: \n' +
    '    at internalConnectMultiple (node:net:1118:18)\n' +
    '    at afterConnectMultiple (node:net:1685:7)',
  errors: [
    {
      stack: 'Error: connect ECONNREFUSED 127.0.0.1:35490\n' +
        '    at createConnectionError (node:net:1648:14)\n' +
        '    at afterConnectMultiple (node:net:1678:16)',
      message: 'connect ECONNREFUSED 127.0.0.1:35490',
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 35490,
      constructor: 'Function<Error>',
      name: 'Error',
      toString: 'Function<toString>'
    },
    {
      stack: 'Error: connect ECONNREFUSED ::1:35490\n' +
        '    at createConnectionError (node:net:1648:14)\n' +
        '    at afterConnectMultiple (node:net:1678:16)',
      message: 'connect ECONNREFUSED ::1:35490',
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '::1',
      port: 35490,
      constructor: 'Function<Error>',
      name: 'Error',
      toString: 'Function<toString>'
    }
  ],
  code: 'ECONNREFUSED',
  stackStr: 'AggregateError: \n' +
    '    at internalConnectMultiple (node:net:1118:18)\n' +
    '    at afterConnectMultiple (node:net:1685:7)',
  nameStr: 'AggregateError',
  expected: 'undefined',
  actual: 'undefined',
  message: '',
  constructor: 'Function<AggregateError>',
  name: 'Caused by: AggregateError',
  toString: 'Function<toString>',
  stacks: []
}
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { errors: [ { stack: 'Error: connect ECONNREFUSED 127.0.0.1:35490\n    at createConnectionError (node:net:1648:14)\n    at afterConnectMultiple (node:net:1678:16)', message: 'connect ECONNREFUSED 127.0.0.1:35490', errno: -61, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 35490, constructor: 'Function<Error>', name: 'Error', toString: 'Function<toString>' }, { stack: 'Error: connect ECONNREFUSED ::1:35490\n    at createConnectionError (node:net:1648:14)\n    at afterConnectMultiple (node:net:1678:16)', message: 'connect ECONNREFUSED ::1:35490', errno: -61, code: 'ECONNREFUSED', syscall: 'connect', address: '::1', port: 35490, constructor: 'Function<Error>', name: 'Error', toString: 'Function<toString>' } ], code: 'ECONNREFUSED' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[26/42]⎯

 FAIL  packages/testcontainers/src/wait-strategies/log-wait-strategy.test.ts > LogWaitStrategy > should wait for log
TypeError: fetch failed
 ❯ checkContainerIsHealthy packages/testcontainers/src/utils/test-helper.ts:13:20
     11| export const checkContainerIsHealthy = async (container: StartedTestContainer): Promise<void> => {
     12|   const url = `http://${container.getHost()}:${container.getMappedPort(8080)}`;
     13|   const response = await fetch(`${url}/hello-world`);
       |                    ^
     14|   expect(response.status).toBe(200);
     15| };
 ❯ packages/testcontainers/src/wait-strategies/log-wait-strategy.test.ts:13:5

{
  stack: 'AggregateError: \n' +
    '    at internalConnectMultiple (node:net:1118:18)\n' +
    '    at afterConnectMultiple (node:net:1685:7)',
  errors: [
    {
      stack: 'Error: connect ECONNREFUSED 127.0.0.1:35677\n' +
        '    at createConnectionError (node:net:1648:14)\n' +
        '    at afterConnectMultiple (node:net:1678:16)',
      message: 'connect ECONNREFUSED 127.0.0.1:35677',
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '127.0.0.1',
      port: 35677,
      constructor: 'Function<Error>',
      name: 'Error',
      toString: 'Function<toString>'
    },
    {
      stack: 'Error: connect ECONNREFUSED ::1:35677\n' +
        '    at createConnectionError (node:net:1648:14)\n' +
        '    at afterConnectMultiple (node:net:1678:16)',
      message: 'connect ECONNREFUSED ::1:35677',
      errno: -61,
      code: 'ECONNREFUSED',
      syscall: 'connect',
      address: '::1',
      port: 35677,
      constructor: 'Function<Error>',
      name: 'Error',
      toString: 'Function<toString>'
    }
  ],
  code: 'ECONNREFUSED',
  stackStr: 'AggregateError: \n' +
    '    at internalConnectMultiple (node:net:1118:18)\n' +
    '    at afterConnectMultiple (node:net:1685:7)',
  nameStr: 'AggregateError',
  expected: 'undefined',
  actual: 'undefined',
  message: '',
  constructor: 'Function<AggregateError>',
  name: 'Caused by: AggregateError',
  toString: 'Function<toString>',
  stacks: []
}
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Serialized Error: { errors: [ { stack: 'Error: connect ECONNREFUSED 127.0.0.1:35677\n    at createConnectionError (node:net:1648:14)\n    at afterConnectMultiple (node:net:1678:16)', message: 'connect ECONNREFUSED 127.0.0.1:35677', errno: -61, code: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 35677, constructor: 'Function<Error>', name: 'Error', toString: 'Function<toString>' }, { stack: 'Error: connect ECONNREFUSED ::1:35677\n    at createConnectionError (node:net:1648:14)\n    at afterConnectMultiple (node:net:1678:16)', message: 'connect ECONNREFUSED ::1:35677', errno: -61, code: 'ECONNREFUSED', syscall: 'connect', address: '::1', port: 35677, constructor: 'Function<Error>', name: 'Error', toString: 'Function<toString>' } ], code: 'ECONNREFUSED' }
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[27/42]⎯


 Test Files  17 failed | 75 passed (92)
      Tests  42 failed | 479 passed | 4 skipped (525)
   Start at  06:48:22
   Duration  516.41s (transform 2.94s, setup 0ms, collect 104.51s, tests 4969.71s, environment 15ms, prepare 8.38s)

[nidomiro]

On my fedora 42 machine, only 10 tests fail

[nidomiro]

I just don't know what is happening 😆
npm test -- packages/modules/postgresql/src/postgresql-container.test.ts works on my fedora machine, but not on my mac.

I added another test case that should fail with my current theory, but this is the only one (except for "should allow custom healthcheck") that is green on mac 🙈

it("should wait until the db is ready to accept client connections", async () => {
    const startUpScriptWaitSeconds = 5;
    const container = new PostgreSqlContainer().withCopyContentToContainer([
      {
        content: ["#!/bin/sh", `sleep ${startUpScriptWaitSeconds}`].join("\n"),
        target: "/docker-entrypoint-initdb.d/init.sh",
        mode: 0o755,
      },
    ]);
    const containerRuntimeClient = await getContainerRuntimeClient();
    await containerRuntimeClient.image.pull(container["imageName"], {
      force: container["pullPolicy"].shouldPull(),
      platform: container["createOpts"].platform,
    });

    const startTime = Date.now();
    const startedContainer = await container.start();
    const endTime = Date.now();
    expect(endTime - startTime).toBeGreaterThanOrEqual(startUpScriptWaitSeconds * 1000);

    const client = new Client({
      host: startedContainer.getHost(),
      port: startedContainer.getPort(),
      database: startedContainer.getDatabase(),
      user: startedContainer.getUsername(),
      password: startedContainer.getPassword(),
    });
    await client.connect();

    //await expect(client.connect()).resolves.toBeUndefined();
    await expect(client.query("SELECT 1").then((result) => result.rows)).resolves.toEqual([{ "?column?": 1 }]);

    await client.end();
    await startedContainer.stop();
  });

[cristianrgreco]

Ah you're using Colima, I guess that's why you're having so many issues. There are several known issues with it. Some main ones documented here https://node.testcontainers.org/supported-container-runtimes/#colima. Try Docker or Podman and see if you still have an issue with the PostgreSQL module.

[nidomiro]

Good to know, thanks for the heads up and sorry for creating noise with my problem.
Unfortunately I cannot use docker desktop, due to company policy and we had problems with podman in the past.

But in that case I would ditch the implementation of the Wait.forAllInSequence() and change the postgres wait strategy to Wait.forAll([Wait.forHealthCheck(), Wait.forListeningPorts()]) because this seems to work.
Should I also change other containers wait strategies or keep it to postgres?

[cristianrgreco]

The PostgreSQL and Cassandra modules follow the same approach. You could change those two. Thanks!