refactor(server): Lazy-initialize Redis, Redlock, and Elasticsearch clients, fix import cycles

[effigies]

The goal here is to be able to dump the graphql schema without using test infrastructure to mock import-type services.

This required deferring initialization of our JWT signing key as well as the elasticsearch, redis and redlock clients. Further, two import cycles had to be removed before we could call the script with yarn tsx.

The script is moved to the monorepo /scripts directory, and can be run with yarn dump-schema, which calls tsx scripts/dump-schema.ts.

[codecov]

Codecov Report

❌ Patch coverage is 40.76923% with 77 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.77%. Comparing base (9a03fe8) to head (611f689).

Files with missing lines	Patch %	Lines
packages/openneuro-server/src/datalad/snapshots.ts	23.52%	13 Missing ⚠️
...openneuro-server/src/libs/authentication/crypto.ts	7.69%	12 Missing ⚠️
packages/openneuro-server/src/datalad/files.ts	37.50%	10 Missing ⚠️
...enneuro-server/src/elasticsearch/elastic-client.ts	27.27%	8 Missing ⚠️
packages/openneuro-server/src/datalad/draft.ts	14.28%	6 Missing ⚠️
...nneuro-server/src/elasticsearch/reindex-dataset.ts	25.00%	6 Missing ⚠️
scripts/dump-schema.ts	16.66%	5 Missing ⚠️
...es/openneuro-server/src/graphql/resolvers/cache.ts	25.00%	3 Missing ⚠️
...uro-server/src/graphql/resolvers/dataset-search.ts	25.00%	3 Missing ⚠️
packages/openneuro-server/src/libs/redis.ts	70.00%	3 Missing ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3910      +/-   ##
==========================================
- Coverage   48.87%   48.77%   -0.11%     
==========================================
  Files         680      682       +2     
  Lines       37665    37710      +45     
  Branches     1867     1868       +1     
==========================================
- Hits        18408    18392      -16     
- Misses      19098    19159      +61     
  Partials      159      159              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[effigies]

Okay, I was wrong and Claude was right, there were two circular dependencies:

  datalad/snapshots.ts                                                                                                                                                                                                                                        
    import { snapshotCreationComparison, updateDatasetName } from resolvers/dataset.ts                                                                                                                                                                       
      ↓                                                                                                                                                                                                                                                       
  resolvers/dataset.ts                                                                                                                                                                                                                                        
    import { latestSnapshot, snapshots } from resolvers/snapshots.ts                                                                                                                                                                                         
      ↓                                                                                                                                                                                                                                                       
  resolvers/snapshots.ts                                                  
    import * as datalad from datalad/snapshots.ts
    import { analytics, dataset, snapshotCreationComparison } from "./dataset.js"

[nellh]

The goal here is to be able to dump the graphql schema without using test infrastructure to mock import-type services. The refactor seems good, but I'm not sure how to run this script without vitest.

I think this is just an issue with pnp + esm with typescript under node. Seems to work fine if you build it, run with tsx, or disable pnp.

yarn tsx scripts/dump-schema.ts

[effigies]

@nellh Sorry, it is working now in this branch, but not off of master:

git clone https://github.com/openneuroorg/openneuro
cd openneuro
git remote add effigies https://github.com/effigies/openneuro
git fetch effigies
git cherry-pick 397c1c8
yarn install
yarn build
git commit -a 'chore: Update lockfiles'

The first error I get is:

󰅂 yarn tsx scripts/dump-schema.ts
TypeError: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
    at Hash.update (node:internal/crypto/hash:142:11)
    at /var/home/chris/Projects/openneuro/on-freshdesk-fix/packages/openneuro-server/src/libs/authentication/crypto.ts:8:4
    at Object.<anonymous> (/var/home/chris/Projects/openneuro/on-freshdesk-fix/packages/openneuro-server/src/libs/authentication/crypto.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)
    at Object.j (/var/home/chris/Projects/openneuro/on-freshdesk-fix/.yarn/cache/tsx-npm-3.14.0-0e7fee0a4e-7147970975.zip/node_modules/tsx/dist/cjs/index.cjs:1:1197)
    at Module.load (node:internal/modules/cjs/loader:1552:32)
    at Module._load (node:internal/modules/cjs/loader:1354:12)
    at Module.require$$0.Module._load (/var/home/chris/Projects/openneuro/on-freshdesk-fix/.pnp.cjs:31530:31)
    at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)
    at Module.require (node:internal/modules/cjs/loader:1575:12)
    at require (node:internal/modules/helpers:191:16)
    at /var/home/chris/Projects/openneuro/on-freshdesk-fix/packages/openneuro-server/src/libs/authentication/jwt.ts:6:25
    at Object.<anonymous> (/var/home/chris/Projects/openneuro/on-freshdesk-fix/packages/openneuro-server/src/libs/authentication/jwt.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)
    at Object.j (/var/home/chris/Projects/openneuro/on-freshdesk-fix/.yarn/cache/tsx-npm-3.14.0-0e7fee0a4e-7147970975.zip/node_modules/tsx/dist/cjs/index.cjs:1:1197)
    at Module.load (node:internal/modules/cjs/loader:1552:32)

So we pull in the fix for libs/authentication/crypto.ts:

git checkout effigies/refactor/lazy-clients packages/openneuro-server/src/libs/authentication/crypto.ts
git commit -m 'fix: Defer initialization of key for JWT to first use'
yarn install
yarn build

Error:

󰅂 yarn tsx scripts/dump-schema.ts

/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/graphql/resolvers/snapshots.ts:1
import * as datalad from "../../datalad/snapshots"
^
ReferenceError: Cannot access 'snapshots' before initialization
    at Object.snapshots (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/graphql/resolvers/snapshots.ts:1:1)
    at Object.get [as snapshots] (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/graphql/resolvers/snapshots.ts:2:648)
    at /var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/graphql/resolvers/dataset.ts:333:3
    at Object.<anonymous> (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/graphql/resolvers/dataset.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)
    at Object.j (/var/home/chris/Projects/tmp/openneuro/.yarn/cache/tsx-npm-3.14.0-0e7fee0a4e-7147970975.zip/node_modules/tsx/dist/cjs/index.cjs:1:1197)
    at Module.load (node:internal/modules/cjs/loader:1552:32)
    at Module._load (node:internal/modules/cjs/loader:1354:12)
    at Module.require$$0.Module._load (/var/home/chris/Projects/tmp/openneuro/.pnp.cjs:31529:31)
    at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)
    at Module.require (node:internal/modules/cjs/loader:1575:12)
    at require (node:internal/modules/helpers:191:16)
    at /var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/datalad/snapshots.ts:12:8
    at Object.<anonymous> (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/datalad/snapshots.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)
    at Object.j (/var/home/chris/Projects/tmp/openneuro/.yarn/cache/tsx-npm-3.14.0-0e7fee0a4e-7147970975.zip/node_modules/tsx/dist/cjs/index.cjs:1:1197)

Handle the cyclic dependencies:

git cherry-pick f04c9b1
yarn install
yarn build

Error:

󰅂 yarn tsx scripts/dump-schema.ts

/var/home/chris/Projects/tmp/openneuro/.yarn/cache/jsonwebtoken-npm-9.0.0-36fd1594c0-769ea563e9.zip/node_modules/jsonwebtoken/sign.js:105
    return failure(new Error('secretOrPrivateKey must have a value'));
                   ^
Error: secretOrPrivateKey must have a value
    at Object.module.exports [as sign] (/var/home/chris/Projects/tmp/openneuro/.yarn/cache/jsonwebtoken-npm-9.0.0-36fd1594c0-769ea563e9.zip/node_modules/jsonwebtoken/sign.js:105:20)
    at indexingToken (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-search/dist/auth.js:10:35)
    at schemaLinkClient (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/elasticsearch/reindex-dataset.ts:13:23)
    at /var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/elasticsearch/reindex-dataset.ts:31:16
    at Object.<anonymous> (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/elasticsearch/reindex-dataset.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)
    at Object.j (/var/home/chris/Projects/tmp/openneuro/.yarn/cache/tsx-npm-3.14.0-0e7fee0a4e-7147970975.zip/node_modules/tsx/dist/cjs/index.cjs:1:1197)
    at Module.load (node:internal/modules/cjs/loader:1552:32)
    at Module._load (node:internal/modules/cjs/loader:1354:12)
    at Module.require$$0.Module._load (/var/home/chris/Projects/tmp/openneuro/.pnp.cjs:31529:31)
    at wrapModuleLoad (node:internal/modules/cjs/loader:255:19)
    at Module.require (node:internal/modules/cjs/loader:1575:12)
    at require (node:internal/modules/helpers:191:16)
    at /var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/queues/consumer.ts:2:32
    at Object.<anonymous> (/var/home/chris/Projects/tmp/openneuro/packages/openneuro-server/src/queues/consumer.ts:3:3)
    at Module._compile (node:internal/modules/cjs/loader:1829:14)

Defer elastic search initialization:

git checkout effigies/refactor/lazy-clients packages/openneuro-server/src/graphql/resolvers/dataset-search.ts packages/openneuro-server/src/elasticsearch
git commit -m 'fix: Defer initialization of elasticsearch'

This actually does generate the schema file, and then error:

󰅂 yarn tsx scripts/dump-schema.ts
(node:3227426) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:3227426) [MONGOOSE] Warning: Duplicate schema index on {"id":1} found. This is often due to declaring an index using both "index: true" and "schema.index()". Please remove the duplicate index definition.
[ioredis] Unhandled error event: RangeError: Port should be >= 0 and < 65536. Received type number (NaN).
    at lookupAndConnect (node:net:1389:5)
    at Socket.connect (node:net:1344:5)
    at connect (node:net:249:17)
    at /var/home/chris/Projects/tmp/openneuro/.yarn/cache/ioredis-npm-5.6.1-d69383b35a-632186e21f.zip/node_modules/ioredis/built/connectors/StandaloneConnector.js:54:49
    at processTicksAndRejections (node:internal/process/task_queues:85:11)

Defer redis initialization:

git checkout effigies/refactor/lazy-clients packages/openneuro-server/src/
# Unselect some format-only changes
git commit -m 'fix: Defer initialization of redis'

Finally resolution:

󰅂 yarn tsx scripts/dump-schema.ts
(node:3238372) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:3238372) [MONGOOSE] Warning: Duplicate schema index on {"id":1} found. This is often due to declaring an index using both "index: true" and "schema.index()". Please remove the duplicate index definition.

I've gone ahead and pushed those specific commits so the logical sequence is clearer.