loader: implement package maps

[arcanis]

This PR adds a new --experimental-package-map=<path> flag letting Node.js resolve packages using a static JSON file instead of walking node_modules directories.

node --experimental-package-map=./package-map.json app.js

Why?

The node_modules resolution algorithm predates npm and its clear definition of the concept of packages. It works well enough and is widely supported, but has known issues:

• Phantom dependencies - packages can accidentally import things they don't declare, because hoisting makes transitive dependencies visible

• Peer dependency resolution is broken in monorepos - if website-v1 uses react@18 and website-v2 uses react@19, and both use a shared component-lib with React as a peer dep, there's no node_modules layout that resolves correctly. The shared lib always gets whichever React was hoisted.

• Hoisting is lossy - runtimes can't tell if an import is legitimate or accidental

• Resolution requires I/O - you have to hit the filesystem to resolve packages, and that quickly adds up. Even more so in agentic worlds where working with multiple git trees become a common pattern.

Package managers have tried workarounds (pnpm symlinks, Yarn PnP), but are either limited by what the filesystem itself can offer (like symlinks) or by their complexity and lack of standardization (like Yarn PnP). This PR offers a mechanism for such tools to solve the problems listed above in tandem with Node.js.

How it works

A package-map.json declares packages, their locations (relative to the package map), and what each can import:

{
  "packages": {
    "my-app": {
      "path": "./src",
      "dependencies": {
        "lodash": "lodash",
        "react": "react"
      }
    },
    "lodash": {
      "path": "./node_modules/lodash"
    },
    "react": {
      "path": "./node_modules/react"
    }
  }
}

When resolving a bare specifier:

1. Find which package contains the importing file (ideally by keeping track of package IDs during resolution, but for now by checking paths)
2. Look up the specifier in that package's dependencies
3. If found, resolve to the target's path
4. If not found but exists elsewhere in the map → ERR_PACKAGE_MAP_ACCESS_DENIED
5. If not in the map at all → MODULE_NOT_FOUND

Compatibility

An important aspect of the package maps feature that separates it from competing options like Yarn PnP is its builtin compatibility with node_modules installs. Package managers can generate both node_modules folders AND package-map.json files, with the later referencing paths from the former.

Tools that know how to leverage package-map.json can then use this pattern for both static package resolution and strict dependency checks (with optional fallbacks to hoisting if they just wish to use the package map information to emit warnings rather than strict errors), whereas tools that don't will fallback to the classical node_modules resolution.

Differences with import maps

Issue #49443 requested to implement import maps. In practice these aren't a good fit for runtimes like Node.js for reasons described here and which can be summarized as: import maps take full ownership of the resolution pipeline by spec, thus preventing implementing additional runtime-specific behaviours such as exports or imports fields.

This PR comes as close from implementing import maps as possible but with a very light difference in design making it possible to stay compatible with other Node.js resolution features.

Why not a loader?

The ecosystem now has to deal with a variety of third-party resolvers, most of them not implementing the loader API for many different reasons: too complex, turing-complete, or dependent on a JS runtime.

After I've been following this path for more than six years I can confidently say that loaders would work for Node.js itself but wouldn't be standard enough to be included in at least some of those popular third-party tools.

Package manager integration

Package maps are meant to be generated by package managers. For example you could imagine Yarn or pnpm generating a node_modules directory containing a node_modules/.package-map.json file should the feature be enabled (it would likely remain opt-in while experimental).

Then all yarn node calls would inject the flag into the environment when spawning Node.js:

NODE_OPTIONS="--experimental-package-map=$PWD/node_modules/.package-map.json"

Questions

• The current implementation makes package maps strict: if they find an issue, they throw and refuse the resolution. Should we instead delegate to the default resolution unless an additional --experimental-strict-package-maps is set? Or via a strict field in package-map.json.

[nodejs-github-bot]

Review requested:

• @nodejs/config
• @nodejs/loaders

[zkochan]

I like the idea, it would greatly reduce the amount of filesystem operations that pnpm has to do in order to create an isolated node_modules layout using symlinks.

I also suggested arcanis to possibly go one layer deeper and allow to map the individual files of packages. This would allow to map node_modules directly from a content-addressable store (that consists of package files). Of course, that would increase the size of the file several times but it would also make installation even faster.

[codecov]

Codecov Report

❌ Patch coverage is 99.24623% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.71%. Comparing base (d080801) to head (ac9d869).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/modules/cjs/loader.js	96.05%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #62239      +/-   ##
==========================================
- Coverage   91.55%   89.71%   -1.85%     
==========================================
  Files         355      707     +352     
  Lines      149381   218508   +69127     
  Branches    23364    41808   +18444     
==========================================
+ Hits       136765   196029   +59264     
- Misses      12354    14392    +2038     
- Partials      262     8087    +7825     

Files with missing lines	Coverage Δ
lib/internal/errors.js	97.63% <100.00%> (+4.18%)	⬆️
lib/internal/modules/esm/resolve.js	99.05% <100.00%> (+11.20%)	⬆️
lib/internal/modules/package_map.js	100.00% <100.00%> (ø)
src/node_options.cc	76.63% <100.00%> (ø)
src/node_options.h	98.00% <ø> (ø)
lib/internal/modules/cjs/loader.js	98.24% <96.05%> (+18.68%)	⬆️

... and 468 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aduh95]

This seems quite close to the importmap HTML feature, but using a different syntax. Have you considered reusing the same syntax, or at least a compatible JSON structure?

[arcanis]

I did, but felt that the semantics were too different; import maps have two fields:

• the imports field is a global resolution map, keyed by bare identifiers. It wouldn't work for packages as that field is a flat map of all packages in the project, and thus must be keyed by arbitrary package IDs to allow for multiple packages sharing the same name (ie multiple versions of a same package in the same dependency tree).

• the scopes field is keyed by filesystem path. This is a problem because it precludes a same folder from having multiple package IDs each with their own dependency set, necessary to represent peer dependencies with workspaces.

Neither of those match the semantics we need, and reusing them just for their name but with different semantics would have been imo misleading for third-party resolver implementors.

[jasnell]

Love it. I'll try to give a detailed review on the flight home today if the in flight wifi treats me kindly.

[bakkot]

What happens if two packages define the exact same path? An error, presumably, given that the algorithm relies on being able to determine for each path which package it belongs to? Needs a test in any case.

[arcanis]

What happens if two packages define the exact same path? An error, presumably, given that the algorithm relies on being able to determine for each path which package it belongs to? Needs a test in any case.

I'll follow-up with a separate improvement to key module instances per both their path and their package IDs (it's key to solve the peer dependency problem I mentioned in the PR description), but for this iteration only one package ID per path is supported. I updated the code to throw an error accordingly.

[arcanis]

@guybedford @avivkeller can I get another review? once this land I'll start looking at a prototype for generating package maps in package managers, and the package ID follow-up

[ljharb]

How does this work given the module cache? Wont the first one to load’s react version “win”?

[arcanis]

Yes, that's why it's not implemented in this PR (it throws should this case happen), and will be implemented as a follow-up: #62239 (comment) (still I want it to be documented as third-party tools reading package maps better know about this early in the implementation)

[bakkot]

What happens when there's symlinks in the specified paths?

From the current implementation it looks like it basically breaks. getKeyForPath relies on being able to determine which package in the package map contains a given file, which it does by walking up from the file and checking if each ancestor directory appears as a path in the package map. But if the path in the package map points to a symlink, it will never resolve.

You could sort of fix this by resolving symlinks in the package map (at the cost of a lot of file system operations during startup), but then you have the problem that the "longest path containing a file" heuristic is now ambiguous: you could have a file /a/b/c/index.js and symlinks /foo/bar -> /a and /baz -> /a/b/c, and package map

{
  "packages": {
    "my-app": {
      "path": "./src",
      "dependencies": {
        "foo": "foo",
        "baz": "baz"
      }
    },
    "foo": {
      "path": "/foo/bar"
    },
    "baz": {
      "path": "/baz"
    }
  }
}

and then /a/b/c/index.js is in foo if "longest path" is defined according to the package map but in bar if "longest path" is defined according to the file system.

I don't have a good intuition for what the right thing to do here is.

[arcanis]

What happens when there's symlinks in the specified paths?

It depends where the symlink is:

• if the symlink is an ancestor of the package map file it should be realpath'd, just like Node.js calls realpath on paths it uses for resolution. Adding a test to make sure of that.

• if the symlink is listed in path (for example "path": "./path/to/symlink"), it'll never match since Node.js calls realpath on each module path (so it would never pass ./path/to/symlink to the resolve function, but instead ./path/to/real/path, which wouldn't match). I'd also tend not to realpath the path field as:

  1. it changes in unpredictable ways the paths as initially intended
  2. whichever tool created the package map can realpath them itself
  3. in the event we later support loading package maps from urls, realpath wouldn't make sense

[ruyadorno]

Is it possible for @arcanis or someone else familiar with the work to come to the next TSC meeting to talk a little about this implementation? I have two concerns with landing this at the moment:

• Awareness: like mentioned in the "Why?" section of its description, this PR is introducing a complete new alternative to the more-than-a-decade old module resolution system, have APM folks been looped in? were the other adjacents WG consulted? @nodejs/package-maintenance @nodejs/tooling
• Import maps: Is the current implementation going to create an irremediable compatibility with that standard? Can this use some help from the @nodejs/wintercg folks to help standardize this so that other runtimes may also implement support in the future?

That said, thanks @arcanis for taking the time to work on it! I feel like the runtime definitely needs to innovate on this and sorry to be potentially slowing things down in the immediate time but hopefully it's to make sure we land on a solution that will last! ✌️

[wesleytodd]

I have been looking for someone who wanted to take over and help get #49443 over the finish line. Maybe this would be a good opportunity to start a thread in with @nodejs/wintercg, come up with the technical and UX goals we have for this feature. I think we need to decide if we need spec changes (to pursue via the standards process) or if we can find workarounds that maintain spec compliance. @arcanis happy to work with you on this if you are interested.

[arcanis]

Is it possible for @arcanis or someone else familiar with the work to come to the next TSC meeting to talk a little about this implementation?

Sure, happy to, please invite me to the next meeting.

I also discussed it a little while ago with folks at the WinterTC and it wasn't clear at the time whether it was worth a multi-vendor discussion at that stage. So I figured that since this is behind --experimental-* we'd have room to iterate, but let me open a thread nonetheless.

Is the current implementation going to create an irremediable compatibility with that standard

The designs serve different goals. The way import maps are keyed makes it impossible to support very common dependency graphs, causing package managers to generate semantically incorrect installs or rely on features like inject or virtual paths which add complexity and come with their own trade-offs.

That's not to say import maps can't be implemented in Node.js as well, but their design limits them to solve more web-specific use cases that wouldn't satisfy the practical problems our local package managers set out to address.

[naugtur]

Using the example in "How it works" as base, some compatibility/security questions:

• What would it look like if react depended on its own copy of lodash that could not be deduplicated? Would the structure be then nested under dependencies or would the independent copies be given separate IDs?
• Same, but react depends on lodash installed straight from git url, so it can be a different package despite the same version number in pkgjson
• Same, but react has lodash (alleged lodash) as a bundled dependency

Context: I'm looking at it from the experience of having to map the dependencies for LavaMoat policy files and these are some of the reasons we were forced to invent canonical names that represent the package as the shortest path in dependency tree leading to it, and we use that as keys in a flat structure.
Package managers would have an easier time figuring out which packages are called the same name and same version but from different sources. I'd love to be able to rely on the information stored in a package map instead.

[arcanis]

What would it look like if react depended on its own copy of lodash that could not be deduplicated? Would the structure be then nested under dependencies or would the independent copies be given separate IDs?

Separate arbitrary IDs. The graph is stored flat in packages, so for example this would be valid:

{
  "packages": {
    "my-app": {
      "path": "./src",
      "dependencies": {
        "lodash": "lodash@1",
        "react": "react"
      }
    },
    "lodash@1": {
      "path": "./node_modules/lodash"
    },
    "lodash@2": {
      "path": "./node_modules/react/node_modules/lodash"
    },
    "react": {
      "path": "./node_modules/react",
      "dependencies": {
        "lodash": "lodash@2"
      }
    }
  }
}

Same, but react depends on lodash installed straight from git url, so it can be a different package despite the same version number in pkgjson

Separate IDs as well; from my experience with Yarn we give different identities to packages that come from different sources (git, npm, http, etc). As an example we also don't deduplicate them during hoisting.

Same, but react has lodash (alleged lodash) as a bundled dependency

I'd treat bundleDependencies as their own source, so supporting package managers would give them their own ID.

[bcomnes]

I like this idea generally!

How does it differ from a lockfile? Could it replace the lockfile pattern, or serve both purposes? It seems like this could potentially merge the lockfile and node_modules into a single lightweight resolution/lock file, eliminating a huge number of stateful edge cases that exist with the lockfile+node_modules pattern.

What is the DX for breaking out a local override / edit / vendor on specific packages?

[arcanis]

How does it differ from a lockfile? Could it replace the lockfile pattern, or serve both purposes?

Lockfiles are package-manager specific files and contain arbitrary information that are only relevant to the package manager that generated them (checksums, whether the package contains scripts, and more).

Depending on the package manager they also store fewer information that are needed for runtime execution. As an example Yarn doesn't store the peer dependency resolutions in the lockfile; they are always recomputed at install time.

Package maps are an install byproduct, same as node_modules directories, but don't replace lockfiles.

[bcomnes]

What's the intention around committing the resolution file to source control?

It seems like if the intention is to have it committed, the a package manager could merge its lock-file state with resolution state into that single file and reduce/eliminate an entire class of post install mutation and drift defects that can happen between lock files and node_modules

If it's not the intention to capture the resolution file to source control, why not?

My goal was to give it more time to raise awareness and I'm satisfied with the extra time and reach out effort we've done since - I'm happy to lift the block now and let the rest of the collaborator base work on landing the best solution here :)

[Tobbe]

I'm the maintainer of a "create"-style package (think create-react-app, create-tsrouter-app etc)
To speed up installation for end users we ship with a pre-generated yarn.lock file, so that yarn can skip the full dependency analysis step and just get on with downloading and installing packages.

This worked fine when we only supported yarn. But now I've added support for npm and pnpm as well. And doing the lock file generation (by performing a full install, grabbing the package manager specific lock file, and then deleting node_modules) now takes around three minutes, significantly slowing down both local testing and CI.

Would shipping a package map in our "create" package provide similar speed gains as we currently achieve by shipping the lock files?

If it does speed up that dependency analysis/resolution in a package manager agnostic way, that's another win worth highlighting :) Not to mention the reduced complexity compared to having generate multiple lock files.

[aduh95]

Not a blocker for landing a PR, but probably something to iron out before making stable: using process.cwd() feels weird here, that would mean a package-map could load completely different files depending on the CWD (i.e. node --experimental-package-map=subdir/package-map.json subdir/index.js and cd subdir && node --experimental-package-map=package-map.json index.js would behave differently). A less surprising behavior would be to use the dirname of package-map.json instead.

[arcanis]

The process.cwd() here is only used when an require call is performed by something that doesn't have a parent. Usually that's when evaluating a script with node -e.

As for --experimental-package-map, if provided a relative path it'll always be relative to the cwd since we call pathResolve here. I think that's the expected behavior from a user point of view.

[aduh95]

Resolving the path to the JSON file is certainly fine, I agree. My point is more that the paths inside that JSON file should not be CWD-dependant.

The process.cwd() here is only used when an require call is performed by something that doesn't have a parent. Usually that's when evaluating a script with node -e.

That still doesn't seem right, evalScript does define its own module.filename using the CWD. Also any entry point script doesn't have a parent, and in general we never use the "parent" (which is not reliable, see https://nodejs.org/api/deprecations.html#dep0144-moduleparent) to resolve require calls

[arcanis]

Resolving the path to the JSON file is certainly fine, I agree. My point is more that the paths inside that JSON file should not be CWD-dependant.

They are not, they are relative to the location of the package map:
https://github.com/nodejs/node/pull/62239/changes#diff-dc08f64ce5a7c16618ae8b4ee0e179f2a0362e31a0c7884969b2c820070d3092R95

That still doesn't seem right, evalScript does define its own module.filename using the CWD.

I was slightly off - the "no parent" case is when calling node -r some-package (not -e). In that case the parent module that's passed to _resolveFilename has its filename property set to null, in which case we need to fallback to the cwd.

Also any entry point script doesn't have a parent, and in general we never use the "parent" (which is not reliable, see https://nodejs.org/api/deprecations.html#dep0144-moduleparent) to resolve require calls

In the context of the resolution pipeline (_resolveFilename), the parent is whichever module performs the require call, it's not the same as module.parent. It's reliable in that context.

[aduh95]

Let's use the existing util:

node/lib/internal/modules/cjs/loader.js

Lines 609 to 621 in d0fa608

	function trySelfParentPath(parent) {
	if (!parent) { return false; }
	if (parent.filename) {
	return parent.filename;
	} else if (parent.id === '<repl>' || parent.id === 'internal/preload') {
	try {
	return process.cwd() + path.sep;
	} catch {
	return false;
	}
	}
	}

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/72681/