Add Yarn PnP support#1966
Conversation
|
@microsoft-github-policy-service agree company="Datadog" |
|
I see my tests are not passing in CI, looking into it 🙇 |
| var ( | ||
| isPnpApiInitialized atomic.Uint32 | ||
| cachedPnpApi *PnpApi | ||
| pnpMu sync.Mutex | ||
| // testPnpCache stores per-goroutine PnP APIs for test isolation | ||
| // Key is goroutine ID (as int) | ||
| testPnpCache sync.Map // map[int]*PnpApi | ||
| ) | ||
|
|
||
| // getGoroutineID returns the current goroutine ID | ||
| // It is usually not recommended to work with goroutine IDs, but it is the most non-intrusive way to setup a parallel testing environment for PnP API | ||
| func getGoroutineID() int { | ||
| var buf [64]byte | ||
| n := runtime.Stack(buf[:], false) | ||
| idField := strings.Fields(strings.TrimPrefix(string(buf[:n]), "goroutine "))[0] | ||
| id, _ := strconv.Atoi(idField) | ||
| return id | ||
| } | ||
|
|
There was a problem hiding this comment.
All of this global / goroutine local storage is something that won't work. If we are storing something, it needs to be attached to a Program, Project, a Host, etc, not global. Parsing out the goroutine ID is definitely a bad idea, especially given our LSP can handle multiple requests at the same time from multiple goroutines and so on.
There was a problem hiding this comment.
@jakebailey Thank you for having a first look at this PR!
Since your comments, we changed how we initialize the PnpApi by attaching it to the Host directly on initialization. The host can then provide it through host.PnpApi() which will return nil or the pnpapi instance if in a yarn project
|
|
||
| pnpApi := &PnpApi{fs: fs, url: filePath} | ||
|
|
||
| manifestData, err := pnpApi.findClosestPnpManifest() |
There was a problem hiding this comment.
I don't really understand this init. Surely one needs to be able to load multiple projects with differing pnp info at the same time? All of this info really does need to be handled differently.
There was a problem hiding this comment.
With the last changes mentioned in this comment, we initialize the PnP api with the Host. This means we assume that we have one yarn PnP project per host
I don't think we could initialize it on a smaller scope (like Project or Program) from looking at the changes we had to do to support PnP, but let me know what you think!
d70cc8a to
854be04
Compare
e92efeb to
e4a3c4b
Compare
36c114c to
9aa7f3e
Compare
* add pnpapi test * remove cycle dependency * add error handling * add error message * apply error message * add error bubbling * code clean up * remove useless test * add empty findBrokenPeerDependencies * early return * change function name * apply code review * merge duplicate logic * apply collect usage * apply codereview * apply fromConfig
440f2f8 to
478d4d9
Compare
478d4d9 to
f3c6477
Compare
a98f2f6 to
c343dca
Compare
e8885f5 to
4f9181b
Compare
Yarn-pnp: Attach to session and hoist filesystem overrides
8e79c4f to
daac702
Compare
There was a problem hiding this comment.
Pull request overview
Adds native Yarn Plug’n’Play (PnP) support across the TypeScript-Go compiler and language server, including zip/VFS handling, module/type resolution, and test harness updates to validate behavior in PnP environments.
Changes:
- Introduces a PnP API + manifest parser and wires it through compiler/LSP/module-resolution hosts.
- Adds a PnP-aware VFS that supports virtual paths and reading from zip archives, plus LSP URI support for zip paths.
- Extends resolution/type-roots/auto-import logic and adds new compiler + fourslash tests and baselines.
Reviewed changes
Copilot reviewed 72 out of 76 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| testdata/tests/cases/compiler/pnpTypeRootsResolution.ts | New compiler test for resolving @types/* via PnP. |
| testdata/tests/cases/compiler/pnpTransitiveDependencies.ts | New compiler test for enforcing direct-dependency rules under PnP. |
| testdata/tests/cases/compiler/pnpSimpleTest.ts | New compiler test for basic PnP package resolution. |
| testdata/tests/cases/compiler/pnpDeclarationEmitWorkspace.ts | New compiler test for declaration emit with workspace exports under PnP. |
| testdata/fixtures/pnp/test-expectations.json | Adds/updates PnP resolution expectation fixtures. |
| testdata/baselines/reference/compiler/pnpTypeRootsResolution.types | Baseline for PnP type analysis output. |
| testdata/baselines/reference/compiler/pnpTypeRootsResolution.symbols | Baseline for PnP symbol output. |
| testdata/baselines/reference/compiler/pnpTypeRootsResolution.js | Baseline for PnP JS emit output. |
| testdata/baselines/reference/compiler/pnpTransitiveDependencies.types | Baseline for transitive-deps type output under PnP. |
| testdata/baselines/reference/compiler/pnpTransitiveDependencies.symbols | Baseline for transitive-deps symbol output under PnP. |
| testdata/baselines/reference/compiler/pnpTransitiveDependencies.js | Baseline for transitive-deps JS emit output under PnP. |
| testdata/baselines/reference/compiler/pnpTransitiveDependencies.errors.txt | Baseline for expected missing-dependency error under PnP. |
| testdata/baselines/reference/compiler/pnpSimpleTest.types | Baseline for basic PnP type output. |
| testdata/baselines/reference/compiler/pnpSimpleTest.symbols | Baseline for basic PnP symbol output. |
| testdata/baselines/reference/compiler/pnpSimpleTest.js | Baseline for basic PnP JS emit output. |
| testdata/baselines/reference/compiler/pnpDeclarationEmitWorkspace.types | Baseline for PnP declaration-emit type output. |
| testdata/baselines/reference/compiler/pnpDeclarationEmitWorkspace.symbols | Baseline for PnP declaration-emit symbol output. |
| testdata/baselines/reference/compiler/pnpDeclarationEmitWorkspace.js | Baseline for PnP declaration-emit JS output. |
| internal/vfs/pnpvfs/pnpvfs.go | Adds PnP VFS: virtual path support + zip file access with caching. |
| internal/vfs/pnpvfs/pnpvfs_test.go | Tests for PnP VFS behavior (virtual paths, zip access, errors). |
| internal/tspath/path.go | Adds IsZipPath helper. |
| internal/tsoptions/tsoptionstest/vfsparseconfighost.go | Updates test ParseConfigHost to include PnP API/VFS behavior. |
| internal/tsoptions/tsconfigparsing.go | Extends ParseConfigHost to expose PnP API. |
| internal/transformers/tstransforms/importelision_test.go | Updates test host to satisfy new PnP API surface. |
| internal/testutil/tsbaseline/js_emit_baseline.go | Treats .pnp.cjs as a preserved harness file for baselines. |
| internal/testutil/harnessutil/harnessutil.go | Initializes PnP API + wraps FS in harness compilation pipeline. |
| internal/project/snapshot.go | Threads PnP API through LSP snapshot lifecycle. |
| internal/project/session.go | Stores PnP API in session; exposes via resolution host interface. |
| internal/project/projectcollectionbuilder.go | Passes PnP API into project/config registry builders. |
| internal/project/extendedconfigcache_test.go | Updates ParseConfigHost test implementation for PnP. |
| internal/project/configfileregistrybuilder.go | Adds PnP API exposure to config file registry builder. |
| internal/project/compilerhost.go | Exposes PnP API via project compiler host. |
| internal/project/autoimport.go | Threads PnP API into auto-import registry clone host. |
| internal/pnp/pnpapi.go | Implements PnP API (manifest loading, locator lookup, resolution, type roots). |
| internal/pnp/pnpapi_test.go | Adds unit tests for manifest parsing + resolution behavior. |
| internal/pnp/pnp.go | Adds PnP initialization + .pnp.cjs detection helper. |
| internal/pnp/manifestparser.go | Parses .pnp.data.json / extracts payload from .pnp.cjs. |
| internal/modulespecifiers/types.go | Extends module specifier host interface with PnP API. |
| internal/modulespecifiers/specifiers.go | Adds PnP-aware module specifier generation logic. |
| internal/modulespecifiers/specifiers_test.go | Updates mock host for new PnP API method. |
| internal/module/types.go | Extends module ResolutionHost interface with PnP API. |
| internal/module/resolver.go | Adds PnP resolution paths, PnP type roots, and PnP external-library detection. |
| internal/module/resolver_test.go | Updates resolver test host stub for PnP API. |
| internal/lsp/server.go | Initializes PnP API for LSP sessions and wraps FS with PnP VFS when present. |
| internal/lsp/lsproto/lsp.go | Supports zip: document URIs in URI->filename conversion. |
| internal/ls/string_completions.go | Notes potential PnP impacts on string completions (TODO). |
| internal/ls/lsconv/converters.go | Emits zip: URIs for zip-backed paths. |
| internal/ls/lsconv/converters_test.go | Adds tests for zip: URI round-tripping. |
| internal/ls/host.go | Extends LS host interface with PnP API. |
| internal/ls/autoimport/util.go | Plumbs PnP API into autoimport module resolver host wrapper. |
| internal/ls/autoimport/registry.go | Adjusts autoimport indexing to account for PnP-managed packages. |
| internal/ls/autoimport/aliasresolver.go | Extends checker.Program stub with PnP API method. |
| internal/fourslash/tests/pnpAutoImportCompletion_test.go | Adds fourslash test validating PnP auto-import completions. |
| internal/execute/watcher.go | Passes PnP API into compiler host construction for watch mode. |
| internal/execute/tsctests/sys.go | Adds PnP API + PnP VFS wrapping to test system. |
| internal/execute/tsc/compile.go | Extends tsc.System interface with PnpApi(). |
| internal/execute/tsc.go | Passes PnP API into cached compiler hosts for builds. |
| internal/execute/build/orchestrator.go | Passes PnP API into build orchestrator compiler host. |
| internal/execute/build/host.go | Exposes PnP API from build host wrapper. |
| internal/execute/build/compilerHost.go | Exposes PnP API from build compiler host wrapper. |
| internal/diagnostics/loc_generated.go | Import ordering adjustment in generated diagnostics localization file. |
| internal/diagnostics/extraDiagnosticMessages.json | Adds PnP-specific diagnostic messages. |
| internal/diagnostics/diagnostics_generated.go | Adds generated constants for new PnP diagnostics. |
| internal/core/compileroptions.go | Refactors effective type-root computation to support PnP type roots. |
| internal/compiler/projectreferencedtsfakinghost.go | Propagates PnP API through project-reference host wrapper. |
| internal/compiler/program.go | Exposes PnP API via checker.Program from compiler Program. |
| internal/compiler/program_test.go | Updates host construction for new PnP API parameter. |
| internal/compiler/host.go | Extends compiler host interface/constructors to carry PnP API. |
| internal/compiler/emitHost.go | Exposes PnP API via emit host interface. |
| internal/compiler/emit_test.go | Updates emit benchmarks for new host constructor signature. |
| internal/checker/checker_test.go | Updates checker tests for new host constructor signature. |
| internal/api/server.go | Initializes PnP API + wraps FS for the API server. |
| cmd/tsgo/sys.go | Initializes PnP API + wraps FS for CLI runs; exposes PnpApi(). |
| _extension/src/client.ts | Adds zip scheme support to VS Code extension document selector. |
Files not reviewed (2)
- internal/diagnostics/diagnostics_generated.go: Language not supported
- internal/diagnostics/loc_generated.go: Language not supported
| // TODO: implement this from yarn sourcecode | ||
| // https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-pnp/sources/loader/makeApi.ts#L458 | ||
| func findBrokenPeerDependencies(specifier string, parent *Locator) []Locator { | ||
| return []Locator{} | ||
| } |
There was a problem hiding this comment.
This function is only used for error reports on peer dependencies, and might require other changes in the manifest parser
I can either:
- Implement properly this function and the required changes
- Remove all mentions of
findBrokenPeerDependenciesand this stub - Do nothing in this PR and implement the changes in a followup PR
| // If pnp is available, node_modules buckets won't be built as all packages are located in `.yarn/cache` | ||
| // This is why we need to handle all files, except the ones that are not importable in the project | ||
| pnpApi := b.host.PnpApi() | ||
| if pnpApi != nil { | ||
| if !pnpApi.IsImportable(string(projectPath), file.FileName()) { | ||
| continue | ||
| } |
There was a problem hiding this comment.
pnpApi.IsImportable also returns true for files within the same package because the pnp manifest always adds the package itself as the first entry of PackageDependencies. I can add a defensive check in case the manifest is malformed but that doesn't seem necessary to me
Also, when PnP is enabled, we only have PnP-managed package locations, so passing all source files through IsImportable is valid here
## What's the problem this PR addresses? Add editor SDK support for oxc. This should enable optional editor integration with both oxfmt and oxlint. ## How did you fix it? Added oxlint, oxfmt and oxlint-tsgolint as supported integration and documented it. Oxlint and oxlint-tsgolint addition is a bit hacky and I don't know whether this is the right direction or not. Rationale can be read from the comment there. Loader option added to sdk setupEnv to prevent oxfmt tinypool from being broken due to failing to resolve tinypool/dist/process.js. I haven't tried this with vim yet since I don't use it so if anyone would kindly help test whether this will work on vim or not, thanks! ### oxlint type-aware feature (tsgolint) The biggest challenge here is the type-aware feature (tsgolint) support. The rust side of oxlint spawn tsgolint a bit different from how the oxlint itself is spawned: https://github.com/oxc-project/oxc/blob/d3dcf5bc9718ebb4839be27062b5d82da2118e2e/crates/oxc_linter/src/tsgolint.rs#L265-L270 Instead of using js entrypoint, it directly spawn executable so the wrapper approach is not directly applicable here. Luckily, there are some strategies that the oxc_linter uses to resolve tsgolint: - via PATH - via node_modules search - direct path via OXLINT_TSGOLINT_PATH (which is what oxc.path.tsgolint does basically) The only realistic strategy is to use PATH. For the sdks, it will generate a windows executable shim (cmd) that will basically just spawn tsgolint.js, else should fallback to tsgolint. Be noted however that this will not fix fundamental issue with typescript-go module resolution, which requires this to be merged first: - microsoft/typescript-go#1966 This PR just enable the usage of tsgo regardless of whether it can resolve with pnp or not in the type-checking. ### Reference Coc config: https://github.com/oxc-project/coc-oxc/blob/7a7f0d8f503d5cdb65da37232da6b865f159b42a/src/common.ts#L27-L36 Vscode oxc binary resolution reference: https://github.com/oxc-project/oxc-vscode/blob/main/client/findBinary.ts Oxlint-tsgolint executable resolution: https://github.com/oxc-project/oxc/blob/d3dcf5bc9718ebb4839be27062b5d82da2118e2e/crates/oxc_linter/src/tsgolint.rs#L1164-L1225 ## Checklist - [x] I have read the [Contributing Guide](https://yarnpkg.com/advanced/contributing). - [x] I have set the packages that need to be released for my changes to be effective. - [x] I will check that all automated PR checks pass before the PR gets reviewed.
|
@RyanCavanaugh The issue is linked in the But perhaps you would like me to link it to this PR differently? |
|
I'd assume he wants it linked properly, not just a mention. To do this, you can add the following in the PR description: |
See: - https://devblogs.microsoft.com/typescript/announcing-typescript-7-0-rc/ Ok, so this is a bit fiddly, so let me explain what's going on: - We used to have `tsc` from `typescript` (v6), written in JS, and `tsgo` from `@typescript/native-preview`, so they could both be installed at the same time. - With v7, the RC package takes over the `tsc` executable; it's now the Go implementation and there is no separate `tsgo`; so... - We change our global install to `7.0-rc`, and ditto for the in-project install. We no longer have a v6 install on disk. - Rename (the project-local) `bin/tsgo` to `bin/tsc`, and make it run `tsc` instead of `tsgo`. - We update our Neovim config to assume `tsc` is now the Go version instead of the JS version; but note that we still have to use the "tsgo" config from nvim-lspconfig, because that hasn't been renamed yet, it's just that we configure it to use the `tsc` executable now instead of `tsgo`. - The Go implementation still doesn't support Yarn PnP and the corresponding PR (microsoft/typescript-go#1966) is still open, so we maintain our hack wherein we use the `ts_ls` config from nvim-lspconfig to use typescript-language-server with a wired-in path to a Yarn-managed shim. This looks fragile, and it probably is, but it has been working for me ever since I added it a couple of months ago (in 011cee5, "feat(nvim): point ts_ls at Yarn PnP tsserver shim when available", 2026-05-12). - The Neovim config will use a repo-local `tsc` if one exists, but note that it assumes `tsc` is the Go implementation. So if you use it in a repo with v6 or older, it won't work, because v6 doesn't support `--lsp --stdio`. I'm not worried about this; I have only a few such repos, and I will update them next time I have to work in them. - Update references to `tsgo` in docs to reference `tsc` instead. - Note: the `_tsc` completions in the third-party zsh-completions repo still match the JS implementation instead of the Go one; presumably they will eventually update. I am not sure exactly where the command line options diverge, other than having seen that while v7 supports LSP via `tsc --lsp --stdio`, v6 does not. Finally, and most importantly, merely updating my dotfiles on other machines won't make all this magically start working: I will need to manually uninstall the old `typescript` package and install the new one: ``` npm uninstall -g typescript @typescript/native-preview npm install -g typescript@rc --min-release-age=0 ``` Change-Id: rmoqwxkqwoywkxrlwspqqmwwmktwqrvz
## Summary - Bound the generated TypeScript compatibility patch to TypeScript versions before 7.0.0, and committed the matching generated patch artifact so the compat patch generator stays reproducible. - Fixed optional patch handling so `optional!` patches fall back when a package no longer ships a target file (for example `lib/_tsc.js` in TypeScript 7 / TypeScript compatibility shims), instead of failing the install with `ENOENT`. - Added regression coverage for both direct `typescript@7.0.1-rc` installs and the TypeScript 7 side-by-side recommendation where the `typescript` ident aliases to `npm:@typescript/typescript6@^6.0.0`. - Stabilized the scoped `plugin-typescript` acceptance test by moving it from `@babel/traverse` (now reported by npm search metadata as having included types) to a fixture package that still exercises DefinitelyTyped scoped package insertion. - Added deferred version metadata for the changed workspaces. ## Test plan - `yarn workspace @yarnpkg/plugin-compat test:plugin-compat` - `yarn test:unit packages/plugin-patch` - `node ./scripts/run-yarn.js test:integration packages/acceptance-tests/pkg-tests-specs/sources/plugins/plugin-typescript.test.ts` - `yarn version check` ## Notes TypeScript 7 ships a native compiler package layout and no longer has the legacy JS compiler files patched by the existing PnP compatibility diff, such as `lib/_tsc.js`. Bounding the TypeScript compat patch makes this intent explicit, while the optional patch fallback fixes the broader bug that `optional!` patch failures caused by missing target files were still fatal. TypeScript 7 support should ship with microsoft/typescript-go#1966. Made with Cursor --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: John Doe <you@example.com> Co-authored-by: Maël Nison <mael.nison@mistral.ai>
Closes #460
Co-authored with @gun-yu as a result of their changes (#1876) being merged in this PR
Motivation
This PR adds Plug'n'Play support natively to Typescript Go, following this issue: #460
It has been reviewed and supported by @arcanis, the lead maintainer of Yarn, and the original author of Yarn PnP.
Datadog has a frontend monorepo using yarn with over 6k packages, and seeing how TS Strada struggles with our current scaling, we decided to invest time in adding a native Yarn PnP support for Typescript Go.
This PnP implementation has been actively used in the IDE of more than 230 engineers at Datadog, and we're committed to fixing all issues reported to us.
Challenges
We did not integrate it in our CI yet as we still have several packages failing on build mode (most errors seem to be reported in the issues section of TS Corsa). Because the TS Corsa API is not available yet, we also couldn't integrate it properly with a fast lage setup unlike with the TS Strada API.
Changes
It's based on the main changes from the original yarn patch (microsoft/TypeScript@99f3e13) that the community has been maintaining for years throughout Typescript Strada updates, except that we implemented the official PnP specification so it doesn't depend on third-party code.
Implemented features:
Hostis initialized for both build and LSP modesinternal/module/resolver.gointernal/modulespecifiers/specifiers.gointernal/core/compileroptions.goMissing features:
.pnp.cjschangesTests