Skip to content

Commit 9a9e93a

Browse files
authored
switch to optimistic + lazy registration for pipenv, pyenv and poetry managers (#1423)
most failures for pipenv/poetry/pyenv are at `nativeFinderRefresh` stage in startup. Most of these machines failing on these env manager are NOT using that manager. They're overwhelmingly venv users who just happen to have pipenv/poetry/pyenv installed on their system. Therefore this PR loads these three managers in an optimistic + lazy manner. Instead of trying to find if they exist on the machine at startup (which potentially causes PET to be called during manager registration) let them get registered. If they are registered, the search for environments will then be done later when the user hard-refreshes managers or clicks on the manager in the sidebar where at that point it will check for any envs of these manager types. It may find them (weirdly placed environments for any of these managers but PETs extensive finding succeeds) or not. If not the behavior is fine and just shows no data in the dropdown if no environments end up being found.
1 parent 3829315 commit 9a9e93a

File tree

13 files changed

+292
-346
lines changed

13 files changed

+292
-346
lines changed

docs/startup-flow.md

Lines changed: 70 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -19,104 +19,79 @@ python environments extension begins activation
1919
**ASYNC (setImmediate callback, still in extension.ts):**
2020
1. spawn PET process (`createNativePythonFinder`)
2121
1. sets up a JSON-RPC connection to it over stdin/stdout
22-
2. register all built-in managers + shell env init in parallel (Promise.all):
23-
- `shellStartupVarsMgr.initialize()`
24-
- for each manager (system, conda, pyenv, pipenv, poetry):
25-
1. check if tool exists (e.g. `getConda(nativeFinder)` asks PET for the conda binary)
26-
2. if tool not found → log, return early (manager not registered)
27-
3. if tool found → create manager, call `api.registerEnvironmentManager(manager)`
28-
- this adds it to the `EnvironmentManagers` map
29-
- fires `onDidChangeEnvironmentManager``ManagerReady` deferred resolves for this manager
30-
3. all registrations complete (Promise.all resolves)
22+
2. register all built-in managers in parallel (Promise.all):
23+
- system: create SysPythonManager + VenvManager + PipPackageManager, register immediately (✅ NO PET call, sets up file watcher)
24+
- conda: `getConda(nativeFinder)` checks settings → cache → persistent state → PATH
25+
- pyenv & pipenv & poetry: create PyEnvManager, register immediately
26+
- ✅ NO PET call — always registers unconditionally (lazy discovery)
27+
- shellStartupVars: initialize
28+
- all managers fire `onDidChangeEnvironmentManager` → ManagerReady resolves
29+
3. all registrations complete (Promise.all resolves) — fast, typically milliseconds
30+
3131

3232
**--- gate point: `applyInitialEnvironmentSelection` ---**
3333

34-
📊 TELEMETRY: ENV_SELECTION.STARTED { duration (activation→here), registeredManagerCount, registeredManagerIds, workspaceFolderCount }
35-
36-
1. for each workspace folder + global scope (no workspace case), run `resolvePriorityChainCore` to find manager:
37-
- P1: pythonProjects[] setting → specific manager for this project
38-
- P2: user-configured defaultEnvManager setting
39-
- P3: user-configured python.defaultInterpreterPath → nativeFinder.resolve(path)
40-
- P4: auto-discovery → try venv manager, fall back to system python
41-
- for workspace scope: call `venvManager.get(scope)`
42-
- if venv found (local .venv/venv) → use venv manager with that env
43-
- if no local venv → venv manager may still return its `globalEnv` (system Python)
44-
- if venvManager.get returns undefined → fall back to system python manager
45-
- for global scope: use system python manager directly
46-
47-
2. get the environment from the winning priority level:
48-
49-
--- fork point: `result.environment ?? await result.manager.get(folder.uri)` ---
50-
left side truthy = envPreResolved | left side undefined = managerDiscovery
51-
52-
envPreResolved — P3 won (interpreter → manager):
53-
`resolvePriorityChainCore` calls `tryResolveInterpreterPath()`:
54-
1. `nativeFinder.resolve(path)` — single PET call, resolves just this one binary
55-
2. find which manager owns the resolved env (by managerId)
56-
3. return { manager, environment } — BOTH are known
57-
→ result.environment is set → the `??` short-circuits
58-
→ no `manager.get()` called, no `initialize()`, no full discovery
59-
60-
managerDiscovery — P1, P2, or P4 won (manager → interpreter):
61-
`resolvePriorityChainCore` returns { manager, environment: undefined }
62-
→ falls through to `await result.manager.get(scope)`
63-
64-
**--- inner fork: fast path vs slow path (tryFastPathGet in fastPath.ts) ---**
65-
Conditions checked before entering fast path:
66-
a. `_initialized` deferred is undefined (never created) OR has not yet completed
67-
b. scope is a `Uri` (not global/undefined)
68-
69-
FAST PATH (background init kickoff + optional early return):
70-
**Race-condition safety (runs before any await):**
71-
1. if `_initialized` doesn't exist yet:
72-
- create deferred and **register immediately** via `setInitialized()` callback
73-
- this blocks concurrent callers from spawning duplicate background inits
74-
- kick off `startBackgroundInit()` as fire-and-forget
75-
- this happens as soon as (a) and (b) are true, **even if** no persisted path exists
76-
2. get project fsPath: `getProjectFsPathForScope(api, scope)`
77-
- prefers resolved project path if available, falls back to scope.fsPath
78-
- shared across all managers to avoid lambda duplication
79-
3. read persisted path (only if scope is a `Uri`; may return undefined)
80-
4. if a persisted path exists:
81-
- attempt `resolve(persistedPath)`
82-
- failure (no env, mismatched manager, etc.) → fall through to SLOW PATH
83-
- success → return env immediately (background init continues in parallel)
84-
**Failure recovery (in startBackgroundInit error handler):**
85-
- if background init throws: `setInitialized(undefined)` — clear deferred so next `get()` call retries init
86-
87-
SLOW PATH — fast path conditions not met, or fast path failed:
88-
4. `initialize()` — lazy, once-only per manager (guarded by `_initialized` deferred)
89-
**Once-only guarantee:**
90-
- first caller creates `_initialized` deferred (if not already created by fast path)
91-
- concurrent callers see the existing deferred and await it instead of re-running init
92-
- deferred is **not cleared on failure** here (unlike in fast-path background handler)
93-
so only one init attempt runs, but subsequent calls still await the same failed init
94-
**Note:** In the fast path, if background init fails, the deferred is cleared to allow retry
95-
a. `nativeFinder.refresh(hardRefresh=false)`:
96-
→ internally calls `handleSoftRefresh()` → computes cache key from options
97-
- on reload: cache is empty (Map was destroyed) → cache miss
98-
- falls through to `handleHardRefresh()`
99-
→ `handleHardRefresh()` adds request to WorkerPool queue (concurrency 1):
100-
1. run `configure()` to setup PET search paths
101-
2. run `refresh` — PET scans filesystem
102-
- PET may use its own on-disk cache
103-
3. returns NativeInfo[] (all envs of all types)
104-
- result stored in in-memory cache so subsequent managers get instant cache hit
105-
b. filter results to this manager's env type (e.g. conda filters to kind=conda)
106-
c. convert NativeEnvInfo → PythonEnvironment objects → populate collection
107-
d. `loadEnvMap()` — reads persisted env path from workspace state
108-
→ matches path against PET discovery results
109-
→ populates `fsPathToEnv` map
110-
5. look up scope in `fsPathToEnv` → return the matched env
111-
112-
📊 TELEMETRY: ENV_SELECTION.RESULT (per scope) { duration (priority chain + manager.get), scope, prioritySource, managerId, path, hasPersistedSelection }
113-
114-
3. env is cached in memory (no settings.json write)
115-
4. Python extension / status bar can now get the selected env via `api.getEnvironment(scope)`
116-
117-
📊 TELEMETRY: EXTENSION.MANAGER_REGISTRATION_DURATION { duration (activation→here), result, failureStage?, errorType? }
118-
119-
**POST-INIT:**
34+
📊 TELEMETRY: ENV_SELECTION.STARTED { duration, registeredManagerCount, registeredManagerIds, workspaceFolderCount }
35+
36+
**Step 1 — pick a manager** (`resolvePriorityChainCore`, per workspace folder + global):
37+
38+
| Priority | Source | Returns |
39+
|----------|--------|---------|
40+
| P1 | `pythonProjects[]` setting | manager only |
41+
| P2 | `defaultEnvManager` setting | manager only |
42+
| P3 | `python.defaultInterpreterPath``nativeFinder.resolve(path)` | manager **+ environment** |
43+
| P4 | auto-discovery: venv → system python fallback | manager only |
44+
45+
**Step 2 — get the environment** (`result.environment ?? await result.manager.get(scope)`):
46+
47+
- **If P3 won:** environment is already resolved → done, no `get()` call needed.
48+
- **Otherwise:** calls `manager.get(scope)`, which has two internal paths:
49+
50+
**Fast path** (`tryFastPathGet` in `fastPath.ts`) — entered when `_initialized` hasn't completed and scope is a `Uri`:
51+
1. Synchronously create `_initialized` deferred + kick off `startBackgroundInit()` (fire-and-forget full PET discovery)
52+
2. Read persisted env path from workspace state
53+
3. If persisted path exists → `resolve(path)` → return immediately (background init continues in parallel)
54+
4. If no persisted path or resolve fails → fall through to slow path
55+
- *On background init failure:* clears `_initialized` so next `get()` retries
56+
57+
**Slow path** — fast path skipped or failed:
58+
1. `initialize()` — lazy, once-only (guarded by `_initialized` deferred, concurrent callers await it)
59+
- `nativeFinder.refresh(false)` → PET scan (cached across managers after first call)
60+
- Filter results to this manager's type → populate `collection`
61+
- `loadEnvMap()` → match persisted paths against discovered envs
62+
2. Look up scope in `fsPathToEnv` → return matched env
63+
64+
📊 TELEMETRY: ENV_SELECTION.RESULT (per scope) { duration, scope, prioritySource, managerId, path, hasPersistedSelection }
65+
66+
**Step 3 — done:**
67+
- env cached in memory (no settings.json write)
68+
- available via `api.getEnvironment(scope)`
69+
70+
📊 TELEMETRY: EXTENSION.MANAGER_REGISTRATION_DURATION { duration, result, failureStage?, errorType? }
71+
72+
---
73+
74+
### Other entry points to `initialize()`
75+
76+
All three trigger `initialize()` lazily (once-only, guarded by `_initialized` deferred). After the first call completes, subsequent calls are no-ops.
77+
78+
**`manager.get(scope)`** — environment selection (Step 2 above):
79+
- Called during `applyInitialEnvironmentSelection` or when settings change triggers re-selection
80+
- Fast path may resolve immediately; slow path awaits `initialize()`
81+
82+
**`manager.getEnvironments(scope)`** — sidebar / listing:
83+
- Called when user expands a manager node in the Python environments panel
84+
- Also called by any API consumer requesting the full environment list
85+
- If PET cache populated from earlier `get()` → instant hit; otherwise warm PET call
86+
87+
**`manager.resolve(context)`** — path resolution:
88+
- Called when resolving a specific Python binary path to check if it belongs to this manager
89+
- Used by `tryResolveInterpreterPath()` in the priority chain (P3) and by external API consumers
90+
- Awaits `initialize()`, then delegates to manager-specific resolve (e.g., `resolvePipenvPath`)
91+
92+
---
93+
94+
POST-INIT:
12095
1. register terminal package watcher
12196
2. register settings change listener (`registerInterpreterSettingsChangeListener`) — re-runs priority chain if settings change
12297
3. initialize terminal manager

src/common/telemetry/constants.ts

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,18 @@ export enum EventNames {
101101
* - hasPersistedSelection: boolean (whether a persisted env path existed in workspace state)
102102
*/
103103
ENV_SELECTION_RESULT = 'ENV_SELECTION.RESULT',
104+
/**
105+
* Telemetry event fired when a lazily-registered manager completes its first initialization.
106+
* Replaces MANAGER_REGISTRATION_SKIPPED and MANAGER_REGISTRATION_FAILED for managers
107+
* that now register unconditionally (pipenv, poetry, pyenv).
108+
* Properties:
109+
* - managerName: string (e.g. 'pipenv', 'poetry', 'pyenv')
110+
* - result: 'success' | 'tool_not_found' | 'error'
111+
* - envCount: number (environments discovered)
112+
* - toolSource: string (how the CLI was found: 'settings', 'local', 'pet', 'none')
113+
* - errorType: string (classified error category, on failure only)
114+
*/
115+
MANAGER_LAZY_INIT = 'MANAGER.LAZY_INIT',
104116
}
105117

106118
// Map all events to their properties
@@ -373,4 +385,22 @@ export interface IEventNamePropertyMapping {
373385
resolutionPath: string;
374386
hasPersistedSelection: boolean;
375387
};
388+
389+
/* __GDPR__
390+
"manager.lazy_init": {
391+
"managerName": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "eleanorjboyd" },
392+
"result": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "eleanorjboyd" },
393+
"envCount": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "owner": "eleanorjboyd" },
394+
"toolSource": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "eleanorjboyd" },
395+
"errorType": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "owner": "eleanorjboyd" },
396+
"<duration>": { "classification": "SystemMetaData", "purpose": "FeatureInsight", "isMeasurement": true, "owner": "eleanorjboyd" }
397+
}
398+
*/
399+
[EventNames.MANAGER_LAZY_INIT]: {
400+
managerName: string;
401+
result: 'success' | 'tool_not_found' | 'error';
402+
envCount: number;
403+
toolSource: string;
404+
errorType?: string;
405+
};
376406
}

src/managers/common/fastPath.ts

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
import { Uri } from 'vscode';
55
import { GetEnvironmentScope, PythonEnvironment, PythonEnvironmentApi } from '../../api';
6-
import { traceError, traceWarn } from '../../common/logging';
6+
import { traceError, traceVerbose, traceWarn } from '../../common/logging';
77
import { createDeferred, Deferred } from '../../common/utils/deferred';
88

99
/**
@@ -62,7 +62,6 @@ export async function tryFastPathGet(opts: FastPathOptions): Promise<FastPathRes
6262

6363
let deferred = opts.initialized;
6464
if (!deferred) {
65-
// Register deferred before any await to avoid concurrent callers starting duplicate inits.
6665
deferred = createDeferred<void>();
6766
opts.setInitialized(deferred);
6867
const deferredRef = deferred;
@@ -93,8 +92,13 @@ export async function tryFastPathGet(opts: FastPathOptions): Promise<FastPathRes
9392
return { env: resolved };
9493
}
9594
} catch (err) {
96-
traceWarn(`[${opts.label}] Fast path resolve failed for '${persistedPath}', falling back to full init:`, err);
95+
traceWarn(
96+
`[${opts.label}] Fast path resolve failed for '${persistedPath}', falling back to full init:`,
97+
err,
98+
);
9799
}
100+
} else {
101+
traceVerbose(`[${opts.label}] Fast path: no persisted path, falling through to slow path`);
98102
}
99103

100104
return undefined;

src/managers/pipenv/main.ts

Lines changed: 3 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,19 @@
11
import { Disposable } from 'vscode';
22
import { PythonEnvironmentApi } from '../../api';
33
import { traceInfo } from '../../common/logging';
4-
import { EventNames } from '../../common/telemetry/constants';
5-
import { classifyError } from '../../common/telemetry/errorClassifier';
6-
import { sendTelemetryEvent } from '../../common/telemetry/sender';
74
import { getPythonApi } from '../../features/pythonApi';
85
import { PythonProjectManager } from '../../internal.api';
96
import { NativePythonFinder } from '../common/nativePythonFinder';
107
import { PipenvManager } from './pipenvManager';
11-
import { getPipenv, hasPipenvEnvironments } from './pipenvUtils';
12-
13-
import { notifyMissingManagerIfDefault } from '../common/utils';
148

159
export async function registerPipenvFeatures(
1610
nativeFinder: NativePythonFinder,
1711
disposables: Disposable[],
1812
projectManager: PythonProjectManager,
1913
): Promise<void> {
20-
let stage = 'getPythonApi';
2114
const api: PythonEnvironmentApi = await getPythonApi();
2215

23-
try {
24-
stage = 'getPipenv';
25-
const pipenv = await getPipenv(nativeFinder);
26-
27-
// Register the manager if the CLI is found, or if there are existing pipenv environments.
28-
// This allows users with existing pipenv environments to still see and use them.
29-
stage = 'hasPipenvEnvironments';
30-
const hasPipenvEnvs = !pipenv && (await hasPipenvEnvironments(nativeFinder));
31-
32-
if (pipenv || hasPipenvEnvs) {
33-
stage = 'createManager';
34-
const mgr = new PipenvManager(nativeFinder, api);
35-
stage = 'registerManager';
36-
disposables.push(mgr, api.registerEnvironmentManager(mgr));
37-
if (!pipenv) {
38-
traceInfo(
39-
'Pipenv CLI not found, but pipenv environments were discovered. Registering manager for read-only environment management. To enable full pipenv features, set the "python.pipenvPath" setting to the path of your pipenv executable.',
40-
);
41-
}
42-
} else {
43-
traceInfo(
44-
'Pipenv not found, turning off pipenv features. If you have pipenv installed in a non-standard location, set the "python.pipenvPath" setting.',
45-
);
46-
sendTelemetryEvent(EventNames.MANAGER_REGISTRATION_SKIPPED, undefined, {
47-
managerName: 'pipenv',
48-
reason: 'tool_not_found',
49-
});
50-
await notifyMissingManagerIfDefault('ms-python.python:pipenv', projectManager, api);
51-
}
52-
} catch (ex) {
53-
const failureStage = (ex as Error & { failureStage?: string })?.failureStage ?? stage;
54-
traceInfo(
55-
'Pipenv not found, turning off pipenv features. If you have pipenv installed in a non-standard location, set the "python.pipenvPath" setting.',
56-
ex,
57-
);
58-
sendTelemetryEvent(EventNames.MANAGER_REGISTRATION_FAILED, undefined, {
59-
managerName: 'pipenv',
60-
errorType: classifyError(ex),
61-
failureStage,
62-
});
63-
await notifyMissingManagerIfDefault('ms-python.python:pipenv', projectManager, api);
64-
}
16+
traceInfo('Registering pipenv manager (environments will be discovered lazily)');
17+
const mgr = new PipenvManager(nativeFinder, api, projectManager);
18+
disposables.push(mgr, api.registerEnvironmentManager(mgr));
6519
}

0 commit comments

Comments
 (0)