Make ms-python.python a soft dependency (#123)

* Make ms-python.python a soft dependency

- Remove `ms-python.python` from `extensionDependencies` so the extension can
  install in VS Code derivatives (e.g. Cursor) whose marketplaces cannot resolve
  the dependency at install time.
- Handle the missing Python extension gracefully at runtime: log a warning on
  startup and show a user-facing error when SDK discovery is attempted without it.

When `ms-python.python` is installed, behavior is unchanged.

* Complete soft Python extension dependency

Builds on 0313b67 (which removed the manifest dep) so the rest of the
extension behaves gracefully when ms-python.python is absent.

- Run monorepo SDK detection before the Python extension check, so
  .derived/ workflows succeed without the Python extension.
- Surface missing Python extension via the status bar with a click-to-
  marketplace action, replacing the modal error.
- React to the Python extension being installed mid-session via
  vscode.extensions.onDidChange, avoiding a required window reload.
- Install ms-python.python in the pixi vscode-test config; previously
  pulled in transitively via extensionDependencies.
- README: scope the Python extension's role to pixi/wheel SDK
  discovery rather than implying a hard requirement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add SDK path override setting

Addresses a reviewer concern that uninstalling the Python extension (or
being unable to install it) silently breaks SDK discovery. The new
`mojo.sdk.path` setting lets users point the extension at a specific SDK
installation regardless of auto-detection.

- New `mojo.sdk.path` setting (window-scoped) declared with a Markdown
  description covering pixi/conda and wheel layouts.
- Detection priority: override → monorepo → Python extension. When the
  override is set but invalid, the extension surfaces an error instead
  of silently falling back to a different SDK.
- Refactor `createSDKFromWheelEnv` to a primitive-args
  `createSDKFromWheelLayout(sysPrefix, major, minor, kind)` so the same
  logic serves Python-ext envs and override paths. For overrides, the
  Python version is detected by globbing `lib/python*` since the Python
  extension may not be available.
- Status bar: new `invalid-sdk-override` error state; clicking opens the
  setting.
- Replace the dead `SDK.additionalSDKs` configWatcher entry with
  `sdk.path` (the dead key was watched but never declared or read).
- README: priority order, override subsection with examples, scope note.

Author: aahrun

.vscode-test.mjs

@@ -21,5 +21,6 @@ export default defineConfig([
     label: 'pixi',
     workspaceFolder: 'fixtures/pixi-workspace/',
     files: 'out/**/*.test.pixi.js',
+    installExtensions: ['ms-python.python'],
   },
 ]);

README.md

@@ -25,18 +25,52 @@ This VS Code extension from the Modular team adds support for the
 
 ### Mojo SDK resolution
 
-The extension relies on the Python extension for locating your Python
-environment. In some cases, this appears to default to your globally-installed
-environment, even when a virtual environment exists.
-
 When the extension detects a Mojo project (a workspace containing `.mojo`
 files, or with a `.mojo` file open), the SDK status appears in the bottom-left
 status bar, showing details of the detected SDK or a clickable notice if no
 SDK was detected.
 
+The extension resolves the active SDK in this priority order:
+
+1. **`mojo.sdk.path` setting** — if set, the extension uses this path as an
+   explicit override and does not fall back to auto-detection. See below.
+2. **Monorepo SDK** — a `.derived/` directory in the open workspace folder.
+3. **Python environment** — discovered via the [Python extension for VS
+   Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python),
+   used to locate SDKs installed in pixi or wheel-based environments.
+
+If the Python extension is not installed, the status bar will prompt you to
+install it; clicking the prompt opens it in the marketplace.
+
+#### `mojo.sdk.path` override
+
+For environments where the Python extension is unavailable or auto-detection
+picks the wrong environment, you can set `mojo.sdk.path` to point at a Mojo
+SDK directly. The setting is available in both the Settings UI and
+`settings.json`, and can be set at either user or workspace scope — workspace
+scope is usually the right choice, since different projects typically use
+different SDKs.
+
+The value must be an absolute path to an environment root:
+
+- **For pixi or conda installs**, point to the environment root that contains
+  `share/max/modular.cfg` — for example,
+  `/path/to/workspace/.pixi/envs/default`.
+- **For wheel installs**, point to the environment root that contains
+  `bin/mojo` and `lib/python*/site-packages/modular/` — for example,
+  `/path/to/.venv`.
+
+When set, this override beats every other detection source. If the path is
+invalid, the extension will surface an error in the status bar rather than
+silently falling back to auto-detection.
+
+#### Troubleshooting
+
 If the Mojo extension cannot find your SDK installation, try invoking the
-`Python: Select Interpreter` command and selecting your virtual
-environment.
+`Python: Select Interpreter` command and selecting the environment that
+contains your Mojo SDK. In some cases, the Python extension defaults to your
+globally-installed environment even when a workspace-local one exists. If
+that doesn't help, set `mojo.sdk.path` directly.
 
 ## Debugger
 

extension/extension.ts

@@ -86,7 +86,15 @@ Activating the Mojo Extension
       const updateStatusBar = async () => {
         statusBar.showLoading();
         const sdk = await this.pyenvManager!.findActiveSDK();
-        statusBar.update(sdk);
+        let reason;
+        if (!sdk) {
+          if (this.pyenvManager!.getOverridePathState() === 'invalid') {
+            reason = 'invalid-sdk-override' as const;
+          } else if (!this.pyenvManager!.isPythonExtensionAvailable()) {
+            reason = 'no-python-extension' as const;
+          }
+        }
+        statusBar.update(sdk, reason);
       };
 
       this.pushSubscription(
@@ -99,7 +107,7 @@ Activating the Mojo Extension
 
       this.pushSubscription(
         await configWatcher.activate({
-          settings: ['SDK.additionalSDKs'],
+          settings: ['sdk.path'],
         }),
       );
 

extension/pyenv.ts

@@ -15,7 +15,6 @@ import * as vscode from 'vscode';
 import * as ini from 'ini';
 import { DisposableContext } from './utils/disposableContext';
 import { PythonExtension, ResolvedEnvironment } from '@vscode/python-extension';
-import assert from 'assert';
 import { Logger } from './logging';
 import path from 'path';
 import * as util from 'util';
@@ -25,7 +24,8 @@ import {
 } from 'child_process';
 import { Memoize } from 'typescript-memoize';
 import { TelemetryReporter } from './telemetry';
-import { fileExists } from './utils/files';
+import { directoryExists, fileExists } from './utils/files';
+import * as config from './utils/config';
 const execFile = util.promisify(callbackExecFile);
 const exec = util.promisify(callbackExec);
 
@@ -141,6 +141,8 @@ class HomeSDK extends SDK {
   }
 }
 
+export type OverridePathState = 'unset' | 'valid' | 'invalid';
+
 export class PythonEnvironmentManager extends DisposableContext {
   private api: PythonExtension | undefined = undefined;
   private logger: Logger;
@@ -150,6 +152,7 @@ export class PythonEnvironmentManager extends DisposableContext {
   private displayedSDKError: boolean = false;
   private lastLoadedEnv: string | undefined = undefined;
   private activeSDK: SDK | undefined = undefined;
+  private overridePathState: OverridePathState = 'unset';
 
   constructor(logger: Logger, reporter: TelemetryReporter) {
     super();
@@ -160,14 +163,56 @@ export class PythonEnvironmentManager extends DisposableContext {
   }
 
   public async init() {
-    this.api = await PythonExtension.api();
+    await this.tryInitApi();
+    // Watch for the Python extension being installed/enabled mid-session so
+    // we can pick it up without requiring a window reload.
+    this.pushSubscription(
+      vscode.extensions.onDidChange(() => this.handleExtensionChange()),
+    );
+  }
+
+  private async tryInitApi() {
+    if (this.api) {
+      return;
+    }
+    if (!vscode.extensions.getExtension('ms-python.python')) {
+      this.logger.warn(
+        'The Python extension is not installed. ' +
+          'Install the Python extension (ms-python.python) to enable automatic SDK discovery.',
+      );
+      return;
+    }
+    try {
+      this.api = await PythonExtension.api();
+    } catch (e) {
+      this.logger.warn('Failed to load the Python extension API:', e);
+      return;
+    }
     this.pushSubscription(
       this.api.environments.onDidChangeActiveEnvironmentPath((p) =>
         this.handleEnvironmentChange(p.path),
       ),
     );
   }
 
+  private async handleExtensionChange() {
+    if (this.api) {
+      return;
+    }
+    if (!vscode.extensions.getExtension('ms-python.python')) {
+      return;
+    }
+    this.logger.info(
+      'Python extension became available, initializing SDK discovery.',
+    );
+    await this.tryInitApi();
+    if (this.api) {
+      // Reset error gating and notify subscribers so the status bar refreshes.
+      this.displayedSDKError = false;
+      this.envChangeEmitter.fire();
+    }
+  }
+
   private async handleEnvironmentChange(newEnv: string) {
     this.logger.debug(
       `Active environment path change: ${newEnv} (current: ${this.lastLoadedEnv})`,
@@ -179,10 +224,35 @@ export class PythonEnvironmentManager extends DisposableContext {
     }
   }
 
-  /// Finds the active SDK from the currently active Python environment, or undefined if one is not present.
+  /// Whether the Python extension API is available. Used by the status bar to
+  /// distinguish "no SDK found" from "Python extension not installed".
+  public isPythonExtensionAvailable(): boolean {
+    return this.api !== undefined;
+  }
+
+  /// State of the `mojo.sdk.path` override setting. Used by the status bar to
+  /// distinguish "user set an invalid override path" from other failure modes.
+  public getOverridePathState(): OverridePathState {
+    return this.overridePathState;
+  }
+
+  /// Finds the active SDK, in priority order:
+  /// 1. `mojo.sdk.path` override (if set; fails loudly without falling back)
+  /// 2. Monorepo `.derived/` SDK
+  /// 3. SDK from the active Python extension environment
   public async findActiveSDK(): Promise<SDK | undefined> {
-    assert(this.api !== undefined);
-    // Prioritize retrieving a monorepo SDK over querying the environment.
+    // 1. User-supplied override path beats every other source. If it's set
+    // but doesn't resolve, do NOT fall back — that would silently violate
+    // the override semantics. The status bar surfaces the failure instead.
+    const overrideSDK = await this.tryGetOverrideSDK();
+    if (overrideSDK) {
+      return overrideSDK;
+    }
+    if (this.overridePathState === 'invalid') {
+      return undefined;
+    }
+
+    // 2. Monorepo SDK — works without the Python extension.
     const monorepoSDK = await this.tryGetMonorepoSDK();
 
     if (monorepoSDK) {
@@ -192,6 +262,13 @@ export class PythonEnvironmentManager extends DisposableContext {
       return monorepoSDK;
     }
 
+    if (!this.api) {
+      this.logger.warn(
+        'Cannot discover SDK: the Python extension (ms-python.python) is not installed.',
+      );
+      return undefined;
+    }
+
     const envPath = this.api.environments.getActiveEnvironmentPath();
     const env = await this.api.environments.resolveEnvironment(envPath);
     this.logger.info('Loading MAX SDK information from Python environment');
@@ -255,11 +332,28 @@ export class PythonEnvironmentManager extends DisposableContext {
   private async createSDKFromWheelEnv(
     env: ResolvedEnvironment,
   ): Promise<SDK | undefined> {
-    const binPath = path.join(env.executable.sysPrefix, 'bin');
-    const libPath = path.join(
+    return this.createSDKFromWheelLayout(
       env.executable.sysPrefix,
+      env.version!.major,
+      env.version!.minor,
+      SDKKind.Environment,
+    );
+  }
+
+  /// Create an SDK from a wheel-style layout rooted at `sysPrefix`, given
+  /// a known Python major/minor version. Used both for Python-extension envs
+  /// and user-supplied override paths.
+  private async createSDKFromWheelLayout(
+    sysPrefix: string,
+    pythonMajor: number,
+    pythonMinor: number,
+    kind: SDKKind,
+  ): Promise<SDK | undefined> {
+    const binPath = path.join(sysPrefix, 'bin');
+    const libPath = path.join(
+      sysPrefix,
       'lib',
-      `python${env.version!.major}.${env.version!.minor}`,
+      `python${pythonMajor}.${pythonMinor}`,
       'site-packages',
       'modular',
       'lib',
@@ -313,7 +407,7 @@ export class PythonEnvironmentManager extends DisposableContext {
     const versionResult = await exec(`"${mojoPath}" --version`);
     return new SDK(
       this.logger,
-      SDKKind.Environment,
+      kind,
       versionResult.stdout,
       lspPath,
       mblackPath,
@@ -398,6 +492,86 @@ export class PythonEnvironmentManager extends DisposableContext {
     }
   }
 
+  /// Attempt to load an SDK from the user-supplied `mojo.sdk.path` setting.
+  /// Updates `overridePathState` as a side effect so callers can distinguish
+  /// "no override set" from "override set but unusable".
+  private async tryGetOverrideSDK(): Promise<SDK | undefined> {
+    const overridePath = config.get<string>('sdk.path', undefined)?.trim();
+    if (!overridePath) {
+      this.overridePathState = 'unset';
+      return undefined;
+    }
+
+    this.logger.info(`Loading SDK from override path: ${overridePath}`);
+
+    if (!(await directoryExists(overridePath))) {
+      this.logger.error(
+        `Override path '${overridePath}' does not exist or is not a directory.`,
+      );
+      this.overridePathState = 'invalid';
+      return undefined;
+    }
+
+    // Try the conda/pixi layout first: <override>/share/max/modular.cfg
+    const homePath = path.join(overridePath, 'share', 'max');
+    if (await fileExists(path.join(homePath, 'modular.cfg'))) {
+      const sdk = await this.createSDKFromHomePath(
+        SDKKind.Custom,
+        homePath,
+        overridePath,
+      );
+      this.overridePathState = sdk ? 'valid' : 'invalid';
+      return sdk;
+    }
+
+    // Fall back to the wheel layout: <override>/lib/python<X>.<Y>/site-packages/modular/...
+    const pythonVersion = await this.detectPythonVersion(overridePath);
+    if (pythonVersion) {
+      const [major, minor] = pythonVersion;
+      const sdk = await this.createSDKFromWheelLayout(
+        overridePath,
+        major,
+        minor,
+        SDKKind.Custom,
+      );
+      this.overridePathState = sdk ? 'valid' : 'invalid';
+      return sdk;
+    }
+
+    this.logger.error(
+      `Override path '${overridePath}' contains neither a 'share/max/modular.cfg' file nor a 'lib/python*' directory.`,
+    );
+    this.overridePathState = 'invalid';
+    return undefined;
+  }
+
+  /// Find a `python<major>.<minor>` directory under `<root>/lib`, returning
+  /// the parsed version. Used to resolve wheel-style install paths whose
+  /// Python version we can't query directly (no Python extension available).
+  private async detectPythonVersion(
+    root: string,
+  ): Promise<[number, number] | undefined> {
+    const libDir = path.join(root, 'lib');
+    let entries: [string, vscode.FileType][];
+    try {
+      entries = await vscode.workspace.fs.readDirectory(
+        vscode.Uri.file(libDir),
+      );
+    } catch {
+      return undefined;
+    }
+    for (const [name, type] of entries) {
+      if (!(type & vscode.FileType.Directory)) {
+        continue;
+      }
+      const match = name.match(/^python(\d+)\.(\d+)$/);
+      if (match) {
+        return [parseInt(match[1], 10), parseInt(match[2], 10)];
+      }
+    }
+    return undefined;
+  }
+
   /// Attempt to load a monorepo SDK from the currently open workspace folder.
   /// Resolves with the loaded SDK, or undefined if one doesn't exist.
   private async tryGetMonorepoSDK(): Promise<SDK | undefined> {