Status: Phases 1-4 implemented (2026-05-06)
All four phases have been implemented with passing tests. Documentation updates (Phase 5/6) remain pending.
| Phase | Status | Tests |
|---|---|---|
1. ModulesInterface |
Done | 19/19 pass (test_modules_interface.py) |
2. PackageModule source type |
Done | 14/14 pass (test_package_module.py) |
3. ModuleContentType |
Done | 7/7 pass (added to test_pkg_content_type.py) |
4. PackageHandlerModules |
Done | 6/6 pass (test_modules_handler.py) |
| Test fixtures | Done | module_leaf1/, module_leaf2/, module_no_ivpm/ |
| Documentation | Pending |
- Handler module specifier fallback: When
type: moduleis used on a non-module source (e.g.src: dir), the handler usespkg.nameas the module specifier fallback, sinceModuleTypeData.moduleis only populated byPackageModule.process_options(). - Site-packages sync: The development environment has a stale copy of
ivpm in site-packages alongside the editable
.pthinstall. New files were symlinked and modified files were copied to site-packages to enable handler entry point discovery. A cleanpip install -e .would eliminate this need. - Entry point registration: The
moduleshandler entry point was added to bothpyproject.tomland the installedentry_points.txtfor immediate test availability.
This plan covers the full implementation of Environment Modules support as
described in design-modules-support.md. The work is split into four
phases:
ModulesInterface— variant-aware subprocess wrapper for querying the module system.PackageModulesource type (src: module) — resolves a module specifier to a root directory and setspkg.path.ModuleContentType(type: module) — validateswith:parameters and signals the handler.PackageHandlerModuleshandler — collects module-typed packages and generatesmodules.envrc/ patchespackages.envrc.
Each phase is self-contained and testable on its own.
src/ivpm/modules_interface.py
Extracted from the variant-aware wrapper in modules_from_python.md
(approach 3/4). This is the single place where IVPM interacts with
modulecmd.tcl / lmod subprocesses.
import enum
import dataclasses as dc
from typing import Optional, Tuple
class ModulesVariant(enum.Enum):
MODULES_3X_TCL = "modules-3x"
MODULES_4X = "modules-4x"
LMOD = "lmod"
UNKNOWN = "unknown"
class ModulesError(Exception):
"""Raised when a modulecmd subprocess fails or returns unexpected output."""
pass
@dc.dataclass
class ModulesInterface:
variant: ModulesVariant = ModulesVariant.UNKNOWN
cmd_path: Optional[str] = None # path to modulecmd.tcl or lmod
tclsh_path: Optional[str] = None # path to tclsh (Modules 3.x/4.x)A module-level function detect_variant() that probes the environment for
the modules installation:
- Check
LMOD_CMDenv var →ModulesVariant.LMOD. - Check
MODULESHOMEenv var → look formodulecmd.tclinside it.- If
modulecmd.tclsupports--versionwith 4.x+ output →MODULES_4X. - Otherwise →
MODULES_3X_TCL.
- If
- Fall back to
shutil.which("modulecmd")orshutil.which("modulecmd.tcl"). - Return
(ModulesVariant, cmd_path, tclsh_path).
All methods call subprocess.run() and parse stdout/stderr. IVPM never
calls exec() or evaluates modulecmd-generated Python code.
| Method | Purpose | Used by |
|---|---|---|
is_avail(module: str) -> bool |
Check module availability | PackageModule.update(), PackageModule.status() |
module_path(module: str) -> Optional[str] |
Return the absolute path to the modulefile | PackageModule.update() |
module_show(module: str) -> str |
Return raw module show output (stderr) |
PackageModule.update() (opt-in resolve_root) |
avail(pattern: str) -> List[str] |
List available modules matching a pattern | Future ivpm search --modules |
Each method raises ModulesError (a new exception class defined in the
same file) on subprocess failure, with the stderr output included in the
message.
Per the variant table in modules_from_python.md:
| Variant | module_path() implementation |
|---|---|
| Modules 4.x | modulecmd.tcl python path <module> → parse stdout |
| Modules 3.x | tclsh modulecmd.tcl python path <module> → parse stdout, handle temp-file quirk |
| Lmod | $LMOD_CMD python show <module> → parse stderr for filename line |
The ModulesInterface is stored on ProjectUpdateInfo as
modules_interface: Optional[ModulesInterface] (see Phase 4, §4.6).
It is created lazily on first access: PackageModule.update() calls
_get_modules_interface(update_info) which runs detect_variant() once
and caches the result.
When handler_configs["modules"] contains variant and/or modulecmd
keys, those override auto-detection. This lets sites with non-standard
installations configure the path explicitly.
src/ivpm/pkg_types/package_module.py
Extends Package directly (like PackageFuseSoC and PackagePyPi).
@dc.dataclass
class PackageModule(Package):
module: str = None # e.g. "gcc/15.2.0"
module_root: str = None # resolved root directory
modulefile_path: str = None # absolute path to the modulefile
root_override: str = None # explicit root: from YAML
resolve_root: bool = False # opt-in module-show parsingdef process_options(self, opts, si):
super().process_options(opts, si)
self.src_type = "module"
if "module" not in opts:
fatal("src: module requires a 'module:' specifier")
self.module = opts["module"]
if "root" in opts:
self.root_override = opts["root"]
if "resolve-root" in opts:
self.resolve_root = bool(opts["resolve-root"])Also implicitly adds ModuleTypeData to pkg.type_data when no explicit
type: is specified — mirroring how src: pypi implicitly gets
PythonTypeData:
# Implicit type assignment (unless user specified type: explicitly)
if not any(td.type_name == "module" for td in self.type_data):
from ..pkg_content_type import ModuleTypeData
td = ModuleTypeData()
td.type_name = "module"
td.module = self.module
self.type_data.append(td)@staticmethod
def create(name, opts, si) -> 'PackageModule':
pkg = PackageModule(name)
pkg.process_options(opts, si)
return pkg
@classmethod
def source_info(cls):
from ..show.info_types import PkgSourceInfo
return PkgSourceInfo(
name="module",
description="Environment Modules (module load) — resolves a module "
"specifier to the modulefile directory on disk",
origin="built-in",
)def update(self, update_info: ProjectUpdateInfo) -> 'ProjInfo':
update_info.report_package(cacheable=False)
mi = _get_modules_interface(update_info)Step 1 — Locate the modulefile:
mf_path = mi.module_path(self.module)
if mf_path is None:
fatal("Module '%s' is not available (module_path returned None). "
"Check MODULEPATH and module availability." % self.module)
self.modulefile_path = mf_pathStep 2 — Determine the root directory (priority order):
if self.root_override:
root = os.path.expandvars(self.root_override)
elif self.resolve_root:
root = self._resolve_root_from_show(mi)
else:
# Default: modulefile directory
root = os.path.dirname(mf_path) if os.path.isfile(mf_path) else mf_pathThe _resolve_root_from_show() helper parses module show output for
setenv *_HOME, prepend-path PATH, or set root directives. It is
only called when resolve_root: true is explicitly set.
Step 3 — Set pkg.path:
Note: PackageUpdater._resolve_pkg() sets pkg.path to
os.path.join(deps_dir, pkg.name) before calling update().
PackageModule.update() must override this default:
self.path = root
self.module_root = rootStep 4 — Load sub-project info:
return ProjInfo.mkFromProj(root)If the root directory contains an ivpm.yaml, its dep-sets, self-types,
handler_configs, and skills declarations are loaded and participate in
dependency resolution — exactly like a git or dir package.
def sync(self, sync_info) -> 'PkgSyncResult':
from ..pkg_sync import PkgSyncResult, SyncOutcome
return PkgSyncResult(
name=self.name,
outcome=SyncOutcome.SKIPPED,
reason="environment module (externally managed)",
)status() reports the module name, resolved root, and whether the module
is still available via mi.is_avail().
In src/ivpm/pkg_types/pkg_type_rgy.py, add to _load():
from .package_module import PackageModule
self.register("module", PackageModule.create, PackageModule.source_info())In src/ivpm/schema/ivpm.json, add "module" to the src field's
oneOf array:
{
"const": "module",
"title": "Environment Module — resolves a module specifier to its modulefile directory"
}Also add "module", "root", and "resolve-root" to the package-dep
properties:
"module": {
"type": "string",
"title": "Module specifier (e.g. gcc/15.2.0). Required when src is 'module'"
},
"root": {
"type": "string",
"title": "Explicit root directory override for src: module"
},
"resolve-root": {
"type": "boolean",
"title": "Opt-in: parse module show output to determine the install prefix",
"default": false
}In src/ivpm/package_lock.py, add a "module" branch to
_entry_from_pkg():
elif src == "module":
entry["module"] = getattr(pkg, "module", None)
entry["modulefile"] = getattr(pkg, "modulefile_path", None)
entry["root"] = getattr(pkg, "module_root", None)And a corresponding branch in _spec_matches_lock():
elif src == "module":
return getattr(pkg, "module", None) == lock_entry.get("module")And in IvpmLockReader.build_packages_info(), add reconstruction:
elif src == "module":
p = PackageModule(name)
p.module = entry.get("module")
p.modulefile_path = entry.get("modulefile")
p.module_root = entry.get("root")
p.path = entry.get("root")
p.src_type = "module"
pkg = psrc/ivpm/pkg_content_type.py — add alongside PythonContentType
and RawContentType:
@dc.dataclass
class ModuleTypeData(TypeData):
load: bool = True # emit 'module load' into envrc
module: str = None # module specifier (copied from PackageModule.module)
class ModuleContentType(PkgContentType):
@property
def name(self) -> str:
return "module"
def create_data(self, with_opts: dict, si) -> ModuleTypeData:
known = {"load"}
for k in with_opts:
if k not in known:
fatal("type 'module' does not accept parameter '%s' "
"(known: %s)" % (k, ", ".join(sorted(known))))
data = ModuleTypeData()
if "load" in with_opts:
data.load = bool(with_opts["load"])
data.type_name = self.name
return dataIn src/ivpm/pkg_content_type_rgy.py, add to _load():
from .pkg_content_type import ModuleContentType
self.register(ModuleContentType())Add "module" to the type field's oneOf array in
src/ivpm/schema/ivpm.json:
{
"const": "module",
"title": "Environment Module — emit module load and/or resolve root directory for handler discovery"
}src/ivpm/handlers/package_handler_modules.py
Follows the structure of PackageHandlerFuseSoC (stateful handler with
leaf discovery, root aggregation, sentinel-based packages.envrc
patching, and output file generation).
@dc.dataclass
class PackageHandlerModules(PackageHandler):
name = "modules"
description = "Generates module load statements into modules.envrc"
leaf_when = None # inspect every package
root_when = None # always run on_root_post_load
phase = 1 # after direnv (0), before python (5)
conditions_summary = "Active when any package carries ModuleTypeData" module_pkgs: Dict[str, str] = dc.field(default_factory=dict)
# Maps pkg_name -> module specifier (e.g. "gcc/15.2.0")For each package, check if it carries ModuleTypeData with load: true:
def on_leaf_post_load(self, pkg: Package, update_info):
from ..package import get_type_data
from ..pkg_content_type import ModuleTypeData
td = get_type_data(pkg, ModuleTypeData)
if td is None:
return
if not td.load:
return
module_spec = td.module or getattr(pkg, 'module', None)
if module_spec is None:
return
with self._lock:
self.module_pkgs[pkg.name] = module_spec-
Generate
packages/modules.envrc:def on_root_post_load(self, update_info): deps_dir = update_info.deps_dir if not self.module_pkgs: self._cleanup(deps_dir) return envrc_path = os.path.join(deps_dir, "modules.envrc") with open(envrc_path, "w") as fp: fp.write("# Generated by IVPM modules handler -- do not edit manually\n") for pkg_name in sorted(self.module_pkgs.keys()): fp.write("module load %s\n" % self.module_pkgs[pkg_name])
Ordering: topological by dependency, then alphabetical (same strategy as the direnv handler). The sorted fallback above is simplified; the real implementation will use the topological ordering available from
update_info. -
Patch
packages/packages.envrcusing the sentinel pattern from the Python and FuseSoC handlers:_MODULES_SENTINEL_BEGIN = "# --- ivpm:modules begin ---" _MODULES_SENTINEL_END = "# --- ivpm:modules end ---"
Insert before the python handler's section (phase 1 < phase 5). If
packages.envrcdoes not exist, skip (direnv handler not active). -
Cleanup: If no module packages remain, remove the sentinel section from
packages.envrcand deletemodules.envrc.
In src/ivpm/project_ops_info.py, add to ProjectUpdateInfo:
modules_interface: Optional['ModulesInterface'] = NoneThis field is lazily populated by PackageModule.update() on first use
and shared across all module packages and the handler.
def get_lock_entries(self, deps_dir: str) -> dict:
if not self.module_pkgs:
return {}
entries = {}
for pkg_name, module_spec in sorted(self.module_pkgs.items()):
entries[pkg_name] = {"module": module_spec}
return {"modules": entries}The handler reads update_info.handler_configs.get("modules", {}) in
on_root_post_load() to obtain user-specified configuration:
package:
name: my-project
with:
modules:
variant: auto # auto | modules-4x | modules-3x | lmod
modulecmd: /path/to/modulecmd.tclThese values are forwarded to ModulesInterface.detect_variant() as
overrides.
In pyproject.toml, add:
[project.entry-points."ivpm.handlers"]
modules = "ivpm.handlers.package_handler_modules:PackageHandlerModules"Create test data in test/unit/data/:
module_leaf1/
ivpm.yaml # name: module_leaf1, declares sub-deps or skills
SKILL.md # optional: verifies handler discovery through pkg.path
module_leaf2/
ivpm.yaml # name: module_leaf2, no sub-deps
module_no_ivpm/
README.md # no ivpm.yaml — verifies ProjInfo.mkFromProj returns None
Additionally, tests that exercise the full PackageModule.update() path
require a mock ModulesInterface (see §5.3) because real modulecmd.tcl
is not available in CI.
File: test/unit/test_modules_interface.py
These tests verify the parsing and detection logic. They mock
subprocess.run to return canned outputs for each variant.
| Test | What it verifies |
|---|---|
test_detect_variant_modules_4x |
MODULESHOME set, modulecmd.tcl --version returns 4.x → MODULES_4X |
test_detect_variant_lmod |
LMOD_CMD set → LMOD |
test_detect_variant_modules_3x |
MODULESHOME set, no 4.x version string → MODULES_3X_TCL |
test_detect_variant_none |
No env vars, no which result → UNKNOWN, methods raise ModulesError |
test_module_path_4x |
Parses modulecmd.tcl python path stdout → correct path |
test_module_path_lmod |
Parses lmod python show stderr → extracts modulefile path |
test_is_avail_true |
Module available → True |
test_is_avail_false |
Module not available → False |
test_module_show_output |
Returns raw stderr for further parsing |
test_explicit_override |
variant and modulecmd config override auto-detection |
test_module_path_not_found |
Module not in MODULEPATH → returns None |
File: test/unit/test_package_module.py
These tests use a FakeModulesInterface stub that returns canned
modulefile paths without calling real subprocesses. The stub is injected
via update_info.modules_interface.
| Test | What it verifies |
|---|---|
test_create_from_options |
PackageModule.create() with module: gcc/15.2.0 → correct fields set |
test_missing_module_field |
create() without module: → fatal() |
test_update_sets_path_to_modulefile_dir |
update() sets pkg.path to the modulefile's parent directory (default) |
test_update_with_root_override |
root: /custom/path → pkg.path set to override, not modulefile dir |
test_update_with_resolve_root |
resolve-root: true → _resolve_root_from_show() called, result used |
test_update_loads_proj_info |
Root dir contains ivpm.yaml → ProjInfo loaded and returned |
test_update_no_ivpm_yaml |
Root dir has no ivpm.yaml → returns None (no error) |
test_update_module_not_available |
module_path() returns None → fatal() with clear message |
test_sync_returns_skipped |
sync() returns SKIPPED with "environment module" reason |
test_implicit_module_type_data |
No explicit type: → ModuleTypeData auto-added to type_data |
test_explicit_type_raw_no_module_data |
type: raw → no ModuleTypeData in type_data |
test_src_type_is_module |
pkg.src_type is "module" after construction |
test_root_override_env_expansion |
root: $TOOL_ROOT/gcc → env var expanded |
File: test/unit/test_pkg_content_type.py (extend existing file)
| Test | What it verifies |
|---|---|
test_module_type_registered |
PkgContentTypeRgy.inst().has("module") is True |
test_module_type_create_data_defaults |
create_data({}) → ModuleTypeData(load=True) |
test_module_type_create_data_load_false |
create_data({"load": False}) → ModuleTypeData(load=False) |
test_module_type_unknown_param |
create_data({"foo": 1}) → fatal() |
File: test/unit/test_modules_handler.py
Uses the TestBase integration pattern with self.mkFile() and
self.ivpm_update(skip_venv=True). Module packages are simulated using
src: dir with type: module to avoid needing a real module system.
| Test | What it verifies |
|---|---|
test_modules_envrc_generated |
Single module dep → packages/modules.envrc created with module load statement |
test_modules_envrc_multiple |
Multiple module deps → all module load statements present, sorted |
test_no_module_deps_no_output |
No module-typed deps → modules.envrc not created |
| Test | What it verifies |
|---|---|
test_appends_to_packages_envrc |
Direnv handler active → packages.envrc contains sentinel-wrapped modules section |
test_sentinel_replaces_old_section |
Second ivpm update → old sentinel section replaced, not duplicated |
test_no_packages_envrc_no_append |
packages.envrc does not exist → handler does not create it |
test_modules_before_python |
Sentinel section appears before python handler's section (phase 1 < 5) |
| Test | What it verifies |
|---|---|
test_load_false_no_module_load |
type: { module: { load: false } } → no module load in envrc |
test_load_false_still_sets_path |
load: false dep → pkg.path still resolved for handler discovery |
| Test | What it verifies |
|---|---|
test_stale_modules_envrc_removed |
Second run with no module deps → modules.envrc deleted, sentinel removed |
test_idempotent_second_run |
Second run with same deps → outputs unchanged |
| Test | What it verifies |
|---|---|
test_lock_entries_recorded |
Module packages appear under "modules" key in package-lock.json |
test_lock_empty_when_no_modules |
No module deps → no "modules" key in lock file |
| Test | What it verifies |
|---|---|
test_agents_finds_skills_in_module_root |
Module root contains SKILL.md → agents handler discovers it |
test_fusesoc_finds_cores_in_module_root |
Module root contains .core files → fusesoc handler discovers them |
These tests require a real module system and are expected to run on developer workstations or EDA-configured CI hosts, not in unit-test CI:
| Test | What it verifies |
|---|---|
test_real_module_resolves |
src: module, module: <available_module> → ivpm update succeeds, pkg.path exists |
test_real_module_envrc_loadable |
Generated modules.envrc can be sourced in a shell without errors |
test_real_module_not_available |
Non-existent module specifier → clear error message |
Mark these tests with @unittest.skipUnless(os.environ.get("IVPM_TEST_MODULES"), "requires module system")
so they are skipped by default.
- Module-level docstring in each new file describing its role in the modules pipeline.
- Docstring on each public method following existing codebase conventions.
handler_info()classmethod onPackageHandlerModulesreturningHandlerInfoforivpm show handler modules.source_info()classmethod onPackageModulereturningPkgSourceInfoforivpm show source module.
Add a module source type section describing:
- YAML syntax with examples (minimal, with
root:override, withresolve-root: true). - Resolution strategy (modulefile directory as default root).
- Relationship to
type: moduleand the handler. - Limitations (read-only roots, no
packages/representation).
Add a modules handler section describing:
- Purpose and use case (EDA/HPC tool management).
- Output artifacts (
modules.envrc, sentinel inpackages.envrc). - Configuration keys (
variant,modulecmd). - Phase ordering relative to direnv and python handlers.
Add a section on Environment Modules integration covering:
- Supported variants (Modules 3.x, 4.x, Lmod).
- How to configure
MODULEPATHandMODULESHOMEfor IVPM. - Interaction with direnv (
module loadin envrc).
Add module to the source type and content type tables.
After implementation, replace design sketches with references to actual files. Mark open issues as resolved where applicable.
| File | Action | Phase |
|---|---|---|
src/ivpm/modules_interface.py |
Create | 1 |
src/ivpm/pkg_types/package_module.py |
Create | 2 |
src/ivpm/pkg_types/pkg_type_rgy.py |
Edit — register "module" source type |
2 |
src/ivpm/schema/ivpm.json |
Edit — add module to src enum, add module/root/resolve-root dep properties |
2 |
src/ivpm/package_lock.py |
Edit — add module branches to _entry_from_pkg, _spec_matches_lock, IvpmLockReader |
2 |
src/ivpm/pkg_content_type.py |
Edit — add ModuleTypeData and ModuleContentType classes |
3 |
src/ivpm/pkg_content_type_rgy.py |
Edit — register ModuleContentType |
3 |
src/ivpm/handlers/package_handler_modules.py |
Create | 4 |
src/ivpm/project_ops_info.py |
Edit — add modules_interface field |
4 |
pyproject.toml |
Edit — add modules handler entry point |
4 |
test/unit/test_modules_interface.py |
Create | 1 |
test/unit/test_package_module.py |
Create | 2 |
test/unit/test_pkg_content_type.py |
Edit — add module content type tests | 3 |
test/unit/test_modules_handler.py |
Create | 4 |
test/unit/data/module_leaf1/ivpm.yaml |
Create | 2 |
test/unit/data/module_leaf1/SKILL.md |
Create | 2 |
test/unit/data/module_leaf2/ivpm.yaml |
Create | 2 |
test/unit/data/module_no_ivpm/README.md |
Create | 2 |
docs/source/package_types.rst |
Edit — add module source type |
docs |
docs/source/handlers.rst |
Edit — add modules handler |
docs |
docs/source/integrations.rst |
Edit — add modules integration section | docs |
docs/packages_and_types.md |
Edit — add module to tables |
docs |
The following files do not need modification because PackageModule
is indistinguishable from any other resolved package once pkg.path is
set:
src/ivpm/ivpm_yaml_reader.py—"module"is handled by the registriessrc/ivpm/package_updater.py— callspkg.update()genericallysrc/ivpm/project_ops.py— unchangedsrc/ivpm/project_sync.py— unchangedsrc/ivpm/handlers/handler_conditions.py—HasType("module")andHasSourceType("module")work with existing infrastructuresrc/ivpm/handlers/package_handler_direnv.py— unchangedsrc/ivpm/handlers/package_handler_python.py— unchangedsrc/ivpm/handlers/package_handler_agents.py— unchangedsrc/ivpm/handlers/package_handler_fusesoc.py— unchanged- All
cmds/files — unchanged
Week 1: Core infrastructure
Day 1 — modules_interface.py + test_modules_interface.py
Day 2 — package_module.py skeleton + process_options + create
Day 3 — package_module.py update() + resolution logic
Day 4 — pkg_type_rgy.py registration + schema update
Day 5 — test_package_module.py (all unit tests with FakeModulesInterface)
Week 2: Content type + handler
Day 1 — ModuleContentType + ModuleTypeData in pkg_content_type.py
Day 2 — pkg_content_type_rgy.py registration + test_pkg_content_type.py additions
Day 3 — package_handler_modules.py skeleton + leaf phase
Day 4 — package_handler_modules.py root phase (envrc generation, sentinel patching)
Day 5 — test_modules_handler.py (handler tests)
Week 3: Integration + documentation
Day 1 — package_lock.py module branches + lock file tests
Day 2 — project_ops_info.py + pyproject.toml entry point
Day 3 — End-to-end integration testing on module-equipped host
Day 4 — Documentation updates (package_types.rst, handlers.rst, integrations.rst)
Day 5 — Update design-modules-support.md + code review