|
| 1 | +# Specification: DepsAnalyser (Dependency Analyzer) |
| 2 | + |
| 3 | +## Overview |
| 4 | +The `DepsAnalyser` is a core module of PyCompiler ARK responsible for identifying all third-party dependencies required by a Python project. It uses a hybrid approach combining configuration file scanning and static code analysis to ensure a complete and accurate `requirements.txt` generation. |
| 5 | + |
| 6 | +## Core Objectives |
| 7 | +- **Precision**: Differentiate between standard library, internal project modules, and third-party dependencies. |
| 8 | +- **Robustness**: Handle various project layouts (standard, `src/`, etc.) and modern Python syntax. |
| 9 | +- **Transparency**: "No Magic" philosophy — explicit classification and predictable output. |
| 10 | +- **Performance**: Use caching and optimized scanning to minimize analysis time. |
| 11 | + |
| 12 | +## 1. Discovery & Analysis Strategy |
| 13 | + |
| 14 | +The analyzer follows a multi-step priority-based discovery process: |
| 15 | + |
| 16 | +### A. Configuration File Scanning (High Priority) |
| 17 | +The analyzer first looks for explicit dependency declarations. This is considered the "ground truth" and often allows skipping the expensive code scan. |
| 18 | +- **`requirements.txt` / `requirements.in`**: Parsed line-by-line, ignoring comments and flags. |
| 19 | +- **`pyproject.toml`**: |
| 20 | + - **PEP 621**: `[project.dependencies]` |
| 21 | + - **Poetry**: `[tool.poetry.dependencies]` and `[tool.poetry.group.dev.dependencies]` |
| 22 | +- **`setup.py`**: Static analysis via regex to match `install_requires=[...]` blocks without executing the script. |
| 23 | +- **`Pipfile`**: Parsed to extract package names from the `[packages]` section. |
| 24 | + |
| 25 | +### B. Static Code Analysis (Deep Scanning) |
| 26 | +When no configuration is found, or if full validation is required, every `.py` file (excluding ignored directories) is analyzed. |
| 27 | + |
| 28 | +#### 1. AST Parsing |
| 29 | +The primary method uses Python's `ast` module to walk the Abstract Syntax Tree. |
| 30 | +- `ast.Import`: Extracts modules from `import x, y as z`. |
| 31 | +- `ast.ImportFrom`: Extracts modules from `from x.y import z`. Handles relative imports by resolving the `level` attribute against the file's position in the workspace. |
| 32 | + |
| 33 | +#### 2. Regex Fallback (The "Summit" Robustness) |
| 34 | +If `ast.parse` fails (e.g., syntax error, or file uses Python 3.14 features while analyzer runs on 3.13), a regex engine takes over: |
| 35 | +- **Pattern `import`**: `^\s*import\s+([\w\.,\s]+)` (Handles comma-separated imports). |
| 36 | +- **Pattern `from`**: `^\s*from\s+([\w\.]+)\s+import` (Extracts the base module). |
| 37 | +- This ensures that a single "broken" or "too new" file doesn't stop the entire dependency discovery process. |
| 38 | + |
| 39 | +#### 3. Dynamic Import Detection |
| 40 | +Specific regex patterns detect: |
| 41 | +- `__import__('module')` |
| 42 | +- `importlib.import_module('module')` |
| 43 | + |
| 44 | +## 2. The Classification Engine |
| 45 | +Every detected string is normalized and passed through the classifier: |
| 46 | + |
| 47 | +| Category | Resolution Logic | Action | |
| 48 | +| :--- | :--- | :--- | |
| 49 | +| **stdlib** | Cross-references `sys.builtin_module_names`, `sysconfig.get_path("stdlib")`, and `importlib.util.find_spec`. | Ignored | |
| 50 | +| **internal** | Resolved if the module root contains an `__init__.py` within the workspace or matches a source root (`src/`, `lib/`). | Ignored | |
| 51 | +| **third_party**| Resolved if found in `site-packages` or if it has a valid distribution metadata via `importlib.metadata`. | Added to Requirements | |
| 52 | +| **unknown** | Fallback for modules that exist in code but aren't installed or local. | Added to Requirements | |
| 53 | + |
| 54 | +### Performance Optimization: LRU Cache |
| 55 | +The `_is_stdlib_module` function uses an `@functools.lru_cache(maxsize=256)` to avoid repeated expensive filesystem and metadata lookups for common modules like `os`, `sys`, or `json`. |
| 56 | + |
| 57 | +## 3. Advanced Exclusion Logic |
| 58 | + |
| 59 | +### Magic & Runtime Modules |
| 60 | +Strict exclusion of modules that are part of the Python runtime environment but should never be in a `requirements.txt`: |
| 61 | +- `__future__`: Compiler directives. |
| 62 | +- `__main__`: The entry point script name. |
| 63 | +- `__builtins__`: Implicitly available objects. |
| 64 | + |
| 65 | +### Directory White-listing |
| 66 | +The analyzer automatically skips: |
| 67 | +- Virtual environments: `.venv`, `venv`, `.env`, `env`. |
| 68 | +- Version control: `.git`. |
| 69 | +- Build artifacts: `build`, `dist`, `__pycache__`. |
| 70 | +- Test suites (optional): `tests`, `test`. |
| 71 | + |
| 72 | +## 4. Workspace Layout Support |
| 73 | +The analyzer is "layout-aware": |
| 74 | +1. **Flat Layout**: Packages at the root. |
| 75 | +2. **Src Layout**: Detects `src/`, `lib/`, or `python/` as root aliases. |
| 76 | +3. **Internal Module Tracking**: Before classifying, it builds a map of all local packages by looking for directories containing `__init__.py`. This prevents a local package named `utils` from being confused with a hypothetical third-party `utils` package. |
| 77 | + |
| 78 | +## 5. Requirements Generation |
| 79 | +The `write_requirements_txt` function: |
| 80 | +- **Normalizes**: Converts all module names to lower-case and replaces `_` with `-` where appropriate for PyPI compatibility. |
| 81 | +- **Sorts**: Alphabetical sorting for clean diffs. |
| 82 | +- **De-duplicates**: Ensures no package is listed twice. |
| 83 | +- **Comments**: Adds a header identifying the file as "Auto-generated by PyCompiler ARK". |
0 commit comments