Skip to content

Commit 5d9a88c

Browse files
houman1359houman1359
authored
Improve documentation. (#134)
## Description <!-- Brief description of the changes --> ## Type of Change - [ ] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation ## Testing - [ ] Tests pass locally - [ ] New tests added (if applicable) ## Related Issues Closes #
2 parents 14b9633 + 16f4395 commit 5d9a88c

25 files changed

Lines changed: 964 additions & 2492 deletions

‎README.md‎

Lines changed: 72 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -2,53 +2,58 @@
22

33
Node and channel metrics for neural network interpretability, importance, and interventions.
44

5-
[![Tests](https://github.com/KempnerInstitute/nodelens/actions/workflows/test.yml/badge.svg)](https://github.com/KempnerInstitute/nodelens/actions/workflows/test.yml)
6-
[![Lint](https://github.com/KempnerInstitute/nodelens/actions/workflows/lint.yml/badge.svg)](https://github.com/KempnerInstitute/nodelens/actions/workflows/lint.yml)
7-
[![Documentation](https://github.com/KempnerInstitute/nodelens/actions/workflows/docs.yml/badge.svg)](https://github.com/KempnerInstitute/nodelens/actions/workflows/docs.yml)
8-
[![Release](https://github.com/KempnerInstitute/nodelens/actions/workflows/release.yml/badge.svg)](https://github.com/KempnerInstitute/nodelens/actions/workflows/release.yml)
5+
[![Tests](https://github.com/KempnerInstitute/NodeLens/actions/workflows/test.yml/badge.svg)](https://github.com/KempnerInstitute/NodeLens/actions/workflows/test.yml)
6+
[![Lint](https://github.com/KempnerInstitute/NodeLens/actions/workflows/lint.yml/badge.svg)](https://github.com/KempnerInstitute/NodeLens/actions/workflows/lint.yml)
7+
[![Documentation](https://github.com/KempnerInstitute/NodeLens/actions/workflows/docs.yml/badge.svg)](https://github.com/KempnerInstitute/NodeLens/actions/workflows/docs.yml)
98
[![Python](https://img.shields.io/badge/python-%3E%3D3.8-3776AB?logo=python&logoColor=white)](pyproject.toml)
10-
[![Artifacts](https://img.shields.io/badge/Hugging%20Face-artifacts-ffcc33)](https://huggingface.co/datasets/hsafaai/supernodes-scar-artifacts)
119
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
1210

1311
NodeLens is a research codebase for studying which channels, neurons, and
14-
features matter most for model behavior. The Python package is imported as
15-
`nodelens`.
16-
17-
The repository supports two related workflows:
18-
19-
- General metric analysis for vision models, transformers, and LLMs.
20-
- Paper-specific releases under `projects/`, including the Supernodes and SCAR
21-
artifact workflow.
12+
features matter most for model behavior. It combines activation capture,
13+
importance metrics, redundancy and information measures, structured
14+
interventions, and report generation in one configuration-driven workflow. The
15+
Python package is imported as `nodelens`.
2216

2317
## What The Code Does
2418

25-
```mermaid
26-
flowchart LR
27-
A[Model + calibration data] --> B[Capture activations and gradients]
28-
B --> C[Compute channel metrics]
29-
C --> D[Identify loss-critical cores]
30-
C --> E[Estimate redundancy and halo structure]
31-
D --> F[Structured pruning and ablation probes]
32-
E --> F
33-
F --> G[Figures, tables, manifests, HF artifacts]
19+
```text
20+
Model + data
21+
|
22+
v
23+
Activation and gradient capture
24+
|
25+
v
26+
Channel and node metrics
27+
|-- activation statistics
28+
|-- Rayleigh quotient and spectral alignment
29+
|-- mutual information, redundancy, and synergy
30+
|-- gradients, curvature, Taylor scores, and loss proxies
31+
|
32+
v
33+
Analysis and interventions
34+
|-- identify outliers or loss-critical cores
35+
|-- cluster channels by metric profile
36+
|-- test ablations, pruning, and sensitivity probes
37+
|-- generate figures, tables, summaries, and manifests
3438
```
3539

3640
Core capabilities:
3741

38-
- Loss-sensitive channel scoring, including SCAR loss-proxy metrics.
39-
- Activation, curvature, Taylor, Rayleigh quotient, and information-theoretic metrics.
40-
- Structured pruning strategies for channel-level model analysis.
41-
- Cluster and halo-style analyses for local redundancy structure.
42-
- Reproducible project folders for paper artifacts and public releases.
43-
44-
Supported model families include MLPs, CNNs, transformer language models, and
45-
LLM backends through Hugging Face causal language models.
42+
- Metric analysis for MLPs, CNNs, transformers, and Hugging Face causal LMs.
43+
- Node and channel scoring with activation, alignment, information,
44+
redundancy, gradient, curvature, and loss-sensitive metrics.
45+
- Structured pruning and ablation tools for testing whether high-scoring
46+
channels are functionally important.
47+
- Clustering and cross-layer analyses for studying local organization,
48+
redundancy, and downstream dependence.
49+
- Project workflows under `projects/` that show how to reproduce concrete
50+
analyses with the shared library.
4651

4752
## Installation
4853

4954
```bash
50-
git clone https://github.com/KempnerInstitute/nodelens.git
51-
cd nodelens
55+
git clone https://github.com/KempnerInstitute/NodeLens.git
56+
cd NodeLens
5257
conda env create -f environment.yml
5358
conda activate nodelens
5459
pip install -e .
@@ -62,40 +67,42 @@ pip install -e .[all]
6267

6368
## Quick Start
6469

70+
Run experiments from YAML configs:
71+
6572
```bash
66-
# Vision model analysis
73+
# Small vision smoke test
6774
python scripts/run_experiment.py --config configs/examples/mnist_basic.yaml
6875

69-
# CNN pruning
76+
# CNN pruning and clustering
7077
python scripts/run_experiment.py --config configs/vision_prune/resnet18_cifar10_full.yaml
7178

72-
# LLM supernode and SCAR analysis
79+
# LLM channel analysis and structured FFN pruning
7380
python scripts/run_experiment.py --config configs/prune_llm/llama3_8b_unified.yaml
7481
```
7582

76-
Package the public Supernodes and SCAR artifacts:
83+
Use metrics directly from Python:
7784

78-
```bash
79-
python projects/supernodes_scar/scripts/prepare_hf_artifacts.py \
80-
--output-dir outputs/supernodes_scar_hf \
81-
--clean
85+
```python
86+
from nodelens.metrics import get_metric, list_metrics
87+
88+
print(list_metrics())
8289

83-
python projects/supernodes_scar/scripts/verify_hf_artifacts.py \
84-
outputs/supernodes_scar_hf
90+
metric = get_metric("rayleigh_quotient")
91+
scores = metric.compute(inputs=layer_inputs, weights=layer_weights)
8592
```
8693

87-
## Paper Releases
94+
## Project Workflows
8895

89-
Paper-specific release material lives under `projects/`. Reusable library code
90-
stays in `src/nodelens`, while each project folder records the exact configs,
91-
artifact layout, reproducibility notes, and release checklist for a paper.
96+
Reusable library code lives in `src/nodelens`. Project folders contain the
97+
configs, small helper scripts, and artifact descriptions needed to reproduce a
98+
specific analysis with the shared package.
9299

93100
Current project:
94101

95-
- `projects/supernodes_scar/`: release material for "Supernodes and Halos:
96-
Loss-Critical Hubs in LLM Feed-Forward Layers".
102+
- `projects/supernodes_scar/`: workflow for the Supernodes and SCAR study of
103+
loss-sensitive FFN channels in LLMs.
97104

98-
Derived artifacts for this project are staged on Hugging Face:
105+
The Supernodes and SCAR project also has a public derived-artifact dataset:
99106

100107
- `https://huggingface.co/datasets/hsafaai/supernodes-scar-artifacts`
101108

@@ -106,28 +113,29 @@ Derived artifacts for this project are staged on Hugging Face:
106113
| Activation metrics | `activation_l2_norm`, `activation_variance`, `activation_outlier_index` |
107114
| Alignment metrics | `rayleigh_quotient`, `delta_alignment` |
108115
| Information metrics | `mutual_information_gaussian`, `pairwise_redundancy_gaussian`, `gaussian_pid_synergy_mmi` |
109-
| SCAR metrics | `scar_activation_power`, `scar_taylor`, `scar_curvature`, `scar_loss_proxy` |
116+
| Loss-sensitive metrics | `scar_activation_power`, `scar_taylor`, `scar_curvature`, `scar_loss_proxy` |
110117
| Pruning strategies | `magnitude`, `alignment`, `composite`, `cluster_aware`, `random` |
111118

112119
## Repository Layout
113120

114121
```text
115-
nodelens/
122+
NodeLens/
116123
|-- configs/
117-
| |-- prune_llm/ # LLM and SCAR configs
118-
| |-- vision_prune/ # Vision pruning configs
119-
| `-- examples/ # Small example configs
120-
|-- projects/ # Paper-specific release material
124+
| |-- examples/ # Small runnable configs
125+
| |-- prune_llm/ # LLM channel-analysis and pruning configs
126+
| `-- vision_prune/ # Vision pruning and clustering configs
127+
|-- projects/ # Reproducible project workflows
121128
|-- scripts/
122129
| |-- run_experiment.py # Main experiment entry point
123-
| `-- run_analysis.py # Post-hoc analysis
130+
| `-- run_analysis.py # Post-hoc analysis entry point
124131
|-- src/nodelens/
125132
| |-- analysis/ # Visualization, clustering, cascade analysis
126133
| |-- experiments/ # Experiment classes
127-
| |-- metrics/ # Importance metrics
134+
| |-- metrics/ # Importance and information metrics
128135
| |-- models/ # Model wrappers
129-
| `-- pruning/ # Pruning strategies
130-
|-- tests/ # Unit tests
136+
| |-- pruning/ # Pruning strategies
137+
| `-- services/ # Activation capture, scoring, and mask utilities
138+
|-- tests/ # Unit and integration tests
131139
`-- docs/ # Documentation
132140
```
133141

@@ -137,7 +145,8 @@ nodelens/
137145
- [API Reference](docs/api_reference.md)
138146
- [LLM Guide](docs/llm_guide.md)
139147
- [Metric Consistency](docs/METRIC_CONSISTENCY.md)
140-
- [Supernodes and SCAR Release Notes](projects/supernodes_scar/README.md)
148+
- [Architecture](docs/ARCHITECTURE.md)
149+
- [Supernodes and SCAR Workflow](projects/supernodes_scar/README.md)
141150

142151
Build the Sphinx docs locally:
143152

@@ -155,8 +164,9 @@ pytest tests/unit/ -v
155164

156165
## Citation
157166

158-
If you use the Supernodes and SCAR release, please cite the paper and the
159-
archived code/artifact versions listed in `CITATION.cff`.
167+
If you use NodeLens, cite the repository metadata in `CITATION.cff`. If you use
168+
a project workflow or public artifact dataset, also cite the associated paper
169+
and artifact record.
160170

161171
## License
162172

‎docs/ARCHITECTURE.md‎

Lines changed: 93 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,101 @@
11
# Architecture
22

3-
NodeLens is organized as a reusable library plus paper-specific project
4-
folders. The library code should remain general; each paper folder should only
5-
contain release notes, configs, and artifact packaging scripts for that paper.
6-
7-
```mermaid
8-
flowchart TB
9-
subgraph Library[src/nodelens]
10-
M[metrics]
11-
P[pruning]
12-
E[experiments]
13-
A[analysis]
14-
S[services]
15-
end
16-
17-
subgraph Inputs[Inputs]
18-
C[configs]
19-
D[calibration data]
20-
N[model checkpoints]
21-
end
22-
23-
subgraph Projects[projects]
24-
R[supernodes_scar]
25-
end
26-
27-
C --> E
28-
D --> S
29-
N --> S
30-
S --> M
31-
M --> P
32-
M --> A
33-
P --> E
34-
A --> E
35-
E --> R
36-
R --> H[Hugging Face artifact bundle]
3+
NodeLens is organized around a reusable Python package and a small set of
4+
configuration-driven workflows. The library code stays general; project folders
5+
show how the same components are combined for a concrete study.
6+
7+
## Data Flow
8+
9+
```text
10+
YAML config
11+
|
12+
v
13+
Experiment runner
14+
|
15+
|-- loads model and dataset
16+
|-- selects tracked layers
17+
|-- captures activations, gradients, weights, and masks
18+
|
19+
v
20+
Metric and scoring layer
21+
|
22+
|-- activation and norm statistics
23+
|-- Rayleigh quotient and spectral metrics
24+
|-- mutual information, redundancy, and synergy
25+
|-- gradient, Taylor, curvature, and loss-proxy scores
26+
|
27+
v
28+
Analysis and intervention layer
29+
|
30+
|-- clustering and cross-layer analyses
31+
|-- ablation and sensitivity probes
32+
|-- structured pruning strategies
33+
|-- plots, tables, JSON summaries, and reports
34+
```
35+
36+
## Package Layout
37+
38+
```text
39+
src/nodelens/
40+
|-- analysis/ # Aggregation, clustering, visualization, reports
41+
|-- configs/ # Config loading and validation
42+
|-- core/ # Registries, protocols, base abstractions
43+
|-- dataops/ # Dataset loading and tensor preprocessing
44+
|-- experiments/ # Config-driven experiment classes
45+
|-- infrastructure/ # Logging, distributed helpers, storage utilities
46+
|-- metrics/ # Node and channel metrics
47+
|-- models/ # Model wrappers and model factory helpers
48+
|-- pruning/ # Pruning configs, masks, and strategies
49+
|-- services/ # Activation capture, scoring, and mask operations
50+
`-- training/ # Training and evaluation helpers
3751
```
3852

3953
## Design Rules
4054

41-
- Keep reusable metrics, services, pruning code, and experiment classes in
42-
`src/nodelens/`.
43-
- Keep paper release instructions and packaging scripts in `projects/`.
55+
- Keep reusable metrics, model wrappers, pruning code, and experiment classes
56+
in `src/nodelens/`.
57+
- Keep runnable experiment settings in `configs/`.
4458
- Keep generated outputs in `outputs/`, which is ignored by git.
45-
- Do not store model weights, raw datasets, cluster logs, or private paths in
46-
the repository.
47-
- Use project manifests and checksums for anything uploaded as an artifact.
48-
49-
## Supernodes and SCAR Flow
50-
51-
```mermaid
52-
sequenceDiagram
53-
participant Config as YAML config
54-
participant Runner as run_experiment.py
55-
participant Capture as activation and gradient capture
56-
participant Metrics as SCAR metrics
57-
participant Prune as structured pruning
58-
participant Artifacts as artifact bundle
59-
60-
Config->>Runner: choose model, calibration data, sparsity, metrics
61-
Runner->>Capture: collect layer-wise activations and gradients
62-
Capture->>Metrics: compute LP, activation, curvature, and Taylor scores
63-
Metrics->>Prune: protect supernode core and rank remaining channels
64-
Prune->>Artifacts: write results, figures, tables, and manifests
59+
- Keep project folders focused on reproducible usage: configs, helper scripts,
60+
artifact descriptions, and notes that connect a study to the shared library.
61+
- Do not store model weights, raw datasets, checkpoints, scheduler logs, access
62+
tokens, or private absolute paths in the repository.
63+
64+
## Common Workflows
65+
66+
### Metric Analysis
67+
68+
```text
69+
model + dataloader
70+
-> activation capture
71+
-> metric computation
72+
-> per-layer channel scores
73+
-> plots or JSON summaries
6574
```
75+
76+
Use this path for activation outliers, Rayleigh quotient scores, information
77+
metrics, redundancy estimates, or loss-proxy ranking.
78+
79+
### Intervention Analysis
80+
81+
```text
82+
channel scores
83+
-> masks or ablation sets
84+
-> model evaluation
85+
-> sensitivity curves
86+
```
87+
88+
Use this path to test whether a metric identifies channels that matter for
89+
accuracy, perplexity, robustness, pruning, or other downstream behavior.
90+
91+
### Project Workflow
92+
93+
```text
94+
shared package + configs
95+
-> experiment outputs
96+
-> aggregation scripts
97+
-> figures, tables, and artifact manifests
98+
```
99+
100+
Project folders under `projects/` should make a study easy to inspect without
101+
turning project-specific scripts into core library code.

‎docs/METRIC_CONSISTENCY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ It exists to prevent subtle drift in:
66
- **Keys** (how values are named/stored),
77
- **Sign conventions** (what "high" means when used for pruning/scoring).
88

9-
It intentionally avoids referencing any paper draft; the canonical sources are the implementations under `src/nodelens/metrics/` and the experiment pipeline that stores per-layer metric arrays.
9+
It intentionally avoids relying on paper-specific wording; the canonical sources are the implementations under `src/nodelens/metrics/` and the experiment pipeline that stores per-layer metric arrays.
1010

1111
---
1212

‎docs/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ NodeLens is the public project name. The Python package is imported as
99
- [API Reference](api_reference.md) - Core classes and functions
1010
- [LLM Guide](llm_guide.md) - LLM-specific analysis and pruning
1111
- [Metric Consistency](METRIC_CONSISTENCY.md) - Theory-code verification
12-
- [Architecture](ARCHITECTURE.md) - Library and project-release layout
12+
- [Architecture](ARCHITECTURE.md) - Library layout and data flow
1313

1414
## Configuration
1515

@@ -25,7 +25,7 @@ NodeLens is the public project name. The Python package is imported as
2525
| Type | Description |
2626
|------|-------------|
2727
| `alignment_analysis` | General alignment metrics for vision models |
28-
| `llm_alignment` | LLM supernode and SCAR analysis |
28+
| `llm_alignment` | LLM channel metrics and structured FFN pruning |
2929
| `cluster_analysis` | Metric-space clustering with halo analysis |
3030

3131
### Key Classes

0 commit comments

Comments
 (0)