Create package (#359)

* convert to pypi package, init

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reorg package struckure to match standard

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix incomplete import migration from src to claymodel

- Update trainer.py imports to use claymodel instead of src
- Fix all notebook files to use claymodel imports
- Fix circular import in claymodel/callbacks_wandb.py
- Add pip installation instructions to README
- All import paths now consistently use claymodel package

Resolves the ModuleNotFoundError issues and makes the package
properly installable via pip.

* Enhance package configuration and initialization

- Improve pyproject.toml with proper metadata, license, and flexible dependencies
- Add comprehensive package documentation and usage examples
- Create structured __init__.py files for better package organization
- Update dependency versions to use >= for better compatibility
- Add proper package metadata for PyPI publication

The package now has professional-grade configuration suitable for
distribution while maintaining all functionality.

* Complete import migration for all example scripts

- Fix remaining finetune imports in classify.py, segment.py, regression.py
- Update internal imports in model files (eurosat_model.py, etc.)
- All scripts now use claymodel.finetune.* imports consistently
- Ready for full testing in proper environment

All import migration work completed:
✅ trainer.py fixed
✅ All 12 notebooks updated
✅ All finetune example scripts fixed
✅ Internal cross-module imports updated

* Add comprehensive testing status documentation

- Documents all completed import migration work
- Lists all fixed files (6 Python scripts + 12 notebooks)
- Provides testing checklist for clean environment
- Ready for validation in proper PyTorch environment

* Add minimum version constraints for better dependency resolution

- Add lightning>=2.0.0 to ensure compatibility
- Add timm>=0.6.0 for stable API
- Maintain flexible upper bounds for PyTorch ecosystem

This ensures users get compatible versions while allowing
for future updates. The flexible bounds prevent the version
conflicts we encountered in testing environments.

* Remove outdated testing status documentation

- Delete the TESTING_STATUS.md file as it is no longer relevant
- This file contained information on completed import migration and testing status
- The removal reflects the transition to a new testing framework or documentation approach

This change simplifies the repository by eliminating obsolete documentation, ensuring that only current and relevant information is maintained.

* """
Update documentation and .gitignore for improved usability

- Add new entries to .gitignore to exclude checkpoint files (*.ckpt) from version control
- Enhance documentation by adding a Quick Start guide and a Migration guide to facilitate user onboarding
- Update installation instructions to reflect the new pip-installable package structure
- Include detailed examples for generating embeddings and using pretrained models

These changes aim to streamline the user experience, making it easier for new users to get started with the Clay Foundation Model while ensuring that unnecessary files are not tracked in the repository.
"""

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* make ruff happy

* ruff-ing

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Refactor function and condition checks for clarity

- Corrected the type check in `initialize_weights` method to use a tuple for `isinstance`, ensuring proper evaluation of both `nn.Linear` and `nn.Conv2d`.
- Fixed a typo in the function name `list_uniqe_ids` to `list_unique_ids` for consistency and accuracy in `preprocess_data.py`.
- Updated the call to the corrected function name in the `process` function.

These changes enhance code readability and maintainability by ensuring correct type checks and function naming conventions.

* Refactor tutorial notebook for clarity and error handling

- Removed unnecessary output cells from the `tutorial_digital_earth_pacific_patch_level.ipynb` notebook, which included warnings and installation logs that cluttered the output.
- Updated the execution count for code cells to `null` to reset the notebook state, ensuring a clean slate for users running the tutorial.
- This refactoring enhances the user experience by providing a cleaner and more focused tutorial without extraneous information.

These changes aim to improve the clarity and usability of the tutorial notebook, making it easier for users to follow along without distractions.

* Update docs/index.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Romeo Kienzler <romeo.kienzler1@ibm.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Author: 4 people

.gitignore

@@ -67,3 +67,7 @@ datadisk/
 
 # vscode
 .vscode
+
+# pip
+.venv
+*.ckpt

.ruff.toml

@@ -0,0 +1,22 @@
+# Ruff configuration for Clay Foundation Model
+target-version = "py39"
+
+[lint]
+# Allow fixes for auto-fixable rules
+fixable = ["ALL"]
+
+# Ignore specific rules for notebooks since they contain tutorial code
+[lint.per-file-ignores]
+# Ignore notebook-specific issues
+"*.ipynb" = [
+    "PLR0913",  # Too many arguments (acceptable in tutorial functions)
+    "PLR2004",  # Magic value comparison (acceptable in demos)
+    "E501",     # Line too long (acceptable for URLs and strings)
+]
+
+[lint.flake8-bugbear]
+extend-immutable-calls = ["fastapi.Depends"]
+
+[format]
+# Use single quotes for strings
+quote-style = "double"

README.md

@@ -21,7 +21,22 @@ Launch into a [JupyterLab](https://jupyterlab.readthedocs.io) environment on
 
 ## Installation
 
-### Basic
+### Pip Installation (Recommended)
+
+The easiest way to install Clay Foundation Model is via pip:
+
+    pip install git+https://github.com/Clay-foundation/model.git
+
+This will install the `claymodel` package and all its dependencies. You can then import and use it in your Python code:
+
+```python
+from claymodel.datamodule import ClayDataModule
+from claymodel.module import ClayMAEModule
+```
+
+### Development Installation
+
+For development or advanced usage, you can set up the full development environment:
 
 To help out with development, start by cloning this [repo-url](/../../)
 
@@ -60,6 +75,10 @@ Finally, double-check that the libraries have been installed.
 
 The neural network model can be ran via
 [LightningCLI v2](https://pytorch-lightning.medium.com/introducing-lightningcli-v2supercharge-your-training-c070d43c7dd6).
+
+> [!NOTE]
+> If you installed via pip, you'll need to clone the repository to access the trainer script and config files.
+
 To check out the different options available, and look at the hyperparameter
 configurations, run:
 

claymodel/__init__.py

@@ -0,0 +1,26 @@
+"""
+Clay Foundation Model - An open source AI model and interface for Earth observation.
+
+This package provides tools for working with satellite imagery and Earth
+observation data using foundation models built on Vision Transformers.
+
+Example usage:
+    from claymodel.datamodule import ClayDataModule
+    from claymodel.module import ClayMAEModule
+
+    # Create data module and model
+    datamodule = ClayDataModule(...)
+    model = ClayMAEModule(...)
+"""
+
+__version__ = "1.5.0"
+
+# Main components available for import
+__all__ = [
+    "ClayMAEModule",
+    "ClayDataModule",
+    "clay_mae_base",
+    "clay_mae_large",
+    "clay_mae_small",
+    "clay_mae_tiny",
+]

src/backbone.py claymodel/backbone.pysrc/backbone.py renamed to claymodel/backbone.py

@@ -10,7 +10,7 @@
 ```
 import lightning as L
 
-from src.callbacks_wandb import LogMAEReconstruction
+from claymodel.callbacks_wandb import LogMAEReconstruction
 
 trainer = L.Trainer(
     ...,

src/callbacks.py claymodel/callbacks.pysrc/callbacks.py renamed to claymodel/callbacks.py

@@ -7,7 +7,7 @@
 import random
 from collections import defaultdict
 from pathlib import Path
-from typing import List, Literal
+from typing import Literal
 
 import lightning as L
 import numpy as np
@@ -27,7 +27,7 @@ class EODataset(Dataset):
     """Reads different Earth Observation data sources from a directory."""
 
     def __init__(
-        self, chips_path: List[Path], size: int, platforms: list, metadata: Box
+        self, chips_path: list[Path], size: int, platforms: list, metadata: Box
     ) -> None:
         super().__init__()
         self.chips_path = chips_path
@@ -322,3 +322,20 @@ def predict_dataloader(self):
             num_workers=self.num_workers,
             shuffle=False,
         )
+
+
+# Import additional DataModules for finetune tasks
+try:
+    from claymodel.finetune.classify.eurosat_datamodule import EuroSATDataModule
+    from claymodel.finetune.regression.biomasters_datamodule import BioMastersDataModule
+    from claymodel.finetune.segment.chesapeake_datamodule import ChesapeakeDataModule
+
+    __all__ = [
+        "ClayDataModule",
+        "EuroSATDataModule",
+        "ChesapeakeDataModule",
+        "BioMastersDataModule",
+    ]
+except ImportError:
+    # Some finetune modules may not be available
+    __all__ = ["ClayDataModule"]

src/callbacks_wandb.py claymodel/callbacks_wandb.pysrc/callbacks_wandb.py renamed to claymodel/callbacks_wandb.py

@@ -9,7 +9,7 @@
 from einops import rearrange
 from torch import nn
 
-from src.utils import posemb_sincos_1d
+from claymodel.utils import posemb_sincos_1d
 
 
 class FCBlock(nn.Module):

src/datamodule.py claymodel/datamodule.pysrc/datamodule.py renamed to claymodel/datamodule.py

@@ -0,0 +1,32 @@
+"""
+Fine-tuning modules for Clay Foundation Model.
+
+This subpackage contains modules for fine-tuning the Clay model on various
+downstream tasks including classification, segmentation, regression, and
+embedding extraction.
+"""
+
+try:
+    # Classification
+    from .classify.eurosat_datamodule import EuroSATDataModule
+    from .classify.eurosat_model import EuroSATClassifier
+
+    # Regression
+    from .regression.biomasters_datamodule import BioMastersDataModule
+    from .regression.factory import Regressor
+
+    # Segmentation
+    from .segment.chesapeake_datamodule import ChesapeakeDataModule
+    from .segment.chesapeake_model import ChesapeakeSegmentor
+
+    __all__ = [
+        "EuroSATDataModule",
+        "EuroSATClassifier",
+        "ChesapeakeDataModule",
+        "ChesapeakeSegmentor",
+        "BioMastersDataModule",
+        "Regressor",
+    ]
+except ImportError:
+    # Allow subpackage to be imported even if specific components fail
+    __all__ = []