Refactor: Use consistent URL representation for all storage paths (including file://)

[dimitri-yatsenko]

Summary

Following review feedback on PR #1311, we should refactor the storage layer to use consistent URL representation for all data sources, including local files.

Current Behavior

• Remote paths use URLs: s3://bucket/path, gs://bucket/path
• Local paths use raw filesystem paths: /path/to/file
• is_remote_url() function distinguishes between the two

Proposed Change

1. Accept both formats from users: /path/to/file and file:///path/to/file
2. Normalize to URLs internally: Convert all paths to URL format (file:// for local)
3. Store URLs consistently in the database
4. Leverage fsspec uniformity: fsspec already treats all backends (including local) uniformly via URLs

Benefits

• Coherent internal representation
• Simpler codebase - no special-casing for local vs remote
• Better alignment with fsspec's design philosophy
• Avoids potential bugs from inconsistent handling

Implementation Notes

• Add file:// to REMOTE_PROTOCOLS (or rename to URL_PROTOCOLS)
• Create helper to normalize user paths to URLs
• Update StorageBackend to work with URLs consistently
• Ensure backward compatibility for existing stored paths

References

• PR DataJoint 2.0 #1311 review discussion: DataJoint 2.0 #1311 (comment)
• Related to fsspec's unified approach to filesystems

[dimitri-yatsenko]

Implementation Complete

Changes to src/datajoint/storage.py

New constants and functions:

# All supported URL protocols
URL_PROTOCOLS = ("file://", "s3://", "gs://", "gcs://", "az://", "abfs://", "http://", "https://")

def is_url(path: str) -> bool:
    """Check if a path is a URL (including file://)."""
    return path.lower().startswith(URL_PROTOCOLS)

def normalize_to_url(path: str) -> str:
    """
    Normalize a path to URL form.
    Converts local filesystem paths to file:// URLs. URLs are returned unchanged.
    
    Examples:
        normalize_to_url("/data/file.dat") -> "file:///data/file.dat"
        normalize_to_url("s3://bucket/key") -> "s3://bucket/key"
    """

def parse_url(url: str) -> tuple[str, str]:
    """
    Parse a URL into (protocol, path).
    Now handles file:// in addition to remote protocols.
    
    Examples:
        parse_url("s3://bucket/key") -> ("s3", "bucket/key")
        parse_url("file:///data/file.dat") -> ("file", "/data/file.dat")
    """

New StorageBackend.get_url() method:

def get_url(self, path: str) -> str:
    """
    Get the full URL for a path in storage.
    Returns consistent URL representation for any backend.
    
    Examples:
        # Local storage
        backend.get_url("schema/table/file.dat") 
        -> "file:///data/schema/table/file.dat"
        
        # S3 storage
        backend.get_url("schema/table/file.dat")
        -> "s3://mybucket/schema/table/file.dat"
    """

Removed (no backward compatibility needed for pre-release):

• REMOTE_PROTOCOLS constant
• is_remote_url() function
• parse_remote_url() function

Tests

Added tests/unit/test_storage_urls.py with 23 unit tests covering:

• URL protocol constants
• is_url() for all protocols and edge cases
• normalize_to_url() path conversion
• parse_url() for all supported protocols

Documentation

Updated in datajoint-docs:

• src/how-to/configure-storage.md - Added "URL Representation" section
• src/reference/specs/type-system.md - Added URL representation table

[dimitri-yatsenko]

Fix implemented in #1328

[dimitri-yatsenko]

Closed by #1328