Moving bionemo-core loads to bionemo-scdl, so that it's fully standalone. (#1133)

The load functions in bionemo-core are moved into bionemo-scdl. This
creates some code duplication, but allows SCDL to be a fully standalone
package. Additionally, this assumes that the downloads are always a
resource, not a model.
An awkward part of this setup is that some scdl resources are listed in
both bionemo-core (when used in geneformer) and in bionemo-scdl.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added SCDL data-loading utilities with caching and support for NGC and
S3 (PBSS); public API to discover and load named resources.

- **Data**
- Added several SCDL test datasets to package resources and removed
duplicate entries from core resources.

- **Tests**
- Added tests for local resource discovery and loader behavior;
simplified fixtures and removed startup delays.

- **Dependencies**
- Updated runtime deps (pooch, tqdm) and test extras (removed one
package, added psutil).

- **Chores**
- Cleaned test setup and moved test data handling into the SCDL package.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: polinabinder1 <pbinder@nvidia.com>
Signed-off-by: Polina Binder <pbinder@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Author: polinabinder1

sub-packages/bionemo-core/src/bionemo/core/data/resources/scdl.yaml

@@ -13,11 +13,3 @@
   sha256: 9020ba336dbfe33bddadba26ca0cde49958cbd73c5ad44f0960a5a4837c9db26 # pragma: allowlist secret
   owner: Savitha Srinivasan <savithas@nvidia.com>
   description: Sample test data for SCDL with feature IDs appended.
-
-- tag: sample_scdl_neighbor
-  ngc: nvidia/clara/scdl_neighbor_testdata:1.0
-  ngc_registry: resource
-  pbss: "s3://bionemo-ci/test-data/scdl_neighbor_test_20250616.tar.gz"
-  sha256: f64a723e5a1d3223d7ad636c2b7601fe5927be47fb1a418a60687ef80eab83d0 # pragma: allowlist secret
-  owner: Camir Ricketts <camirr@nvidia.com>
-  description: Sample test data for SCDL with neighbors.

sub-packages/bionemo-scdl/pyproject.toml

@@ -16,15 +16,17 @@ dependencies = [
     'numpy>=1.24.4',
     'pandas>=2.2.1',
     'pyarrow>=16.0.0',
+    'pooch>=1.6.0',
     'scipy>=1.11.1',
     'torch>=2.2.1',
     'pydantic[email]>=2.2.0',
+    'tqdm>=4.67.1'
 ]
 
 [project.optional-dependencies]
 test = [
-    "bionemo-core>=2.4.5",
-    'pytest>=8.4.1'
+    'pytest>=8.4.1',
+    'psutil>=7.0.0'
 ]
 
 [project.scripts]
@@ -36,6 +38,11 @@ include = ["bionemo.*"]
 namespaces = true
 exclude = ["test*."]
 
+# Make sure that the resource yaml files are being packaged alongside the python files.
+[tool.setuptools.package-data]
+"bionemo.scdl" = ["**/*.yaml"]
+
+
 [tool.setuptools.dynamic]
 version = { file = "VERSION" }
 

sub-packages/bionemo-scdl/src/bionemo/scdl/data/__init__.py

@@ -0,0 +1,21 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: LicenseRef-Apache2
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SCDL data loading utilities."""
+
+from .load import Resource, get_all_resources, load
+
+
+__all__ = ["Resource", "get_all_resources", "load"]