Skip to content

README.md

Latest commit

 

History

History
168 lines (109 loc) · 4.74 KB

README.md

File metadata and controls

168 lines (109 loc) · 4.74 KB

DataCrunch Python SDK

The official DataCrunch.io Python SDK.

The SDK's documentation is available on ReadTheDocs

DataCrunch's Public API documentation is available here.

Getting Started - Using the SDK:

  • Install via pip:

    pip3 install datacrunch

  • Generate your client credentials - instructions in the public API docs.

  • Add your client id and client secret to an environment variable (don't want it to be hardcoded):

    Linux (bash):

    export DATACRUNCH_CLIENT_ID=YOUR_ID_HERE
export DATACRUNCH_CLIENT_SECRET=YOUR_SECRET_HERE

  • To enable sending inference requests from SDK you must generate an inference key - Instructions on inference authorization

  • Add your inference key to an environment variable

    Linux (bash):

    export DATACRUNCH_INFERENCE_KEY=YOUR_API_KEY_HERE

    Other platforms: https://en.wikipedia.org/wiki/Environment_variable

  • Example for creating a new instance:

    import os
from datacrunch import DataCrunchClient

# Get credentials from environment variables
CLIENT_ID = os.environ.get('DATACRUNCH_CLIENT_ID')
CLIENT_SECRET = os.environ['DATACRUNCH_CLIENT_SECRET']

# Create datcrunch client
datacrunch = DataCrunchClient(CLIENT_ID, CLIENT_SECRET)

# Get all SSH keys
ssh_keys = datacrunch.ssh_keys.get()
ssh_keys = list(map(lambda key: key.id, ssh_keys))

# Create a new instance
instance = datacrunch.instances.create(instance_type='1V100.6V',
                                      image='ubuntu-24.04-cuda-12.8-open-docker',
                                      ssh_key_ids=ssh_keys,
                                      hostname='example',
                                      description='example instance')

# Delete instance
datacrunch.instances.action(instance.id, datacrunch.constants.instance_actions.DELETE)

    More examples can be found in the /examples folder or in the documentation.

Development

Setting up the local development environment

  • Clone the repository:

    git clone

  • Create local virtual environment:

    python3 -m venv datacrunch_env && source ./datacrunch_env/bin/activate

    or if using fish shell:

    python3 -m venv datacrunch_env && source ./datacrunch_env/bin/activate.fish

  • Install Dependencies:

    pip3 install -e .[test]
pip3 install -U pytest

Running Tests

We use pytest for testing.

  • To execute all tests

    pytest

  • To execute a single test file

    pytest ./tests/unit_tests/test_file.py

Local Manual Testing

Create this file in the root directory of the project:

from datacrunch.datacrunch import DataCrunchClient

CLIENT_SECRET = 'secret'
CLIENT_ID = 'your-id'

# Create datcrunch client
datacrunch = DataCrunchClient(CLIENT_ID, CLIENT_SECRET, base_url='http://localhost:3001/v1')

Generating the documentation

If added a new service, create a documentation template under api/services for that service.

cd docs
make html

Style Guide

Use autopep8 for auto code formatting:

# Install
pip3 install autopep8

# Apply to an entire directory
autopep8 directory_name --recursive --in-place --pep8-passes 2000 --verbose

# Or a single file
autopep8 file.py --in-place

Contact

You can contact us here, or open an issue in the repo.