Skip to content

Commit 2962fc7

Browse files
lukasbindreiterlukasbindreiter
committed
Update READMEs, add contributing guide (#193)
1 parent b55d9f5 commit 2962fc7

7 files changed

Lines changed: 242 additions & 97 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Contributing to Tilebox Python
2+
3+
Welcome! We're excited that you're interested in contributing to Tilebox Python. This document provides guidelines and information to help you get started.
4+
5+
## Module and package structure
6+
7+
This repository contains multiple python packages. We use python [namespace packages](https://packaging.python.org/en/latest/guides/packaging-namespace-packages/) to
8+
organize our code into multiple packages that then share the common `tilebox` python module name.
9+
10+
## Required tooling
11+
12+
Before starting to work on this repository, make sure you have uv installed. [Install instructions](https://docs.astral.sh/uv/getting-started/installation/).
13+
14+
## Workspace setup
15+
16+
This repository is set up as a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/).
17+
18+
This means it contains multiple packages, which all share the same virtualenv, and may have interdependencies between
19+
them.
20+
21+
## Useful commands for development
22+
23+
### Initial setup
24+
25+
```bash
26+
uv sync
27+
```
28+
29+
### Running tests
30+
31+
```bash
32+
./tests.sh # helper script for running with coverage
33+
34+
# testing a single package
35+
uv run --package tilebox-datasets pytest tilebox-datasets
36+
```
37+
38+
### Running code analysis
39+
40+
```bash
41+
# formatting and linting:
42+
uv run ruff format . && uv run ruff check --fix .
43+
44+
# type checking:
45+
uv run pyright .
46+
```
47+
48+
### Adding dependencies to one of the packages
49+
50+
```bash
51+
uv add --package tilebox-datasets "numpy>=2"
52+
```
53+
54+
### Used code quality tools
55+
56+
- [ruff](https://github.com/astral-sh/ruff) for linting and formatting
57+
- [pyright](https://github.com/microsoft/pyright) for type checking
58+
- [pre-commit](https://pre-commit.com/) for running all of the above automatically on each git commit
59+
60+
## Protobuf usage
61+
62+
We use [protobuf](https://protobuf.dev/) for defining our API contracts. The protobuf files are generated from the messages
63+
defined in the [Tilebox Go repository](https://github.com/tilebox/tilebox-go/tree/main/apis).
64+
65+
### Updating protobuf files
66+
67+
```bash
68+
uv run copy-protobuf
69+
```
70+
71+
## Releasing and deploying a new version
72+
73+
There is a Github CI workflow set up for building the packages and pushing them to the internal `pypi`.
74+
All that is needed is to create a github release for a given tag. This can be done in the github web UI or on the
75+
command line.
76+
77+
For example, to release a version `0.1.0` via the CLI the following commands could be used:
78+
(this uses [parse-changelog](https://github.com/taiki-e/parse-changelog) for release notes)
79+
80+
```bash
81+
git tag "v0.1.0" -m "Tilebox Python Release v0.1.0"
82+
git push -u origin v0.1.0
83+
gh release create v0.1.0 --notes $(parse-changelog CHANGELOG.md)
84+
# wait a bit until CI pipeline is done and voila!
85+
```

‎README.md‎

Lines changed: 86 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -1,92 +1,114 @@
1-
# Tilebox Python Packages
1+
<h1 align="center">
2+
<img src="https://storage.googleapis.com/tbx-web-assets-2bad228/banners/tilebox-banner.svg" alt="Tilebox Logo">
3+
<br>
4+
</h1>
5+
6+
<div align="center">
7+
<a href="https://pypi.org/project/tilebox-workflows/">
8+
<img src="https://img.shields.io/pypi/v/tilebox-workflows.svg?style=flat-square&label=version&color=f43f5e" alt="PyPi Latest Release badge"/>
9+
</a>
10+
<a href="https://pypi.org/project/tilebox-workflows/">
11+
<img src="https://img.shields.io/pypi/pyversions/tilebox-workflows.svg?style=flat-square&logo=python&color=f43f5e&logoColor=f43f5e" alt="Required Python Version badge"/>
12+
</a>
13+
<a href="https://github.com/tilebox/tilebox-python/blob/main/LICENSE">
14+
<img src="https://img.shields.io/github/license/tilebox/tilebox-python.svg?style=flat-square&color=f43f5e" alt="MIT License"/>
15+
</a>
16+
<a href="https://github.com/tilebox/tilebox-python/actions">
17+
<img src="https://img.shields.io/github/actions/workflow/status/tilebox/tilebox-python/main.yml?style=flat-square&color=f43f5e" alt="Build Status"/>
18+
</a>
19+
<a href="https://tilebox.com/discord">
20+
<img src="https://img.shields.io/badge/Discord-%235865F2.svg?style=flat-square&logo=discord&logoColor=white" alt="Join us on Discord"/>
21+
</a>
22+
</div>
23+
24+
<p align="center">
25+
<a href="https://docs.tilebox.com/"><b>Documentation</b></a>
26+
|
27+
<a href="https://console.tilebox.com/"><b>Console</b></a>
28+
|
29+
<a href="https://examples.tilebox.com/"><b>Example Gallery</b></a>
30+
</p>
31+
32+
# Tilebox Python
33+
34+
Python clients for [Tilebox](https://tilebox.com) - a framework for space data management and workflow orchestration.
35+
36+
## Install
237

3-
Mono-repo containing various Tilebox python packages.
4-
5-
## Module and package structure
6-
7-
We use python [namespace packages](https://packaging.python.org/en/latest/guides/packaging-namespace-packages/) to
8-
organize our code into multiple packages that then share a common python module name.
9-
10-
Currently we use the following top level module names:
11-
12-
```
13-
import tilebox # public API
14-
import _tilebox # private, internal modules
38+
```bash
39+
pip install tilebox-datasets tilebox-workflows tilebox-storage
1540
```
1641

17-
There is some inter-dependency across packages in this repository. A quick overview of how they are connected
18-
provides this diagram:
42+
> [!TIP]
43+
> For new projects we recommend using [uv](https://docs.astral.sh/uv/) - `uv add tilebox-datasets tilebox-workflows tilebox-storage`. Additional installation options are available [in our docs](https://docs.tilebox.com/sdks/python/install).
1944
20-
![Package Dependency Overview](.docs/packages.svg)
45+
## Documentation
2146

22-
## Required tooling
47+
Documentation is available at [docs.tilebox.com](https://docs.tilebox.com).
2348

24-
Before starting to work on this repository, make sure you have uv installed. [Install instructions](https://docs.astral.sh/uv/getting-started/installation/).
49+
## Getting started
2550

26-
## Workspace setup
51+
### Tilebox Datasets
2752

28-
This repository is set up as a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/).
53+
Structured and high-performance satellite metadata storage, indexing and querying. [See documentation](https://docs.tilebox.com/datasets/introduction)
2954

30-
This means it contains multiple packages, which all share the same virtualenv, and may have interdependencies between
31-
them.
55+
```python
56+
from tilebox.datasets import Client
57+
from shapely.geometry import shape
3258

33-
## Useful commands for development
59+
# create your API key at https://console.tilebox.com
60+
client = Client(token="YOUR_TILEBOX_API_KEY")
61+
datasets = client.datasets()
62+
print(datasets)
3463

35-
### Initial setup
64+
sentinel2_msi = client.dataset("open_data.copernicus.sentinel2_msi")
65+
collections = sentinel2_msi.collections()
66+
print(collections)
3667

37-
```bash
38-
uv sync
68+
area_of_interest = shape({
69+
"type": "Polygon", # coords in lon, lat
70+
"coordinates": [[[-5, 50], [-5, 56], [-11, 56], [-11, 50], [-5, 50]]]}
71+
)
72+
s2a_l1c = sentinel2_msi.collection("S2A_S2MSI1C")
73+
results = s2a_l1c.query(
74+
temporal_extent=("2022-07-13", "2022-07-13T02:00"),
75+
spatial_extent=area_of_interest,
76+
show_progress=True
77+
)
78+
print(f"Found {results.sizes['time']} datapoints") # Found 979 datapoints
3979
```
4080

41-
### Copying proto files to the correct locations
42-
43-
```bash
44-
uv run copy-protobuf
45-
```
4681

47-
### Running tests
82+
### Tilebox Workflows
4883

49-
```bash
50-
./tests.sh # helper script for running with coverage
84+
A parallel processing engine to simplify the creation of dynamic tasks that can be executed across various computing environments, including on-premise and auto-scaling clusters in public clouds.
5185

52-
# testing a single package
53-
uv run --package tilebox-datasets pytest tilebox-datasets
54-
```
86+
```python
87+
from tilebox.workflows import Client, Task
5588

56-
### Running code analysis
89+
class MyFirstTask(Task):
90+
def execute(self):
91+
print("Hello World from my first Tilebox task!")
5792

58-
```bash
59-
# formatting and linting:
60-
uv run ruff format . && uv run ruff check --fix .
6193

62-
# type checking:
63-
uv run pyright .
64-
```
94+
# create your API key at https://console.tilebox.com
95+
client = Client(token="YOUR_TILEBOX_API_KEY")
6596

66-
### Adding dependencies to one of the packages
97+
# submit a job
98+
jobs = client.jobs()
99+
jobs.submit("my-very-first-job", MyFirstTask())
67100

68-
```bash
69-
uv add --package tilebox-datasets "numpy>=2"
101+
# and run it
102+
runner = client.runner(tasks=[MyFirstTask])
103+
runner.run_all()
70104
```
71105

72-
### Used code quality tools
73-
74-
- [ruff](https://github.com/astral-sh/ruff) for linting and formatting
75-
- [pyright](https://github.com/microsoft/pyright) for type checking
76-
- [pre-commit](https://pre-commit.com/) for running all of the above automatically on each git commit
106+
## Contributing
77107

78-
## Releasing and deploying a new version
108+
Contributions are welcome! Please see the [contributing guide](https://github.com/tilebox/tilebox-python/blob/main/CONTRIBUTING.md) for more information.
79109

80-
There is a Github CI workflow set up for building the packages and pushing them to the internal `pypi`.
81-
All that is needed is to create a github release for a given tag. This can be done in the github web UI or on the
82-
command line.
110+
You can also join us on [Discord](https://tilebox.com/discord) if you have any questions or want to share your ideas.
83111

84-
For example, to release a version `0.1.0` via the CLI the following commands could be used:
85-
(this uses [parse-changelog](https://github.com/taiki-e/parse-changelog) for release notes)
112+
## License
86113

87-
```bash
88-
git tag "v0.1.0" -m "Tilebox Python Release v0.1.0"
89-
git push -u origin v0.1.0
90-
gh release create v0.1.0 --notes $(parse-changelog CHANGELOG.md)
91-
# wait a bit until CI pipeline is done and voila!
92-
```
114+
Distributed under the MIT License (`The MIT License`).

‎tilebox-datasets/README.md‎

Lines changed: 37 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,23 @@
1010
<a href="https://pypi.org/project/tilebox-datasets/">
1111
<img src="https://img.shields.io/pypi/pyversions/tilebox-datasets.svg?style=flat-square&logo=python&color=f43f5e&logoColor=f43f5e" alt="Required Python Version badge"/>
1212
</a>
13+
<a href="https://github.com/tilebox/tilebox-python/blob/main/LICENSE">
14+
<img src="https://img.shields.io/github/license/tilebox/tilebox-python.svg?style=flat-square&color=f43f5e" alt="MIT License"/>
15+
</a>
16+
<a href="https://github.com/tilebox/tilebox-python/actions">
17+
<img src="https://img.shields.io/github/actions/workflow/status/tilebox/tilebox-python/main.yml?style=flat-square&color=f43f5e" alt="Build Status"/>
18+
</a>
19+
<a href="https://tilebox.com/discord">
20+
<img src="https://img.shields.io/badge/Discord-%235865F2.svg?style=flat-square&logo=discord&logoColor=white" alt="Join us on Discord"/>
21+
</a>
1322
</div>
1423

1524
<p align="center">
1625
<a href="https://docs.tilebox.com/datasets/introduction"><b>Documentation</b></a>
1726
|
1827
<a href="https://console.tilebox.com/"><b>Console</b></a>
1928
|
20-
<a href="https://tilebox.com/discord"><b>Discord</b></a>
29+
<a href="https://examples.tilebox.com/"><b>Example Gallery</b></a>
2130
</p>
2231

2332
# Tilebox Datasets
@@ -37,48 +46,48 @@ Instantiate a client:
3746
```python
3847
from tilebox.datasets import Client
3948

40-
# create your API key at
41-
# https://console.tilebox.com
49+
# create your API key at https://console.tilebox.com
4250
client = Client(token="YOUR_TILEBOX_API_KEY")
4351
```
4452

45-
Explore datasets:
53+
Explore datasets and collections:
4654

4755
```python
4856
datasets = client.datasets()
4957
print(datasets)
5058

51-
sentinel1_sar = datasets.open_data.copernicus.sentinel1_sar
52-
collections = sentinel1_sar.collections()
59+
sentinel2_msi = client.dataset("open_data.copernicus.sentinel2_msi")
60+
collections = sentinel2_msi.collections()
5361
print(collections)
5462
```
5563

56-
Load data:
64+
Query data:
5765

5866
```python
59-
s1a_raw = collections["S1A_IW_RAW__0S"]
60-
interval = ("2017-01-01", "2023-01-01")
61-
raw_data = s1a_raw.load(interval, show_progress=True)
62-
print(raw_data)
67+
s2a_l1c = sentinel2_msi.collection("S2A_S2MSI1C")
68+
results = s2a_l1c.query(
69+
temporal_extent=("2017-01-01", "2023-01-01"),
70+
show_progress=True
71+
)
72+
print(f"Found {results.sizes['time']} datapoints") # Found 220542 datapoints
6373
```
6474

65-
```plaintext
66-
<xarray.Dataset> Size: 725MB
67-
Dimensions: (time: 1109597, latlon: 2)
68-
Coordinates:
69-
ingestion_time (time) datetime64[ns] 9MB 2024-06-21T11:03:33.8524...
70-
id (time) <U36 160MB '01595763-bae7-a646-99a5-d7f40d7...
71-
* time (time) datetime64[ns] 9MB 2017-01-01T00:17:50.8230...
72-
* latlon (latlon) <U9 72B 'latitude' 'longitude'
73-
Data variables: (12/30)
74-
granule_name (time) object 9MB 'S1A_IW_RAW__0SSV_20170101T00175...
75-
processing_level (time) <U2 9MB 'L0' 'L0' 'L0' 'L0' ... 'L0' 'L0' 'L0'
76-
satellite (time) object 9MB 'SENTINEL-1' ... 'SENTINEL-1'
77-
flight_direction (time) <U10 44MB 'ASCENDING' ... 'ASCENDING'
78-
product_type (time) object 9MB 'IW_RAW__0S' ... 'IW_RAW__0S'
79-
copernicus_id (time) <U36 160MB 'f3f6ec28-0f72-5d28-9e14-93f96b3...
80-
... ...
81-
acquisition_mode (time) <U2 9MB 'IW' 'IW' 'IW' 'IW' ... 'IW' 'IW' 'IW'
75+
Spatio-temporal queries:
76+
77+
```python
78+
from shapely.geometry import shape
79+
80+
area_of_interest = shape({
81+
"type": "Polygon", # coords in lon, lat
82+
"coordinates": [[[-5, 50], [-5, 56], [-11, 56], [-11, 50], [-5, 50]]]}
83+
)
84+
s2a_l1c = sentinel2_msi.collection("S2A_S2MSI1C")
85+
results = s2a_l1c.query(
86+
temporal_extent=("2022-07-13", "2022-07-13T02:00"),
87+
spatial_extent=area_of_interest,
88+
show_progress=True
89+
)
90+
print(f"Found {results.sizes['time']} datapoints") # Found 979 datapoints
8291
```
8392

8493
## Documentation

‎tilebox-grpc/README.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,23 @@
1010
<a href="https://pypi.org/project/tilebox-grpc/">
1111
<img src="https://img.shields.io/pypi/pyversions/tilebox-grpc.svg?style=flat-square&logo=python&color=f43f5e&logoColor=f43f5e" alt="Required Python Version badge"/>
1212
</a>
13+
<a href="https://github.com/tilebox/tilebox-python/blob/main/LICENSE">
14+
<img src="https://img.shields.io/github/license/tilebox/tilebox-python.svg?style=flat-square&color=f43f5e" alt="MIT License"/>
15+
</a>
16+
<a href="https://github.com/tilebox/tilebox-python/actions">
17+
<img src="https://img.shields.io/github/actions/workflow/status/tilebox/tilebox-python/main.yml?style=flat-square&color=f43f5e" alt="Build Status"/>
18+
</a>
19+
<a href="https://tilebox.com/discord">
20+
<img src="https://img.shields.io/badge/Discord-%235865F2.svg?style=flat-square&logo=discord&logoColor=white" alt="Join us on Discord"/>
21+
</a>
1322
</div>
1423

1524
<p align="center">
1625
<a href="https://docs.tilebox.com/"><b>Documentation</b></a>
1726
|
1827
<a href="https://console.tilebox.com/"><b>Console</b></a>
1928
|
20-
<a href="https://tilebox.com/discord"><b>Discord</b></a>
29+
<a href="https://examples.tilebox.com/"><b>Example Gallery</b></a>
2130
</p>
2231

2332
# Tilebox GRPC

0 commit comments

Comments
 (0)