Skip to content

Commit ac35835

Browse files
committed
(chore): remove unnecessary link + formatting
1 parent c54449a commit ac35835

1 file changed

Lines changed: 36 additions & 14 deletions

File tree

docs/how-to-dask.md

Lines changed: 36 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -6,35 +6,46 @@ Here we will go through some common questions and answers about `dask`, with a s
66

77
**How do I monitor the {doc}`dask dashboard <dask:dashboard>`?**
88

9-
If you are in a jupyter notebook, when you render the `repr` of your `client`, you will see a link, usually something like `http://localhost:8787/status`. If you are working locally, this link alone should suffice.
9+
If you are in a jupyter notebook, when you render the `repr` of your `client`, you will see a link, usually something like `http://localhost:8787/status`.
10+
If you are working locally, this link alone should suffice.
1011

1112
If you are working on some sort of remote notebook from a web browser, you will need to replace `http://localhost:8787` by the root url of the notebook.
1213

1314
If you are in vscode, there is an [`dask` extension] which will allow you to monitor there.
1415

1516
**How do I know how to allocate resources?**
1617

17-
In `dask`, every worker will receive an equal share of the memory available. So if you request e.g., a slurm job with 256GB of RAM, and then start 8 workers, each will have 32 GB of memory.
18+
In `dask`, every worker will receive an equal share of the memory available.
19+
So if you request e.g., a slurm job with 256GB of RAM, and then start 8 workers, each will have 32 GB of memory.
1820

19-
`dask` distributes jobs to each worker generally based on the chunking of the array. So if you have dense chunks of `(30_000, 30_000)` with 32 bit integers, you will need to be have 3.6 GB for each worker, at the minimum to even load the data. Then if you do something like matrix multiplication, you will need double or even more, as an example.
21+
`dask` distributes jobs to each worker generally based on the chunking of the array.
22+
So if you have dense chunks of `(30_000, 30_000)` with 32 bit integers, you will need to be have 3.6 GB for each worker, at the minimum to even load the data.
23+
Then if you do something like matrix multiplication, you will need double or even more, as an example.
2024

2125
**How do I read my data into a `dask` array?**
2226

23-
{func}`anndata.experimental.read_elem_lazy` or {func}`anndata.experimental.read_lazy` can help you if you already have data on-disk that was written to the `anndata` file format. If you use {func}`dask.array.to_zarr`, the data *cannot* be read in using `anndata`'s functionality as `anndata` will look for its {doc}`specified file format metadata <anndata:fileformat-prose>`.
27+
{func}`anndata.experimental.read_elem_lazy` or {func}`anndata.experimental.read_lazy` can help you if you already have data on-disk that was written to the `anndata` file format.
28+
If you use {func}`dask.array.to_zarr`, the data _cannot_ be read in using `anndata`'s functionality as `anndata` will look for its {doc}`specified file format metadata <anndata:fileformat-prose>`.
2429

25-
If you need to implement custom io, generally we found that using {func}`dask.array.map_blocks` provides a nice way. See [our custom h5 io code] for an example.
30+
If you need to implement custom io, generally we found that using {func}`dask.array.map_blocks` provides a nice way.
31+
See [our custom h5 io code] for an example.
2632

2733
## Advanced use and how-to-contribute
2834

2935
**How do `scanpy` and `anndata` handle sparse matrices?**
3036

31-
While there is some {class}`scipy.sparse.csr_matrix` and {class}`scipy.sparse.csc_matrix` support for `dask`, it is not comprehensive and missing key functions like summation, mean etc. We have implemented custom functionality, much of which lives in {mod}`fast_array_utils`, although we have also had to implement custom algorithms like `pca` for sparse-in-dask. In the future, an [`array-api`] compatible sparse matrix like [`finch`] would help us considerably as `dask` supports the [`array-api`].
37+
While there is some {class}`scipy.sparse.csr_matrix` and {class}`scipy.sparse.csc_matrix` support for `dask`, it is not comprehensive and missing key functions like summation, mean etc.
38+
We have implemented custom functionality, much of which lives in {mod}`fast_array_utils`, although we have also had to implement custom algorithms like `pca` for sparse-in-dask.
39+
In the future, an [`array-api`] compatible sparse matrix like [`finch`] would help us considerably as `dask` supports the [`array-api`].
3240

33-
Therefore, if you run into a puzzling error after trying to run a function like {func}`numpy.sum` (or similar) on a sparse-in-dask array, consider checking {mod}`fast_array_utils`. If you need to implement the function yourself, see the next point.
41+
Therefore, if you run into a puzzling error after trying to run a function like {func}`numpy.sum` (or similar) on a sparse-in-dask array, consider checking {mod}`fast_array_utils`.
42+
If you need to implement the function yourself, see the next point.
3443

3544
**Custom block-wise array operations**
3645

37-
Sometimes you may want to do an operation on a an array that is implemented nowhere. Generally, we have found {func}`dask.array.map_blocks` to be versatile enough that most operations can be expressed on it. Take this (simplified) example of calculating a gram matrix from {func}`scanpy.pp.pca` for sparse-in-dask:
46+
Sometimes you may want to do an operation on a an array that is implemented nowhere.
47+
Generally, we have found {func}`dask.array.map_blocks` to be versatile enough that most operations can be expressed on it.
48+
Take this (simplified) example of calculating a gram matrix from {func}`scanpy.pp.pca` for sparse-in-dask:
3849

3950
```python
4051
def gram_block(x_part):
@@ -51,28 +62,39 @@ gram_matrix_dask = da.map_blocks(
5162
).sum(axis=0)
5263
```
5364

54-
This algorithm goes through every `chunk_size` number of rows and calculates the gram matrix for those rows producing a collection of `(n_vars,n_vars)` size matrix. These are the summed together to produce a single `(n_vars,n_vars)` matrix, which is the gram amtrix.
65+
This algorithm goes through every `chunk_size` number of rows and calculates the gram matrix for those rows producing a collection of `(n_vars,n_vars)` size matrix.
66+
These are the summed together to produce a single `(n_vars,n_vars)` matrix, which is the gram amtrix.
5567

56-
Because `dask` does not implement matrix multiplication for sparse-in-dask, we do it ourselves. We use `map_blocks` over a CSR sparse-in-dask array where the chunking looks something like `(chunk_size, n_vars)`. When we compute the invdividual block's gram matrix, we add an axis via `[None, ...]` so that we can sum over that axis i.e., the `da.map_blocks` call produces a `(n_obs // chunk_size, n_vars, n_vars)` sized-matrix which is summed over the first dimension. However, to make this work, we need to be very specific about how `da.map_blocks` expects its result to look like, done via `new_axis` and `chunks`. `new_axis` indicates that we are adding a single new axis at the front. The `chunks` argument specifices that the output of `da.map_blocks` should have `x.blocks.size` number of `(1, n_vars, n_vars)` matrixes. This `chunks` argument thus allows the inferral of the shape of the output.
68+
Because `dask` does not implement matrix multiplication for sparse-in-dask, we do it ourselves.
69+
We use `map_blocks` over a CSR sparse-in-dask array where the chunking looks something like `(chunk_size, n_vars)`.
70+
When we compute the invdividual block's gram matrix, we add an axis via `[None, ...]` so that we can sum over that axis i.e., the `da.map_blocks` call produces a `(n_obs // chunk_size, n_vars, n_vars)` sized-matrix which is summed over the first dimension.
71+
However, to make this work, we need to be very specific about how `da.map_blocks` expects its result to look like, done via `new_axis` and `chunks` `new_axis` indicates that we are adding a single new axis at the front.
72+
The `chunks` argument specifices that the output of `da.map_blocks` should have `x.blocks.size` number of `(1, n_vars, n_vars)` matrixes.
73+
This `chunks` argument thus allows the inferral of the shape of the output.
5774

5875
While this example is a bit complicated it shows how you can go from a matrix of one shape and chunking to another by operating in a clean way over blocks.
5976

6077
## FAQ
6178

6279
**What is `persist` for in RSC noteboooks?**
6380

64-
In the {doc}`multi-gpu showcase notebook for rapids-singlecell <rapids-singlecell:notebooks/06-multi_gpu_show>`, {meth}`dask.array.Array.persist` appears across the notebook. This loads the entire dataset into memory while keeping the representation as a dask array. Thus, lazy computation still works but only necessitates a single read into memory. The catch is that you have enough memory to use `persist`.
81+
In the {doc}`multi-gpu showcase notebook for rapids-singlecell <rapids-singlecell:notebooks/06-multi_gpu_show>`, {meth}`dask.array.Array.persist` appears across the notebook.
82+
This loads the entire dataset into memory while keeping the representation as a dask array.
83+
Thus, lazy computation still works but only necessitates a single read into memory.
84+
The catch is that you have enough memory to use `persist`.
6585

6686
**I'm out of memory, what now?**
6787

68-
You can alawys reduce the number of workers you use, which will cause more memory to be allocated per worker. Some algorithms may have limitations with loading all data onto a single node; see {issue}`dask/dask-ml#985` for an example.
88+
You can alawys reduce the number of workers you use, which will cause more memory to be allocated per worker.
89+
Some algorithms may have limitations with loading all data onto a single node; see {issue}`dask/dask-ml#985` for an example.
6990

7091
**How do I choose chunk sizes?**
7192

72-
Have a look at the {doc}`dask docs for chunking <dask:array-chunks>`, however the general rule of thumb there is to use larger chunks in memory than on disk. In this sense, it is probably a good idea to use the largest chunk size in memory allowable by your memory limits (and the algorithms you use) in order to maximize any thread-level parallelization in algorithms to its fullest. For sparse data, where the chunks in-memory do not map to those on disk, maxing out the memory available by choosing a large chunk size becomes more imperative.
93+
Have a look at the {doc}`dask docs for chunking <dask:array-chunks>`, however the general rule of thumb there is to use larger chunks in memory than on disk.
94+
In this sense, it is probably a good idea to use the largest chunk size in memory allowable by your memory limits (and the algorithms you use) in order to maximize any thread-level parallelization in algorithms to its fullest.
95+
For sparse data, where the chunks in-memory do not map to those on disk, maxing out the memory available by choosing a large chunk size becomes more imperative.
7396

7497
[`dask` extension]: https://marketplace.visualstudio.com/items?itemName=joyceerhl.vscode-das
7598
[our custom h5 io code]: https://github.com/scverse/anndata/blob/089ed929393a02200b389395f278b7c920e5bc4a/src/anndata/_io/specs/lazy_methods.py#L179-L20
7699
[`array-api`]: https://data-apis.org/array-api/latest/index.html
77100
[`finch`]: https://github.com/finch-tensor/finch-tensor-python
78-
[multi-gpu showcase notebook]: https://rapids-singlecell.readthedocs.io/en/latest/notebooks/06-multi_gpu_show.html#transferring-anndata-to-gpu-and-persisting-data

0 commit comments

Comments
 (0)