Skip to content

Commit 30be4d5

Browse files
tomwhitetomwhite
authored
Update slides - August 2025 (#786)
1 parent 3454747 commit 30be4d5

4 files changed

Lines changed: 274 additions & 165 deletions

File tree

‎docs/articles.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Articles and Presentations
22

3-
- [Cubed: an introduction](https://cubed-dev.github.io/cubed/cubed-intro.slides.html) by Tom White, November 2024
3+
- [Cubed: an introduction](https://cubed-dev.github.io/cubed/cubed-intro.slides.html) by Tom White, August 2025
44
- [Optimizing Cubed](https://medium.com/pangeo/optimizing-cubed-7a0b8f65f5b7) by Tom White, April 3, 2024
55
- [Cubed: Bounded-memory serverless array processing in xarray](https://xarray.dev/blog/cubed-xarray) by Tom Nicholas and Tom White, June 28th, 2023

‎docs/slides/intro/cubed-intro.ipynb‎

Lines changed: 71 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
"source": [
1212
"# Cubed: an introduction\n",
1313
"\n",
14-
"Tom White, November 2024"
14+
"Tom White, August 2025"
1515
]
1616
},
1717
{
@@ -238,8 +238,15 @@
238238
"outputs": [
239239
{
240240
"data": {
241-
"image/svg+xml": [
242-
"<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"230pt\" height=\"319pt\" viewBox=\"0.00 0.00 229.75 318.75\">\n",
241+
"text/html": [
242+
"<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
243+
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
244+
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
245+
"<!-- Generated by graphviz version 12.2.1 (20241206.2353)\n",
246+
" -->\n",
247+
"<!-- Pages: 1 -->\n",
248+
"<svg width=\"230pt\" height=\"319pt\"\n",
249+
" viewBox=\"0.00 0.00 229.75 318.75\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
243250
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 314.75)\">\n",
244251
"<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-314.75 225.75,-314.75 225.75,4 -4,4\"/>\n",
245252
"<text text-anchor=\"start\" x=\"8\" y=\"-39.5\" font-family=\"Times,serif\" font-size=\"10.00\">num tasks: 5</text>\n",
@@ -248,132 +255,101 @@
248255
"<text text-anchor=\"start\" x=\"8\" y=\"-5.75\" font-family=\"Times,serif\" font-size=\"10.00\">optimized: True</text>\n",
249256
"<!-- op&#45;001 -->\n",
250257
"<g id=\"node1\" class=\"node\">\n",
251-
"<title>op-001</title>\n",
252-
"<g id=\"a_node1\"><a xlink:title=\"name: op-001\n",
253-
"op: asarray\n",
254-
"calls: &lt;module&gt; -&gt; asarray\n",
255-
"line: 2 in &lt;module&gt;\">\n",
258+
"<title>op&#45;001</title>\n",
259+
"<g id=\"a_node1\"><a xlink:title=\"name: op&#45;001&#10;op: asarray&#10;calls: &lt;module&gt; &#45;&gt; asarray&#10;line: 2 in &lt;module&gt;\">\n",
256260
"<path fill=\"none\" stroke=\"black\" d=\"M43.25,-310.75C43.25,-310.75 13.25,-310.75 13.25,-310.75 7.25,-310.75 1.25,-304.75 1.25,-298.75 1.25,-298.75 1.25,-286.75 1.25,-286.75 1.25,-280.75 7.25,-274.75 13.25,-274.75 13.25,-274.75 43.25,-274.75 43.25,-274.75 49.25,-274.75 55.25,-280.75 55.25,-286.75 55.25,-286.75 55.25,-298.75 55.25,-298.75 55.25,-304.75 49.25,-310.75 43.25,-310.75\"/>\n",
257-
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op-001</text>\n",
261+
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op&#45;001</text>\n",
258262
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-283.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">asarray</text>\n",
259263
"</a>\n",
260264
"</g>\n",
261265
"</g>\n",
262266
"<!-- array&#45;001 -->\n",
263267
"<g id=\"node2\" class=\"node\">\n",
264-
"<title>array-001</title>\n",
265-
"<g id=\"a_node2\"><a xlink:title=\"name: array-001\n",
266-
"variable: a\n",
267-
"shape: (3, 3)\n",
268-
"chunks: (2, 2)\n",
269-
"dtype: int64\n",
270-
"chunk memory: 32 bytes\">\n",
268+
"<title>array&#45;001</title>\n",
269+
"<g id=\"a_node2\"><a xlink:title=\"name: array&#45;001&#10;variable: a&#10;shape: (3, 3)&#10;chunks: (2, 2)&#10;dtype: int64&#10;chunk memory: 32 bytes&#10;nbytes: 72 bytes&#10;nchunks: 4\">\n",
271270
"<polygon fill=\"none\" stroke=\"black\" points=\"56.5,-238.75 0,-238.75 0,-202.75 56.5,-202.75 56.5,-238.75\"/>\n",
272-
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-222.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-001</text>\n",
271+
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-222.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array&#45;001</text>\n",
273272
"<text text-anchor=\"middle\" x=\"28.25\" y=\"-211.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">a</text>\n",
274273
"</a>\n",
275274
"</g>\n",
276275
"</g>\n",
277276
"<!-- op&#45;001&#45;&gt;array&#45;001 -->\n",
278277
"<g id=\"edge1\" class=\"edge\">\n",
279-
"<title>op-001-&gt;array-001</title>\n",
278+
"<title>op&#45;001&#45;&gt;array&#45;001</title>\n",
280279
"<path fill=\"none\" stroke=\"black\" d=\"M28.25,-274.45C28.25,-267.16 28.25,-258.48 28.25,-250.29\"/>\n",
281280
"<polygon fill=\"black\" stroke=\"black\" points=\"31.75,-250.37 28.25,-240.37 24.75,-250.37 31.75,-250.37\"/>\n",
282281
"</g>\n",
283282
"<!-- op&#45;004 -->\n",
284283
"<g id=\"node5\" class=\"node\">\n",
285-
"<title>op-004</title>\n",
286-
"<g id=\"a_node5\"><a xlink:title=\"name: op-004\n",
287-
"op: blockwise\n",
288-
"projected memory: 100.0 MB\n",
289-
"tasks: 4\n",
290-
"num input blocks: (1, 1)\n",
291-
"calls: &lt;module&gt; -&gt; add -&gt; elemwise -&gt; blockwise\n",
292-
"line: 1 in &lt;module&gt;\">\n",
284+
"<title>op&#45;004</title>\n",
285+
"<g id=\"a_node5\"><a xlink:title=\"name: op&#45;004&#10;op: blockwise&#10;projected memory: 100.0 MB&#10;tasks: 4&#10;write chunks: (2, 2)&#10;num input blocks: (1, 1)&#10;num output blocks: (1,)&#10;calls: &lt;module&gt; &#45;&gt; add &#45;&gt; elemwise &#45;&gt; blockwise&#10;line: 1 in &lt;module&gt;\">\n",
293286
"<path fill=\"#dcbeff\" stroke=\"black\" d=\"M80.25,-166.75C80.25,-166.75 50.25,-166.75 50.25,-166.75 44.25,-166.75 38.25,-160.75 38.25,-154.75 38.25,-154.75 38.25,-137 38.25,-137 38.25,-131 44.25,-125 50.25,-125 50.25,-125 80.25,-125 80.25,-125 86.25,-125 92.25,-131 92.25,-137 92.25,-137 92.25,-154.75 92.25,-154.75 92.25,-160.75 86.25,-166.75 80.25,-166.75\"/>\n",
294-
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-153.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op-004</text>\n",
287+
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-153.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op&#45;004</text>\n",
295288
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-142\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">add</text>\n",
296289
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-130.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">tasks: 4</text>\n",
297290
"</a>\n",
298291
"</g>\n",
299292
"</g>\n",
300293
"<!-- array&#45;001&#45;&gt;op&#45;004 -->\n",
301294
"<g id=\"edge2\" class=\"edge\">\n",
302-
"<title>array-001-&gt;op-004</title>\n",
295+
"<title>array&#45;001&#45;&gt;op&#45;004</title>\n",
303296
"<path fill=\"none\" stroke=\"black\" d=\"M37.02,-202.48C40.85,-194.94 45.45,-185.87 49.82,-177.26\"/>\n",
304297
"<polygon fill=\"black\" stroke=\"black\" points=\"52.88,-178.97 54.28,-168.47 46.64,-175.8 52.88,-178.97\"/>\n",
305298
"</g>\n",
306299
"<!-- op&#45;002 -->\n",
307300
"<g id=\"node3\" class=\"node\">\n",
308-
"<title>op-002</title>\n",
309-
"<g id=\"a_node3\"><a xlink:title=\"name: op-002\n",
310-
"op: asarray\n",
311-
"calls: &lt;module&gt; -&gt; asarray\n",
312-
"line: 1 in &lt;module&gt;\">\n",
301+
"<title>op&#45;002</title>\n",
302+
"<g id=\"a_node3\"><a xlink:title=\"name: op&#45;002&#10;op: asarray&#10;calls: &lt;module&gt; &#45;&gt; asarray&#10;line: 1 in &lt;module&gt;\">\n",
313303
"<path fill=\"none\" stroke=\"black\" d=\"M118.25,-310.75C118.25,-310.75 88.25,-310.75 88.25,-310.75 82.25,-310.75 76.25,-304.75 76.25,-298.75 76.25,-298.75 76.25,-286.75 76.25,-286.75 76.25,-280.75 82.25,-274.75 88.25,-274.75 88.25,-274.75 118.25,-274.75 118.25,-274.75 124.25,-274.75 130.25,-280.75 130.25,-286.75 130.25,-286.75 130.25,-298.75 130.25,-298.75 130.25,-304.75 124.25,-310.75 118.25,-310.75\"/>\n",
314-
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op-002</text>\n",
304+
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">op&#45;002</text>\n",
315305
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-283.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">asarray</text>\n",
316306
"</a>\n",
317307
"</g>\n",
318308
"</g>\n",
319309
"<!-- array&#45;002 -->\n",
320310
"<g id=\"node4\" class=\"node\">\n",
321-
"<title>array-002</title>\n",
322-
"<g id=\"a_node4\"><a xlink:title=\"name: array-002\n",
323-
"variable: b\n",
324-
"shape: (3, 3)\n",
325-
"chunks: (2, 2)\n",
326-
"dtype: int64\n",
327-
"chunk memory: 32 bytes\">\n",
311+
"<title>array&#45;002</title>\n",
312+
"<g id=\"a_node4\"><a xlink:title=\"name: array&#45;002&#10;variable: b&#10;shape: (3, 3)&#10;chunks: (2, 2)&#10;dtype: int64&#10;chunk memory: 32 bytes&#10;nbytes: 72 bytes&#10;nchunks: 4\">\n",
328313
"<polygon fill=\"none\" stroke=\"black\" points=\"131.5,-238.75 75,-238.75 75,-202.75 131.5,-202.75 131.5,-238.75\"/>\n",
329-
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-222.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-002</text>\n",
314+
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-222.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array&#45;002</text>\n",
330315
"<text text-anchor=\"middle\" x=\"103.25\" y=\"-211.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">b</text>\n",
331316
"</a>\n",
332317
"</g>\n",
333318
"</g>\n",
334319
"<!-- op&#45;002&#45;&gt;array&#45;002 -->\n",
335320
"<g id=\"edge3\" class=\"edge\">\n",
336-
"<title>op-002-&gt;array-002</title>\n",
321+
"<title>op&#45;002&#45;&gt;array&#45;002</title>\n",
337322
"<path fill=\"none\" stroke=\"black\" d=\"M103.25,-274.45C103.25,-267.16 103.25,-258.48 103.25,-250.29\"/>\n",
338323
"<polygon fill=\"black\" stroke=\"black\" points=\"106.75,-250.37 103.25,-240.37 99.75,-250.37 106.75,-250.37\"/>\n",
339324
"</g>\n",
340325
"<!-- array&#45;002&#45;&gt;op&#45;004 -->\n",
341326
"<g id=\"edge4\" class=\"edge\">\n",
342-
"<title>array-002-&gt;op-004</title>\n",
327+
"<title>array&#45;002&#45;&gt;op&#45;004</title>\n",
343328
"<path fill=\"none\" stroke=\"black\" d=\"M94.24,-202.48C90.31,-194.94 85.58,-185.87 81.09,-177.26\"/>\n",
344329
"<polygon fill=\"black\" stroke=\"black\" points=\"84.24,-175.71 76.51,-168.47 78.03,-178.95 84.24,-175.71\"/>\n",
345330
"</g>\n",
346331
"<!-- array&#45;004 -->\n",
347332
"<g id=\"node6\" class=\"node\">\n",
348-
"<title>array-004</title>\n",
349-
"<g id=\"a_node6\"><a xlink:title=\"name: array-004\n",
350-
"variable: c\n",
351-
"shape: (3, 3)\n",
352-
"chunks: (2, 2)\n",
353-
"dtype: int64\n",
354-
"chunk memory: 32 bytes\n",
355-
"nbytes: 72 bytes\">\n",
333+
"<title>array&#45;004</title>\n",
334+
"<g id=\"a_node6\"><a xlink:title=\"name: array&#45;004&#10;variable: c&#10;shape: (3, 3)&#10;chunks: (2, 2)&#10;dtype: int64&#10;chunk memory: 32 bytes&#10;nbytes: 72 bytes&#10;nchunks: 4\">\n",
356335
"<polygon fill=\"#ffd8b1\" stroke=\"black\" points=\"93.5,-89 37,-89 37,-53 93.5,-53 93.5,-89\"/>\n",
357-
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-72.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array-004</text>\n",
336+
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-72.75\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">array&#45;004</text>\n",
358337
"<text text-anchor=\"middle\" x=\"65.25\" y=\"-61.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">c</text>\n",
359338
"</a>\n",
360339
"</g>\n",
361340
"</g>\n",
362341
"<!-- op&#45;004&#45;&gt;array&#45;004 -->\n",
363342
"<g id=\"edge5\" class=\"edge\">\n",
364-
"<title>op-004-&gt;array-004</title>\n",
343+
"<title>op&#45;004&#45;&gt;array&#45;004</title>\n",
365344
"<path fill=\"none\" stroke=\"black\" d=\"M65.25,-124.58C65.25,-117.19 65.25,-108.7 65.25,-100.73\"/>\n",
366345
"<polygon fill=\"black\" stroke=\"black\" points=\"68.75,-100.74 65.25,-90.74 61.75,-100.74 68.75,-100.74\"/>\n",
367346
"</g>\n",
368347
"<!-- create&#45;arrays -->\n",
369348
"<g id=\"node7\" class=\"node\">\n",
370-
"<title>create-arrays</title>\n",
371-
"<g id=\"a_node7\"><a xlink:title=\"name: create-arrays\n",
372-
"op: create-arrays\n",
373-
"projected memory: 100.0 MB\n",
374-
"tasks: 1\">\n",
349+
"<title>create&#45;arrays</title>\n",
350+
"<g id=\"a_node7\"><a xlink:title=\"name: create&#45;arrays&#10;op: create&#45;arrays&#10;projected memory: 100.0 MB&#10;tasks: 1\">\n",
375351
"<path fill=\"none\" stroke=\"black\" d=\"M209.75,-310.75C209.75,-310.75 160.75,-310.75 160.75,-310.75 154.75,-310.75 148.75,-304.75 148.75,-298.75 148.75,-298.75 148.75,-286.75 148.75,-286.75 148.75,-280.75 154.75,-274.75 160.75,-274.75 160.75,-274.75 209.75,-274.75 209.75,-274.75 215.75,-274.75 221.75,-280.75 221.75,-286.75 221.75,-286.75 221.75,-298.75 221.75,-298.75 221.75,-304.75 215.75,-310.75 209.75,-310.75\"/>\n",
376-
"<text text-anchor=\"middle\" x=\"185.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">create-arrays</text>\n",
352+
"<text text-anchor=\"middle\" x=\"185.25\" y=\"-294.5\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">create&#45;arrays</text>\n",
377353
"<text text-anchor=\"middle\" x=\"185.25\" y=\"-283.25\" font-family=\"Helvetica,sans-Serif\" font-size=\"10.00\">tasks: 1</text>\n",
378354
"</a>\n",
379355
"</g>\n",
@@ -389,15 +365,15 @@
389365
"</g>\n",
390366
"<!-- create&#45;arrays&#45;&gt;arrays -->\n",
391367
"<g id=\"edge6\" class=\"edge\">\n",
392-
"<title>create-arrays-&gt;arrays</title>\n",
368+
"<title>create&#45;arrays&#45;&gt;arrays</title>\n",
393369
"<path fill=\"none\" stroke=\"black\" d=\"M185.25,-274.45C185.25,-267.16 185.25,-258.48 185.25,-250.29\"/>\n",
394370
"<polygon fill=\"black\" stroke=\"black\" points=\"188.75,-250.37 185.25,-240.37 181.75,-250.37 188.75,-250.37\"/>\n",
395371
"</g>\n",
396372
"</g>\n",
397-
"</svg>"
373+
"</svg>\n"
398374
],
399375
"text/plain": [
400-
"<IPython.core.display.SVG object>"
376+
"<IPython.core.display.HTML object>"
401377
]
402378
},
403379
"execution_count": 4,
@@ -455,6 +431,20 @@
455431
"More details in [Optimizing Cubed](https://medium.com/pangeo/optimizing-cubed-7a0b8f65f5b7)\n"
456432
]
457433
},
434+
{
435+
"cell_type": "markdown",
436+
"id": "08a78436",
437+
"metadata": {
438+
"slideshow": {
439+
"slide_type": "slide"
440+
}
441+
},
442+
"source": [
443+
"# Quadratic Means timeline\n",
444+
"\n",
445+
"![QM timeline](qm-timeline.png)"
446+
]
447+
},
458448
{
459449
"cell_type": "markdown",
460450
"id": "ba00f446",
@@ -551,60 +541,24 @@
551541
},
552542
{
553543
"cell_type": "markdown",
554-
"id": "d5a1fddd",
555-
"metadata": {
556-
"slideshow": {
557-
"slide_type": "fragment"
558-
}
559-
},
560-
"source": [
561-
"* __Lithops__: multi-cloud serverless computing framework\n",
562-
" * Slightly more work to get started since you have to build a runtime environment first\n",
563-
" * Tested on AWS Lambda and Google Cloud Functions with ~1000 workers"
564-
]
565-
},
566-
{
567-
"cell_type": "markdown",
568-
"id": "b1fb4379",
569-
"metadata": {
570-
"slideshow": {
571-
"slide_type": "fragment"
572-
}
573-
},
574-
"source": [
575-
"* __Modal__: a commercial serverless platform\n",
576-
" * Very easy to set up since it builds the runtime automatically\n",
577-
" * Tested with ~300 workers"
578-
]
579-
},
580-
{
581-
"cell_type": "markdown",
582-
"id": "edcd5b33",
583-
"metadata": {
584-
"slideshow": {
585-
"slide_type": "fragment"
586-
}
587-
},
588-
"source": [
589-
"* __Apache Beam (Google Cloud Dataflow)__: fully managed pipeline processing service\n",
590-
" * Slow to spin up\n",
591-
" * Only tested with ~20 workers, but very mature so will scale out"
592-
]
593-
},
594-
{
595-
"cell_type": "markdown",
596-
"id": "08a78436",
597-
"metadata": {
598-
"slideshow": {
599-
"slide_type": "slide"
600-
}
601-
},
544+
"id": "aac6fd84-b914-40d0-b3f5-1adc0de210fe",
545+
"metadata": {},
602546
"source": [
603-
"# Example timeline\n",
604-
"\n",
605-
"Adding two 20GB arrays on Lithops (AWS Lambda)\n",
606-
"\n",
607-
"![Lithops timeline](cubed-lithops-timeline.png)"
547+
"* __Local machine__\n",
548+
" * Start here to process hundreds of GB on your laptop using all available cores\n",
549+
"* __Serverless__\n",
550+
" * __Lithops__: multi-cloud serverless computing framework\n",
551+
" * Slightly more work to get started since you have to build a runtime environment first\n",
552+
" * Tested on AWS Lambda and Google Cloud Functions with ~1000 workers\n",
553+
" * __Modal__: a commercial serverless platform\n",
554+
" * Very easy to set up since it builds the runtime automatically\n",
555+
" * Tested with ~300 workers\n",
556+
" * __Coiled Functions__\n",
557+
"* __Cluster/HPC__\n",
558+
" * __Ray__\n",
559+
" * __Apache Beam (Google Cloud Dataflow)__\n",
560+
" * __Globus Compute__\n",
561+
" * __Apache Spark__"
608562
]
609563
},
610564
{
@@ -679,7 +633,7 @@
679633
"name": "python",
680634
"nbconvert_exporter": "python",
681635
"pygments_lexer": "ipython3",
682-
"version": "3.11.0"
636+
"version": "3.12.11"
683637
}
684638
},
685639
"nbformat": 4,

0 commit comments

Comments
 (0)