Commit 75737ad
committed
Thread-safe reads via reference-counted mmap cache
Multiple threads reading the same file now share a single read-only
mmap instead of each opening their own. A module-level _MmapCache
protected by a threading.Lock manages reference counts per file path.
The mmap is closed when the last reader releases it.
Read-only mmap slicing (which is what the strip/tile readers do) is
thread-safe at the OS level -- no seek or file position involved.
Tested with 16 concurrent threads reading different windows from the
same deflate+tiled file, and a stress test of 400 reads across 8
threads. Zero errors, cache drains properly.
For dask lazy reads, this means all chunk-read tasks for the same
file share one mmap instead of opening/closing the file per chunk.1 parent 9cf43ab commit 75737ad
1 file changed
+62
-12
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
26 | 84 | | |
27 | | - | |
| 85 | + | |
28 | 86 | | |
29 | 87 | | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
| 88 | + | |
| 89 | + | |
38 | 90 | | |
39 | 91 | | |
40 | 92 | | |
| |||
52 | 104 | | |
53 | 105 | | |
54 | 106 | | |
55 | | - | |
56 | | - | |
57 | | - | |
| 107 | + | |
58 | 108 | | |
59 | 109 | | |
60 | 110 | | |
| |||
0 commit comments