|
| 1 | +# OPeNDAP and NetCDF |
| 2 | + |
| 3 | +Notes on how NetCDF files are laid out and how OPeNDAP lets you read arbitrary |
| 4 | +slices of them over HTTP without downloading the whole file. |
| 5 | + |
| 6 | +## NetCDF in one paragraph |
| 7 | + |
| 8 | +A NetCDF file is a self-describing binary container for N-dimensional arrays |
| 9 | +("variables") plus their coordinate axes ("dimensions") and metadata |
| 10 | +("attributes"). The on-disk format comes in a few flavors: |
| 11 | + |
| 12 | +- **NetCDF-3 "classic"** — a single flat binary file. Variables are stored |
| 13 | + either contiguously (fixed-size dims) or as record-major stripes (one |
| 14 | + unlimited dim). No compression, no chunking. |
| 15 | +- **NetCDF-4** — HDF5 under the hood. Variables can be **chunked** (stored as |
| 16 | + a grid of rectangular blocks) and each chunk can be independently compressed |
| 17 | + (zlib, szip, zstd, etc.). |
| 18 | +- **NetCDF Classic via netCDF-4** — classic data model, HDF5 container. |
| 19 | + |
| 20 | +The key practical difference: NetCDF-3 is linear on disk, so reading "just a |
| 21 | +corner" still requires a lot of seeks and can't be done efficiently over plain |
| 22 | +HTTP range requests (the bytes for a lat/lon box are scattered across the |
| 23 | +file). NetCDF-4 chunking helps, but HTTP range reads still require the client |
| 24 | +to know the HDF5 b-tree layout. Neither format is friendly to "give me this |
| 25 | +bounding box" over a naive HTTP GET. |
| 26 | + |
| 27 | +This is why THREDDS ships OPeNDAP. |
| 28 | + |
| 29 | +## OPeNDAP in one paragraph |
| 30 | + |
| 31 | +OPeNDAP (Open-source Project for a Network Data Access Protocol) is a |
| 32 | +server-side protocol that speaks the NetCDF data model over HTTP. The server |
| 33 | +reads the file, the client asks for a subset using a URL query string, and |
| 34 | +only the requested bytes are sent back — already unpacked from chunks, |
| 35 | +decompressed, and laid out as a simple binary blob the client can splat |
| 36 | +straight into an array. |
| 37 | + |
| 38 | +The client doesn't need to know anything about how the file is stored on the |
| 39 | +server. Chunked HDF5, classic NetCDF-3, aggregated multi-file datasets — the |
| 40 | +protocol hides all of it. |
| 41 | + |
| 42 | +## Anatomy of an OPeNDAP URL |
| 43 | + |
| 44 | +THREDDS exposes the same dataset under several services. The service name is |
| 45 | +the second path segment: |
| 46 | + |
| 47 | +``` |
| 48 | +https://host/thredds/fileServer/path/to/file.nc # whole-file HTTP download |
| 49 | +https://host/thredds/dodsC/path/to/file.nc # OPeNDAP |
| 50 | +https://host/thredds/wcs/path/to/file.nc # Web Coverage Service |
| 51 | +https://host/thredds/wms/path/to/file.nc # Web Map Service |
| 52 | +``` |
| 53 | + |
| 54 | +`dodsC` is the OPeNDAP endpoint (the "C" is for "constrained"). Appending |
| 55 | +suffixes to that base URL retrieves protocol documents: |
| 56 | + |
| 57 | +| Suffix | Returns | |
| 58 | +| ------- | ----------------------------------------------------------- | |
| 59 | +| `.dds` | Dataset Descriptor Structure — variable names, types, shape | |
| 60 | +| `.das` | Dataset Attribute Structure — all attributes | |
| 61 | +| `.dmr` | DAP4 metadata (newer, XML) | |
| 62 | +| `.dods` | Binary data response (DAP2) | |
| 63 | +| `.dap` | Binary data response (DAP4) | |
| 64 | +| `.html` | Human-readable data request form | |
| 65 | +| `.info` | Combined DDS + DAS in HTML | |
| 66 | + |
| 67 | +Visiting the `.html` form in a browser is the fastest way to sanity-check a |
| 68 | +variable's name, shape, and dimension order before coding against it. |
| 69 | + |
| 70 | +## Hyperslab syntax |
| 71 | + |
| 72 | +The request itself is a **constraint expression** appended after `?`. The |
| 73 | +syntax mirrors Fortran/C array slicing: |
| 74 | + |
| 75 | +``` |
| 76 | +<var>[start:stride:stop] |
| 77 | +``` |
| 78 | + |
| 79 | +All indices are **inclusive** and **zero-based**. Multiple variables are |
| 80 | +comma-separated. Examples against an ETOPO-like `(lat, lon)` elevation grid: |
| 81 | + |
| 82 | +``` |
| 83 | +.../file.nc.dods?z # whole variable |
| 84 | +.../file.nc.dods?z[0:1:99][0:1:199] # top-left 100x200 block |
| 85 | +.../file.nc.dods?z[100:1:200][0:10:3600] # middle rows, every 10th col |
| 86 | +.../file.nc.dods?lat,lon # just the coordinate axes |
| 87 | +.../file.nc.dods?z[0:99][0:199],lat[0:99],lon[0:199] # slab + its coords |
| 88 | +``` |
| 89 | + |
| 90 | +You ask in **index space, not coordinate space.** To request a lat/lon bbox |
| 91 | +you first fetch `lat` and `lon` (they're small), find the index range that |
| 92 | +covers your extent, then request `z` with those indices. That two-step — read |
| 93 | +coords, compute indices, read slab — is the fundamental OPeNDAP access |
| 94 | +pattern. |
| 95 | + |
| 96 | +## What the client actually does |
| 97 | + |
| 98 | +Julia's `NCDatasets.jl` (and Python's `netCDF4`, `xarray`, etc.) are built on |
| 99 | +**netCDF-C**, which has a DAP client compiled in. When you open a `dodsC` URL: |
| 100 | + |
| 101 | +```julia |
| 102 | +ds = NCDataset("https://host/thredds/dodsC/.../file.nc") |
| 103 | +z = ds["z"][1000:2000, 500:1500] # Julia 1-based, inclusive |
| 104 | +``` |
| 105 | + |
| 106 | +the library: |
| 107 | + |
| 108 | +1. Fetches `.dds` + `.das` once, on open, and builds an in-memory schema that |
| 109 | + looks exactly like a local NetCDF file. |
| 110 | +2. On `getindex`, translates the Julia slice into a DAP2/DAP4 constraint |
| 111 | + expression and issues a single HTTP GET for `.dods` / `.dap`. |
| 112 | +3. Parses the returned binary payload straight into the output array. |
| 113 | + |
| 114 | +There is no temp file, no second format boundary, and no "download then open" |
| 115 | +step. From the caller's perspective it *is* a NetCDF file; the network is |
| 116 | +invisible until you look at wall-clock time. |
| 117 | + |
| 118 | +Caveats worth knowing: |
| 119 | + |
| 120 | +- **No random-access caching by default.** Every new slice is a new HTTP |
| 121 | + request. Reading in a tight loop over small windows is slow; reading one |
| 122 | + big slab is fast. Size your requests accordingly. |
| 123 | +- **Server-side subset limits.** Many THREDDS deployments cap a single |
| 124 | + response at ~500 MB. Split very large requests into tiles and concatenate |
| 125 | + client-side. |
| 126 | +- **DAP2 can't represent all NetCDF-4 types.** Groups, compound types, and |
| 127 | + strings-of-strings may fall back to DAP4 (`.dmr` / `.dap`) or fail. For |
| 128 | + ordinary gridded float/int arrays this never matters. |
| 129 | +- **Axis order matches the file.** If the file stores `(time, lat, lon)`, |
| 130 | + your Julia slice is `ds["z"][t_idx, lat_idx, lon_idx]` — NCDatasets does |
| 131 | + not reorder axes for you. |
| 132 | +- **Auth and cookies.** OPeNDAP URLs can be behind Earthdata Login or similar. |
| 133 | + netCDF-C honors `~/.netrc` and `~/.dodsrc` for credentials; set those up |
| 134 | + once rather than munging URLs. |
| 135 | + |
| 136 | +## When to reach for OPeNDAP vs alternatives |
| 137 | + |
| 138 | +- **Whole file, no subsetting needed** — use `fileServer` (plain HTTP). |
| 139 | + Simpler, cacheable by CDNs, no server CPU cost. |
| 140 | +- **Bounding-box slice of a gridded variable** — OPeNDAP. Lowest overhead, |
| 141 | + transparent through NCDatasets. |
| 142 | +- **Reprojection or format conversion as part of the fetch** — WCS. You pay |
| 143 | + for server-side GDAL machinery but get back a ready-to-use GeoTIFF. |
| 144 | +- **Cloud-native object stores (S3/GCS)** — Zarr or COG, not OPeNDAP. THREDDS |
| 145 | + is a stateful server; cloud datasets are served as range-readable blobs |
| 146 | + with no server in the loop. |
| 147 | + |
| 148 | +## Further reading |
| 149 | + |
| 150 | +- OPeNDAP DAP2 spec: https://www.opendap.org/pdf/ESE-RFC-004v1.2.pdf |
| 151 | +- DAP4 spec: https://docs.opendap.org/index.php/DAP4:_Specification_Volume_1 |
| 152 | +- THREDDS services reference: https://docs.unidata.ucar.edu/tds/current/userguide/services_ref.html |
| 153 | +- NCDatasets.jl: https://alexander-barth.github.io/NCDatasets.jl/stable/ |
0 commit comments