Add Analysis design doc by brian-oneill · Pull Request #446 · E3SM-Project/Omega

brian-oneill · 2026-06-25T23:38:53Z

Add Analysis module design document. Compiled here.

Describes the configurable operator framework for in-situ analysis computation, including:

Templated operators with type-safe runtime dispatch
Operator factory and registration system
Compute scheduling driven by dependency-based alarms
Bundled analysis groups integrated with IOStream output
Future fully composable operator chains (target design)

Checklist

Documentation:
- Design document has been generated and added to the docs
- Documentation has been built locally and changes look as expected

cbegeman · 2026-06-26T15:46:19Z

+```
+
+The factory registers all combinations of scalar type (I4/I8/R4/R8), rank
+(1–5), and memory location (Device/Host/Both) for each operator template:


For my understanding, why is it important to support both device and host locations? Is it that the actual operations would be conducted on the device but we would accept inputs that originate on the host and are copied to device within the operator?

Actually, after thinking it over, I'm leaning towards not supporting HostArray types. Given that the compute work is currently done almost exclusively with ArrayDDTT types (ArrayMemLoc::Device arrays for GPU builds and ArrayMemLoc::Both for CPU-only builds), supporting HostArrayDDTT (ArrayMemLoc::Host) seems unnecessary right now, especially given deadlines.

Adding HostArray support to the analysis code would significantly increase code bloat and complexity. This would require either:

memory transfers (as you mention). This would also complicate the operator templating, since output arrays could no longer be straightforwardly templated off the input arrays

branching compute methods. Each operator compute method would require if-else branches, with duplicate code for kokkos kernels or ordinary loops depending on ArrayMemLoc.

If a compelling need comes up later, we can add this support then. The analysis module will continue to evolve after the immediate delivery, so I think we can avoid adding in this extra complexity prematurely.

Sounds good. I support this simpler path.

cbegeman · 2026-06-28T22:38:42Z

+- `NormalVelocity_SpatialMean_TimeMean1day` — 1-day time average of the
+  spatial mean of NormalVelocity


This suggests that the order of operations is generally reversed in the name. Is that correct (or is the convention stated somewhere in this doc)?

The order of operations here is parsed from left to right. So this translates to:

Fetch the NormalVelocity field -> Apply the SpatialMeanOp across the mesh -> Accumulate the average NormalVelocity value into the TimeMeanOp

By default for now the accumulation interval is the time step, so this whole chain is computed each time step. After a day passes the accumulated values in the TimeMeanOp are finalized by dividing by number of accumulation steps.

This is implied by the pattern: FieldName_Op1[Params]_Op2[Params]... in 4.2, but I could add a statement that makes the order more explicit.

I think your pattern is probably sufficient for the design doc and we can just make this explicit when we write the user doc. Thanks!

This also answers my question above -- we accumulate every time step for now. Make sense to me.

cbegeman

@brian-oneill It's clear you've given this a lot of thought! I don't have much feedback to offer, it's looking like a good plan to me.

sbrus89

@brian-oneill, this looks great. I just had a few comments mainly for my own understanding. Thanks for putting this capability together!

sbrus89 · 2026-06-29T18:21:38Z

+      \mathcal{O}_m\} \leftarrow \texttt{parseOperatorChain}(a)$
+    - **For** $i = 1$ to $m$:
+        - Compute signature: $s \leftarrow \text{sig}(\mathcal{O}_i)$
+        - **If** $s \in \Sigma$ (operator already exists):


I don't think \Sigma has been defined, is it the same as sig?

It's the signature cache, the parenthesis in line 160 first references it. How exactly this will be ultimately be implemented is to be determined, it's essentially a list of strings that fully define unique operator instances.

sbrus89 · 2026-06-29T18:22:18Z

+2. Extract `ArrayDataType`, rank, and `ArrayMemLoc` from Field metadata
+3. Build fully-qualified type key: `OpType + "_" + ArrayTypeName + "_" + MemLoc`
+4. Look up constructor in registry; abort if not found
+5. Invoke constructor with `(UpstreamNames, Options)` and return result


What are UpstreamNames and Options?

The example SpatailMaxOp in 4.1.2 shows an example of the constructor for the derived operators. UpstreamNames is a vector of strings, the names of any input fields to the operator. Options is a Config object. It can be the actual Config read from omega.yml, or I also describe a method for creating temporary Configs for passing parameters needed by an operator during construction, for example on line 466:
// Usage: makeOpConfig(opParam("Period", "1day"), opParam("Layer", 10))

sbrus89 · 2026-06-29T18:28:09Z

+                - **If** $\text{alarm} \notin u.\text{ComputeAlarms}$:
+                    - $u.\text{ComputeAlarms} \leftarrow
+                      u.\text{ComputeAlarms} \cup \{\text{Alarm}\}$
+                     (upstream nodes observe all downstream alarms)


I probably just missed it somewhere, but how/when are the additional output alarms assigned to terminal operators?

Each AnalysisGroup can have 1 or more streams based on configuration, and each stream has a separate output alarm. The example GlobalStats config in the doc has time-averaging periods of 1 day and 1 month, as well as instantaneous snapshots every hour. So three streams get created with alarms with those frequencies, and the terminal operators in the group all get those alarms, which are then propagated to upstream operators.

sbrus89 · 2026-06-29T18:31:46Z

+      Fields: ["NormalVelocity", "Temperature", "Salinity"]
+      SpatialStats: ["Max", "Min", "Mean", "StdDev"]
+      ReductionPeriod: ["1Day", "1Month"]


Just to check if I'm understanding this correctly, but this GlobalStats group would produce 24 different output variables?

Each stream would have 12 outputs, and the group config would create 3 streams: 1 Day averages, 1 Month averages, and 1 Hour Snapshots, so 36 total outputs.

The SampleFreq: ["1Hour"] option is intended to mean discrete sampling i.e., instantaneous snapshots output every hour. The label SampleFreq was one thing I was unsure of. It is a little ambigous because it could be interpreted as an accumulation interval for the temporal averages. Maybe SnapshotFreq would be better?

This may just be an example for generality. But I would expect that we don't want any temporal statistic done on global stats, just spatial stats. They are snapshots in time.

I agree that SnapshotFreq is better than SampleFreq because I assumed that this was indeed the computation interval for temporal stats.

How is the computation interval set (if it is not every time step)?

sbrus89 · 2026-06-29T18:41:02Z

+- `PseudoThickness_SpatialStdDev` — spatial standard deviation of
+  PseudoThickness (implicitly requires `PseudoThickness_SpatialMean` as a
+  shared intermediate)
+


I'm assuming that PseudoThickness_SpatialMean would get created automatically if only PseudoThickness_SpatialStdDev is specified. Would this happen because the SpatialStdDev operator class has a SpatialMean member? Or does the orchestrator handle adding adding a SpatialMean node/dependency?

Ultimately yes, the dependencies would automatically be created. In my current implementation however, the mean needs to be explicitly specified first in order to use the std dev. This is because the PseudoThickness_SpatialMean operator needs to exist when the PseudoThickness_SpatialStdDev is being created. Otherwise, an error during initialization will occur when trying to fetch the PseudoThickness_SpatialMean Field. How to implement these kinds of more complex dependencies automatically is still to be worked out.

sbrus89 · 2026-06-29T18:46:02Z

+Field does not already exist — enabling natural sharing of intermediate
+results without an explicit signature cache.
+
+> **Note on operator chain syntax**: The exact form of operator chain strings


Are there checks in place for invalid chains?

Yes, if a substring from the chain doesn't exist as a Field, or there is no corresponding operator with that name, ABORT_ERROR will be called during initialization.

xylar · 2026-07-01T18:42:12Z

Sorry @brian-oneill, I didn't get to this one today. I'll get to it tomorrow.

xylar

This looks good to me. Thank you @brian-oneill for the hard work on this. I think the current design is exactly the right combination of practical for now and forward-looking.

xylar · 2026-07-02T11:57:15Z

+      Fields: ["NormalVelocity", "Temperature", "Salinity"]
+      SpatialStats: ["Max", "Min", "Mean", "StdDev"]
+      ReductionPeriod: ["1Day", "1Month"]


This may just be an example for generality. But I would expect that we don't want any temporal statistic done on global stats, just spatial stats. They are snapshots in time.

xylar · 2026-07-02T11:58:46Z

+      Fields: ["NormalVelocity", "Temperature", "Salinity"]
+      SpatialStats: ["Max", "Min", "Mean", "StdDev"]
+      ReductionPeriod: ["1Day", "1Month"]


I agree that SnapshotFreq is better than SampleFreq because I assumed that this was indeed the computation interval for temporal stats.

xylar · 2026-07-02T11:59:16Z

+      Fields: ["NormalVelocity", "Temperature", "Salinity"]
+      SpatialStats: ["Max", "Min", "Mean", "StdDev"]
+      ReductionPeriod: ["1Day", "1Month"]


How is the computation interval set (if it is not every time step)?

xylar · 2026-07-02T12:05:23Z

+- `NormalVelocity_SpatialMean_TimeMean1day` — 1-day time average of the
+  spatial mean of NormalVelocity


This also answers my question above -- we accumulate every time step for now. Make sense to me.

add analysis design doc

9ffe4a9

brian-oneill added the Design label Jun 25, 2026

brian-oneill requested review from cbegeman, sbrus89 and xylar June 25, 2026 23:42

cbegeman reviewed Jun 26, 2026

View reviewed changes

cbegeman reviewed Jun 28, 2026

View reviewed changes

cbegeman approved these changes Jun 28, 2026

View reviewed changes

sbrus89 reviewed Jun 29, 2026

View reviewed changes

xylar approved these changes Jul 2, 2026

View reviewed changes

		- `NormalVelocity_SpatialMean_TimeMean1day` — 1-day time average of the
		spatial mean of NormalVelocity

Uh oh!

Conversation

brian-oneill commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbegeman left a comment

Choose a reason for hiding this comment

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xylar commented Jul 1, 2026

Uh oh!

xylar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants