Skip to content

Commit efb6abb

Browse files
coreyjadamsgreptile-apps[bot]
coreyjadams
and
greptile-apps[bot]
authored
Pnm mesh datapipes (#1512)
* Adding mesh datapipe temporarily * first, very early, draft of mesh-based datapipe * Update datapipes for physicsnemo mesh: now can read and transform surface and volumetric meshes directly from physicsnemo mesh files. * Add a functionally working - but not yet cxonverging - xaero recipe with multiple mesh based datapipes * Add functional physicsnemo mesh pipeline * Update README * Snapshotting training recipe * Remove Ahmed dataset with physicsnemo mesh, it's not needed. * Simplify and consolidate, or remove statistics work for this example. * Remove inspect_data, and ensure readme is updated * Remove AGENT.md * Training recipe is basically complete * Update physicsnemo/datapipes/readers/mesh.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update physicsnemo/datapipes/transforms/mesh/transforms.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update physicsnemo/datapipes/readers/mesh.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Addressing some PR comments. * make sure rotations are flexible, add docstrings * Update distribution handling per PR review. * This commit enablers the datapipes to be reproducible * Add testing for the datapipes * Add missing rng updates * Update training python codes * Add geotransolver configs with obscured paths * Remove shift suv configs for now * Adding configs for flare and geotransolver with a flare backend. * Update README.md * Add datapipes.md. Describes some of the computational design of the datapipes at a high level. * Update datapipes to use non-blocking stream syncs when streams are active. * Fix cuda memory access errors arising from incorrect fake tensor ops. fix torch compile bug from incorrect arg list. GeoTransovler compiles again with local features enabled. * Adding datapipe tests for new functionality for mesh datapipe loader * Trying to close down github comments * Use apply instead of _map_meshes * Make sure poisson subsampling has no duplicates if replacement=false * merging * Update comments to .pmsh or .pdmsh instead of .pt for physicsnemo dataformats * Migrate generators ad distributions in transforms.base too * Add tests for distribution and generator migration * Update rmse to relative mse naming, and make sure huber is default in loss function * Fix DDP bug with drop last, and consolidate data paths to unified location to manage file paths much more simply * Use match-case for loud and explicit failure, and default to uniform, in mesh rotation augmentation * Move re-dimensionalization to utils until inference script is added in a future PR * Use a simpler parametrization to create random rotation matrices * undo uv.lock changes * Revert uv.lock to main * Add ruff corrections * Update change log and fix loss function. --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
1 parent 613d768 commit efb6abb

70 files changed

Lines changed: 11730 additions & 376 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.markdownlintignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
CODE_OF_CONDUCT.md
2+
physicsnemo/datapipes/transforms/mesh/DISTRIBUTIONS.md

‎CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
6868
- In PhysicsNeMo-Mesh, `physicsnemo.mesh.geometry` now publicly exposes
6969
`stable_angle_between_vectors` and `compute_triangle_angles` (previously
7070
only available via the private `physicsnemo.mesh.curvature._utils`).
71+
- PhysicsNeMo Datapipes enables reproducability through `torch.generator`
72+
utilities.
73+
- PhysicsNeMo Datapipes now supports `physicsnemo.mesh.Mesh` and
74+
`physicsnemo.mesh.DomainMesh` objects for deserialization, with
75+
transformations and utilities for mesh-based datasets.
76+
- PhysicsNeMo Datapipes now support `MultiDataset` construction,
77+
allowing on-the-fly construction of multi-source composite datasets
78+
that can be sampled and processed efficiently and coherently
79+
as one dataset.
80+
- PhysicsNeMo Datapipes also support random augmentations for
81+
mesh-based datapipes, leveraging `torch.distributions` for
82+
broad random distribution support. Mesh and DomainMesh
83+
datasets allow random translation, scaling, and rotation
84+
of mesh data in coherent ways, compatible with reproducability
85+
features of physicsnemo datapipes.
86+
- Adds a new *unified* training recipe for external aerodynamics
87+
that supports training on multiple datasets (DrivaerML, ShiftSUV,
88+
HighLiftAeroML, or more, bring your own, mix and match), supports
89+
training several different models (Domino, Transolver, GeoTransolver,
90+
Flare, GeoTransolver with Flare-attention, bring your own!). Leverages
91+
mesh datasets and non-dimensionalization to enable dataset mixing and
92+
matching at runtime. Train with surface or volume data.
7193

7294
### Changed
7395

@@ -149,6 +171,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
149171
`smooth_laplacian` and `compute_quality_metrics` have been replaced
150172
with the dtype-aware `.clamp(min=safe_eps(dtype))` to avoid silently
151173
zeroing fp16 weights.
174+
- Fixed issues with physicsnemo.nn.functional's `radius_search` that
175+
caused crashes when used with torch.compile.
152176

153177
### Security
154178

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/.gitignore‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
*.sh
2+
runs/
3+
mlruns/
4+
*.err
5+
*.out
6+
outputs/
7+
stats/
8+
checkpoints/
9+
*.mdlus
10+
*.tfevents*
11+
*.parquet

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/README.md‎

Lines changed: 543 additions & 0 deletions
Large diffs are not rendered by default.

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/conf/dataset/dataset_paths.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES.
2+
# SPDX-FileCopyrightText: All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# Dataset root directories.
18+
# Fill in the paths below to point to your local dataset locations.
19+
# Use ??? as a placeholder for paths you have not yet configured.
20+
# OmegaConf will raise MissingMandatoryValue at runtime for any
21+
# path that is still ???.
22+
23+
drivaer_ml: ???
24+
highlift_aero_ml: ???
25+
shift_suv: ???

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/conf/dataset/drivaer_ml_surface.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES.
2+
# SPDX-FileCopyrightText: All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# DrivaerML surface dataset
18+
# Reads the boundary surface Mesh directly from DomainMesh .pdmsh files
19+
# by navigating into the on-disk tensordict directory structure.
20+
# Triangulated surface mesh. Fields live in cell_data.
21+
# Splits are controlled by manifest.json + train_split/val_split in the training config.
22+
23+
name: drivaer_ml_surface
24+
25+
train_datadir: ${dataset_paths.drivaer_ml}
26+
27+
metadata:
28+
U_inf: [30.0, 0.0, 0.0] # freestream velocity
29+
p_inf: 0.0 # reference pressure, drivaerml is gauge pressure
30+
rho_inf: 1.225 # freestream density
31+
nu: 1
32+
L_ref: 5.0 # reference length [m]
33+
34+
pipeline:
35+
reader:
36+
_target_: ${dp:MeshReader}
37+
path: ${train_datadir}
38+
pattern: "**/*.pdmsh/_tensordict/boundaries/surface"
39+
subsample_n_cells: ${sampling_resolution}
40+
augmentations:
41+
- _target_: ${dp:RandomRotateMesh}
42+
axes: ["z"]
43+
transform_cell_data: true
44+
transform_global_data: true
45+
- _target_: ${dp:RandomTranslateMesh}
46+
distribution:
47+
_target_: torch.distributions.Uniform
48+
low: [-1.0, -1.0, 0.0]
49+
high: [1.0, 1.0, 0.0]
50+
transforms:
51+
- _target_: ${dp:DropMeshFields}
52+
global_data: [TimeValue]
53+
- _target_: ${dp:CenterMesh}
54+
use_area_weighting: false
55+
- _target_: ${dp:NonDimensionalizeByMetadata}
56+
fields:
57+
pMeanTrim: pressure
58+
wallShearStressMeanTrim: stress
59+
section: cell_data
60+
- _target_: ${dp:RenameMeshFields}
61+
cell_data:
62+
pMeanTrim: pressure
63+
wallShearStressMeanTrim: wss
64+
- _target_: ${dp:NormalizeMeshFields}
65+
section: cell_data
66+
fields:
67+
wss: {type: vector, mean: [0.0, 0.0, 0.0], std: 0.00313}
68+
- _target_: ${dp:ComputeSurfaceNormals}
69+
store_as: cell_data
70+
field_name: normals
71+
- _target_: ${dp:SubsampleMesh}
72+
n_cells: ${sampling_resolution}
73+
- _target_: ${dp:MeshToTensorDict}
74+
- _target_: ${dp:ComputeCellCentroids}
75+
- _target_: ${dp:RestructureTensorDict}
76+
groups:
77+
input:
78+
points: cell_centroids
79+
normals: cell_data.normals
80+
U_inf: global_data.U_inf
81+
output:
82+
pressure: cell_data.pressure
83+
wss: cell_data.wss
84+
85+
targets:
86+
pressure: scalar
87+
wss: vector
88+
89+
metrics: [l1, l2, mae]

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/conf/dataset/drivaer_ml_volume.yaml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES.
2+
# SPDX-FileCopyrightText: All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# DrivaerML volume dataset (DomainMesh)
18+
# Loads domain_*.pmsh files containing interior volume mesh + surface boundary + global_data.
19+
# Interior: tetrahedral volume mesh with fields in point_data.
20+
# Boundary: triangulated surface mesh ("surface").
21+
# global_data: U_inf, rho_inf (baked into .pmsh); p_inf, nu, L_ref injected from metadata below.
22+
# Splits are controlled by manifest.json + train_split/val_split in the training config.
23+
24+
name: drivaer_ml_volume
25+
26+
train_datadir: ${dataset_paths.drivaer_ml}
27+
28+
metadata:
29+
U_inf: [30.0, 0.0, 0.0] # freestream velocity
30+
p_inf: 0.0 # reference pressure, drivaerml is gauge pressure
31+
rho_inf: 1.225 # freestream density
32+
nu: 1
33+
L_ref: 5.0 # reference length [m]
34+
35+
pipeline:
36+
reader:
37+
_target_: ${dp:DomainMeshReader}
38+
path: ${train_datadir}
39+
pattern: "**/domain_*.pdmsh"
40+
subsample_n_points: ${sampling_resolution}
41+
subsample_n_cells: ${sampling_resolution}
42+
extra_boundaries:
43+
stl_geometry:
44+
pattern: "*_single_solid.stl.pmsh"
45+
augmentations:
46+
- _target_: ${dp:RandomRotateMesh}
47+
axes: ["z"]
48+
transform_point_data: true
49+
transform_global_data: true
50+
- _target_: ${dp:RandomTranslateMesh}
51+
distribution:
52+
_target_: torch.distributions.Uniform
53+
low: [-1.0, -1.0, 0.0]
54+
high: [1.0, 1.0, 0.0]
55+
transforms:
56+
- _target_: ${dp:DropMeshFields}
57+
global_data: [TimeValue]
58+
- _target_: ${dp:CenterMesh}
59+
use_area_weighting: false
60+
- _target_: ${dp:NonDimensionalizeByMetadata}
61+
fields:
62+
UMeanTrim: velocity
63+
pMeanTrim: pressure
64+
nutMeanTrim: identity
65+
section: point_data
66+
- _target_: ${dp:ComputeSDFFromBoundary}
67+
boundary_name: stl_geometry
68+
sdf_field: sdf
69+
normals_field: sdf_normals
70+
use_winding_number: true
71+
- _target_: ${dp:DropBoundary}
72+
names: [stl_geometry]
73+
- _target_: ${dp:RenameMeshFields}
74+
point_data:
75+
UMeanTrim: velocity
76+
pMeanTrim: pressure
77+
nutMeanTrim: nut
78+
- _target_: ${dp:NormalizeMeshFields}
79+
section: point_data
80+
fields:
81+
nut: {type: scalar, mean: 4.8e-4, std: 9.4e-4}
82+
- _target_: ${dp:MeshToTensorDict}
83+
- _target_: ${dp:RestructureTensorDict}
84+
groups:
85+
input:
86+
points: interior.points
87+
U_inf: global_data.U_inf
88+
sdf: interior.point_data.sdf
89+
sdf_normals: interior.point_data.sdf_normals
90+
output:
91+
velocity: interior.point_data.velocity
92+
pressure: interior.point_data.pressure
93+
nut: interior.point_data.nut
94+
95+
targets:
96+
velocity: vector
97+
pressure: scalar
98+
nut: scalar
99+
100+
metrics: [l1, l2, mae]

‎examples/cfd/external_aerodynamics/unified_external_aero_recipe/conf/dataset/highlift_surface.yaml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES.
2+
# SPDX-FileCopyrightText: All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
# High-lift airplane surface dataset (compressible, cell-centric)
18+
# Reads the boundary surface Mesh directly from DomainMesh .pdmsh files
19+
# by navigating into the on-disk tensordict directory structure.
20+
# Boundary mesh from HiLiftAeroML. Physics fields live in cell_data
21+
# (derived from vertex-projected point_data during conversion).
22+
# Model inputs use cell centroids and face normals; topology (triangles)
23+
# is preserved for normal computation and cell-based subsampling.
24+
# Splits are controlled by manifest.json + train_split/val_split in the training config.
25+
26+
name: highlift_surface
27+
28+
train_datadir: ${dataset_paths.highlift_aero_ml}
29+
30+
# Freestream conditions — slug-inch-second-Rankine unit system
31+
metadata:
32+
U_inf: [2672.95, 0.0, 186.92] # 2679.505 in/s at AoA=4°
33+
p_inf: 176.352 # 14.696 psi × 12 → slug/(in·s²)
34+
rho_inf: 1.3756e-6 # 0.002377 slug/ft³ → slug/in³
35+
T_inf: 518.67 # [°R]
36+
L_ref: 1156.75 # [in] — span
37+
38+
pipeline:
39+
reader:
40+
_target_: ${dp:MeshReader}
41+
path: ${train_datadir}
42+
pattern: "geo_LHC*/*.pdmsh/_tensordict/boundaries/boundary"
43+
subsample_n_cells: ${sampling_resolution}
44+
augmentations:
45+
- _target_: ${dp:RandomRotateMesh}
46+
axes: ["z"]
47+
transform_point_data: true
48+
transform_cell_data: true
49+
transform_global_data: true
50+
- _target_: ${dp:RandomTranslateMesh}
51+
distribution:
52+
_target_: torch.distributions.Uniform
53+
low: [-1.0, -1.0, 0.0]
54+
high: [1.0, 1.0, 0.0]
55+
transforms:
56+
- _target_: ${dp:CenterMesh}
57+
use_area_weighting: false
58+
- _target_: ${dp:ComputeSurfaceNormals}
59+
store_as: cell_data
60+
field_name: normals
61+
- _target_: ${dp:NonDimensionalizeByMetadata}
62+
fields:
63+
PROJ(AVG(P)): pressure
64+
PROJ(AVG(T)): temperature
65+
PROJ(AVG(RHO)): density
66+
PROJ(AVG(U)): velocity
67+
AVG(TAU_WALL): stress
68+
section: cell_data
69+
- _target_: ${dp:RenameMeshFields}
70+
cell_data:
71+
PROJ(AVG(P)): pressure
72+
PROJ(AVG(T)): temperature
73+
PROJ(AVG(RHO)): density
74+
PROJ(AVG(U)): velocity
75+
AVG(TAU_WALL): tau_wall
76+
# z-score normalization (dataset-wide stats)
77+
# Stats are post-nondimensionalization values.
78+
# TODO: recompute stats for cell-centered data (these are vertex-projected estimates)
79+
- _target_: ${dp:NormalizeMeshFields}
80+
section: cell_data
81+
fields:
82+
pressure: {type: scalar, mean: -0.5, std: 1.0}
83+
temperature: {type: scalar, mean: 1.00, std: 0.002}
84+
density: {type: scalar, mean: 1.0, std: 0.02}
85+
velocity: {type: vector, mean: [0.00, 0.00, 0.00], std: 0.20}
86+
tau_wall: {type: scalar, mean: [0.00, 0.00, 0.00], std: 0.003}
87+
- _target_: ${dp:MeshToTensorDict}
88+
- _target_: ${dp:ComputeCellCentroids}
89+
- _target_: ${dp:RestructureTensorDict}
90+
groups:
91+
input:
92+
points: cell_centroids
93+
normals: cell_data.normals
94+
U_inf: global_data.U_inf
95+
output:
96+
pressure: cell_data.pressure
97+
temperature: cell_data.temperature
98+
density: cell_data.density
99+
velocity: cell_data.velocity
100+
tau_wall: cell_data.tau_wall
101+
102+
targets:
103+
pressure: scalar
104+
temperature: scalar
105+
density: scalar
106+
velocity: vector
107+
tau_wall: vector
108+
109+
metrics: [l1, l2, mae]

0 commit comments

Comments
 (0)