|
1 | 1 | # Common Pitfalls |
2 | 2 |
|
3 | | -## Array Bounds & Ghost Cells |
4 | | -- Grid dimensions: `m`, `n`, `p` (cells in x, y, z). 1D: n=p=0. 2D: p=0. |
5 | | -- Interior domain: `0:m`, `0:n`, `0:p` |
6 | | -- Buffer/ghost region: `-buff_size:m+buff_size` (similar for n, p in multi-D) |
7 | | -- `buff_size` is **not** a single formula: it's set per reconstruction scheme (WENO/MUSCL/IGR) in |
8 | | - `s_configure_coordinate_bounds` (`m_helper_basic.fpp`) and floored higher for Lagrange bubbles and IB. |
9 | | - Read that routine for the current value rather than assuming one. |
10 | | -- Domain bounds: `idwint(1:3)` (interior `0:m`), `idwbuff(1:3)` (with ghost cells) |
11 | | -- Cell-center coords: `x_cc(-buff_size:m+buff_size)`, `y_cc(...)`, `z_cc(...)` |
12 | | -- Cell-boundary coords: `x_cb(-1-buff_size:m+buff_size)` |
13 | | -- Riemann solver indexing: left state at `j`, right state at `j+1` |
14 | | -- Off-by-one errors in ghost cell regions are a common source of bugs |
15 | | - |
16 | | -## Field Variable Indexing |
17 | | -- Conserved variables: `q_cons_vf(1:sys_size)`. Primitive: `q_prim_vf(1:sys_size)`. |
18 | | -- All equation indices live in the unified `eqn_idx` struct (`eqn_idx_info` type in `m_derived_types.fpp`). |
19 | | - Index ranges depend on `model_eqns` and enabled features (set in `m_global_parameters.fpp`): |
20 | | - - `eqn_idx%cont` — continuity range (partial densities, one per fluid) |
21 | | - - `eqn_idx%mom` — momentum range |
22 | | - - `eqn_idx%E` — total energy (scalar) |
23 | | - - `eqn_idx%adv` — volume fractions (advection equations) |
24 | | - - `eqn_idx%bub`, `eqn_idx%stress`, `eqn_idx%xi`, `eqn_idx%species`, `eqn_idx%B` — optional |
25 | | - - `eqn_idx%gamma`, `eqn_idx%pi_inf`, `eqn_idx%alf`, `eqn_idx%int_en` — additional scalars/ranges |
26 | | -- Use `eqn_idx%cont%beg`/`eqn_idx%cont%end`, `eqn_idx%mom%beg`/`eqn_idx%mom%end`, etc. (old `contxb`/`contxe`, `momxb`/`momxe` shorthands are gone) |
27 | | -- `sys_size` = total number of conserved variables (computed at startup) |
28 | | -- Changing `model_eqns` or enabling features changes ALL index positions |
29 | | - |
30 | | -## Blast Radius |
31 | | -- `src/common/` is shared by ALL three executables (pre_process, simulation, post_process) |
32 | | -- Any change to common/ requires testing all three targets |
33 | | -- Public subroutine signature changes affect all callers across all targets |
34 | | -- Parameter default changes affect all existing case files |
35 | | - |
36 | | -## Physics Consistency |
37 | | -- Pressure formula MUST match `model_eqns` setting |
38 | | -- Model-specific conservative ↔ primitive conversion paths exist |
39 | | -- Volume fractions must sum to 1.0 |
40 | | -- Boundary condition symmetry requirements must be maintained |
41 | | - |
42 | | -## Compiler-Specific Issues |
43 | | -- See the compiler-backend matrix in `.claude/rules/gpu-and-mpi.md` for which compilers |
44 | | - are CI-gated and which backends each supports. |
45 | | -- Each compiler has different strictness levels and warning behavior |
46 | | -- Fypp macros must expand correctly for both GPU and CPU builds |
47 | | - |
48 | | -## Test System |
49 | | -- Tests are generated **programmatically** in `toolchain/mfc/test/cases.py`, not standalone files |
50 | | -- Each test is a parameter modification on top of `BASE_CFG` defaults |
51 | | -- Test UUID = CRC32 hash of the test's trace string; `./mfc.sh test -l` lists all |
52 | | -- To add a test: modify `cases.py` using `CaseGeneratorStack` push/pop pattern |
53 | | -- Golden files: `tests/<UUID>/golden.txt` — tolerance-based comparison, not exact match |
54 | | -- If your change intentionally modifies output, regenerate golden files: |
55 | | - `./mfc.sh test --generate --only <affected_tests> -j 8` |
56 | | -- Do not regenerate ALL golden files unless you understand every output change |
57 | | - |
58 | | -## PR Checklist |
59 | | -The base loop (format → precheck → build → test → one logical commit) is the |
60 | | -Development Workflow Contract in `CLAUDE.md`. Beyond it, watch for: |
61 | | -- [ ] If adding parameters: definitions.py (_r + _nv) updated; cmake reconfigured; case_validator.py if constraints |
62 | | -- [ ] If modifying `src/common/`: all three targets tested |
63 | | -- [ ] If changing output: golden files regenerated for affected tests (`./mfc.sh test --generate --only <tests>`) |
| 3 | +Traps that live in no other doc, collected because their failure modes are silent — |
| 4 | +wrong answers and wrong indices, not error messages. General development pitfalls are |
| 5 | +covered in `docs/documentation/contributing.md`. |
| 6 | + |
| 7 | +## Indexing and Ghost Cells |
| 8 | + |
| 9 | +- Grid dimensions `m`, `n`, `p` (cells in x, y, z); 1D: n=p=0, 2D: p=0. Interior `0:m`; |
| 10 | + ghost region `-buff_size:m+buff_size`; bounds structs `idwint(1:3)` (interior) and |
| 11 | + `idwbuff(1:3)` (with ghosts); cell boundaries `x_cb(-1-buff_size:m+buff_size)`. |
| 12 | +- `buff_size` is **not** a single formula: it's set per reconstruction scheme in |
| 13 | + `s_configure_coordinate_bounds` (`m_helper_basic.fpp`) and floored higher for Lagrange |
| 14 | + bubbles and IB. Read that routine for the current value rather than assuming one. |
| 15 | +- Riemann solvers: left state at `j`, right state at `j+1`. |
| 16 | +- All equation indices live in the `eqn_idx` struct (`eqn_idx_info` in |
| 17 | + `m_derived_types.fpp`, populated by `s_initialize_eqn_idx` in `m_global_parameters_common.fpp`): `%cont`, `%mom`, `%E`, |
| 18 | + `%adv`, plus optional ranges (`%bub`, `%stress`, `%species`, `%B`, ...). The old |
| 19 | + `contxb`/`momxb` shorthands are gone. Index positions depend on `model_eqns` and |
| 20 | + enabled features — changing either moves ALL indices; never hard-code one. |
| 21 | + |
| 22 | +## GPU |
| 23 | + |
| 24 | +- WARNING: do NOT wrap `GPU_LOOP` in `GPU_PARALLEL` for spatial loops — `GPU_LOOP` emits |
| 25 | + empty directives on Cray and AMD, causing silent serial execution. Spatial loops always |
| 26 | + use `GPU_PARALLEL_LOOP`/`END_GPU_PARALLEL_LOOP`. Macro API: |
| 27 | + `docs/documentation/gpuParallelization.md`; signatures: |
| 28 | + `src/common/include/parallel_macros.fpp`. Never call the `ACC_*`/`OMP_*` |
| 29 | + implementation layers directly. |
| 30 | +- Only `src/simulation/` is GPU-accelerated. Backends: OpenACC (nvfortran primary, Cray) |
| 31 | + and OpenMP offload (Cray primary, AMD flang, nvfortran). The CPU-only build must always |
| 32 | + work — every `#ifdef` needs a path for all configurations (CPU, ACC, OMP, with/without |
| 33 | + MPI). Gates: `MFC_GPU`, `MFC_OpenACC`, `MFC_OpenMP`, `MFC_MPI`, `MFC_DEBUG`, |
| 34 | + `MFC_SINGLE_PRECISION`/`MFC_MIXED_PRECISION`, `MFC_PRE_PROCESS`/`MFC_SIMULATION`/ |
| 35 | + `MFC_POST_PROCESS`, and compiler macros (`_CRAYFTN`, `__PGI`, ...). |
| 36 | +- `@:ACC_SETUP_VFs(...)`/`@:ACC_SETUP_SFs(...)` GPU pointer setup compiles only under |
| 37 | + Cray. Around MPI: `GPU_UPDATE(host=...)` before send, `GPU_UPDATE(device=...)` after |
| 38 | + receive. |
| 39 | + |
| 40 | +## Parameters |
| 41 | + |
| 42 | +- Adding one: `_r()` definition + `_nv()` `NAMELIST_VARS` registration in |
| 43 | + `toolchain/mfc/params/definitions.py`; `case_validator.py` only if physics-constrained |
| 44 | + (with a `PHYSICS_DOCS` entry). Fortran declarations and namelist bindings are |
| 45 | + auto-generated at build time (ninja-tracked custom command) — re-run cmake (or `./mfc.sh build`) after editing. |
| 46 | +- Still manual: derived-type `TYPE` member definitions in `src/common/m_derived_types.fpp`; |
| 47 | + default-value assignments in `s_assign_default_values_to_user_inputs`; the |
| 48 | + `CASE_OPT_EXTRA_LINES` literal in `toolchain/mfc/params/generators/fortran_gen.py` (covers `num_dims`, |
| 49 | + `num_vels`, `weno_polyn`, `muscl_polyn`, `weno_num_stencils`, `wenojs`); |
| 50 | + multi-variable declaration lines (`bc_x/y/z`, `x/y/z_domain`, `x/y/z_output`, post's |
| 51 | + `G`); and the MPI broadcast residue in `m_mpi_proxy` (computed variables that are not |
| 52 | + namelist-bound: `m_glb`/`n_glb`/`p_glb`, `cfl_dt`, `bc_io`, and complex struct-member |
| 53 | + array loops — these cannot be auto-generated and stay hand-listed). Everything else — scalar declarations, plain arrays (`FORTRAN_ARRAY_DIMS` |
| 54 | + table in `definitions.py`), derived-type namelist declarations including `GPU_DECLARE` |
| 55 | + lines and Doxygen descs (`TYPED_DECLS` table in `definitions.py`), the simulation |
| 56 | + case-optimization declaration block, and the per-target MPI broadcast lists for all |
| 57 | + namelist-registry scalars (`generated_bcast.fpp`) — is regenerated at build time by a |
| 58 | + ninja-tracked custom command (editing `params/*.py` triggers regeneration automatically). |
| 59 | + Gotcha: ADDING a new file under `toolchain/mfc/params/` needs one reconfigure |
| 60 | + (the custom command's DEPENDS list is globbed at configure time). Under `--case-optimization` the baked-in constants are dropped from the |
| 61 | + namelist, so changing one needs a *rebuild*, not a case edit. |
| 62 | +- Shared-state pattern: namelist declarations (`#:include 'generated_decls.fpp'`), the |
| 63 | + `eqn_idx`/`sys_size`/`b_size`/`tensor_size` state variables, and the common defaults |
| 64 | + core all live in `src/common/m_global_parameters_common.fpp`. Each per-target |
| 65 | + `m_global_parameters.fpp` does `use m_global_parameters_common` (default-public), so |
| 66 | + `use m_global_parameters` continues to work for all downstream modules without change. |
| 67 | + Sim-only declarations (GPU_DECLARE, Re_idx allocation) stay in |
| 68 | + `m_global_parameters_common` behind `#ifdef MFC_SIMULATION`. Generated includes |
| 69 | + (`generated_decls.fpp`, `generated_bcast.fpp`, `generated_case_opt_decls.fpp`) must exist for every target — the build |
| 70 | + emits stubs where the content is sim-only, so a common file that includes one will |
| 71 | + compile for pre/post too. |
| 72 | +- Runtime checks (`@:PROHIBIT`) go where they run: shared → |
| 73 | + `src/common/m_checker_common.fpp`; simulation-only → `src/simulation/m_checker.fpp`; |
| 74 | + pre/post-only → `src/{pre,post}_process/m_checker.fpp` (their `s_check_inputs` are |
| 75 | + currently empty — that IS the right place, not m_checker_common). |
| 76 | +- Analytic ICs are compiled into the binary. Expressions are AST-validated at case load |
| 77 | + (syntax errors and unknown variables are immediate, named errors; bare `e` is not a |
| 78 | + variable — write `exp(1.0)`). |
| 79 | + Each IC variable maps to an `eqn_idx%…` expression in `QPVF_IDX_VARS` |
| 80 | + (`toolchain/mfc/case.py`); a new patch-settable conserved variable means updating that |
| 81 | + map AND the Fortran `eqn_idx` builder to agree — a mismatch is a silent wrong index. |
| 82 | + Variables available in expressions: `docs/documentation/case.md`. |
| 83 | + |
| 84 | +## Tests |
| 85 | + |
| 86 | +- Tests are generated programmatically in `toolchain/mfc/test/cases.py` (parameter |
| 87 | + modifications on `BASE_CFG` via the `CaseGeneratorStack` push/pop pattern); test UUID = |
| 88 | + CRC32 of the trace string; `./mfc.sh test -l` lists all. |
| 89 | +- Golden files are tolerance-compared. Regenerate only the affected tests |
| 90 | + (`./mfc.sh test --generate --only <tests>`) — an unexplained golden-file diff is a bug |
| 91 | + report, not noise to be regenerated away. |
0 commit comments