Fix Float8CurrentScaling NaN for CodonFM: init TE layers on CUDA (#1539)

## Summary

Fix nightly CI failure in `unit-tests-recipes.yml` ([run
#23790357242](https://github.com/NVIDIA/bionemo-framework/actions/runs/23790357242)).

### Root Cause

`CodonFMEncoder` and `CodonFMLMHead` initialized TransformerEngine
layers on `"cpu"` instead of `"cuda"` (unlike ESM2 and all other
models). In `test_legacy_quantized_model_init_forward_and_backward`, the
model is created inside a `quantized_model_init(Float8CurrentScaling)`
context then moved with `model.to("cuda")`. Moving FP8-quantized tensors
from CPU→CUDA corrupts `Float8CurrentScaling`'s scale metadata,
producing NaN loss.

### Fix

Changed CodonFM's TE layer init device from `"cpu"` to `"cuda"`
(matching ESM2), which is a 2-line change in `modeling_codonfm_te.py`.
The initial xfail approach (commit 1) was too broad — only codonfm was
affected.

### Files Changed

- `bionemo-recipes/models/codonfm/modeling_codonfm_te.py` — Fix device
init (`"cpu"` → `"cuda"`)
- `bionemo-recipes/recipes/codonfm_native_te/modeling_codonfm_te.py` —
Synced copy
- 5× `test_modeling_common.py` — Removed unnecessary xfail (net -1 line
each)

---
*Automated fix by OpenClaw + Claude Code*

Signed-off-by: svc-bionemo <267129667+svc-bionemo@users.noreply.github.com>
Co-authored-by: svc-bionemo <267129667+svc-bionemo@users.noreply.github.com>

Author: svc-bionemo

bionemo-recipes/models/codonfm/modeling_codonfm_te.py

@@ -224,7 +224,7 @@ def __init__(
         if self.config.layer_precision is not None and "fp4" in self.config.layer_precision and fp4_recipe is None:
             raise RuntimeError("layer_precision contains 'fp4' entries but no fp4_recipe was provided.")
 
-        device = "meta" if torch.get_default_device() == torch.device("meta") else "cpu"
+        device = "meta" if torch.get_default_device() == torch.device("meta") else "cuda"
 
         layers: list[transformer_engine.pytorch.TransformerLayer] = []
         for i in range(config.num_hidden_layers):
@@ -362,7 +362,7 @@ def __init__(self, config: CodonFMConfig):
             config: Model configuration.
         """
         super().__init__()
-        device = "meta" if torch.get_default_device() == torch.device("meta") else "cpu"
+        device = "meta" if torch.get_default_device() == torch.device("meta") else "cuda"
         _act_fns = {
             "gelu": torch.nn.functional.gelu,
             "relu": torch.nn.functional.relu,

bionemo-recipes/recipes/codonfm_native_te/modeling_codonfm_te.py

@@ -230,7 +230,7 @@ def __init__(
         if self.config.layer_precision is not None and "fp4" in self.config.layer_precision and fp4_recipe is None:
             raise RuntimeError("layer_precision contains 'fp4' entries but no fp4_recipe was provided.")
 
-        device = "meta" if torch.get_default_device() == torch.device("meta") else "cpu"
+        device = "meta" if torch.get_default_device() == torch.device("meta") else "cuda"
 
         layers: list[transformer_engine.pytorch.TransformerLayer] = []
         for i in range(config.num_hidden_layers):
@@ -368,7 +368,7 @@ def __init__(self, config: CodonFMConfig):
             config: Model configuration.
         """
         super().__init__()
-        device = "meta" if torch.get_default_device() == torch.device("meta") else "cpu"
+        device = "meta" if torch.get_default_device() == torch.device("meta") else "cuda"
         _act_fns = {
             "gelu": torch.nn.functional.gelu,
             "relu": torch.nn.functional.relu,