Implement formatter to ensure Nemotron VoiceChat speech decoder reproducibility, speed up training and support half precision inference (#15583)

* Add new formatter and force codec to runs in the same device of TTS model

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Set find_unused_parameters=False as default to speedup training

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Fix model input data type

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Add support for half precision inference

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Fix RMSnorm

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Update debug_dtype default parameter

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Add unit tests for duplex formatters

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Replace fastcopy with deepcopy

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

* Remove unecessary comments

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Update unit tests

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Add function for dtype debug and create a method to check codec precision

Signed-off-by: Edresson Casanova <edresson1@gmail.com>

* Apply isort and black reformatting

Signed-off-by: Edresson <Edresson@users.noreply.github.com>

---------

Signed-off-by: Edresson Casanova <edresson1@gmail.com>
Signed-off-by: Edresson <Edresson@users.noreply.github.com>
Co-authored-by: Edresson <Edresson@users.noreply.github.com>

Author: Edresson

examples/speechlm2/conf/duplex_eartts.yaml

@@ -127,7 +127,7 @@ trainer:
   strategy:
     _target_: lightning.pytorch.strategies.DDPStrategy
     gradient_as_bucket_view: true
-    find_unused_parameters: true
+    find_unused_parameters: false
 
 data:
   # data loader configs