Skip to content

Commit 4ee7547

Browse files
Mr-Neutr0nTimDettmers
authored andcommitted
Add LARS to str2optimizer32bit dictionary
LARS optimizer was missing from str2optimizer32bit, causing KeyError when using LARS32bit optimizer. LARS uses momentum-based kernels since it's essentially SGD with momentum plus layerwise adaptive learning rates. Fixes #1810
1 parent a2c92f7 commit 4ee7547

File tree

1 file changed

+4
-0
lines changed
  • bitsandbytes/backends/cuda

1 file changed

+4
-0
lines changed

bitsandbytes/backends/cuda/ops.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -574,6 +574,10 @@ def _gemv_4bit_impl(
574574
lib.cademamix32bit_grad_fp16,
575575
lib.cademamix32bit_grad_bf16,
576576
),
577+
"lars": (
578+
lib.cmomentum32bit_grad_32,
579+
lib.cmomentum32bit_grad_16,
580+
),
577581
}
578582

579583
str2optimizer8bit_blockwise = {

0 commit comments

Comments
 (0)