Commit 1e3d29b
Fix GEGLU docstring: Sigmoid -> GELU (#8696)
## Summary
- Fixed GEGLU docstring which incorrectly stated the activation function
was Sigmoid
- The code correctly uses GELU, as specified in the original GEGLU paper
## Details
- GLU uses Sigmoid: GLU(x) = σ(xW) ⊗ xV
- GEGLU uses GELU: GEGLU(x) = GELU(xW) ⊗ xV
Reference: https://arxiv.org/abs/2002.05202
---------
Signed-off-by: Soumya Snigdha Kundu <soumya_snigdha.kundu@kcl.ac.uk>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>1 parent 342bd7a commit 1e3d29b
1 file changed
+9
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
168 | 168 | | |
169 | 169 | | |
170 | 170 | | |
171 | | - | |
| 171 | + | |
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
180 | 188 | | |
181 | 189 | | |
182 | 190 | | |
| |||
0 commit comments