Commit 72b2505
fix: pass skipSpecialTokens param to Tokenizer in _TokenizerModule (#175)
## Description
Currently, the native module accepts two params, namely tokens to decode
and a flag whether we should skip special tokens. The _TokenizerModule
wrapper's decode called Tokenizer.decode with just the tokens to decode
param, resulting in wrong parameters being passed to native side. This
PR solves that problem.
### Type of change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update (improves or adds clarity to existing
documentation)
### Tested on
- [ ] iOS
- [ ] Android
### Testing instructions
<!-- Provide step-by-step instructions on how to test your changes.
Include setup details if necessary. -->
### Screenshots
<!-- Add screenshots here, if applicable -->
### Related issues
<!-- Link related issues here using #issue-number -->
### Checklist
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have updated the documentation accordingly
- [ ] My changes generate no new warnings
### Additional notes
<!-- Include any additional information, assumptions, or context that
reviewers might need to understand this PR. -->1 parent 0f270a7 commit 72b2505
1 file changed
+2
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
279 | 279 | | |
280 | 280 | | |
281 | 281 | | |
282 | | - | |
283 | | - | |
| 282 | + | |
| 283 | + | |
284 | 284 | | |
285 | 285 | | |
286 | 286 | | |
| |||
0 commit comments