Skip to content

feat: add support for hugging face tokenizers, add executorch and tokenizers as submodules#144

Merged
NorbertKlockiewicz merged 9 commits intov0.4.0-rc1from
@nk/hf-tokenizers-support
Mar 27, 2025
Merged

feat: add support for hugging face tokenizers, add executorch and tokenizers as submodules#144
NorbertKlockiewicz merged 9 commits intov0.4.0-rc1from
@nk/hf-tokenizers-support

Conversation

@NorbertKlockiewicz
Copy link
Copy Markdown
Contributor

@NorbertKlockiewicz NorbertKlockiewicz commented Mar 20, 2025

Description

This PR introduces support for hugging face tokenizers by utilizing cpp bridge that connects native code with hf tokenizers library. It also features configured submodules which point to our executorch and tokenizers-cpp repositories that are stored in software mansion labs organization.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update (improves or adds clarity to existing documentation)

Tested on

  • iOS
  • Android

Testing instructions

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@NorbertKlockiewicz NorbertKlockiewicz changed the title feat: add support for hugging face tokenizers(ios) feat: add support for hugging face tokenizers, add executorch and tokenizers as submodules Mar 25, 2025
@NorbertKlockiewicz NorbertKlockiewicz marked this pull request as ready for review March 25, 2025 08:39
Comment thread .gitmodules
Comment thread third-party/ios/ExecutorchLib/ExecutorchLib/Exported/HuggingFaceTokenizer.mm Outdated
Comment thread android/src/main/java/com/swmansion/rnexecutorch/Tokenizer.kt
Comment thread ios/RnExecutorch/Tokenizer.mm Outdated
@NorbertKlockiewicz NorbertKlockiewicz changed the base branch from main to v0.4.0-rc1 March 27, 2025 11:28
@NorbertKlockiewicz NorbertKlockiewicz merged commit 346f74d into v0.4.0-rc1 Mar 27, 2025
3 checks passed
@NorbertKlockiewicz NorbertKlockiewicz deleted the @nk/hf-tokenizers-support branch March 27, 2025 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants