Skip to content

feat: port TokenizerModule to C++#393

Merged
chmjkb merged 23 commits intomainfrom
@chmjkb/tokenizers-cpp-port
Jun 17, 2025
Merged

feat: port TokenizerModule to C++#393
chmjkb merged 23 commits intomainfrom
@chmjkb/tokenizers-cpp-port

Conversation

@chmjkb
Copy link
Copy Markdown
Collaborator

@chmjkb chmjkb commented Jun 13, 2025

Description

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update (improves or adds clarity to existing documentation)

Tested on

  • iOS
  • Android

Testing instructions

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@chmjkb chmjkb marked this pull request as draft June 13, 2025 08:16
@chmjkb chmjkb force-pushed the @chmjkb/tokenizers-cpp-port branch from c5eb0e6 to b676862 Compare June 13, 2025 12:21
@chmjkb chmjkb requested a review from pweglik June 17, 2025 05:17
@chmjkb chmjkb marked this pull request as ready for review June 17, 2025 05:17
@chmjkb chmjkb self-assigned this Jun 17, 2025
@chmjkb chmjkb added this to the v0.5.0 milestone Jun 17, 2025
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.h Outdated
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.h Outdated
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.h Outdated
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.cpp Outdated
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.cpp Outdated
@pweglik pweglik requested review from jakmro and mkopcins June 17, 2025 10:21
Comment thread packages/react-native-executorch/common/rnexecutorch/TokenizerModule.cpp Outdated
@chmjkb chmjkb merged commit 72b1014 into main Jun 17, 2025
2 checks passed
@chmjkb chmjkb deleted the @chmjkb/tokenizers-cpp-port branch June 17, 2025 14:24
mkopcins pushed a commit that referenced this pull request Oct 15, 2025
## Description

<!-- Provide a concise and descriptive summary of the changes
implemented in this PR. -->

### Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update (improves or adds clarity to existing
documentation)

### Tested on

- [ ] iOS
- [ ] Android

### Testing instructions

<!-- Provide step-by-step instructions on how to test your changes.
Include setup details if necessary. -->

### Screenshots

<!-- Add screenshots here, if applicable -->

### Related issues

<!-- Link related issues here using #issue-number -->

### Checklist

- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have updated the documentation accordingly
- [ ] My changes generate no new warnings

### Additional notes

<!-- Include any additional information, assumptions, or context that
reviewers might need to understand this PR. -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants