Feat/coreml#308
Open
Skyline-23 wants to merge 40 commits intoazooKey:mainfrom
Open
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+120
to
+123
| #else | ||
| package func getModel(modelURL: URL) -> Zenz? { | ||
| self.zenzStatus = "zenz-v2 model unavailable on this platform" | ||
| return nil |
There was a problem hiding this comment.
When ZenzaiCoreML is not set, getModel now always marks the model as unavailable and returns nil, but the rest of the pipeline still checks zenzaiMode.enabled. On Zenzai/ZenzaiCPU builds this means the llama backend is never loaded and convertToLattice falls through to the non-zenz path, so enabling zenzaiMode no longer produces any zenz candidates or next-character predictions even though the trait and weight URL are provided—a regression from the previous behavior.
Useful? React with 👍 / 👎.
Skyline-23
force-pushed
the
feat/coreml
branch
from
5a07f7d to
ddfa88d
Compare
Skyline-23
force-pushed
the
feat/coreml
branch
3 times, most recently
from
aaf2dd8 to
32560bc
Compare
Adds support for CoreML-based Zenzai conversion, leveraging the `ZenzCoreMLBackend` for improved performance and efficiency on compatible devices. This includes: - Integration of the CoreML model for stateful generation. - A new `ZenzContext+CoreML.swift` to manage the CoreML inference process. - Updates to `Package.swift` to include `swift-transformers` dependency. - Conditional compilation to switch between CoreML and CPU/Metal implementations based on the `ZenzaiCoreML` flag. - Adds tokenizer files and marks mlpackage for LFS. The changes aim to provide a faster and more resource-efficient Zenzai conversion method on Apple platforms with Neural Engine support.
This reverts commit 63f31ac.
Cause of swift-transformers package uses combine, and it not compatible with windows
Enables building the application with CoreML support for enhanced performance on Apple devices. This allows users to leverage CoreML acceleration by using the `--zenzai-coreml` flag during installation.
Removes the explicit LFS bootstrap step from CI workflows as it is no longer required due to changes in how dependencies are managed. This simplifies the CI configuration and reduces build times. Adds documentation for development setup in multiple languages, covering build instructions, testing and devcontainer usage. This enhances the developer experience by providing clear and comprehensive instructions in English, Japanese and Korean. Improves Zenzai integration, adding CoreML support and cross-platform compatibility with Swift 6 concurrency. This allows for high-precision neural Kana-Kanji conversion on Apple devices using CoreML, while ensuring compatibility and performance on both Darwin and Linux platforms.
- replace the blockingAsync result capture with a lock-protected sendable box so Task.detached no longer trips Swift 6.3 data-race checks - align the CoreML macOS linker platform_version hint with the 15.5 XCFramework minimum to remove stale version mismatch warnings - keep the existing CoreML sync bridge behavior intact while restoring a successful ZenzaiCoreML build
- reintroduce the CoreML-facing request option shims and converter entry points that upstream main now expects from AncoSession and prediction flows - reconnect the Zenz prompt/context helpers after rebasing so llama-only helpers stay out of CoreML builds while shared APIs remain available - remove the duplicate fixed-size heap implementation and expose tokenizer APIs needed by the CoreML context so both default and ZenzaiCoreML builds succeed
- remove the zenz-CoreML Swift package dependency and load the stateful 8-bit CoreML assets directly from the Skyline23/zenz-coreml Hugging Face repo - download tokenizer and mlpackage assets into a local cache, compile the mlpackage to mlmodelc, and reuse the compiled model on subsequent launches - stop install_cli.sh from expecting a copied CoreML framework and document that CoreML assets are fetched from Hugging Face at runtime
- switch the default HF CoreML artifact selection to the working stateful FP16 package while keeping 8-bit available via environment override - make the loader follow the current Hugging Face main artifact layout and materialize mlpackage resources into a local cache before compiling them to mlmodelc - align the CoreML stateful decode path with the current app reference so fresh-cache zenz_evaluate produces stable text instead of collapsing into repeated replacement tokens
- move CoreML and llama-specific model lifecycle helpers out of KanaKanjiConverter.swift so the main converter flow reads as backend-agnostic logic - switch ZenzCoreMLService from NSLock-based state management to an actor to make the CoreML path easier to reason about during review - keep behavior unchanged while making the backend split more explicit for follow-up cleanup around shared Zenz APIs
- reintroduce the upstream session-scoped predictive input and stable prediction candidate caches inside KanaKanjiConverter so prediction view behavior matches main again - wire the existing CoreML backend path into the restored session state flow instead of bypassing the cache-aware converter logic - fix the failing AncoSession and Scenario prediction stability tests that were breaking the latest macOS and ubuntu CI runs
- pin swift-transformers to the revision with the Android localized string fallback\n- make Android CI resolve a deterministic compatibility commit instead of a moving branch head\n- preserve the existing CoreML and llama.cpp package structure
- pin swift-transformers to the latest main revision e5e227b\n- keep the repo on the upstream Android-compatible Hub/Tokenizer stack\n- retain the existing CoreML and llama.cpp integration in this branch
- remove CoreML-only stub returns from the Zenz wrapper type\n- gate unsupported predictive input and typo generation at the converter call sites\n- keep backend capability differences local to the integration layer for easier review
- revert the latest-main pin that regressed non-mac CI\n- keep the known-good Android-compatible revision for this PR\n- defer the upstream swift-transformers main upgrade until its cross-platform issues are resolved
Author
|
Follow-up notes after the latest cleanup pass: Future cleanup items that still fit this PR's scope:
I also tested upgrading Exact failure reasons from the latest-main attempt (
Because of that, the branch stays pinned to the known-good compatibility revision for |
Author
|
@ensan-hcl Could you please re-review my CoreML branch? I removed the actor-based refactoring and the README changes from this PR to keep the scope focused. I will submit those improvements in a separate follow-up PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Summary (feat/coreml)
Overview
This branch isolates CoreML integration (no actor refactor) and makes it buildable/usable via the CLI with a CoreML-only flag. Sync APIs remain primary; async wrappers are thin bridges.
Key changes
ZenzandZenzCoreMLServicemarked@unchecked Sendableto unblock async bridging.blockingAsyncnow accepts@Sendableclosures viaTask.detached; CoreML personalization handle is mapped explicitly to a tuple.ZenzaiCoreMLtrait is used, macOS linker gets a-platform_version macos 15.0 15.0hint to silence xcframework version warnings while keeping the default macOS 13 baseline for non-CoreML builds.--zenzai-coreml(plus optional--debug).AzooKeyKanaKanjiConverter_KanaKanjiConverterModuleWithDefaultDictionary.bundleinto/usr/local/bin/.llama.framework; for Zenzai/ZenzaiCPU builds, still expects it.Add Support Page for English/Korean
How to build and install (CoreML)
Known warnings / notes
updateIfRequired(options:)inKanaKanjiConverteris deprecated but behavior is unchanged; can be refactored later.intentioninInputTable.swift, redundanttryinZenzContext+CoreML.swift) with no behavioral impact.