|
| 1 | +--- |
| 2 | +name: core-ai |
| 3 | +description: Run on-device AI models in iOS, iPadOS, macOS, and visionOS apps with Core AI. Use when bundling .aimodel files, loading an AIModel, running inference over NDArray tensors, or compiling models ahead of time with coreai-build. |
| 4 | +--- |
| 5 | + |
| 6 | +# Core AI: On-Device AI Models |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +Core AI runs AI models **on device** inside your app. Inference stays private, works |
| 11 | +offline, and has no per-inference cost. You start from a `.aimodel` file (converted |
| 12 | +from a model or already in the correct format) that contains one or more named |
| 13 | +inference functions, bundle it, load it as an `AIModel`, and run those functions |
| 14 | +over `NDArray` tensors (image inputs use a pixel-buffer value). |
| 15 | + |
| 16 | +Available on **Apple Intelligence** devices: iPhone/iPad with A17 Pro or later, Mac |
| 17 | +with M1 or later, Apple Vision Pro (M2+). Building requires the **Metal Toolchain** |
| 18 | +in Xcode (not installed by default) — without it, builds that include `.aimodel` |
| 19 | +files fail with a missing Metal compiler error. |
| 20 | + |
| 21 | +> Core AI runs your own model files at the tensor level. For Apple's high-level |
| 22 | +> on-device LLM API (chat sessions, tool calling, guided generation), use the |
| 23 | +> Foundation Models framework — a separate framework, not covered here. |
| 24 | +
|
| 25 | +## Workflow |
| 26 | + |
| 27 | +1. **Inspect the model** in Xcode's model viewer: |
| 28 | + - _General_ — size (parameter count and on-disk storage), numeric precision split |
| 29 | + into **compute** (used during inference) and **storage** (weights on disk), |
| 30 | + operation distribution, and editable metadata. |
| 31 | + - _Functions_ — each inference function's input/output names, types, and shapes. |
| 32 | + A `?` in a dimension means it is dynamic (supplied or determined at runtime). |
| 33 | +2. **Bundle the model** — add the `.aimodel` file to the Xcode target (it appears in |
| 34 | + the _Compile Sources_ build phase). Install the Metal Toolchain first. |
| 35 | +3. **Load the model** — `AIModel(contentsOf:)` is asynchronous because Core AI |
| 36 | + **specializes** the model for the current device and selects the compute units |
| 37 | + that deliver the best performance. For large models this can take significant |
| 38 | + time, so consider ahead-of-time compilation (below). |
| 39 | +4. **Load a function** — `model.loadFunction(named:)` returns the function or `nil` |
| 40 | + when no function with that name exists (it throws on other load failures). Use |
| 41 | + `functionNames` when a model has multiple functions. The same inference function |
| 42 | + is safe to call from concurrent tasks. |
| 43 | +5. **Prepare inputs** — match each input's shape and scalar type from |
| 44 | + `function.descriptor`, then write data through a mutable view. |
| 45 | +6. **Run and read outputs** — `function.run(inputs:)` returns outputs keyed by name; |
| 46 | + pull each result with `outputs.remove(_:)` and read it through a view. |
| 47 | + |
| 48 | +## Core API |
| 49 | + |
| 50 | +```swift |
| 51 | +import CoreAI |
| 52 | + |
| 53 | +// 1. Specialize for this device and load the model. |
| 54 | +let model = try await AIModel(contentsOf: urlOfModel) |
| 55 | + |
| 56 | +// 2. Load the inference function. Returns nil if the name is absent. |
| 57 | +guard let function = try model.loadFunction(named: "main") else { |
| 58 | + // Handle a missing function. |
| 59 | +} |
| 60 | + |
| 61 | +// 3. Verify the input's shape and scalar type from the descriptor. |
| 62 | +let descriptor = function.descriptor |
| 63 | +guard let valueDescriptor = descriptor.inputDescriptor(of: "input"), |
| 64 | + case .ndArray(let arrayDescriptor) = valueDescriptor, |
| 65 | + arrayDescriptor.shape == [3, 4], |
| 66 | + arrayDescriptor.scalarType == .float32 else { |
| 67 | + // Handle an unexpected type or shape. |
| 68 | +} |
| 69 | + |
| 70 | +// 4. Create the input tensor and write data through a mutable view. |
| 71 | +var input = NDArray(shape: [3, 4], scalarType: .float32) |
| 72 | +var mutableView = input.mutableView(as: Float.self) |
| 73 | +guard let elements = mutableView.contiguousElements else { |
| 74 | + // Handle a non-contiguous memory layout. |
| 75 | +} |
| 76 | +writeInputData(into: elements) |
| 77 | + |
| 78 | +// 5. Run inference and extract the named output. |
| 79 | +var outputs = try await function.run(inputs: ["input": input]) |
| 80 | +guard let value = outputs.remove("prediction"), |
| 81 | + let prediction = value.ndArray else { |
| 82 | + // Handle a missing or unexpected output. |
| 83 | +} |
| 84 | +processOutput(prediction.view()) |
| 85 | +``` |
| 86 | + |
| 87 | +### Tensors and values |
| 88 | + |
| 89 | +- `NDArray` — an n-dimensional tensor. Build it with `NDArray(shape:scalarType:)`. |
| 90 | + It is **read-only by default**: use `mutableView(as:)` → `contiguousElements` to |
| 91 | + write, and `view()` to read. Swift enforces read vs. write access at compile time. |
| 92 | +- `scalarType` — the element type, e.g. `.float32`. Shape is an `[Int]` matching the |
| 93 | + model's expectation; a `?` dimension in the viewer is dynamic. |
| 94 | +- **Images** — values marked as images at conversion time use a pixel-buffer value |
| 95 | + rather than `NDArray`. |
| 96 | +- `ValueDescriptor` — `.ndArray(ArrayDescriptor)` vs. image cases. Inspect |
| 97 | + `descriptor.inputDescriptor(of:)` / `outputDescriptor(of:)` at runtime so the app |
| 98 | + can adapt if a function's signature changes between deployments without code edits. |
| 99 | + |
| 100 | +## Ahead-of-Time (AOT) Compilation |
| 101 | + |
| 102 | +On-device specialization can delay first load. Move the most expensive part — model |
| 103 | +compilation — to the build machine with the **`coreai-build`** CLI. It converts |
| 104 | +`MyModel.aimodel` into one `MyModel.<arch>.aimodelc` asset per device architecture. |
| 105 | +At runtime the app picks the asset for the current architecture and loads it with the |
| 106 | +**same** `AIModel` API, so loading code does not change. |
| 107 | + |
| 108 | +```bash |
| 109 | +# 1. Install the Metal Toolchain (also: Xcode > Settings > Components > Get). |
| 110 | +xcodebuild -downloadComponent MetalToolchain |
| 111 | + |
| 112 | +# 2. Compile one .aimodelc per architecture. |
| 113 | +xcrun coreai-build compile MyModel.aimodel --platform iOS --output compiled/ |
| 114 | + |
| 115 | +# Override compute units, deployment version, target arch, and more: |
| 116 | +xcrun coreai-build compile MyModel.aimodel --platform macOS \ |
| 117 | + --preferred-compute gpuAndNeuralEngine --output compiled/ |
| 118 | +xcrun coreai-build compile --help |
| 119 | +``` |
| 120 | + |
| 121 | +```swift |
| 122 | +// Select the compiled asset for this device, then load normally. |
| 123 | +let arch = AIModel.deviceArchitectureName |
| 124 | +let assetName = "MyModel.\(arch).aimodelc" |
| 125 | +let model = try await AIModel(contentsOf: bundledURL(for: assetName)) |
| 126 | +``` |
| 127 | + |
| 128 | +Notes: |
| 129 | + |
| 130 | +- `coreai-build` emits one `.<arch>.aimodelc` per architecture; the filename prefix |
| 131 | + comes from the input model. `AIModel.deviceArchitectureName` is the identifier that |
| 132 | + matches `<arch>` at runtime. |
| 133 | +- Compute units default to best performance. Pass `--preferred-compute` to override, |
| 134 | + and use matching load options. |
| 135 | +- A compiled asset still requires **some** on-device specialization — AOT removes the |
| 136 | + bulk of compilation, not all of it. AOT only targets Apple Intelligence devices. |
| 137 | + |
| 138 | +## Checklist |
| 139 | + |
| 140 | +- [ ] Metal Toolchain installed (Xcode, or `xcodebuild -downloadComponent MetalToolchain`). |
| 141 | +- [ ] `.aimodel` added to the target and visible in _Compile Sources_. |
| 142 | +- [ ] `AIModel(contentsOf:)` awaited; slow first load handled (or AOT-compiled). |
| 143 | +- [ ] `loadFunction(named:)` nil-checked; `functionNames` inspected for multi-function models. |
| 144 | +- [ ] Input `shape` / `scalarType` verified against `function.descriptor`. |
| 145 | +- [ ] Mutable views for writes, read-only views for reads. |
| 146 | +- [ ] Outputs extracted by name with `outputs.remove(_:)`. |
| 147 | +- [ ] For large models: per-architecture `.aimodelc` built via `coreai-build`, and the |
| 148 | + correct asset selected at runtime using `AIModel.deviceArchitectureName`. |
| 149 | + |
| 150 | +## Resources |
| 151 | + |
| 152 | +- Apple — _Integrating on-device AI models in your app with Core AI_ |
| 153 | +- Apple — _Compiling Core AI models ahead of time_ |
| 154 | +- Prefer Apple docs for up-to-date API details; web-search the current Core AI |
| 155 | + documentation alongside this skill. |
0 commit comments