|
| 1 | +--- |
| 2 | +name: apple-intelligence |
| 3 | +description: Apple Intelligence participation layer. Wire an app's actions and content into Siri, Spotlight, Writing Tools, Image Playground, and summaries via App Intents, entities, and schemas. Use before adding App Intents/schemas, Spotlight indexing, Writing Tools, or Image Playground; defer custom generative features and on-device model files to the sibling skills. |
| 4 | +--- |
| 5 | + |
| 6 | +# Apple Intelligence |
| 7 | + |
| 8 | +## Overview |
| 9 | + |
| 10 | +Apple Intelligence is the personal intelligence system behind built-in capabilities |
| 11 | +across Apple platforms. It runs on-device on Apple silicon and in Private Cloud |
| 12 | +Compute. Your app participates by teaching the system about its **actions** and |
| 13 | +**content** so system features — Siri, Spotlight, Shortcuts, Writing Tools, Image |
| 14 | +Playground, Visual Intelligence, Genmoji, and summaries — can use them. |
| 15 | + |
| 16 | +The participation contract is the **App Intents** framework: an `AppIntent` wraps one |
| 17 | +of your app's actions; an `AppEntity` represents the data those actions operate on. |
| 18 | +Apple Intelligence uses donated intents/entities, your Spotlight index, and declared |
| 19 | +schemas to find and act on your content — even when described vaguely. |
| 20 | + |
| 21 | +Two sibling skills handle the custom-model side and are out of scope here: |
| 22 | + |
| 23 | +- **`foundation-models`** — the Foundation Models framework for custom generative |
| 24 | + features (prompting, guided generation, tool calling, Private Cloud Compute). |
| 25 | +- **`core-ai`** — running your own deep-learning model files (`.aimodel`) on-device. |
| 26 | + |
| 27 | +This skill covers only the built-in intelligence surfaces an app adopts. |
| 28 | + |
| 29 | +## Device eligibility and availability |
| 30 | + |
| 31 | +Apple Intelligence requires Apple silicon (iPhone 15 Pro and later, iPad with A17 Pro |
| 32 | +or M-series, Mac with M-series). It is opt-in via the Apple Intelligence toggle in |
| 33 | +Settings. Before exposing intelligence features, branch on the system model's |
| 34 | +`availability` rather than assuming it is on: |
| 35 | + |
| 36 | +```swift |
| 37 | +import FoundationModels |
| 38 | + |
| 39 | +private var model = SystemLanguageModel.default |
| 40 | + |
| 41 | +switch model.availability { |
| 42 | +case .available: |
| 43 | + // Show your intelligence UI. |
| 44 | +case .unavailable(.deviceNotEligible): |
| 45 | + // Show an alternative UI. |
| 46 | +case .unavailable(.appleIntelligenceNotEnabled): |
| 47 | + // Ask the person to turn on Apple Intelligence. |
| 48 | +case .unavailable(.modelNotReady): |
| 49 | + // Model is downloading or not ready. |
| 50 | +case .unavailable(let other): |
| 51 | + // Unknown reason. |
| 52 | +} |
| 53 | +``` |
| 54 | + |
| 55 | +## How an app participates: App Intents, entities, schemas |
| 56 | + |
| 57 | +The single biggest lever. Make the system aware of your actions and data via the App |
| 58 | +Intents framework: |
| 59 | + |
| 60 | +- **App intents** — a custom type encapsulating one action. Ship in your app or an app |
| 61 | + extension so the action runs even when the app isn't open. Each intent performs an |
| 62 | + action in `perform()`, returns a result/error, declares parameters, and provides a |
| 63 | + localized title Siri and Shortcuts can display. |
| 64 | +- **App entities** — lightweight versions of your data objects. Define them only for |
| 65 | + the subset of data people see and might reference with Siri ("this photo", "this |
| 66 | + album"). Real data objects stay the source of truth. |
| 67 | +- **App enums** — enumerate fixed choices (e.g. repeat options) so the system can |
| 68 | + resolve parameters. |
| 69 | +- **Donations** — when someone performs an action in your UI, donate the matching |
| 70 | + intent/entity. Donations give Apple Intelligence behavioral cues to predict and |
| 71 | + disambiguate. Donate only direct UI interactions, not actions Siri or Shortcuts |
| 72 | + initiated. |
| 73 | +- **Schemas (domains)** — the predefined intent/entity shapes for common actions (mail, |
| 74 | + messaging, files, etc.). Prefer building from a schema over a custom intent; schemas |
| 75 | + are the contract Apple Intelligence uses to match everyday phrases. |
| 76 | + |
| 77 | +Grounded minimal `AppIntent` shape: |
| 78 | + |
| 79 | +```swift |
| 80 | +struct OrderAlbum: AppIntent { |
| 81 | + static var title: LocalizedStringResource { "Order Album" } |
| 82 | + static var description = IntentDescription("Order a vinyl record album.") |
| 83 | + |
| 84 | + @Parameter(title: "Album", description: "The name of the album to order.") |
| 85 | + var albumName: String |
| 86 | + |
| 87 | + @Dependency |
| 88 | + private var albumManager: AlbumDataManager |
| 89 | + |
| 90 | + func perform() async throws -> some IntentResult { |
| 91 | + // Perform the action... |
| 92 | + return .result() |
| 93 | + } |
| 94 | + |
| 95 | + static var parameterSummary: some ParameterSummary { |
| 96 | + Summary("Order \(\.$albumName)") |
| 97 | + } |
| 98 | +} |
| 99 | +``` |
| 100 | + |
| 101 | +> For full intent/entity/schema implementation patterns, see the sibling |
| 102 | +> `ios-app-intents` skill. |
| 103 | +
|
| 104 | +## App Shortcuts |
| 105 | + |
| 106 | +An `AppShortcut` bundles an `AppIntent` with a title, image, preconfigured parameters, |
| 107 | +and spoken phrases so it appears polished in Shortcuts and other system experiences. |
| 108 | +The compiler generates system metadata, so shortcuts are available the moment someone |
| 109 | +installs your app — no registration needed. Define them in a type adopting the |
| 110 | +`AppShortcutProvider` protocol, alongside the intents they use. Surface them in-app |
| 111 | +with tip views. |
| 112 | + |
| 113 | +## Apple Intelligence + Siri AI: the extra steps |
| 114 | + |
| 115 | +Beyond a basic App Intents adoption, make content discoverable by Apple Intelligence |
| 116 | +specifically: |
| 117 | + |
| 118 | +- **Index entities into Spotlight** — Apple Intelligence uses Spotlight's semantic |
| 119 | + search to find your content even when described vaguely. This is what makes "the |
| 120 | + email from Mei about the Q3 plan" work. |
| 121 | +- **Choose transferable types** — conform entities/values to transferable types so the |
| 122 | + system can move content across apps for cross-app Siri tasks. |
| 123 | +- **Adopt schemas** — the contract the system uses to identify, query, and understand |
| 124 | + actions and content, and to match them to conversational phrases. |
| 125 | +- **Associate entities with views / user activities** — gives Apple Intelligence |
| 126 | + onscreen context so someone can refer to what's visible ("this photo"). |
| 127 | +- **Donate actions and content** — behavioral cues for prediction and disambiguation. |
| 128 | + |
| 129 | +## Spotlight / semantic indexing |
| 130 | + |
| 131 | +Indexing your app's content makes it findable in search and lets Siri and other system |
| 132 | +features locate app-specific data. If you define app entities for your data, index |
| 133 | +those entities with the rest of your content. When a search finds your content, the |
| 134 | +system uses the associated entity to open your app and navigate to it. Apple |
| 135 | +Intelligence retrieves entities it finds in your Spotlight index to interact with your |
| 136 | +content. |
| 137 | + |
| 138 | +## Writing Tools and Genmoji |
| 139 | + |
| 140 | +Writing Tools (proofreading, rewrite, summaries) integrate into standard system text |
| 141 | +views automatically: |
| 142 | + |
| 143 | +- Adopt `NSAttributedString` / attributed strings as the backing store for text content. |
| 144 | +- Use standard text views whenever possible and customize via their configuration |
| 145 | + options. |
| 146 | +- Use the Writing Tools API directly only if you have a custom text-editing engine or |
| 147 | + can't use system views. |
| 148 | +- Genmoji is built into system text views. In custom views, add Genmoji text |
| 149 | + attachments and read/write them correctly when persisting to a custom file format. |
| 150 | + |
| 151 | +## Image Playground |
| 152 | + |
| 153 | +The ImagePlayground framework gives a system interface to generate images from a text |
| 154 | +description, an optional source image, and a style. Present a system sheet (SwiftUI) or |
| 155 | +view controller (UIKit/AppKit); the system manages all interaction and delivers the |
| 156 | +resulting image. Key API surface: |
| 157 | + |
| 158 | +- `ImagePlaygroundViewController` — the standard system interface (UIKit/AppKit). |
| 159 | +- `ImagePlaygroundConcept` — text elements specifying content to include. |
| 160 | +- `ImagePlaygroundStyle` / `ImagePlaygroundOptions` — style and generation options. |
| 161 | +- `ImageCreator` — generate images programmatically without the UI. |
| 162 | +- SwiftUI sheet modifier: |
| 163 | + |
| 164 | +```swift |
| 165 | +imagePlaygroundSheet( |
| 166 | + isPresented: Binding<Bool>, |
| 167 | + concept: String, // or concepts: [ImagePlaygroundConcept] |
| 168 | + sourceImage: Image?, // or sourceImageURL: URL |
| 169 | + onCompletion: (URL) -> Void, |
| 170 | + onCancellation: (() -> Void)? |
| 171 | +) |
| 172 | +``` |
| 173 | + |
| 174 | +## Visual Intelligence |
| 175 | + |
| 176 | +For object/scene scanning via Camera Control, adopt Visual Intelligence. The framework |
| 177 | +detects content and exchanges information with your app using App Intents — so your App |
| 178 | +Intents participation is also what makes you a destination for Visual Intelligence |
| 179 | +results. |
| 180 | + |
| 181 | +## Checklist |
| 182 | + |
| 183 | +- [ ] Check `SystemLanguageModel.default.availability` before showing intelligence UI; |
| 184 | + handle `.deviceNotEligible` and `.appleIntelligenceNotEnabled`. |
| 185 | +- [ ] Identified the user-visible actions and data; modeled them as `AppIntent` and |
| 186 | + `AppEntity`. |
| 187 | +- [ ] Built common actions from built-in schema domains (not bespoke intents) where one |
| 188 | + exists. |
| 189 | +- [ ] Each intent has `perform()`, a result/error, declared `@Parameter`s, and a |
| 190 | + localized `title`. |
| 191 | +- [ ] Entities are lightweight and cover only what people see/reference. |
| 192 | +- [ ] Donating intents/entities on direct UI interactions (not Siri/Shortcuts ones). |
| 193 | +- [ ] App entities indexed into Spotlight (enables semantic search by Apple |
| 194 | + Intelligence). |
| 195 | +- [ ] Entities associated with views/user activities for onscreen context. |
| 196 | +- [ ] Conformed entities to transferable types for cross-app Siri tasks. |
| 197 | +- [ ] App Shortcuts defined via `AppShortcutProvider` for headline actions. |
| 198 | +- [ ] Text content backed by attributed strings; Writing Tools and Genmoji handled in |
| 199 | + custom text views. |
| 200 | +- [ ] Image Playground adopted via `imagePlaygroundSheet` / `ImagePlaygroundViewController` |
| 201 | + (or `ImageCreator` for programmatic generation). |
| 202 | +- [ ] Custom generative features delegated to `foundation-models`; on-device model files |
| 203 | + to `core-ai`. |
| 204 | +- [ ] Tested intents end-to-end (Siri, Spotlight, Shortcuts) on a real eligible device. |
| 205 | + |
| 206 | +## Resources |
| 207 | + |
| 208 | +- Apple Intelligence overview — <https://developer.apple.com/documentation/technologyoverviews/apple-intelligence> |
| 209 | +- Built-in intelligence (on-device vision/speech/NLP) — <https://developer.apple.com/documentation/technologyoverviews/built-in-intelligence> |
| 210 | +- Generative models overview — <https://developer.apple.com/documentation/technologyoverviews/generative-models> |
| 211 | +- Apple Intelligence and Siri AI — <https://developer.apple.com/documentation/appintents/apple-intelligence-and-siri-ai> |
| 212 | +- Spotlight integration — <https://developer.apple.com/documentation/appintents/spotlight> |
| 213 | +- Donations and discovery — <https://developer.apple.com/documentation/appintents/donations-and-discovery> |
| 214 | +- Getting started with App Intents — <https://developer.apple.com/documentation/appintents/getting-started-with-the-app-intents-framework> |
| 215 | +- App entities — <https://developer.apple.com/documentation/appintents/app-entities> |
| 216 | +- App Shortcuts — <https://developer.apple.com/documentation/appintents/app-shortcuts> |
| 217 | +- AppIntent protocol — <https://developer.apple.com/documentation/appintents/appintent> |
| 218 | +- Image Playground — <https://developer.apple.com/documentation/imageplayground> |
| 219 | +- Foundation Models quick start (eligibility code) — <https://developer.apple.com/documentation/foundationmodels/generating-content-and-performing-tasks-with-foundation-models> |
| 220 | +- Sibling skills: `foundation-models`, `core-ai`, `ios-app-intents` |
0 commit comments