Skip to content

Commit c28a788

Browse files
duyetduyetbot
andauthored
feat(skills): add Foundation Models, Apple Intelligence, macOS best practices (#75)
- build-ios-apps/foundation-models: on-device LLM via LanguageModelSession (SystemLanguageModel availability, @generable guided generation, GenerationSchema, tool calling, system/PCC/custom backends). - build-ios-apps/apple-intelligence: participation layer (App Intents, entities, schemas, Spotlight, Writing Tools, Image Playground); cross-references sibling skills. - build-macos-apps/macos-app-best-practices: scene/window/menu architecture, SwiftUI vs AppKit vs Mac Catalyst, from Apple SwiftUI/AppKit/Mac Catalyst docs. Co-authored-by: duyetbot <duyetbot@users.noreply.github.com>
1 parent b92b2c4 commit c28a788

3 files changed

Lines changed: 661 additions & 0 deletions

File tree

  • build-ios-apps/skills
  • build-macos-apps/skills/macos-app-best-practices
Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
---
2+
name: apple-intelligence
3+
description: Apple Intelligence participation layer. Wire an app's actions and content into Siri, Spotlight, Writing Tools, Image Playground, and summaries via App Intents, entities, and schemas. Use before adding App Intents/schemas, Spotlight indexing, Writing Tools, or Image Playground; defer custom generative features and on-device model files to the sibling skills.
4+
---
5+
6+
# Apple Intelligence
7+
8+
## Overview
9+
10+
Apple Intelligence is the personal intelligence system behind built-in capabilities
11+
across Apple platforms. It runs on-device on Apple silicon and in Private Cloud
12+
Compute. Your app participates by teaching the system about its **actions** and
13+
**content** so system features — Siri, Spotlight, Shortcuts, Writing Tools, Image
14+
Playground, Visual Intelligence, Genmoji, and summaries — can use them.
15+
16+
The participation contract is the **App Intents** framework: an `AppIntent` wraps one
17+
of your app's actions; an `AppEntity` represents the data those actions operate on.
18+
Apple Intelligence uses donated intents/entities, your Spotlight index, and declared
19+
schemas to find and act on your content — even when described vaguely.
20+
21+
Two sibling skills handle the custom-model side and are out of scope here:
22+
23+
- **`foundation-models`** — the Foundation Models framework for custom generative
24+
features (prompting, guided generation, tool calling, Private Cloud Compute).
25+
- **`core-ai`** — running your own deep-learning model files (`.aimodel`) on-device.
26+
27+
This skill covers only the built-in intelligence surfaces an app adopts.
28+
29+
## Device eligibility and availability
30+
31+
Apple Intelligence requires Apple silicon (iPhone 15 Pro and later, iPad with A17 Pro
32+
or M-series, Mac with M-series). It is opt-in via the Apple Intelligence toggle in
33+
Settings. Before exposing intelligence features, branch on the system model's
34+
`availability` rather than assuming it is on:
35+
36+
```swift
37+
import FoundationModels
38+
39+
private var model = SystemLanguageModel.default
40+
41+
switch model.availability {
42+
case .available:
43+
// Show your intelligence UI.
44+
case .unavailable(.deviceNotEligible):
45+
// Show an alternative UI.
46+
case .unavailable(.appleIntelligenceNotEnabled):
47+
// Ask the person to turn on Apple Intelligence.
48+
case .unavailable(.modelNotReady):
49+
// Model is downloading or not ready.
50+
case .unavailable(let other):
51+
// Unknown reason.
52+
}
53+
```
54+
55+
## How an app participates: App Intents, entities, schemas
56+
57+
The single biggest lever. Make the system aware of your actions and data via the App
58+
Intents framework:
59+
60+
- **App intents** — a custom type encapsulating one action. Ship in your app or an app
61+
extension so the action runs even when the app isn't open. Each intent performs an
62+
action in `perform()`, returns a result/error, declares parameters, and provides a
63+
localized title Siri and Shortcuts can display.
64+
- **App entities** — lightweight versions of your data objects. Define them only for
65+
the subset of data people see and might reference with Siri ("this photo", "this
66+
album"). Real data objects stay the source of truth.
67+
- **App enums** — enumerate fixed choices (e.g. repeat options) so the system can
68+
resolve parameters.
69+
- **Donations** — when someone performs an action in your UI, donate the matching
70+
intent/entity. Donations give Apple Intelligence behavioral cues to predict and
71+
disambiguate. Donate only direct UI interactions, not actions Siri or Shortcuts
72+
initiated.
73+
- **Schemas (domains)** — the predefined intent/entity shapes for common actions (mail,
74+
messaging, files, etc.). Prefer building from a schema over a custom intent; schemas
75+
are the contract Apple Intelligence uses to match everyday phrases.
76+
77+
Grounded minimal `AppIntent` shape:
78+
79+
```swift
80+
struct OrderAlbum: AppIntent {
81+
static var title: LocalizedStringResource { "Order Album" }
82+
static var description = IntentDescription("Order a vinyl record album.")
83+
84+
@Parameter(title: "Album", description: "The name of the album to order.")
85+
var albumName: String
86+
87+
@Dependency
88+
private var albumManager: AlbumDataManager
89+
90+
func perform() async throws -> some IntentResult {
91+
// Perform the action...
92+
return .result()
93+
}
94+
95+
static var parameterSummary: some ParameterSummary {
96+
Summary("Order \(\.$albumName)")
97+
}
98+
}
99+
```
100+
101+
> For full intent/entity/schema implementation patterns, see the sibling
102+
> `ios-app-intents` skill.
103+
104+
## App Shortcuts
105+
106+
An `AppShortcut` bundles an `AppIntent` with a title, image, preconfigured parameters,
107+
and spoken phrases so it appears polished in Shortcuts and other system experiences.
108+
The compiler generates system metadata, so shortcuts are available the moment someone
109+
installs your app — no registration needed. Define them in a type adopting the
110+
`AppShortcutProvider` protocol, alongside the intents they use. Surface them in-app
111+
with tip views.
112+
113+
## Apple Intelligence + Siri AI: the extra steps
114+
115+
Beyond a basic App Intents adoption, make content discoverable by Apple Intelligence
116+
specifically:
117+
118+
- **Index entities into Spotlight** — Apple Intelligence uses Spotlight's semantic
119+
search to find your content even when described vaguely. This is what makes "the
120+
email from Mei about the Q3 plan" work.
121+
- **Choose transferable types** — conform entities/values to transferable types so the
122+
system can move content across apps for cross-app Siri tasks.
123+
- **Adopt schemas** — the contract the system uses to identify, query, and understand
124+
actions and content, and to match them to conversational phrases.
125+
- **Associate entities with views / user activities** — gives Apple Intelligence
126+
onscreen context so someone can refer to what's visible ("this photo").
127+
- **Donate actions and content** — behavioral cues for prediction and disambiguation.
128+
129+
## Spotlight / semantic indexing
130+
131+
Indexing your app's content makes it findable in search and lets Siri and other system
132+
features locate app-specific data. If you define app entities for your data, index
133+
those entities with the rest of your content. When a search finds your content, the
134+
system uses the associated entity to open your app and navigate to it. Apple
135+
Intelligence retrieves entities it finds in your Spotlight index to interact with your
136+
content.
137+
138+
## Writing Tools and Genmoji
139+
140+
Writing Tools (proofreading, rewrite, summaries) integrate into standard system text
141+
views automatically:
142+
143+
- Adopt `NSAttributedString` / attributed strings as the backing store for text content.
144+
- Use standard text views whenever possible and customize via their configuration
145+
options.
146+
- Use the Writing Tools API directly only if you have a custom text-editing engine or
147+
can't use system views.
148+
- Genmoji is built into system text views. In custom views, add Genmoji text
149+
attachments and read/write them correctly when persisting to a custom file format.
150+
151+
## Image Playground
152+
153+
The ImagePlayground framework gives a system interface to generate images from a text
154+
description, an optional source image, and a style. Present a system sheet (SwiftUI) or
155+
view controller (UIKit/AppKit); the system manages all interaction and delivers the
156+
resulting image. Key API surface:
157+
158+
- `ImagePlaygroundViewController` — the standard system interface (UIKit/AppKit).
159+
- `ImagePlaygroundConcept` — text elements specifying content to include.
160+
- `ImagePlaygroundStyle` / `ImagePlaygroundOptions` — style and generation options.
161+
- `ImageCreator` — generate images programmatically without the UI.
162+
- SwiftUI sheet modifier:
163+
164+
```swift
165+
imagePlaygroundSheet(
166+
isPresented: Binding<Bool>,
167+
concept: String, // or concepts: [ImagePlaygroundConcept]
168+
sourceImage: Image?, // or sourceImageURL: URL
169+
onCompletion: (URL) -> Void,
170+
onCancellation: (() -> Void)?
171+
)
172+
```
173+
174+
## Visual Intelligence
175+
176+
For object/scene scanning via Camera Control, adopt Visual Intelligence. The framework
177+
detects content and exchanges information with your app using App Intents — so your App
178+
Intents participation is also what makes you a destination for Visual Intelligence
179+
results.
180+
181+
## Checklist
182+
183+
- [ ] Check `SystemLanguageModel.default.availability` before showing intelligence UI;
184+
handle `.deviceNotEligible` and `.appleIntelligenceNotEnabled`.
185+
- [ ] Identified the user-visible actions and data; modeled them as `AppIntent` and
186+
`AppEntity`.
187+
- [ ] Built common actions from built-in schema domains (not bespoke intents) where one
188+
exists.
189+
- [ ] Each intent has `perform()`, a result/error, declared `@Parameter`s, and a
190+
localized `title`.
191+
- [ ] Entities are lightweight and cover only what people see/reference.
192+
- [ ] Donating intents/entities on direct UI interactions (not Siri/Shortcuts ones).
193+
- [ ] App entities indexed into Spotlight (enables semantic search by Apple
194+
Intelligence).
195+
- [ ] Entities associated with views/user activities for onscreen context.
196+
- [ ] Conformed entities to transferable types for cross-app Siri tasks.
197+
- [ ] App Shortcuts defined via `AppShortcutProvider` for headline actions.
198+
- [ ] Text content backed by attributed strings; Writing Tools and Genmoji handled in
199+
custom text views.
200+
- [ ] Image Playground adopted via `imagePlaygroundSheet` / `ImagePlaygroundViewController`
201+
(or `ImageCreator` for programmatic generation).
202+
- [ ] Custom generative features delegated to `foundation-models`; on-device model files
203+
to `core-ai`.
204+
- [ ] Tested intents end-to-end (Siri, Spotlight, Shortcuts) on a real eligible device.
205+
206+
## Resources
207+
208+
- Apple Intelligence overview — <https://developer.apple.com/documentation/technologyoverviews/apple-intelligence>
209+
- Built-in intelligence (on-device vision/speech/NLP) — <https://developer.apple.com/documentation/technologyoverviews/built-in-intelligence>
210+
- Generative models overview — <https://developer.apple.com/documentation/technologyoverviews/generative-models>
211+
- Apple Intelligence and Siri AI — <https://developer.apple.com/documentation/appintents/apple-intelligence-and-siri-ai>
212+
- Spotlight integration — <https://developer.apple.com/documentation/appintents/spotlight>
213+
- Donations and discovery — <https://developer.apple.com/documentation/appintents/donations-and-discovery>
214+
- Getting started with App Intents — <https://developer.apple.com/documentation/appintents/getting-started-with-the-app-intents-framework>
215+
- App entities — <https://developer.apple.com/documentation/appintents/app-entities>
216+
- App Shortcuts — <https://developer.apple.com/documentation/appintents/app-shortcuts>
217+
- AppIntent protocol — <https://developer.apple.com/documentation/appintents/appintent>
218+
- Image Playground — <https://developer.apple.com/documentation/imageplayground>
219+
- Foundation Models quick start (eligibility code) — <https://developer.apple.com/documentation/foundationmodels/generating-content-and-performing-tasks-with-foundation-models>
220+
- Sibling skills: `foundation-models`, `core-ai`, `ios-app-intents`

0 commit comments

Comments
 (0)