Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
155 changes: 155 additions & 0 deletions build-ios-apps/skills/core-ai/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
name: core-ai
description: Run on-device AI models in iOS, iPadOS, macOS, and visionOS apps with Core AI. Use when bundling .aimodel files, loading an AIModel, running inference over NDArray tensors, or compiling models ahead of time with coreai-build.
---

# Core AI: On-Device AI Models

## Overview

Core AI runs AI models **on device** inside your app. Inference stays private, works
offline, and has no per-inference cost. You start from a `.aimodel` file (converted
from a model or already in the correct format) that contains one or more named
inference functions, bundle it, load it as an `AIModel`, and run those functions
over `NDArray` tensors (image inputs use a pixel-buffer value).

Available on **Apple Intelligence** devices: iPhone/iPad with A17 Pro or later, Mac
with M1 or later, Apple Vision Pro (M2+). Building requires the **Metal Toolchain**
in Xcode (not installed by default) — without it, builds that include `.aimodel`
files fail with a missing Metal compiler error.

> Core AI runs your own model files at the tensor level. For Apple's high-level
> on-device LLM API (chat sessions, tool calling, guided generation), use the
> Foundation Models framework — a separate framework, not covered here.

## Workflow

1. **Inspect the model** in Xcode's model viewer:
- _General_ — size (parameter count and on-disk storage), numeric precision split
into **compute** (used during inference) and **storage** (weights on disk),
operation distribution, and editable metadata.
- _Functions_ — each inference function's input/output names, types, and shapes.
A `?` in a dimension means it is dynamic (supplied or determined at runtime).
2. **Bundle the model** — add the `.aimodel` file to the Xcode target (it appears in
the _Compile Sources_ build phase). Install the Metal Toolchain first.
3. **Load the model** — `AIModel(contentsOf:)` is asynchronous because Core AI
**specializes** the model for the current device and selects the compute units
that deliver the best performance. For large models this can take significant
time, so consider ahead-of-time compilation (below).
4. **Load a function** — `model.loadFunction(named:)` returns the function or `nil`
when no function with that name exists (it throws on other load failures). Use
`functionNames` when a model has multiple functions. The same inference function
is safe to call from concurrent tasks.
5. **Prepare inputs** — match each input's shape and scalar type from
`function.descriptor`, then write data through a mutable view.
6. **Run and read outputs** — `function.run(inputs:)` returns outputs keyed by name;
pull each result with `outputs.remove(_:)` and read it through a view.

## Core API

```swift
import CoreAI

// 1. Specialize for this device and load the model.
let model = try await AIModel(contentsOf: urlOfModel)

// 2. Load the inference function. Returns nil if the name is absent.
guard let function = try model.loadFunction(named: "main") else {
// Handle a missing function.
}

// 3. Verify the input's shape and scalar type from the descriptor.
let descriptor = function.descriptor
guard let valueDescriptor = descriptor.inputDescriptor(of: "input"),
case .ndArray(let arrayDescriptor) = valueDescriptor,
arrayDescriptor.shape == [3, 4],
arrayDescriptor.scalarType == .float32 else {
// Handle an unexpected type or shape.
}

// 4. Create the input tensor and write data through a mutable view.
var input = NDArray(shape: [3, 4], scalarType: .float32)
var mutableView = input.mutableView(as: Float.self)
guard let elements = mutableView.contiguousElements else {
// Handle a non-contiguous memory layout.
}
writeInputData(into: elements)

// 5. Run inference and extract the named output.
var outputs = try await function.run(inputs: ["input": input])
guard let value = outputs.remove("prediction"),
let prediction = value.ndArray else {
// Handle a missing or unexpected output.
}
processOutput(prediction.view())
```

### Tensors and values

- `NDArray` — an n-dimensional tensor. Build it with `NDArray(shape:scalarType:)`.
It is **read-only by default**: use `mutableView(as:)` → `contiguousElements` to
write, and `view()` to read. Swift enforces read vs. write access at compile time.
- `scalarType` — the element type, e.g. `.float32`. Shape is an `[Int]` matching the
model's expectation; a `?` dimension in the viewer is dynamic.
- **Images** — values marked as images at conversion time use a pixel-buffer value
rather than `NDArray`.
- `ValueDescriptor` — `.ndArray(ArrayDescriptor)` vs. image cases. Inspect
`descriptor.inputDescriptor(of:)` / `outputDescriptor(of:)` at runtime so the app
can adapt if a function's signature changes between deployments without code edits.

## Ahead-of-Time (AOT) Compilation

On-device specialization can delay first load. Move the most expensive part — model
compilation — to the build machine with the **`coreai-build`** CLI. It converts
`MyModel.aimodel` into one `MyModel.<arch>.aimodelc` asset per device architecture.
At runtime the app picks the asset for the current architecture and loads it with the
**same** `AIModel` API, so loading code does not change.

```bash
# 1. Install the Metal Toolchain (also: Xcode > Settings > Components > Get).
xcodebuild -downloadComponent MetalToolchain

# 2. Compile one .aimodelc per architecture.
xcrun coreai-build compile MyModel.aimodel --platform iOS --output compiled/

# Override compute units, deployment version, target arch, and more:
xcrun coreai-build compile MyModel.aimodel --platform macOS \
--preferred-compute gpuAndNeuralEngine --output compiled/
xcrun coreai-build compile --help
```

```swift
// Select the compiled asset for this device, then load normally.
let arch = AIModel.deviceArchitectureName
let assetName = "MyModel.\(arch).aimodelc"
let model = try await AIModel(contentsOf: bundledURL(for: assetName))
```

Notes:

- `coreai-build` emits one `.<arch>.aimodelc` per architecture; the filename prefix
comes from the input model. `AIModel.deviceArchitectureName` is the identifier that
matches `<arch>` at runtime.
- Compute units default to best performance. Pass `--preferred-compute` to override,
and use matching load options.
- A compiled asset still requires **some** on-device specialization — AOT removes the
bulk of compilation, not all of it. AOT only targets Apple Intelligence devices.

## Checklist

- [ ] Metal Toolchain installed (Xcode, or `xcodebuild -downloadComponent MetalToolchain`).
- [ ] `.aimodel` added to the target and visible in _Compile Sources_.
- [ ] `AIModel(contentsOf:)` awaited; slow first load handled (or AOT-compiled).
- [ ] `loadFunction(named:)` nil-checked; `functionNames` inspected for multi-function models.
- [ ] Input `shape` / `scalarType` verified against `function.descriptor`.
- [ ] Mutable views for writes, read-only views for reads.
- [ ] Outputs extracted by name with `outputs.remove(_:)`.
- [ ] For large models: per-architecture `.aimodelc` built via `coreai-build`, and the
correct asset selected at runtime using `AIModel.deviceArchitectureName`.

## Resources

- Apple — _Integrating on-device AI models in your app with Core AI_
- Apple — _Compiling Core AI models ahead of time_
- Prefer Apple docs for up-to-date API details; web-search the current Core AI
documentation alongside this skill.
25 changes: 25 additions & 0 deletions build-ios-apps/skills/swiftui-liquid-glass/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,31 @@ description: Implement and review iOS 26+ SwiftUI Liquid Glass UI. Use when adop

Use this skill to build or review SwiftUI features that fully align with the iOS 26+ Liquid Glass API. Prioritize native APIs (`glassEffect`, `GlassEffectContainer`, glass button styles) and Apple design guidance. Keep usage consistent, interactive where needed, and performance aware.

## Adoption Guidance

Liquid Glass is a dynamic material that forms a distinct functional layer for
controls and navigation. Adopting it well is mostly about getting out of the way:

- **Prefer standard components.** Bars, sheets, popovers, and controls from SwiftUI
(and UIKit) adopt Liquid Glass automatically when you build with the latest SDK.
Reach for the explicit `glassEffect` API only when a standard component does not
already give you the treatment you need.
- **Remove custom backgrounds on controls and navigation.** Custom backgrounds in
split views, tab bars, and toolbars can overlay or interfere with Liquid Glass and
the scroll-edge effect. Let the system determine background appearance; remove
custom effects unless they are essential.
- **Test accessibility and display settings.** Reduce Transparency, Reduce Motion,
and the user's preferred Liquid Glass look can remove or modify effects. Standard
components adapt automatically; verify your custom elements, colors, and
animations under these configurations.
- **Use glass on custom controls sparingly.** Overusing it distracts from content.
Limit glass to the most important functional elements.
- **App icons are now layered and dynamic.** They respond to lighting and visual
effects, and use a standardized, concentric icon grid. Provide default (light),
dark, clear, and tinted appearances.
- **New navigation affordances.** `TabView` gains `.tabBarMinimizeBehavior(.onScrollDown)`
and a search role via `Tab(role: .search)` (UIKit: `UISearchTab`).

## Workflow Decision Tree

Choose the path that matches the request:
Expand Down
1 change: 1 addition & 0 deletions build-macos-apps/skills/appkit-interop/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ the source of truth, while AppKit handles the imperative edge.
- `references/window-panels.md`: window access, utility windows, and open/save panels.
- `references/responder-menus.md`: first responder, command routing, and menu validation.
- `references/drag-drop-pasteboard.md`: pasteboard, file URLs, and desktop drag/drop edges.
- `references/appkit-overview.md`: AppKit topic map and notable symbols for orienting before bridging.

## Guardrails

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# AppKit Overview

AppKit provides the objects to build a macOS app's user interface and to manage
interaction between the app, the person, and macOS: windows, panels, buttons, text
fields, event handling, drawing, printing, animation, documents, drag and drop,
menus, accessibility, and localization. It interops with SwiftUI in both directions
— AppKit views and view controllers embed into SwiftUI via representables, and
SwiftUI views can be hosted inside AppKit.

## When to reach for AppKit

Prefer SwiftUI for new macOS UI. Reach for AppKit directly only when SwiftUI lacks a
capability (window behavior, responder-chain control, a specific control, text-system
behavior, menus). See the parent `appkit-interop` skill for the smallest-bridge
approach.

## Topic map

- **App and Environment** — app lifecycle, run loop, environment, appearance.
- **Windows, Panels, and Screens** — `NSWindow`, panels, full-screen, restoration.
- **Views and Controls** — buttons, fields, sliders, segmented controls, toolbars.
- **Text System** — text storage, selection, layout, attachments, view providers.
- **Documents, Data, and Pasteboard** — document architecture, pasteboard, undo.
- **Drawing** — `NSBezierPath`, graphics contexts, images and PDF.
- **Color and Fonts** — color management, font descriptors, typography.
- **Menus, Cursors, and the Dock** — menu bars, menu items, dock tiles.
- **Mouse, Keyboard, and Trackpad** — event handling, gestures, responders.
- **Drag and Drop** — pasteboard-based dragging sessions.
- **Animation** — `NSAnimationContext`, animator proxies, layer-backed views.
- **Accessibility** — accessibility elements, traits, and actions.
- **Cocoa Bindings** — KVO/bindings to connect UI to models.
- **Appearance Customization** — light/dark, accent, vibrancy.
- **Mac Catalyst** — bring iPad apps to the Mac.

## Notable symbols from the AppKit reference

These appear in the current AppKit reference; confirm availability and semantics in
the Apple Developer docs before relying on them.

- `NSRefreshController` — a refresh control for AppKit scroll views.
- `NSTextSelectionManager` (plus `.Delegate`, `.Mode`) — unified text selection.
- `NSMenuItem.ImageVisibility` — control menu-item icon visibility.
- `NSSegmentedControl.Role` — per-segment roles.
- `NSStatusItemExpandedInterfaceDelegate` / `NSStatusItemExpandedInterfaceSession` —
expanded status items.
- `NSTextViewportRenderingSurface` / `NSTextViewportRenderingSurfaceKey` — text
viewport rendering hooks.
10 changes: 10 additions & 0 deletions build-macos-apps/skills/liquid-glass/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,16 @@ scrims, and clipping before adding more effects.
Liquid Glass behavior; missing foreground activation can make a design bug
look like a rendering bug.

## App Icons and Accessibility

- App icons are now layered and dynamic on macOS too. They respond to lighting and
system effects and follow a standardized, concentric icon grid. Ship the default,
dark, clear, and tinted appearances rather than a single flat icon.
- Verify custom glass, colors, and animations under Reduce Transparency, Reduce
Motion, and the user's preferred Liquid Glass look. Standard components adapt
automatically; custom surfaces and animations may need explicit handling so they
stay legible and calm under these settings.

## When To Use Other Skills

- Use `swiftui-patterns` when the main question is scene architecture,
Expand Down
Loading