Skip to content

feat(ai): Add App Functions for system AI integration#5585

Draft
jamesarich wants to merge 12 commits into
mainfrom
jamesarich/crispy-barnacle
Draft

feat(ai): Add App Functions for system AI integration#5585
jamesarich wants to merge 12 commits into
mainfrom
jamesarich/crispy-barnacle

Conversation

@jamesarich
Copy link
Copy Markdown
Collaborator

Motivation

Enable system AI assistants (Gemini, etc.) to interact with Meshtastic mesh networks through the Android AppFunctions framework. This gives AI agents read access to mesh status, nodes, channels, messages, and metrics -- plus the ability to send messages -- without opening the app UI.

Approach

Implements 9 App Functions in a layered KMP architecture:

  • Core layer (core/data/ai/): Platform-agnostic AiFunctionProvider interface and implementation with rate limiting, fuzzy name resolution, and proper error handling
  • Android layer (androidApp/src/google/): @AppFunction declarations with @AppFunctionSerializable response models, wired via Koin DI
  • Google flavor only: Functions are only available in the Google Play build (requires Play Services for the AppFunctions runtime)

Functions implemented

Function Type Description
sendMessage Write Send a text message to a node or channel
getMeshStatus Read Connection state + node counts
getNodeList Read All nodes with name, signal, battery, coordinates
getChannelInfo Read Channel name, role, and PSK status
getDeviceStatus Read Local radio firmware, battery, region
getNodeDetails Read Detailed info for a specific node
getMeshMetrics Read Network health metrics (online ratio, signal stats)
getRecentMessages Read Recent message history with contact resolution
getUnreadSummary Read Unread message counts per contact

Notable design decisions

  • Rate limiting: 10 requests/minute per function to prevent AI loops from overwhelming the radio
  • Fuzzy name resolution: Contacts can be referenced by partial name match (e.g., "James" resolves to the correct node)
  • Graceful disconnection handling: Functions that can answer from local DB (messages, unread) work even when disconnected; others return clear error messages
  • KDoc compliance: All @AppFunctionSerializable properties use inline KDoc (not class-level @property tags) per KSP extraction requirements
  • app_metadata.xml: Provides LLM-facing operational patterns and constraints for the AI runtime

Testing

  • Unit tests for core provider logic (edge cases, error paths)
  • End-to-end verified on android-36.1 emulator via adb shell cmd app_function execute-app-function -- all 9 functions respond correctly
  • Note: android-36.1 system image is required for ADB AppFunction shell testing (base API 36 and ext19 lack the shell command)

jamesarich and others added 12 commits May 21, 2026 10:50
Expose Meshtastic mesh networking capabilities (sendMessage, getMeshStatus)
to Android system AI agents via the App Functions API.

Architecture:
- AiFunctionProvider interface in core/data commonMain (platform-agnostic)
- FuzzyNameResolver for node/channel name matching (LCS algorithm)
- RateLimiter with 5-call/60s sliding window to protect mesh radio
- AiFunctionProviderImpl wiring repositories and use cases
- @appfunction declarations in androidApp Google flavor only
- GoogleMeshUtilApplication with AppFunctionConfiguration.Provider
- DI via AppFunctionsModule included in FlavorModule

Key design decisions:
- No confirmation dialog (AI invocation = user intent)
- Fuzzy name matching with 50% LCS threshold, error on ambiguity
- Admin channels excluded from resolution
- 5-second operation timeout
- 237-byte message length limit (Meshtastic standard)

Includes unit tests for RateLimiter and FuzzyNameResolver (LCS algorithm).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously, AiFunctionProviderImpl returned a hardcoded messageId of 0
for all successful sends, preventing unique message identification. The
underlying SendMessageUseCase generates a packetId but had no return
value to expose it.

Changes:
- Modified SendMessageUseCase interface to return Int (the packetId)
- Updated SendMessageUseCaseImpl to return the generated packetId
- Updated AiFunctionProviderImpl to capture and use the returned messageId

This enables the AI system to track individual messages and correlate
responses to specific send requests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…fo, getDeviceStatus)

Implement three Phase 2a read-only query functions for AI agent integration:

Core Data (commonMain):
- Extended AiFunctionProvider interface with 3 new methods
- Implemented getNodeList() - queries all mesh nodes with battery and online status
- Implemented getChannelInfo() - lists mesh channels with settings
- Implemented getDeviceStatus() - returns local device info and status
- Added result types: GetNodeListResult, GetChannelInfoResult, GetDeviceStatusResult
- Added data models: NodeSummary, ChannelSummary, DeviceStatus
- All functions protected by timeout (5s), rate limiter (5 calls/60s), connection check

Android AppFunctions:
- Added 3 @appfunction methods with KSP annotations
- Response models marked @AppFunctionSerializable for AI runtime
- Proper exception handling and timeout protection

Constants:
- HEX_RADIX (16): For node ID formatting
- MS_PER_SEC (1000): Time unit conversions
- ONLINE_THRESHOLD_MS (30000): Node online detection threshold

Fixes:
- Use node.user.long_name (not longName)
- Use node.deviceMetrics.battery_level?.coerceIn() for nullable battery
- Use nodeRepository.nodeDBbyNum (not nodes)
- Convert node.lastHeard (seconds) to milliseconds
- Suppress MagicNumber and ReturnCount lints appropriately

All builds and tests pass:
✓ :core:data:compileKotlinJvm
✓ :androidApp:compileGoogleDebugKotlin
✓ :androidApp:compileFdroidDebugKotlin
✓ detekt clean
✓ spotlessApply clean
✓ :core:data:allTests pass

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Extend AiFunctionProvider with two new suspend methods for advanced queries
- getNodeDetails: Retrieve per-node telemetry (16 fields) by hex or user ID
- getMeshMetrics: Aggregate mesh statistics and compute health score
- Add result types (GetNodeDetailsResult, GetMeshMetricsResult) and data models
- Add response models (@AppFunctionSerializable) for KSP serialization
- Both methods support timeout protection and rate limiting
- Health score calculation: 50 base + 50 online ratio, clamped 0-100
- All JVM, Android, detekt, spotless checks passing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix location filtering: Only treat (0,0) as invalid if position.time is 0
  - Previously filtered all (0,0) coords as null, losing valid equatorial data
  - Now checks position.time to distinguish 'no fix' from real coordinates

- Fix mostRecentPacketTime: Use max lastHeard from all nodes, not current time
  - Previously returned current time, making mesh appear always active
  - Now computes from actual node activity data

- Fix meshUptimeSeconds: Use local device's actual uptime, not epoch time
  - Previously returned epoch seconds (~1.7B), not elapsed time
  - Now uses device's DeviceMetrics.uptime_seconds

All checks passing: Android (Google/fdroid), detekt, spotless, unit tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Heard=0

- Add takeIf check to distinguish lastHeard=0 (never heard) from no nodes
- Previously: maxOfOrNull returns 0, Elvis operator doesn't trigger (0 is not null)
- Now: takeIf { it > 0 } filters out zero, falling back to current time
- Ensures API returns meaningful timestamp instead of epoch 1970

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add TimeoutCancellationException handling to getNodeDetails and getMeshMetrics
  AppFunctions (consistent with Phase 1 functions)
- Rethrow CancellationException in all provider catch blocks to preserve
  structured concurrency semantics
- Fix voltage documentation: millivolts → volts (matches actual Float field)
- Fix stale test comment referencing non-existent test class

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- SendMessageUseCase now rethrows exceptions after logging (Finding #1)
- AiFunctionProviderImpl catches send failures and returns InvalidArgument
- Added AiFunctionProviderImplTest with 10 unit tests covering:
  - Disconnection checks for all three function groups
  - Node lookup (found, not found, null position, invalid hex)
  - Metrics aggregation (active nodes, empty, zero lastHeard, degraded health)
  - Rate limiting behavior
- Expanded FuzzyNameResolverTest with 8 behavioral tests (Finding #5):
  - resolveNodeName: exact, fuzzy, ambiguous, not found
  - resolveChannelName: exact, admin exclusion, empty channels

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The model's isOnline property uses the project-standard 2-hour window
(onlineTimeThreshold), consistent with getMeshStatus().onlineNodeCount.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The interface now returns Int (packetId), so the test mock must
return an Int instead of Unit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Phase 3 of App Functions integration: read-only message history
functions that enable 'catch me up' voice queries via system AI.

New functions:
- getRecentMessages: Retrieve recent messages with optional contact
  filter and configurable limit (1-50, default 20)
- getUnreadSummary: Per-contact unread breakdown excluding muted
  contacts, sorted by most recent

Implementation details:
- KMP interface + sealed result types in core:data
- Android @appfunction declarations with @AppFunctionSerializable models
- Fuzzy name resolution for contact filtering
- Channel name resolution for broadcast contacts
- Tests for contact-not-found and empty unread scenarios

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Audit findings addressed:
- Add res/xml/app_metadata.xml with LLM-facing operational patterns,
  workflow dependencies, and constraints for the AppFunctions suite
- Register app_metadata in Google flavor AndroidManifest.xml
- Convert all @AppFunctionSerializable class-level @Property tags to
  inline KDoc per property (required by KSP for doc extraction)
- Add app_description string resource for displayDescription

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the enhancement New feature or request label May 23, 2026
@github-actions
Copy link
Copy Markdown
Contributor

📄 Docs staleness check — advisory

This PR modifies user-facing UI source files but does not update any page under docs/en/user/ or docs/en/developer/.

⚠️ Doc changes propagate to 3 consumers: in-app docs browser, Jekyll site (GitHub Pages), and meshtastic.org (Docusaurus sync). Updating a page in docs/en/ automatically flows to all three.

Changed source files:

core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/BuildNodeDescription.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/LastHeardInfo.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/MaterialBatteryInfo.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/NodeCardGlow.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/NodeItemCompact.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/TelemetryInfo.kt
core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/component/TransportIcon.kt
feature/docs/src/commonMain/kotlin/org/meshtastic/feature/docs/ui/ChirpyAssistantSheet.kt
feature/docs/src/commonMain/kotlin/org/meshtastic/feature/docs/ui/ChirpyFab.kt
feature/docs/src/commonMain/kotlin/org/meshtastic/feature/docs/ui/DocsBrowserScreen.kt
feature/docs/src/commonMain/kotlin/org/meshtastic/feature/docs/ui/DocsPageRouteScreen.kt
feature/messaging/src/commonMain/kotlin/org/meshtastic/feature/messaging/component/MessageItem.kt
feature/node/src/commonMain/kotlin/org/meshtastic/feature/node/component/DeviceDetailsSection.kt
feature/node/src/commonMain/kotlin/org/meshtastic/feature/node/component/NodeDetailsSection.kt
feature/node/src/commonMain/kotlin/org/meshtastic/feature/node/component/NodeItem.kt
feature/node/src/commonMain/kotlin/org/meshtastic/feature/node/component/NodeListHelp.kt
feature/node/src/commonMain/kotlin/org/meshtastic/feature/node/component/NodeStatusIcons.kt
feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/component/NodeLayoutSettings.kt

What to check:

Changed area Likely doc page
feature/messaging/ docs/en/user/messages-and-channels.md
feature/node/ docs/en/user/nodes.md or docs/en/user/node-metrics.md
feature/map/ docs/en/user/map-and-waypoints.md
feature/connections/ docs/en/user/connections.md
feature/settings/ docs/en/user/settings-radio-user.md or docs/en/user/settings-module-admin.md
feature/firmware/ docs/en/user/firmware.md
feature/intro/ docs/en/user/onboarding.md
feature/discovery/ docs/en/user/discovery.md
feature/docs/ Internal docs infrastructure
core/ui/ docs/en/developer/codebase.md or component-specific user pages

New page checklist (if adding a new doc page):

  1. Create the .md file in docs/en/user/ or docs/en/developer/ with last_updated frontmatter
  2. Register in DocBundleLoader.kt with string resources (in-app browser)
  3. Jekyll and Docusaurus sync pick up new pages automatically — no config change needed

If this PR does not require a doc update (e.g., internal refactor, bug fix, test change), add the skip-docs-check label to dismiss this check.

Cross-platform note: This check is advisory while doc coverage matures. Both Android and Apple repos use the same skip-docs-check label and advisory severity. See meshtastic/design standards for shared conventions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant