Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions engines/llamacpp/rac_vlm_llamacpp.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ enum class VLMModelType {
SmolVLM, // SmolVLM uses "User:" / "Assistant:" format
Qwen2VL, // Qwen2-VL uses chatml with <|im_start|>user format
LLaVA, // LLaVA uses "USER:" / "ASSISTANT:" format
LFM2VL, // Liquid LFM2-VL uses <|startoftext|> + ChatML + <image>
Generic // Generic chatml fallback
};

Expand Down Expand Up @@ -226,6 +227,17 @@ std::string format_vlm_prompt_manual(const std::string& user_content, const char
formatted += user_content;
formatted += "\nASSISTANT:";
break;
case VLMModelType::LFM2VL:
formatted = "<|startoftext|>";
if (effective_system) {
formatted += "<|im_start|>system\n";
formatted += effective_system;
formatted += "<|im_end|>\n";
}
formatted += "<|im_start|>user\n";
formatted += user_content;
formatted += "<|im_end|>\n<|im_start|>assistant\n";
break;
case VLMModelType::Qwen2VL:
case VLMModelType::Generic:
case VLMModelType::Unknown:
Expand Down Expand Up @@ -293,6 +305,11 @@ VLMModelType detect_vlm_model_type(llama_model* model) {
RAC_LOG_DEBUG(LOG_CAT, "Detected SmolVLM model type from architecture");
return VLMModelType::SmolVLM;
}
if (arch.find("lfm2") != std::string::npos &&
(arch.find("vl") != std::string::npos || arch.find("vision") != std::string::npos)) {
RAC_LOG_DEBUG(LOG_CAT, "Detected LFM2-VL model type from architecture");
return VLMModelType::LFM2VL;
}
}

// Fallback: model name metadata when architecture is missing or generic
Expand All @@ -319,6 +336,11 @@ VLMModelType detect_vlm_model_type(llama_model* model) {
RAC_LOG_DEBUG(LOG_CAT, "Detected LLaVA model type from name");
return VLMModelType::LLaVA;
}
if (name.find("lfm2-vl") != std::string::npos || name.find("lfm2vl") != std::string::npos ||
(name.find("lfm2") != std::string::npos && name.find("vl") != std::string::npos)) {
RAC_LOG_DEBUG(LOG_CAT, "Detected LFM2-VL model type from name");
return VLMModelType::LFM2VL;
}
}

// Final fallback: chat-template sniffing for SmolVLM-style models that
Expand Down
7 changes: 1 addition & 6 deletions examples/ios/RunAnywhereAI/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ See `docs/RELEASE_INSTRUCTIONS.md`. Key step: after building, run `./scripts/pat
| 0 | `ChatInterfaceView` | LLM chat with tool calling, LoRA, analytics |
| 1 | `VisionHubView` | VLM camera |
| 2 | `VoiceAssistantView` | Full voice agent (STT + LLM + TTS pipeline) |
| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, Solutions, Voice Keyboard |
| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, Voice Keyboard |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Mark Voice Keyboard as iOS-only in the navigation table.

Line 67 currently reads like MoreHubView exposes Voice Keyboard on every platform, but the runtime view gates that entry behind #if os(iOS). The doc is inaccurate for macOS.

✏️ Suggested doc fix
-| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, Voice Keyboard |
+| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, iOS-only Voice Keyboard |
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, Voice Keyboard |
| 3 | `MoreHubView` | RAG, STT, TTS, VAD, Storage, iOS-only Voice Keyboard |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/ios/RunAnywhereAI/AGENTS.md` at line 67, Update the navigation table
entry for MoreHubView so Voice Keyboard is clearly marked as iOS-only. In
AGENTS.md, adjust the MoreHubView row to reflect the same platform gating used
in the runtime view’s `#if` os(iOS) logic, so the docs do not imply Voice Keyboard
appears on macOS.

| 4 | `CombinedSettingsView` | Generation params, API keys, tools, storage |

> **Deferred backends.** MetalRT and Diffusion (image generation)
Expand Down Expand Up @@ -126,7 +126,6 @@ RunAnywhereAI/
│ ├── Models/ # Model browser/downloader (7 files)
│ ├── Storage/ # Disk usage management (2 files)
│ ├── Settings/ # App configuration (3 files)
│ └── Solutions/ # YAML pipeline demo (1 file)
├── Shared/
│ ├── SharedConstants.swift # IPC keys, Darwin notification names, URL scheme
│ └── SharedDataBridge.swift # App Group UserDefaults + Darwin CFNotificationCenter
Expand Down Expand Up @@ -262,10 +261,6 @@ Deterministic performance testing across 4 modalities (LLM, STT, TTS, VLM). Each

`ToolSettingsViewModel`: registers/clears demo tools via `RunAnywhere.registerTool(definition:executor:)`. Includes `SafeMathEvaluator` (recursive-descent parser) for the `calculate` tool.

### 13. Solutions (`Features/Solutions/`)

Minimal demo of `RunAnywhere.solutions.run(yaml:)` — the SDK's declarative pipeline API. Two hardcoded YAML strings (voice agent, RAG) submitted to SDK, lifecycle callbacks logged.

---

## Markdown Rendering Pipeline
Expand Down
11 changes: 0 additions & 11 deletions examples/ios/RunAnywhereAI/RunAnywhereAI/App/ContentView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -167,17 +167,6 @@ struct MoreHubView: View {
)
}

NavigationLink {
SolutionsView()
} label: {
FeatureRow(
icon: "square.stack.3d.up",
iconColor: .teal,
title: "Solutions",
subtitle: "Run prepackaged pipelines from YAML"
)
}

#if os(iOS)
NavigationLink {
VoiceDictationManagementView()
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -561,22 +561,47 @@ final class VoiceAgentViewModel: ObservableObject {
case let .error(err):
logger.error("Voice agent error: \(err.message)")
errorMessage = err.message
sessionState = .error(err.message)
currentStatus = "Error"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turn error ends active UI

Medium Severity

Every .error voice event now sets sessionState to .error, so isActive becomes false while streamVoiceAgent() and the SDK mic driver keep running. A single failed turn can show a dead session in the UI even though capture continues.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 6bf113c. Configure here.


case let .sessionError(err):
logger.error("Voice session error: \(err.message)")
errorMessage = err.message
if !err.recoverable {
sessionState = .error(err.message)
currentStatus = "Error"
}

case .sessionStopped:
sessionState = .disconnected
currentStatus = "Ready"
audioLevel = 0
isSpeechDetected = false

case .sessionStarted:
sessionState = .listening
currentStatus = "Listening..."

case .agentResponseStarted:
assistantResponse = ""
currentTranscript = ""

Comment on lines +585 to +588

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Don’t clear the finalized user transcript when the response starts.

.userSaid sets currentTranscript, but .agentResponseStarted can arrive right after and erase it before the user sees what was transcribed. Clear only the assistant response here.

Proposed fix
         case .agentResponseStarted:
             assistantResponse = ""
-            currentTranscript = ""
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
case .agentResponseStarted:
assistantResponse = ""
currentTranscript = ""
case .agentResponseStarted:
assistantResponse = ""
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@examples/ios/RunAnywhereAI/RunAnywhereAI/Features/Voice/VoiceAgentViewModel.swift`
around lines 585 - 588, The .agentResponseStarted handling in
VoiceAgentViewModel is clearing both assistantResponse and currentTranscript,
which removes the finalized user transcript too early. Update the
.agentResponseStarted case to clear only assistantResponse and leave
currentTranscript intact so the .userSaid transcript remains visible. Use the
VoiceAgentViewModel state handling around .userSaid and .agentResponseStarted to
locate the change.

case .agentResponseCompleted:
sessionState = .listening
currentStatus = "Listening..."

case let .audioLevel(level):
audioLevel = min(max(level.rms, 0), 1)

case .wakewordDetected:
sessionState = .listening
currentStatus = "Listening..."
isSpeechDetected = false

case .interrupted, .metrics, .none:
// No UX-visible effect for these arms today.
break

// The regenerated RAVoiceEvent payload added new arms; we do not
// surface them in the UI yet, so they are intentionally folded into
// the same no-op bucket as .interrupted / .metrics.
case .componentStateChanged, .sessionError, .sessionStarted,
.sessionStopped, .agentResponseStarted, .agentResponseCompleted,
.speechTurnDetection, .turnLifecycle:
case .componentStateChanged, .speechTurnDetection, .turnLifecycle:
break
default:
break
Expand Down

This file was deleted.

Loading
Loading