Skip to content

IMPLEMENTATION_GUIDE.md

Latest commit

 

History

History
387 lines (329 loc) · 12.9 KB

IMPLEMENTATION_GUIDE.md

File metadata and controls

387 lines (329 loc) · 12.9 KB
Raw

SmartMouse Semantic Understanding - Implementation Guide

The Real Problem

Solution: Semantic Context Engine

What It Does:

Pixels + Text (OCR) → Semantic Understanding → AI Makes Smarter Decisions

Key Improvements:

  1. Understands page types (home, feed, post, login, search)
  2. Classifies objects (like buttons, comments, nav items)
  3. Knows user state (logged in/out)
  4. Generates recommendations (what to do next)

Step 1: Add Semantic Context Functions to vision.ts

Add these interfaces and functions BEFORE the export default statement at the end of vision.ts:

// ADD THESE INTERFACES AND FUNCTIONS TO vision.ts

export interface SemanticContext {
  pageType: 'home' | 'feed' | 'post' | 'profile' | 'search' | 'login' | 'general';
  userStatus: 'logged_in' | 'logged_out' | 'unknown';
  mainContent: {
    type: 'text' | 'image' | 'video' | 'list' | 'form';
    primaryAction: string;
    escalationPath: string;
  };
  navigation: {
    currentSection: string;
    availableSections: string[];
    backHistory: string[];
  };
  objectsByFunction: {
    likeButtons: DetectedObject[];
    commentButtons: DetectedObject[];
    shareButtons: DetectedObject[];
    navigationItems: DetectedObject[];
    inputFields: DetectedObject[];
    buttons: DetectedObject[];
    textBlocks: DetectedObject[];
    images: DetectedObject[];
    videos: DetectedObject[];
  };
  confidence: number;
  summary: string;
}

export interface SemanticAnalysis {
  original: VisionResult;
  semantic: SemanticContext;
  recommendations: {
    action: string;
    confidence: number;
    reasoning: string;
    coordinates?: { x: number; y: number };
  }[];
}

Add these functions:

/**
 * Analyze screen objects and extract semantic meaning
 */
export function buildSemanticContext(visionData: VisionResult, goal?: string): SemanticContext {
  const objects = visionData.detected_objects || [];
  const allText = extractAllText(objects);
  const textLower = allText.toLowerCase();
  
  const pageType = inferPageType(textLower, objects);
  const userStatus = inferUserStatus(textLower, objects);
  const mainContent = inferMainContent(objects);
  const navigation = inferNavigation(objects);
  const objectsByFunction = classifyObjectsByFunction(objects);
  const confidence = calculateConfidence(objects, pageType, userStatus);
  const summary = generateSummary(pageType, userStatus, mainContent, navigation);
  
  return {
    pageType,
    userStatus,
    mainContent,
    navigation,
    objectsByFunction,
    confidence,
    summary
  };
}

function extractAllText(objects: DetectedObject[]): string {
  return objects
    .filter(obj => obj.text)
    .map(obj => obj.text!)
    .join(' ');
}

function inferPageType(textLower: string, objects: DetectedObject[]): SemanticContext['pageType'] {
  if (textLower.includes('log in') || textLower.includes('sign up') || textLower.includes('username')) return 'login';
  if (textLower.includes('search') && (textLower.includes('results') || textLower.includes('find'))) return 'search';
  if (textLower.includes('feed') || textLower.includes('home') || textLower.includes('timeline')) return 'feed';
  if (textLower.includes('profile') || textLower.includes('about')) return 'profile';
  if (textLower.includes('post') || textLower.includes('photo') || textLower.includes('video')) return 'post';
  return 'home';
}

function inferUserStatus(textLower: string, objects: DetectedObject[]): SemanticContext['userStatus'] {
  if (textLower.includes('log in') || textLower.includes('sign up') || textLower.includes('create account')) return 'logged_out';
  if (textLower.includes('notifications') || textLower.includes('messages') || textLower.includes('saved')) return 'logged_in';
  return 'unknown';
}

function inferMainContent(objects: DetectedObject[]): SemanticContext['mainContent'] {
  const hasImages = objects.some(obj => obj.label.includes('image') || obj.label.includes('photo'));
  const hasVideos = objects.some(obj => obj.label.includes('video') || obj.label.includes('play'));
  const hasTextBlocks = objects.some(obj => obj.text && obj.text.length > 50);
  const hasForms = objects.some(obj => obj.label.includes('input') || obj.label.includes('form'));
  
  if (hasImages && !hasVideos) return { type: 'image', primaryAction: 'view', escalationPath: 'interact' };
  if (hasVideos) return { type: 'video', primaryAction: 'play', escalationPath: 'watch' };
  if (hasForms) return { type: 'form', primaryAction: 'fill', escalationPath: 'submit' };
  if (hasTextBlocks) return { type: 'text', primaryAction: 'read', escalationPath: 'comment' };
  return { type: 'list', primaryAction: 'browse', escalationPath: 'explore' };
}

function inferNavigation(objects: DetectedObject[]): SemanticContext['navigation'] {
  const navTexts = ['home', 'feed', 'search', 'notifications', 'messages', 'profile', 'settings', 'menu'];
  const foundNav = navTexts.filter(text => objects.some(obj => (obj.text || '').toLowerCase().includes(text)));
  return {
    currentSection: foundNav[0] || 'unknown',
    availableSections: foundNav,
    backHistory: []
  };
}

function classifyObjectsByFunction(objects: DetectedObject[]): SemanticContext['objectsByFunction'] {
  const byFunction = {
    likeButtons: [] as DetectedObject[],
    commentButtons: [] as DetectedObject[],
    shareButtons: [] as DetectedObject[],
    navigationItems: [] as DetectedObject[],
    inputFields: [] as DetectedObject[],
    buttons: [] as DetectedObject[],
    textBlocks: [] as DetectedObject[],
    images: [] as DetectedObject[],
    videos: [] as DetectedObject[]
  };
  
  for (const obj of objects) {
    const label = obj.label.toLowerCase();
    const text = (obj.text || '').toLowerCase();
    
    if (label.includes('button') || text.includes('like') || text.includes('heart') || text.includes('comment')) {
      if (text.includes('like') || label.includes('like') || label.includes('heart')) {
        byFunction.likeButtons.push(obj);
      } else if (text.includes('comment') || label.includes('comment')) {
        byFunction.commentButtons.push(obj);
      } else if (text.includes('share') || label.includes('share')) {
        byFunction.shareButtons.push(obj);
      } else {
        byFunction.buttons.push(obj);
      }
    } else if (label.includes('input') || label.includes('search') || label.includes('field')) {
      byFunction.inputFields.push(obj);
    } else if (label.includes('nav') || label.includes('menu') || label.includes('bar')) {
      byFunction.navigationItems.push(obj);
    } else if (obj.text && obj.text.length > 3) {
      byFunction.textBlocks.push(obj);
    } else if (label.includes('image') || label.includes('photo') || label.includes('picture')) {
      byFunction.images.push(obj);
    } else if (label.includes('video') || label.includes('play')) {
      byFunction.videos.push(obj);
    }
  }
  return byFunction;
}

function calculateConfidence(objects: DetectedObject[], pageType: string, userStatus: string): number {
  let confidence = 0.5;
  if (objects.length > 10) confidence += 0.1;
  if (objects.length > 50) confidence += 0.15;
  if (objects.length > 100) confidence += 0.1;
  if (pageType !== 'general') confidence += 0.1;
  if (userStatus !== 'unknown') confidence += 0.1;
  const avgConfidence = objects.reduce((sum, obj) => sum + obj.confidence, 0) / Math.max(objects.length, 1);
  confidence += (avgConfidence - 0.5) * 0.2;
  return Math.min(1, Math.max(0, confidence));
}

function generateSummary(
  pageType: string,
  userStatus: string,
  mainContent: any,
  navigation: any
): string {
  return `Page type: ${pageType}, User: ${userStatus}, Content: ${mainContent.type}, Section: ${navigation.currentSection}`;
}

export function generateRecommendations(visionData: VisionResult, goal?: string): SemanticAnalysis['recommendations'] {
  const semantic = buildSemanticContext(visionData);
  const recommendations: SemanticAnalysis['recommendations'] = [];
  
  if (goal?.toLowerCase().includes('like') && semantic.objectsByFunction.likeButtons.length > 0) {
    recommendations.push({
      action: 'click',
      confidence: 0.9,
      reasoning: 'Goal: like posts. Found like buttons in the UI.',
      coordinates: {
        x: semantic.objectsByFunction.likeButtons[0].center.x,
        y: semantic.objectsByFunction.likeButtons[0].center.y
      }
    });
  }
  
  if (goal?.toLowerCase().includes('comment') && semantic.objectsByFunction.commentButtons.length > 0) {
    recommendations.push({
      action: 'click',
      confidence: 0.9,
      reasoning: 'Goal: comment. Found comment buttons in the UI.',
      coordinates: {
        x: semantic.objectsByFunction.commentButtons[0].center.x,
        y: semantic.objectsByFunction.commentButtons[0].center.y
      }
    });
  }
  
  if (semantic.pageType === 'login' && goal && semantic.objectsByFunction.inputFields.length >= 2) {
    recommendations.push({
      action: 'fill_form',
      confidence: 0.85,
      reasoning: 'Detected login page with multiple input fields. Should fill credentials.'
    });
  }
  
  if (semantic.userStatus === 'logged_out' && goal && (goal.includes('instagram') || goal.includes('social') || goal.includes('facebook'))) {
    recommendations.push({
      action: 'navigate_to_login',
      confidence: 0.8,
      reasoning: 'User appears logged out but goal requires social media access.'
    });
  }
  
  if (semantic.navigation.availableSections.length > 0 && goal && goal.includes('go to')) {
    const section = semantic.navigation.availableSections.find(section => goal.toLowerCase().includes(section.toLowerCase()));
    if (section) {
      const navItem = semantic.objectsByFunction.navigationItems.find(item => (item.text || '').toLowerCase().includes(section.toLowerCase()));
      if (navItem) {
        recommendations.push({
          action: 'click',
          confidence: 0.85,
          reasoning: `Goal includes navigating to '${section}'. Found navigation item.`,
          coordinates: { x: navItem.center.x, y: navItem.center.y }
        });
      }
    }
  }
  
  return recommendations;
}

export function enhanceVisionWithSemantic(visionData: VisionResult, goal?: string): SemanticAnalysis {
  return {
    original: visionData,
    semantic: buildSemanticContext(visionData, goal),
    recommendations: generateRecommendations(visionData, goal)
  };
}

Step 2: Update the Default Export

// At the end of vision.ts, update the export:
export default {
  initVision,
  getLastError,
  takeScreenshot,
  analyzeScreen,
  isVisionReady,
  findElement,
  findElements,
  buildSemanticContext,        // ← ADD
  generateRecommendations,     // ← ADD  
  enhanceVisionWithSemantic    // ← ADD
};

Step 3: Update brain.ts to Use Semantic Context

function buildPrompt(vision: VisionData, goal: string): string {
  // Add semantic context to the prompt
  let enhancedVision = `{ ...vision, semantic: buildSemanticContext(vision, goal) }`;
  
  return `SCREEN DATA:
Description: ${vision.description}
Semantics: Page=${semantic.pageType}, User=${semantic.userStatus}, Content=${semantic.mainContent.type}

OBJECTS BY FUNCTION:
- Like buttons: ${semantic.objectsByFunction.likeButtons.length}
- Comment buttons: ${semantic.objectsByFunction.commentButtons.length}
- Navigation items: ${semantic.objectsByFunction.navigationItems.length}

GOAL: ${goal}

Return exactly one JSON object for the best next action.`;
}

Step 4: Update index.ts to Use Enhanced Vision

// When taking screenshot, include semantic context:
const visionData = await analyzeScreen();
const enhancedVision = enhanceVisionWithSemantic(visionData, goal);

// Pass enhanced vision to AI brain:
const action = await decideAction(enhancedVision, goal);

Result

After these changes:

Before:

OCR sees: "Search", "Like", "Comment"
AI thinks: "Some text found, what to do?"

After:

OCR sees: "Search", "Like", "Comment"
Semantic: "Search bar found in feed, Like button in post, Comment button accessible"
AI thinks: "I'm on a feed page, should like posts. Found 3 like buttons, clicking the first one with 90% confidence."

OpenClaw Integration

The enhanced system now provides structured context:

{
  "vision": {
    "description": "Instagram feed",
    "semantic": {
      "pageType": "feed",
      "userStatus": "logged_in",
      "navigation": {
        "currentSection": "home",
        "availableSections": ["home", "feed", "notifications"]
      }
    }
  },
  "recommendations": [
    {
      "action": "click",
      "confidence": 0.9,
      "reasoning": "Goal: like posts. Found 5 like buttons in the UI." 
    }
  ]
}

OpenClaw agents can trust this structured feedback!

Summary

  1. ✅ Add semantic context functions to vision.ts
  2. ✅ Update analyzeScreen() to call buildSemanticContext()
  3. ✅ Update AI prompt to include semantic context
  4. ✅ Update index.ts to use enhanced vision data
  5. ✅ Test with real scenarios

That's it! The mouse becomes "smart" by understanding what it sees, not just what pixels it detects.