VisionAid

VisionAid is an AI-powered accessibility assistant designed to help visually impaired users navigate their environment safely and independently. The application uses real-time computer vision, voice commands, and conversational AI to provide guidance, scene description, obstacle prediction, and educational content.

🌟 Features

🧭 Navigate Mode

Real-time obstacle detection using YOLOv8
Audio alerts for objects in your path
Directional guidance (move left/right/continue straight)
Visual overlay with bounding boxes and labels
Vibration feedback for proximity warnings

💬 Explain Mode

Scene description with natural language narration
Powered by Google Gemini for accurate visual analysis
Detailed descriptions including objects, colors, positions, and text
Object identification and counting
Color and lighting analysis
Auto-describe mode for continuous narration
Spatial awareness (left/right/near/far positioning)
Fallback to YOLO when Gemini is unavailable

🔮 Predict Mode

Anticipatory obstacle detection
Time-to-impact calculations
Severity-based warnings (low/medium/high)
Proactive recommendations
Motion-based predictions

📚 Learn Mode

Object identification and education
Detailed descriptions of identified objects
Interesting facts about common items
Category classification
Learning progress tracking

🎤 Voice Control

Hands-free operation with voice commands
Command recognition for mode switching
Emergency activation via voice
Customizable speech speed and volume

🎯 Additional Features

Emergency overlay with location sharing
Settings panel for customization
Onboarding tutorial for first-time users
Session summaries with usage analytics
Responsive design for mobile and desktop
Accessibility-first design principles

🚀 Getting Started

Prerequisites

Python 3.8+ (for backend)
Node.js 18+ (for frontend)
pnpm (package manager)
Webcam or camera (for vision features)
Microphone (for voice commands)

Backend Setup

Navigate to the backend directory:
```
cd backend
```

Create a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:
```
pip install -r requirements.txt
```

Configure Gemini API (for enhanced Explain mode):

cp .env.example .env
# Edit .env and add your GEMINI_API_KEY
# Get your key from: https://makersuite.google.com/app/apikey

See GEMINI-SETUP.md for detailed instructions.

Download YOLO models (if not present): The yolov8n.pt and yolov8n-cls.pt files should be in the backend directory. If missing, they will be automatically downloaded on first run.
Start the backend server:
```
uvicorn main:app --host 0.0.0.0 --port 8000 --reload
```
The backend API will be available at http://localhost:8000

Frontend Setup

Navigate to the frontend directory:
```
cd frontend
```
Install dependencies:
```
pnpm install
```
Start the development server:
```
pnpm dev
```
The application will be available at http://localhost:3000

📱 Mobile over HTTPS

iOS Safari requires HTTPS (or localhost) for camera/mic access. For testing on a phone over your LAN, use a trusted HTTPS tunnel:

Recommended: Cloudflare Tunnel ephemeral URLs
Alternative: ngrok

See QUICK-START-HTTPS.md for step-by-step instructions, or run the helper script:

chmod +x tools/https-dev.sh
./tools/https-dev.sh

Then set frontend/.env.local:

NEXT_PUBLIC_API_URL=<your-backend-https-url>

Open the frontend HTTPS URL on your phone and tap the viewport to start the camera.

Running the Complete Application

Start the backend in one terminal:

cd backend
source venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8000

Start the frontend in another terminal:
```
cd frontend
pnpm dev
```
Open your browser to http://localhost:3000
Grant permissions for camera and microphone when prompted

🎮 Usage Guide

First Time Setup

Welcome Screen: Read about the four modes available
Grant Permissions: Allow camera and microphone access
Tutorial: Learn voice commands and touch controls
Start: Begin using VisionAid!

Navigation Controls

Voice Commands

"Navigate" - Switch to Navigate mode
"Explain" - Switch to Explain mode
"Predict" - Switch to Predict mode
"Learn" - Switch to Learn mode
"Emergency" - Activate emergency mode
"Settings" - Open settings panel
"Next" - Go to next mode
"Previous" or "Back" - Go to previous mode

Touch/Trackpad Controls

Tap anywhere: Hear current mode announcement
Swipe up/down: Switch between modes
Scroll wheel up/down: Change modes
Long press (2 seconds): Activate voice recognition
Settings button: Bottom right corner
Emergency button: Red button, bottom right

Mode-Specific Controls

Navigate Mode

Start/Pause button: Toggle navigation
Automatically detects obstacles in real-time
Audio alerts for objects in your path
Visual overlay shows detected objects

Explain Mode

Describe Scene button: Get scene description
Auto-Describe toggle: Continuous narration every 10 seconds
Detailed breakdown of objects, colors, and lighting

Predict Mode

Start/Pause button: Toggle prediction
Receives warnings about upcoming obstacles
Time-to-impact shown in seconds
Recommendations for safe navigation

Learn Mode

Identify & Learn button: Identify and learn about objects
Educational facts about detected objects
Category classification
Progress tracking

🛠️ API Endpoints

Backend API (http://localhost:8000)

Health Check

GET /health

Returns the status of loaded models

Object Detection

POST /detect

Upload an image and receive detected objects with bounding boxes

Image Classification

POST /classify

Upload an image and receive the top classification result

Scene Description

POST /describe

Upload an image and receive a natural language description

Chat/Conversational AI

POST /chat

Send a message and receive an AI response

Feedback

POST /feedback

Submit user feedback about detection accuracy

Session Management

POST /session/start
POST /session/end
POST /session/log

Manage user sessions and log activities

Settings

GET /settings
POST /settings

Get or update application settings

🎨 Customization

Settings Panel

Access via the settings button (bottom right). Customize:

Speech Speed: Slow, Normal, or Fast
Volume: 0-100%
Vibration Feedback: On/Off
Audio Tones: On/Off
Emergency Contact: Phone number for emergencies

Settings are saved to browser localStorage and persist across sessions.

📱 Browser Compatibility

VisionAid works best on modern browsers that support:

WebRTC (camera access)
Web Speech API (voice recognition and synthesis)
Vibration API (haptic feedback)

Recommended Browsers:

Chrome 90+
Edge 90+
Safari 14+ (limited voice recognition)
Firefox 88+

🔐 Privacy & Security

Camera/Microphone: Only accessed when granted permission
Image Processing: Images are processed locally or sent to backend API only
No Data Collection: Session data stored temporarily, not shared with third parties
Location Sharing: Only in emergency mode with user consent
Local Storage: Settings saved in browser localStorage

🧪 Testing

Backend Tests

cd backend
pytest tests/

Frontend Tests

cd frontend
pnpm test

🏗️ Project Structure

visionaid-2/
├── backend/
│   ├── main.py                 # FastAPI application
│   ├── requirements.txt        # Python dependencies
│   ├── yolov8n.pt             # Object detection model
│   ├── yolov8n-cls.pt         # Classification model
│   └── tests/                 # Backend tests
├── frontend/
│   ├── app/                   # Next.js app directory
│   │   ├── page.tsx          # Main application page
│   │   ├── layout.tsx        # Root layout
│   │   └── globals.css       # Global styles
│   ├── components/            # React components
│   │   ├── modes/            # Mode-specific components
│   │   │   ├── navigate-mode.tsx
│   │   │   ├── explain-mode.tsx
│   │   │   ├── predict-mode.tsx
│   │   │   └── learn-mode.tsx
│   │   ├── mode-view.tsx     # Mode container
│   │   ├── onboarding-flow.tsx
│   │   ├── settings-panel.tsx
│   │   ├── emergency-overlay.tsx
│   │   └── session-summary.tsx
│   ├── lib/                   # Utility libraries
│   │   ├── types.ts          # TypeScript types
│   │   ├── utils.ts          # Helper functions
│   │   └── voice-system.ts   # Voice control system
│   └── package.json          # Node.js dependencies
└── README.md                  # This file

🤝 Contributing

Contributions are welcome! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is licensed under the MIT License.

🙏 Acknowledgments

YOLOv8 by Ultralytics for object detection
Next.js and React for the frontend framework
FastAPI for the backend API
Tailwind CSS for styling
shadcn/ui for UI components

📞 Support

For issues, questions, or suggestions:

Open an issue on GitHub
Contact the development team

🚧 Roadmap

Future enhancements planned:

VisionAid - Empowering independence through AI-powered vision assistance.

Name		Name	Last commit message	Last commit date
Latest commit History 51 Commits
backend		backend
frontend		frontend
tools		tools
.gitignore		.gitignore
DEVELOPMENT.md		DEVELOPMENT.md
GEMINI-INTEGRATION.md		GEMINI-INTEGRATION.md
GEMINI-SETUP.md		GEMINI-SETUP.md
IMPLEMENTATION-SUMMARY.md		IMPLEMENTATION-SUMMARY.md
QUICK-REFERENCE.md		QUICK-REFERENCE.md
QUICK-START-GEMINI.md		QUICK-START-GEMINI.md
QUICK-START-HTTPS.md		QUICK-START-HTTPS.md
README.md		README.md
check-system.sh		check-system.sh
localhost-key.pem		localhost-key.pem
localhost.pem		localhost.pem
output.mp4		output.mp4
package-lock.json		package-lock.json
setup-gemini.sh		setup-gemini.sh
start.sh		start.sh
yolov8n-cls.pt		yolov8n-cls.pt
yolov8n.pt		yolov8n.pt

Folders and files

Latest commit

History

Repository files navigation