Sentient v2 (#57)

List of commit messages for all commits that are being merged:

* feat: advanced voice mode sandbox script.

expanded implementation details
1. insanely fast whisper for STT with Google Speech Recognition as a fallback
2. WebRTCVAD for voice activity detection. the mic will only stream to the backend when voice activity is detected
3. Sesame CSM for TTS - incorporated SOTA TTS for life-like speech capabilities
4. integrated with existing chat backend - ensures seamless integration with existing agent and memory pipelines

next steps - move VAD logic to Electron frontend, setup electron app with proper frontend for voice chats, modify sandboxed script and convert it to module that we can import into the main app server for usage, add websocket to main app server for voice streaming from electron.

* feat[Gsuite]: added gslides functions

* feat (gsuite): removed token

* feat [Gsuite]: gslides improvements

* feat: voice mode switched transcription to faster whisper, added transcription test script

* feat: orpheus TTS quantized with faster-whisper for STT, real-time voice conversations

* feat[Gsuite]: integrated gdocs functions

* feat[Gsuite]:integrated gsheet functions

* feat: voice mode (WIP)

* fix (optimization): removed model switching feature

* fix (optimization): removed redundant code

* chore (optimization): dev prod toggle in index

* fix: voice mode debuggings (still not working)

* fix (optimization): migrated graph loading logic to server

* feat (optimization): added notifications tab

* fix (optimization): duplicate context issue in intent

* fix (optimization): resolved notifications db issue

* fix (optimization): update progress, data sources for intent

* feat: advanced voice mode gradio sandboxing (DONE) faster-whisper + llama 3.2 3b + orpheus TTS with FastRTC

* fix: trying voice mode fastapi backend with websocket instead of webrtc

* fix (optimization): resolved get graph data issue

* chore (optimization): added print statements

* chore (optimization): renamed endpoints, removed redundant files

* feat: voice mode

FINALLY COMPLETED

completed voice mode integration with existing backend - user messages are saved to the chatdb, actions are added to the task queue and memory functions also work as expected. all low-latency on RTX A5000.

transcription uses faster-whisper base on CPU for now. can be moved to GPU with a larger model for better accuracy

TTS uses a 4 bit quant of orpheus

RTC functionality is supported by FastRTC

* fix: datetime fix in chat history retrieval

* fix (optimization): switched to phi4-mini, changed unified classification prompt, resolved minor backend issues

* fix (optimization): resolved app lifecycle events

* fix (optimization): npm package updates, switched to 4bit orpheus, removed unnecessary fallback in index, updated requirements (freeze versions)

* fix (optimization): memory ops

* feat (optimizations): frontend changes to voice mode components

* fix (optimization): memory crud

* feat (optimization): frontend optimizations - updating visualizations, call control buttons, etc

* feat (optimization): task page optimization, fixing webrtc client functionality, added microphone switching logic to the chat page

* fix (optimization): updated chat system prompt

* fix (optimizations): frontend fixes on tasks page

* fix (optimization): resolved task approval

* fix (optimization): fixing react children render error on task page and fixing timestamp logic

* feat (optimization): memory page frontend

* fix (optimization): agents testing

* feat (optimizations): notifications page frontend

* fix (optimization): updated prompts, resolved timestamp issue

* feat (optimizations): settings page frontend and memory type switcher minor changes

* fix[memory] : import format issue

* fix (optimization): fixed memory internet agent icons not appearing on messages

* fix (optimization): removed model selector from onboarding, dual memory fixes for retention days logic

* fix (optimization): resolved agentic message

* fix (optimization): mock mode for settings

* fix (optimization): renamed folders, researched on prompting techniques

* fix (optimization): renamed paths according to new folder structure

* fix[optimization]: prompts optimized

Co-authored-by: Abhijeet Suryawanshi <108229267+abhijeetsuryawanshi12@users.noreply.github.com>

* fix (optimization): testing scrapers

* fix[optimization] : test linkedin scraper

* fix[optimization]: replaced linkedin scraper

* fix[optimization]: minor changes

---------

Co-authored-by: abhijeetsuryawanshi12 <abhijeetsuryawanshi128@gmail.com>
Co-authored-by: itsskofficial <itsskofficial03@gmail.com>
Co-authored-by: Abhijeet Suryawanshi <108229267+abhijeetsuryawanshi12@users.noreply.github.com>

Author: 3 people

.gitignore

@@ -205,7 +205,7 @@ next-env.d.ts
 !.vscode
 .vscode/*
 !.vscode/settings.json
-!.vscode/tasks.json
+!.vscode/agentic_operations.json
 !.vscode/launch.json
 !.vscode/extensions.json
 !.vscode/*.code-snippets
@@ -229,6 +229,7 @@ results
 checkpoints
 chroma_db
 uploads
+outputs
 
 # Custom files
 .env*
@@ -239,8 +240,11 @@ uploads
 token.pickle
 chatsDb.json
 userProfileDb.json
-tasks.json
+agentic_operations.json
 context.json
 memory_operations.json
+notificationsDb.json
 ruff_cache
-.db
+memory.db
+token.json
+.wav

.vscode/tasks.json

@@ -112,7 +112,7 @@ We at [Existence](https://existence.technology) believe that AI won't simply die
 
 ### :dart: Features
 
--   Local-first, with support for Ollama 
+-   Local-first, with support for Ollama
 -   Multi-Model Support (Switch between multiple Ollama/Cloud models)
 -   MBTI Personality Test (Used to collect initial information about the user and their personality to personalize responses)
 -   LinkedIn, Reddit and X Integration for Personal Context