Skip to content

Commit 3e0444d

Browse files
Leon-SanderLeon-Sander
committed
readme updated
1 parent d669f33 commit 3e0444d

4 files changed

Lines changed: 142 additions & 116 deletions

File tree

‎.env‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎.env.example‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Copy this file to .env and fill in values as needed.
2+
# cp .env.example .env
3+
#
4+
# Only required if you use the OpenAI endpoint. Leave empty for local-only (Ollama) use.
5+
OPENAI_API_KEY=

‎.gitignore‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# Environment / secrets
2+
.env
3+
4+
# Python
5+
__pycache__/
6+
*.py[cod]
7+
*.egg-info/
8+
.venv/
9+
venv/
10+
.ipynb_checkpoints/
11+
12+
# App data / generated
13+
chroma_db/
14+
chat_sessions/*.db
15+
temp_audio.*
16+
*.wav
17+
*.webm
18+
19+
# Downloaded model weights (keep the folder, ignore contents)
20+
models/**
21+
!models/.gitkeep
22+
23+
# OS / IDE
24+
.DS_Store
25+
.vscode/
26+
.idea/

‎README.md‎

Lines changed: 111 additions & 115 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,146 @@
11
# Local Multimodal AI Chat
2-
## Getting Started
3-
4-
You can follow my [YouTube-Video](https://youtu.be/eNwvAdem4vo) on setting up the repository on Linux or Windows.
5-
6-
To get started with Local Multimodal AI Chat, clone the repository and follow these simple steps:
7-
8-
### Easiest and Preferred Method: Docker Compose
9-
1. **Set model save path**: Line 21 in the docker-compose.yml file
10-
11-
2. **Enter command in terminal**: ```docker compose up```
12-
13-
Note: If you don't have a GPU, you can remove the deploy section from the docker compose file.
14-
15-
3. **Optional**:
16-
- Check the config.yaml file and change accordingly to your needs.
17-
- Place your user_image.png and/or bot_image.png inside the chat_icons folder and remove the old ones.
18-
19-
4. **Open the app**: Open [0.0.0.0:8501](http://0.0.0.0:8501) in the Browser
20-
21-
5. **Pull Models**: Go to https://ollama.com/library and choose the models you want to use. Enter ```/pull MODEL_NAME``` in the chat bar.
22-
You need one embedding model e.g. [nomic-embed-text](https://ollama.com/library/nomic-embed-text) to embed pdf files (change embedding model in config if you choose another). You also need a model which undertands
23-
images e.g. [llava](https://ollama.com/library/llava)
24-
25-
6. **Optional**:
26-
- Check the config.yaml file and change accordingly to your needs.
27-
- Place your user_image.png and/or bot_image.png inside the chat_icons folder and remove the old ones.
28-
29-
### Recommendation for Windows
30-
Using ollama docker container results in very slow loading times for the models due to system calls being translated between two kernels. Installing Ollama locally works best here.
31-
32-
33-
1. **Install [Ollama](https://ollama.com/download) desktop**
34-
35-
2. **Change Docker Compose file**: remove docker-compose.yml and rename docker-compose_without_ollama.yml to docker-compose.yml
362

37-
3. **Change Ollama Base URL in config.yaml**: Use line 4 in the config.yaml file and remove line 3
3+
A self-hostable, multimodal chat application that runs open-source models locally. Chat with **text, images, PDFs, and voice** in one interface, powered by local models via **Ollama** (with an optional **OpenAI API** fallback), retrieval-augmented generation over your own PDFs, and speech-to-text with **Whisper**.
384

39-
3. **Enter command in terminal**: ```docker compose up```
5+
[![Stars](https://img.shields.io/github/stars/Leon-Sander/Local-Multimodal-AI-Chat?style=flat)](https://github.com/Leon-Sander/Local-Multimodal-AI-Chat/stargazers)
6+
[![Forks](https://img.shields.io/github/forks/Leon-Sander/Local-Multimodal-AI-Chat?style=flat)](https://github.com/Leon-Sander/Local-Multimodal-AI-Chat/forks)
7+
[![License: GPL-3.0](https://img.shields.io/badge/License-GPLv3-blue.svg)](./LICENSE.md)
8+
![Python](https://img.shields.io/badge/Python-3.10-blue)
409

41-
4. **Open the app**: Open [0.0.0.0:8501](http://0.0.0.0:8501) in the Browser
10+
**Demo / setup walkthrough:** [YouTube video](https://youtu.be/eNwvAdem4vo)
4211

43-
5. **Pull Models**: Go to https://ollama.com/library and choose the models you want to use. Enter ```/pull MODEL_NAME``` in the chat bar.
44-
You need one embedding model e.g. [nomic-embed-text](https://ollama.com/library/nomic-embed-text) to embed pdf files (change embedding model in config if you choose another). You also need a model which undertands
45-
images e.g. [llava](https://ollama.com/library/llava)
46-
47-
6. **Optional**:
48-
- Check the config.yaml file and change accordingly to your needs.
49-
- Place your user_image.png and/or bot_image.png inside the chat_icons folder and remove the old ones.
50-
51-
### Complete Manual Install
12+
## Overview
5213

53-
1. **Install [Ollama](https://github.com/ollama/ollama)**
14+
Local Multimodal AI Chat integrates several AI models behind a single Streamlit interface, with a focus on **data privacy** (everything can run on your own machine) and **modularity** (each capability lives in its own handler).
5415

55-
2. **Create a Virtual Environment**: I am using Python 3.10.12
16+
- **Chat** with local models through the Ollama API, or switch to the OpenAI API at runtime.
17+
- **Talk to your PDFs** via retrieval-augmented generation (RAG) backed by a Chroma vector database.
18+
- **Send images** to vision-capable models (e.g. LLaVA).
19+
- **Speak instead of type**: audio is transcribed with Whisper.
20+
- **Keep your history**: chat sessions, messages, and settings are persisted in SQLite.
5621

57-
3. **Install Requirements**:
58-
- ```pip install --upgrade pip```
59-
- ```pip install -r requirements.txt```
60-
- ```pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu```
22+
## Features
6123

62-
4. **Enter commands in terminal**:
63-
1. ```python3 database_operations.py``` This will initialize the sqlite database for the chat sessions.
64-
2. ```streamlit run app.py```
24+
- **Local-first inference (Ollama):** run models on your own hardware, no cloud dependency required.
25+
- **Optional OpenAI API:** switch endpoint and model at runtime from the sidebar.
26+
- **PDF chat (RAG):** upload PDFs, chunk and embed them (Ollama `nomic-embed-text`), and query them through Chroma.
27+
- **Image chat:** pass images to multimodal models via the same chat bar.
28+
- **Voice input (Whisper):** record or upload audio; it is transcribed and fed into the conversation.
29+
- **Persistent, multi-session history:** SQLite-backed sessions, messages (text/image/audio), and user settings.
30+
- **Configurable retrieval:** adjust chunk size, overlap, retrieved-chunk count, and chat-memory length in the UI.
31+
- **Dockerized:** `docker compose up` for the full stack, with a variant for a local Ollama install.
32+
33+
## Architecture
34+
35+
```mermaid
36+
flowchart TD
37+
U["User"] --> UI["Streamlit UI (app.py)"]
38+
UI -->|"record / upload audio"| W["Whisper ASR (audio_handler)"]
39+
W --> R["ChatAPIHandler router"]
40+
UI -->|"text / image / PDF / commands"| R
41+
UI -->|"upload PDF"| P["PDF handler: pypdfium2 + chunking"]
42+
P --> VDB[("Chroma vector DB")]
43+
EMB["Ollama embeddings (nomic-embed-text)"] --> VDB
44+
R -->|"PDF chat: retrieve context"| VDB
45+
R -->|"endpoint = ollama"| OL["Ollama (local models, incl. LLaVA)"]
46+
R -->|"endpoint = openai"| OA["OpenAI API"]
47+
UI --> DB[("SQLite: sessions, messages, settings")]
48+
R --> DB
49+
```
50+
51+
**Module layout**
52+
53+
| File | Responsibility |
54+
|------|----------------|
55+
| `app.py` | Streamlit UI, session state, orchestration |
56+
| `chat_api_handler.py` | Endpoint router (Ollama / OpenAI), text + image + RAG calls |
57+
| `vectordb_handler.py` | Chroma client + Ollama embeddings |
58+
| `pdf_handler.py` | PDF text extraction, chunking, indexing |
59+
| `audio_handler.py` | Whisper transcription (webm→wav via ffmpeg) |
60+
| `database_operations.py` | SQLite repositories (messages, settings) |
61+
| `utils.py` / `prompt_templates.py` / `html_templates.py` | Helpers, prompts, UI styling |
62+
63+
## Tech Stack
64+
65+
Python · Streamlit · Ollama · OpenAI API · LangChain · Chroma · Whisper (Transformers) · SQLite · Docker
6566

66-
5. **Pull Models**: Go to https://ollama.com/library and choose the models you want to use. Enter ```/pull MODEL_NAME``` in the chat bar.
67-
You need one embedding model e.g. [nomic-embed-text](https://ollama.com/library/nomic-embed-text) to embed pdf files and one model which undertands
68-
images e.g. [llava](https://ollama.com/library/llava)
67+
## Getting Started
6968

70-
6. **Optional**:
71-
- Check the config.yaml file and change accordingly to your needs.
72-
- Place your user_image.png and/or bot_image.png inside the chat_icons folder and remove the old ones.
69+
You can also follow the [setup video](https://youtu.be/eNwvAdem4vo) (Linux/Windows).
7370

71+
First, copy the example environment file and add your key only if you plan to use the OpenAI endpoint:
7472

75-
## Overview
73+
```bash
74+
cp .env.example .env
75+
# edit .env and set OPENAI_API_KEY=... (optional; leave empty for local-only use)
76+
```
7677

77-
Local Multimodal AI Chat is a multimodal chat application that integrates various AI models to manage audio, images, and PDFs seamlessly within a single interface. This application is ideal for those passionate about AI and software development, offering a comprehensive solution that employs Whisper AI for audio processing, LLaVA for image management, and Chroma DB for handling PDFs.
78+
### Easiest and preferred: Docker Compose
7879

79-
The application has been enhanced with the Ollama server and the OpenAI API, boosting its functionality and performance. You can find a detailed tutorial on the development of this repository on my [youtube channel](https://youtu.be/CUjO8b6_ZuM). While significant advancements have been made, the project is still open to further development and refinement.
80+
1. **Set the model save path**: line 21 in `docker-compose.yml`.
81+
2. **Start it**: `docker compose up`
82+
*No GPU? Remove the `deploy` section from the compose file.*
83+
3. **Open the app**: [http://0.0.0.0:8501](http://0.0.0.0:8501)
84+
4. **Pull models**: browse [ollama.com/library](https://ollama.com/library) and enter `/pull MODEL_NAME` in the chat bar. You need an embedding model (e.g. [nomic-embed-text](https://ollama.com/library/nomic-embed-text)) for PDFs and a vision model (e.g. [llava](https://ollama.com/library/llava)) for images.
85+
5. **Optional**: adjust `config.yaml`; replace the avatars in `chat_icons/`.
8086

81-
I welcome contributions of all forms. Whether you’re introducing new features, optimizing the code, or correcting bugs, your participation is valued. This project thrives on community collaboration and aims to serve as a robust resource for those interested in the practical application of multimodal AI technologies.
87+
### Recommendation for Windows
8288

89+
Running Ollama inside Docker is slow on Windows (system calls are translated between kernels). Install Ollama locally instead:
8390

84-
## Features
91+
1. **Install [Ollama](https://ollama.com/download) desktop.**
92+
2. Delete `docker-compose.yml` and rename `docker-compose_without_ollama.yml` to `docker-compose.yml`.
93+
3. In `config.yaml`, use the `host.docker.internal` base URL (line 4) and remove line 3.
94+
4. `docker compose up`, then open [http://0.0.0.0:8501](http://0.0.0.0:8501) and pull models as above.
8595

86-
- **Local Model Processing with Ollama**: This app utilizes the Ollama server for running local instances of models, providing a powerful and customizable AI experience without the need for external cloud dependencies. This setup is ideal for maintaining data privacy and improving response times.
96+
### Complete manual install
8797

88-
- **Integration with OpenAI API**: For broader AI capabilities, this application also connects to the OpenAI API, enabling access to a wide range of cutting-edge AI models hosted externally. This feature ensures the app remains versatile and capable of handling a variety of tasks and queries efficiently.
98+
1. **Install [Ollama](https://github.com/ollama/ollama).**
99+
2. **Create a virtual environment** (developed on Python 3.10.12).
100+
3. **Install requirements**:
101+
```bash
102+
pip install --upgrade pip
103+
pip install -r requirements.txt
104+
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
105+
```
106+
4. **Run**:
107+
```bash
108+
python3 database_operations.py # initializes the SQLite database
109+
streamlit run app.py
110+
```
111+
5. **Pull models** as described above.
89112

90-
- **Audio Chatting with Whisper AI**: Leveraging Whisper AI's robust transcription capabilities, this app offers a sophisticated audio messaging experience. The integration of Whisper AI allows for accurate interpretation and response to voice inputs, enhancing the natural flow of conversations.
91-
[Whisper models](https://huggingface.co/collections/openai/whisper-release-6501bba2cf999715fd953013)
113+
## Configuration
92114

93-
- **PDF Chatting with Chroma DB**: The app is tailored for both professional and academic uses, integrating Chroma DB as a vector database for efficient PDF interactions. This feature allows users to engage with their own PDF files locally on their device. This makes it a valuable tool for personal use, where one can extract insights, summaries, and engage in a unique form of dialogue with the text in their PDF files. [Chroma website](https://docs.trychroma.com/)
115+
Key settings live in `config.yaml` (Ollama base URL and embedding model, Whisper model, Chroma path/collection, chat-sessions DB path). Runtime options (endpoint, model, chunk size/overlap, retrieved chunks, chat memory) are adjustable in the sidebar and persisted per user.
94116

95-
## Changelog
117+
## Roadmap
96118

97-
### 16.09.2024:
98-
- **Big Update**: Model Serving based on Ollama API now. Added Openai API.
119+
- [ ] Additional model providers (Gemini, others)
120+
- [ ] Image generation
121+
- [ ] Authentication
122+
- [ ] Theming
99123

100124
<details>
101-
<summary>Click to see more!</summary>
102-
103-
### 24.08.2024:
104-
- **Docker Compose Added**
105-
106-
### 17.02.2024:
107-
- **Input Widget Update**: Replaced st.text_input with st.chat_input to enhance interaction by leveraging a more chat-oriented UI, facilitating user engagement.
108-
- **Sidebar Adjustment**: Relocated the audio recording button to the sidebar for a cleaner and more organized user interface, improving accessibility and user experience.
109-
110-
### 10.02.2024:
111-
- **License Added**: Implemented the GNU General Public License v3.0 to ensure the project is freely available for use, modification, and distribution under the terms of this license. A comprehensive copyright and license notice has been included in the main file (app.py) to clearly communicate the terms under which the project is offered. This addition aims to protect both the contributors' and users' rights, fostering an open and collaborative development environment. For full license details, refer to the LICENSE file in the project repository.
112-
- **Caching for Chat Model**: Introduced caching for the chat model to prevent it from being reloaded with every script execution. This optimization significantly improves performance by reducing load times
113-
- **Config File Expansion**: Expanded the configuration file to accommodate new settings and features, providing greater flexibility and customization options for the chat application.
125+
<summary>Changelog (highlights)</summary>
114126

127+
- **2025-05** (v2.5.0): file upload via the chat bar.
128+
- **2024-09**: Model serving moved to the Ollama API; OpenAI API added.
129+
- **2024-08**: Docker Compose added.
130+
- **2024-02**: SQLite chat history; model caching; images/audio in history; config expansion; GPL-3.0 license.
115131

116-
### 09.02.2024:
132+
See [Releases](https://github.com/Leon-Sander/Local-Multimodal-AI-Chat/releases) for the full history.
117133

118-
- **SQLite Database for Chat History**: Implemented a SQLite database to store the chat history.
119-
- **Displaying Images and Audio Files in Chat**: Chat history now supports displaying images and audio files.
120-
- **Added Button to delete Chat History**
121-
- **Updated langchain**: Runs now with the current langchain version 0.1.6
122-
123-
### 16.01.2024:
124-
- **Windows User DateTime Format Issue:** Windows users seemed to have problems with the datetime format of the saved JSON chat histories. I changed the format in the `ultis.py` file to `"%Y_%m_%d_%H_%M_%S"`, which should solve the issue. Feel free to change it to your liking.
125-
- **UI Adjustment for Chat Scrolling:** Scrolling down in the chat annoyed me, so the text input box and the latest message are at the top now.
126-
127-
### 12.01.2024:
128-
- **Issue with Message Sending:** After writing in the text field and pressing the send button, the LLM would not generate a response.
129-
- **Cause of the Issue:** This happened because the `clear_input_field` callback from the button changes the text field value to an empty string after saving the user question. However, changing the text field value triggers the callback from the text field widget, setting the `user_question` to an empty string again. As a result, the LLM is not called.
130-
- **Implemented Workaround:** As a workaround, I added a check before changing the `user_question` value.
131134
</details>
132135

136+
## Contributing
137+
138+
Contributions are welcome: features, optimizations, or bug fixes. Please check existing [issues](https://github.com/Leon-Sander/Local-Multimodal-AI-Chat/issues) first.
133139

134-
## Possible Improvements
135-
- ~~Add Model Caching.~~
136-
- ~~Add Images and Audio to Chat History Saving and Loading.~~
137-
- ~~Use a Database to Save the Chat History.~~
138-
- Integrate ~~Ollama, OpenAI,~~ Gemini, or Other Model Providers.
139-
- Add Image Generator Model.
140-
- Authentication Mechanism.
141-
- Change Theme.
142-
- ~~Separate Frontend and Backend Code for Better Deployment.~~
140+
## License
143141

144-
## Contact Information
142+
Licensed under the GNU GPL-3.0. See [LICENSE.md](./LICENSE.md).
145143

146-
If you're interested in working with me, feel free to contact me via email.
147-
Before contacting me because of errors you're encountering, make sure to check the github issues first: https://github.com/Leon-Sander/Local-Multimodal-AI-Chat/issues?q=
144+
## Contact
148145

149-
- Email: leonsander.consulting@gmail.com
150-
- Twitter: [@leonsanderai](https://twitter.com/leonsanderai)
146+
Built by Leon Sander. [GitHub](https://github.com/Leon-Sander) · [LinkedIn](https://www.linkedin.com/in/leon-sander-8292b7153/) · leonsander.consulting@gmail.com

0 commit comments

Comments
 (0)