Skip to content

Commit 370d469

Browse files
initial import (#410)
1 parent 10f5a76 commit 370d469

1 file changed

Lines changed: 70 additions & 0 deletions

File tree

integrations/faiss.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,76 @@ results = query_pipeline.run({"text_embedder": {"text": "your query"}})
114114
documents = results["retriever"]["documents"]
115115
```
116116

117+
### Fixing OpenMP Runtime Conflicts on macOS
118+
119+
#### Symptoms
120+
121+
You may encounter one or both of the following errors at runtime:
122+
123+
```
124+
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.
125+
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.
126+
```
127+
128+
```
129+
resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
130+
```
131+
132+
If setting `OMP_NUM_THREADS=1` prevents the crash, the root cause is **multiple OpenMP runtimes loaded simultaneously**. Each runtime maintains its own thread pool and thread-local storage (TLS). When two runtimes spin up worker threads at the same time, they corrupt each other's memory — causing segfaults at `N > 1` threads.
133+
134+
---
135+
136+
#### Diagnosis
137+
138+
First, find how many copies of `libomp.dylib` exist in your virtual environment:
139+
140+
```bash
141+
find /path/to/your/.venv -name "libomp.dylib" 2>/dev/null
142+
```
143+
144+
If you see more than one, e.g.:
145+
146+
```
147+
.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib
148+
.venv/lib/pythonX.Y/site-packages/sklearn/.dylibs/libomp.dylib
149+
.venv/lib/pythonX.Y/site-packages/faiss/.dylibs/libomp.dylib
150+
```
151+
152+
you need to consolidate them into a single runtime.
153+
154+
---
155+
156+
#### Fix
157+
158+
The solution is to pick one canonical `libomp.dylib` (torch's is a good choice) and replace all other copies with symlinks pointing to it.
159+
160+
For each duplicate, delete the copy and replace it with a symlink:
161+
162+
```bash
163+
# Delete the duplicate
164+
rm /path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
165+
166+
# Replace with a symlink to the canonical copy
167+
ln -s /path/to/.venv/lib/pythonX.Y/site-packages/torch/lib/libomp.dylib \
168+
/path/to/.venv/lib/pythonX.Y/site-packages/<package>/.dylibs/libomp.dylib
169+
```
170+
171+
Repeat for every duplicate found. Because these packages use `@loader_path`-relative references to load `libomp.dylib`, the symlink will be transparently resolved to the single canonical runtime at load time.
172+
173+
---
174+
175+
#### Verify
176+
177+
After applying the fix, confirm only one unique `libomp.dylib` is being referenced:
178+
179+
```bash
180+
find /path/to/your/.venv -name "*.so" | xargs otool -L 2>/dev/null | grep libomp | sort -u
181+
```
182+
183+
All entries should resolve to the same canonical path. You should now be able to run without `OMP_NUM_THREADS=1`.
184+
185+
186+
117187
## License
118188

119189
`faiss-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.

0 commit comments

Comments
 (0)