Faiss handler with flat index

[ea-rus]

Description

Continue of #11750

Query examples:

CREATE DATABASE db_faiss
WITH
    ENGINE = 'duckdb_faiss';

create knowledge base kb_faiss
using storage = db_faiss.kb_faiss,
embedding_model={"provider": "openai", "model_name": "text-embedding-3-small"};

insert into kb_faiss (id, content, legs) values (1, 'duck', 2);
insert into kb_faiss (id, content, legs) values (2, 'cat', 4);

select * from kb_faiss where
 legs=4
 and content = 'cat'
 and hybrid_search=true;

Keyword search

Implemented by using duckdb fts extension

How it works: when keyword search is used and FTS index doesn't exist - it is created
But this index is removed when any record is inserted into KB (because FTS index isn't updated after inserts in duckdb)

Other updates:

• added lock for faiss index file to prevent from opening and modifying index file from different KBs or mindsdb instances
  • disabled on windows, looks like faiss apply this lock there already
• checking RAM usage:
  • forecast the size of inserted data in RAM: at least 1Gb RAM should be be free. make this check on every 10k inserted records
  • forecast the size in RAM before index is loaded, if index size > 1Gb: at least 1Gb RAM should be be free after loading
• If inserted dataset is greater than MAX_INSERT_BATCH_SIZE, it is split by this size and inserted in the batches
  User request is kept sync (user waits till insert is completed)

Fixes https://linear.app/mindsdb/issue/FQE-1830/faiss-handler-v1-flat-index-kw-search

Type of change

• ⚡ New feature (non-breaking change which adds functionality)

Verification Process

To ensure the changes are working as expected:

• Test Location: Specify the URL or path for testing.
• Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

• I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

• My code follows the style guidelines(PEP 8) of MindsDB.
• I have appropriately commented on my code, especially in complex areas.
• Necessary documentation updates are either made or tracked in issues.
• Relevant unit and integration tests are updated or added.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
mindsdb/integrations/handlers/bigquery_handler
__init__.py	14	3	79%	7–9
bigquery_handler.py	140	21	85%	53, 57, 62–65, 68, 88, 109–113, 119, 141–143, 159–161, 313–314
mindsdb/integrations/handlers/file_handler
file_handler.py	117	17	85%	25–27, 56, 59, 62, 88–89, 113, 131–132, 155–158, 166–167
mindsdb/integrations/handlers/mssql_handler
__init__.py	14	3	79%	10–12
mssql_handler.py	284	41	86%	39–58, 79, 99, 106, 110, 121, 196, 244, 290–294, 306–308, 336, 363, 365–368, 402–407, 421, 444, 482, 524, 568, 600, 671, 710
mindsdb/integrations/handlers/mysql_handler
__init__.py	14	3	79%	10–12
mysql_handler.py	201	11	95%	33–37, 79–84, 100, 164
settings.py	81	37	54%	30, 37–42, 49, 57, 71–81, 87–111, 117
mindsdb/integrations/handlers/oracle_handler
__init__.py	14	3	79%	10–12
oracle_handler.py	267	47	82%	79–80, 113, 115, 117, 194, 205–206, 229–235, 239, 242–245, 253–255, 261–263, 295–297, 303, 361–378, 516–517, 622
mindsdb/integrations/handlers/postgres_handler
__init__.py	14	3	79%	8–10
postgres_handler.py	326	25	92%	96–102, 196, 240, 278–279, 313–318, 345, 389–392, 455, 514, 545–546, 695
mindsdb/integrations/handlers/redshift_handler
__init__.py	14	3	79%	8–10
mindsdb/integrations/handlers/salesforce_handler
__init__.py	14	3	79%	10–12
salesforce_handler.py	122	5	96%	102–104, 146, 341
salesforce_tables.py	88	12	86%	140, 184–185, 231, 249–267
mindsdb/integrations/handlers/slack_handler
__init__.py	13	3	77%	10–12
slack_handler.py	136	11	92%	57, 59, 122–124, 310, 314, 319, 323, 327, 342
slack_tables.py	270	65	76%	58, 64–65, 71–72, 101–103, 156, 163–165, 251–265, 285, 293–295, 310, 340–344, 375–377, 384–387, 394, 404–408, 438–440, 447–450, 459–463, 549–551, 556, 577, 585–587, 599, 630–634, 698, 705–707
mindsdb/integrations/handlers/snowflake_handler
__init__.py	14	3	79%	8–10
auth_types.py	44	12	73%	12, 38, 44, 68–77
snowflake_handler.py	324	42	87%	33–34, 111, 116, 118, 122, 157, 159–163, 210, 294, 328, 383–399, 432, 577–594, 620–623, 637, 658, 693
mindsdb/integrations/handlers/timescaledb_handler
__init__.py	13	3	77%	7–9
TOTAL	2724	376	86%

Tests	Skipped	Failures	Errors	Time
549	53 💤	0 ❌	0 🔥	9.072s ⏱️