Feature/elasticsearch handler update: fix array support and standardize format

[richardokonicha]

Description

Improve the Elasticsearch handler by fixing array field handling, adding Data Catalog support for enterprise integration, and standardizing documentation format to match MindsDB conventions.

This PR addresses the "Arrays not supported" error that users encounter when querying Elasticsearch indices with array fields, implements the three required Data Catalog methods
(get_column_statistics, get_primary_keys, get_foreign_keys), and ensures the handler follows MindsDB's standard documentation format.

Demo Video:
Array support showcase and feature overview
[Elastic search demo with data catalog feature]
(https://youtu.be/FGfXqtKxdWY)

Type of change

• 🐛 Bug fix (non-breaking change which fixes an issue)
• ⚡ New feature (non-breaking change which adds functionality)
• 📄 This change requires a documentation update

Verification Process

To ensure the changes are working as expected:

• Test Location: /mindsdb/integrations/handlers/elasticsearch_handler/tests/
• Test Coverage: 48 tests (6 handler + 10 Data Catalog + 32 query tests)
• Verification Steps:
  1. Connect to Elasticsearch cluster with array fields
  2. Run SELECT * FROM elasticsearch_conn.products WHERE product_id = '12345' (arrays converted to JSON strings)
  3. Test Data Catalog: SELECT * FROM mindsdb.get_column_statistics('elasticsearch_conn', 'products')
  4. Use SELECT table_name FROM information_schema.tables WHERE table_schema = 'elasticsearch_conn' for schema discovery
  5. Verify handler automatically falls back to Search API when SQL API fails
• CI Testing: mindsdb-handlers-monitor PR #13

Additional Media:

• I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

• My code follows the style guidelines (PEP 8) of MindsDB.
• I have appropriately commented on my code, especially in complex areas.
• Necessary documentation updates are either made or tracked in issues.
• Relevant unit and integration tests are updated or added.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[entelligence-ai-pr-reviews]

🔒 Entelligence AI Vulnerability Scanner

❌ Security analysis failed: Security analysis failed: 403: blocked: error while getting the team: team is blocked

[richardokonicha]

I have read the CLA Document and I hereby sign the CLA

[ZoranPandovski]

Thanks, please check my comments

[richardokonicha]

@MinuraPunchihewa All comments addressed! Here's a summary:

Removed Data Catalog documentation sections from README
Changed inheritance to MetaDatabaseHandler
Updated all column names to uppercase specification
Renamed methods with meta_ prefix and updated signatures
Added meta_get_tables and meta_get_columns methods
Consolidated all tests to tests/unit/handlers/test_elasticsearch.py

[richardokonicha]

I have consolidated all tests to tests/unit/handlers/test_elasticsearch.py

[MinuraPunchihewa]

@richardokonicha This looks good. The only other ask that I have is for you to test this with our open-source agents. This will be a good smoke test for whether it will actually work with Minds, which is what we want. You can follow these steps:

*As a prerequisite, set the MINDSDB_DATA_CATALOG_ENABLED environment variable to true before starting MindsDB:
export MINDSDB_DATA_CATALOG_ENABLED=1

1. Create a MindsDB database for Elasticsearch.
2. Create an Agent connected to the Elasticsearch database:

CREATE AGENT my_agent
USING
    model = {
        "provider": "openai",
        "model_name" : "gpt-4o",
        "api_key": "<your-api-key>"
    },
    data = {
         "tables": ["elasticsearch_conn.<table-1>", "elasticsearch_conn.<table-2>"]
    },
    prompt_template='
        <an-explanation-of-the-data-contained-in-your-tables>
    ';

3. Ask a few questions and evaluate if the answers are accurate.

If you can post a few screenshots of this working well, I think we should be good to go.

[MinuraPunchihewa]

@richardokonicha For the tests to run, you will also need to add a line to install the dependencies for the Elasticsearch integration to the workflow here. You just need to add .[elasticsearch] to the end (also add a forward slash to the previous line).

[richardokonicha]

@MinuraPunchihewa this branch has been updated and now all tests pass gracefully, here is a short video i made to demo this elastic search functionality https://youtu.be/FGfXqtKxdWY

[tino097]

@richardokonicha hey man, can you please resolve the issues with this PR ?

[richardokonicha]

@tino097 Hey! Branch is synced with releases/26.1.0 — all conflicts resolved and elasticsearch handler is intact.

Changes in this PR:

• ✅ Array field handling (convert arrays to JSON strings)
• ✅ Data Catalog support (meta_get_tables, meta_get_columns, get_column_statistics, get_primary_keys, get_foreign_keys)
• ✅ Elasticsearch added to tests_unit.yml HANDLERS_TO_INSTALL for CI
• ✅ MetaDatabaseHandler inheritance
• ✅ Consolidated tests at tests/unit/handlers/test_elasticsearch.py
• ✅ Demo video: https://youtu.be/FGfXqtKxdWY

Ready for review.

[richardokonicha]

@tino097 Done — removed elasticsearch from HANDLERS_TO_INSTALL in the unit test workflow (1463a17).

Regarding run_tests.py — that file doesn't exist on this branch. Tests were already moved to tests/unit/handlers/test_elasticsearch.py per the earlier review. The comment may be referencing the old diff.

Let me know if anything else needs adjusting!