🚀 Feature Request: Higher Dimensions support for vectorize

[Aslemammad]

Describe the solution

Thank you so much for the amazing stack y'all built.

Right now, the dimensions range is [32, 1536] while more advanced models like https://huggingface.co/nomic-ai/nomic-embed-code or https://huggingface.co/Qodo/Qodo-Embed-1-7B require way higher dimensions like 3584.

I wonder if it's possible to consider higher dimensions since I'm building a new package/dependency manager that relies on better code retrieval mechanisms.

[Aslemammad]

I have been researching other tools meanwhile, but nothing comes close to the DX of cloudflare workers and Vectorize. Here are two pictures to give you an idea of the possible dimension of a vector in other toolings like Pinecone and Milvus, in case you'd raise the maximum in the range.

[sejoker]

@Aslemammad I understand that support of high dimensionality would be beneficial. Due to resource constraints we won't be able prioritize the work this year.

[sejoker]

@petebacondarwin feel free to close it.