[Feature] The length of vectorized chunks supports customization

[devnotperfect]

MaxKB Version

v2.0.1

Please describe your needs or suggestions for improvements

目前默认向量化支持的是按照256字符前最完整的一句话进行截断后向量化，在bge-m3、qwen-embedding这种支持长token输入的向量模型下测试检索召回的效果不理想，是否可以支持在知识库或者文档向量化时自定义这个截断的长度限制

Please describe the solution you suggest

No response

Additional Information

No response

[baixin513]

感谢反馈， 2.4会支持知识库工作流，知识库写入时会支持写入子分块的长度，即向量化的最小分块长度。

[shaohuzhang1]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Thanks for the feedback, you can consider putting the parameter settings like the big language model in the future.

[baixin513]

v2.4.0已支持工作流知识库，支持自定义子分块长度，请更新试用！

[shaohuzhang1]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

v2.4.0 already supports workflow knowledge base and custom sub-block length, please update to try it out!