refactor!: vendor the tokenizer stack into lance (#6512)

This PR vendors the tokenizer stack Lance actually uses into a new
`rust/lance-tokenizer` crate and rewires FTS and inverted-index code to
depend on it instead of `tantivy` and `lindera-tantivy`. It keeps the
existing document and query tokenization semantics in-tree, renames the
old FTS document adapter module to `document_tokenizer`, and preserves
upstream license headers on vendored code.

Author: Xuanwo

.typos.toml

@@ -23,5 +23,6 @@ nprob = "nprobe"
 extend-exclude = [
     "notebooks/*.ipynb",
     "*_THIRD_PARTY_LICENSES.*",
+    "rust/lance-tokenizer/src/stop_word_filter/stopwords.rs",
 ]
 # If a line ends with # or // and has spellchecker:disable-line, ignore it