Update llama.cpp and support arm64

[rodydavis]

• Updates the llama.cpp submodule to the latest version.
• Adapts the code to the new llama.cpp API.
• Fixes the build process.
• Updates the tests to reflect the changes in the embeddings.
• Adds support for arm64.
• Updates the .gitignore file.

[rodydavis]

Fixes #18

[O-J1]

Worked for me on Ubuntu as a minimal test script. If anyone else sees the size difference and is confused, that seems(?) to be expected (2MB -> 76kb)

I wasnt able to figure out cross-compiling it to windows sadly.

Thanks for this @rodydavis

[vlasky]

Excellent work on this PR @rodydavis ! The API adaptations are spot-on.

One thing worth noting: the llama.cpp update changes the embedding values produced by BERT models. The test uses "alex garcia" with all-MiniLM-L6-v2:

Old llama.cpp (2b33896): first float ≈ -0.092
New llama.cpp (4fd1242): first float ≈ +0.005

I investigated the cause. It appears to be due to ggml-org/llama.cpp@6562e5a ("context: allow cache-less context for embeddings") which optimizes BERT models to skip KV cache allocation. A side effect is that llama_decode() now redirects to llama_encode() for these models, which returns the [CLS] token embedding instead of the last token embedding.

Claude advises me that "this new behavior is actually more correct for BERT models [CLS] is the designated sentence-level representation". Absolute values have changed, but semantic similarity is preserved. Nonetheless, embeddings from old and new versions aren't directly comparable.

The lesson is: users need to be aware that updates to llama.cpp have the potential to affect embeddings, which may require stored embeddings to be regenerated.

Here is a full breakdown

[vlasky]

@rodydavis you might also be interested in checking out PR#21