GLM DSA: reduce indexer cache size

[ikawrakow]

The GLM-5 indexer only operates in a subset of the model layers, the other layers reuse the indexer result from a preceding "full" layer. This is taken into account when preparing the compute graph, but not when allocating memory for the indexer cache.

This PR avoids the unnecessary cache allocation. For a context of 128k tokens and f16 indexer cache, cache size drops from 2.5 GiB to 672 MiB.

[sayap]

This applies to GGUF from unsloth only right? If I understand correctly, they hacked the conversion to duplicate the indexer tensor to all layers, just so it can be loaded without ggml-org/llama.cpp#24770

With a proper GGUF, I don't notice any difference from this PR 😄

[ikawrakow]

This applies to GGUF from unsloth only right? If I understand correctly, they hacked the conversion to duplicate the indexer tensor to all layers, just so it can be loaded without ggml-org/llama.cpp#24770

With a proper GGUF, I don't notice any difference from this PR 😄

If the GGUF doesn't have the indexer tensors for a layer, no indexer cache was created in that layer even without this PR. So, yes, you will only see a difference with the llama.cpp-style "hacked" GGUFs that duplicate the indexer tensors in each layer.