You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
laguna: generalize converter + model to Laguna-M.1 (all full attention)
Laguna-M.1 is the all-full-attention sibling of Laguna-XS.2: 70 layers, no
SWA layers, per-element attention gate, full rotary (~226B). The XS.2-only
converter/model could not load it (empty layer_types arrays, per-head-only
gate with a shape mismatch, hardcoded half-rotary).
- conversion/laguna.py: default layer_types to all full_attention when
absent; write data-driven rope.dimension_count[_swa] from
partial_rotary_factor; write the new attention.gate_per_head KV.
- laguna.cpp: drop the hardcoded n_rot_full /= 2; support per-element
gating (shape + element-wise application); set swa_type from
is_swa_any() and branch the attention input path (build_attn_inp_kv when
there are no SWA layers, required by create_memory's swa_type <=>>
is_swa_any() assert).
- New laguna.attention.gate_per_head KV across gguf-py + llama-arch.h/.cpp
+ llama-hparams.h (default true = per-head, keeps old XS.2 GGUFs working).
Verified: test-llama-archs laguna still OK (XS.2 mixed-SWA path); a 420 GiB
f16 Laguna-M.1 GGUF converts cleanly, has correct metadata (gate_per_head=
false, dimension_count=128, gate tensor 4096x8192), and loads + runs.
Pre-existing XS.2 GGUFs now need reconversion (they predate the required
rope.dimension_count).
Co-Authored-By: Claude <noreply@anthropic.com>
# Task: Add Laguna (Poolside) Support to llama.cpp
6
6
7
-
**Goal:** Implement `LLM_ARCH_LAGUNA`targeting `poolside/Laguna-XS.2`— a 33B-A3B MoE model with 40 layers, 256 routed + 1 shared expert, head-wise sigmoid attention gate, per-layer head counts, dual RoPE (YaRN for global, default for SWA), and per-layer-type partial rotary.
7
+
**Goal:** Implement `LLM_ARCH_LAGUNA`covering the Laguna family. Initially targeted `poolside/Laguna-XS.2`(33B-A3B: 40 layers, 256 routed + 1 shared expert, head-wise softplus attention gate, per-layer head counts, dual RoPE (YaRN for global, default for SWA), per-layer-type partial rotary). Later generalized to also support **`poolside/Laguna-M.1`** (~226B: 70 layers, all full attention, per-element attention gate, full rotary) — see "Laguna-M.1 generalization" below.
-**Step 1 (bonus) — Model saver**: Added laguna to unsupported arch list in `llama-model-saver.cpp` (same as step35) since roundtrip serialization loses SWA pattern keys.
-**Chat template + streaming parser**: `models/templates/poolside-Laguna.jinja` added; name registration and auto-detection via `laguna_glm_thinking_v5` marker in `src/llama-chat.cpp`/`.h`. Autoparser workaround in `common/chat-diff-analyzer.cpp` fixes two streaming bugs (see learnings 9–10).
26
+
-**Laguna-M.1 generalization**: Extended converter + `laguna.cpp` from XS.2-only to the all-full-attention sibling `poolside/Laguna-M.1` (~226B). (1) `conversion/laguna.py` defaults `layer_types` to all-`full_attention` when absent and writes data-driven `rope.dimension_count`/`rope.dimension_count_swa` plus a new `attention.gate_per_head` KV. (2) `laguna.cpp` removed the hardcoded `n_rot_full /= 2`, supports per-element gating, sets `swa_type` from `is_swa_any()`, and branches the attention input path. A 420 GiB f16 GGUF was produced and verified to load + run; XS.2 regression (`test-llama-archs laguna`) still `OK`.
7.**EOT token from `eos_token_id` list**: `config.json` has `eos_token_id: [2, 24]`. SpecialVocab only registers index 0 as EOS; token 24 (`</assistant>`) must be written as EOT via `add_eot_token_id` so it gets added to `special_eog_ids`. Without this the model loops past turn boundaries.
41
42
8.**ABI mismatch after cparams change**: Adding fields to `llama_cparams` (Step 2) requires a full rebuild of all binaries that link against `libllama-common`. A partial rebuild will crash with unexpected behavior.
42
43
9.**Prefilled thinking token breaks autoparser**: Laguna prefills `<think>` as the generation prompt prefix (thinking=on). Auto-detection finds `<think>` as `reasoning.start` from template diffs, but it never appears in the generated stream — the parser never enters reasoning mode. Fix: set `reasoning.start = ""` (delimiter-style) in the workaround. The `analyze_reasoning::build_parser` was extended to return `p.eps()` when `start.empty() && !ctx.inputs.enable_thinking`, preventing the streaming parser from misclassifying all plain content as reasoning when thinking is disabled (`--reasoning off`).
43
-
10.**EOT token as regular vocab token**: `</assistant>` (token 24) is a regular vocabulary token, not a special token. `preserved_tokens` only controls special-token decode and doesn't suppress it. Use `additional_stops` instead — this feeds into `task->params.antiprompt` which the `process_token` stop-word erase logic checks, stripping the text before it's sent. Added `additional_stops` field to `autoparser` struct; apply in `cli.cpp` as `task.params.antiprompt` additions.
44
+
10.**EOT token as regular vocab token**: `</assistant>` (token 24) is a regular vocabulary token, not a special token. `preserved_tokens` only controls special-token decode and doesn't suppress it. Use `additional_stops` instead — this feeds into `task->params.antiprompt` which the `process_token` erase logic checks, stripping the text before it's sent. Added `additional_stops` field to `autoparser` struct; apply in `cli.cpp` as `task.params.antiprompt` additions.
45
+
11.**`swa_type` must match `is_swa_any()` (`create_memory` assert)**: `llama_model::create_memory` asserts `swa_type != NONE iff is_swa_any()` — the iswa KV cache is only allocated when there are SWA layers. Laguna-M.1 has zero SWA layers, so `swa_type` must be `NONE` and the graph builder must use `build_attn_inp_kv()` (+ its `build_attn` overload), not `build_attn_inp_kv_iswa()`. `laguna.cpp` sets `swa_type = is_swa_any() ? STANDARD : NONE` and branches the attention input on `has_swa`. (Leaving `swa_type = STANDARD` with zero SWA layers was the first M.1 smoke-test crash.)
46
+
12.**Per-element vs per-head attention gate**: Laguna-M.1 uses `gating: "per-element"` (g_proj output = `num_heads*head_dim`, element-wise `attn *= softplus(g_proj(x))`); XS.2 uses per-head (g_proj output = `num_heads`, broadcast across head_dim). Signal via a new `laguna.attention.gate_per_head` bool KV (default `true` = per-head, so old XS.2 GGUFs keep working). Declare the gate tensor shape from it (`{n_embd, n_head_l}` vs `{n_embd, n_head_l*n_embd_head_v}`) and branch the application — per-element is a plain `ggml_mul`.
47
+
13.**`layer_types` defaults to all full_attention (Laguna-M); rotary is data-driven**: `configuration_laguna.py` defaults `layer_types` to `["full_attention"]*n_layers` when absent (Laguna-M); Laguna-XS ships an explicit mix. The converter must mirror this or the per-layer `head_count`/`sliding_window_pattern` arrays come out empty. Partial rotary is now data-driven via `rope.dimension_count`/`rope.dimension_count_swa` (written from each rope config's `partial_rotary_factor`: XS.2 0.5 global / 1.0 SWA; M.1 1.0 everywhere), replacing the old hardcoded `n_rot_full /= 2`.
| Checkpoint layout | split-per-expert, singular `shared_expert`, `e_score_correction_bias` under `experts` not `gate`|
65
69
70
+
> **Laguna-M.1** differs from the XS.2 values above: 70 layers (3 dense + 67 sparse), uniform 64 heads / 8 KV heads (no per-layer head counts), `sliding_window = 0` with **no SWA layers** (`layer_types` all `full_attention`), **per-element** attention gate on all layers, **full rotary** (partial_rotary 1.0), top-16 routing, 3 dense-lead layers, YaRN factor 64 / context 262144. Same tensor layout and MoE structure as XS.2.
71
+
66
72
---
67
73
68
74
## Key Design Decisions
69
75
70
76
**YaRN per-layer approach:** Use **Path A** — add per-layer-type YaRN siblings to hparams/cparams (`yarn_ext_factor_swa`, `yarn_attn_factor_swa`, `yarn_beta_fast_swa`, `yarn_beta_slow_swa`). Initialize them to non-SWA values so other architectures are unaffected. In the graph builder, select via `hparams.is_swa(il)`.
**Partial rotary:**Data-driven via `rope.dimension_count` (global layers) and `rope.dimension_count.swa` (SWA layers), written by the converter from each rope config's `partial_rotary_factor` (XS.2: 0.5 global / 1.0 SWA; M.1: 1.0 everywhere). The earlier hardcoded `n_rot_full /= 2` was removed — it only worked for XS.2.
73
79
74
80
**Dense lead:** Layer 0 is dense FFN; layers 1–39 are MoE. Branch on `static_cast<uint32_t>(i) < hparams.n_layer_dense_lead`.
75
81
76
82
**Attention gate:**`TENSOR_NOT_REQUIRED` on `wqkv_gate`; guard with `if (model.layers[il].wqkv_gate)` in the graph builder.
77
83
84
+
**Attention gate mode (M.1):** Per-element (`gate_per_head=false`) applies a plain element-wise `ggml_mul(attn_out, gate)`; per-head (XS.2) reshapes the gate to `[1, n_head, n_tokens]` and broadcasts. Mode comes from the `laguna.attention.gate_per_head` KV.
85
+
86
+
**SWA path selection (M.1):**`swa_type` and the attention input builder are chosen from `is_swa_any()` — iswa path for XS.2 (mixed SWA), plain `build_attn_inp_kv()` for M.1 (no SWA). Required by `create_memory`'s `swa_type ⇔ is_swa_any()` assert.
17.**`tools/cli/cli.cpp`** ✅ — Apply `chat_params.additional_stops` to `task.params.antiprompt` so the CLI stop-word erase logic strips them from streaming output.
115
125
126
+
### Laguna-M.1 generalization (additional)
127
+
128
+
18.**`conversion/laguna.py`** ✅ — Default `layer_types` to all-`full_attention` when absent; write `rope.dimension_count`/`dimension_count_swa` from `partial_rotary_factor`; write `attention.gate_per_head`.
129
+
19.**`src/models/laguna.cpp`** ✅ — Removed hardcoded `n_rot_full /= 2`; per-element vs per-head gate (shape + application); `swa_type` from `is_swa_any()` with branched attention input path (`build_attn_inp_kv` vs `_iswa`).
Greedy decoding on `"The capital of France is"` → `" Paris.\nThe capital of Germany is"` (from sgl-project empirical reproducer).
137
+
Greedy decoding on `"The capital of France is"` → `" Paris.\nThe capital of Germany is"` (from sgl-project empirical reproducer).*(XS.2 only; M.1 reference output not captured — the 226B MoE is I/O-bound on 60 GB RAM, ~50 s/token via mmap.)*
120
138
121
139
## Open Items
122
-
None. `moe_router_logit_softcapping` is absent from `poolside/Laguna-XS.2` config.json — no plumbing needed.
140
+
-`moe_router_logit_softcapping` is absent from both Laguna-XS.2 and Laguna-M.1 `config.json` — no plumbing needed.
141
+
-**Laguna-M.1 chat template**: ships `laguna_glm_thinking_v4`, but auto-detection in `src/llama-chat.cpp` keys on the `laguna_glm_thinking_v5` marker, so the built-in matcher rejects it (`custom template not supported`). Workaround: pass `--jinja`. Follow-up: add the v4 marker to auto-detection.
142
+
-**Step 5 / Step 7** still deferred — numerical validation needs CUDA; the 226B M.1 is also impractical for the quant top-1 check on current hardware.
143
+
-**Pre-existing XS.2 GGUFs need reconversion** — they predate the now-required `rope.dimension_count` and would otherwise get full rotary instead of half.
0 commit comments