Z2M won't start: NCP RESET_ASSERT / ASH_NCP_FATAL_ERROR during init — identical on TWO different ember coordinators (over TCP)

[devincornell7]

What happened?

Summary
After a coordinator migration, Zigbee2MQTT can no longer start. The ember NCP fatally asserts (RESET_ASSERT → ASH_NCP_FATAL_ERROR) partway through init, preceded by Frame(s) in progress cancelled and (in some runs) ROUTE_ERROR_ADDRESS_CONFLICT for "0", then ASH_ERROR_TIMEOUTS. The exact same failure occurs on two completely different radios — the original SLZB‑06M (EFR32MG21) and a brand‑new SLZB‑06Mg24U (EFR32MG24) with freshly‑cleared NVM — both connected over TCP. The crash point varies between runs (GET_EUI64, SET_SOURCE_ROUTE_DISCOVERY_MODE, GET_NETWORK_PARAMETERS/ezspExportKey), which suggests an unstable/aborting link rather than one deterministic logic error.

Setup

• Host: Home Assistant Green (HA OS), Zigbee2MQTT add‑on 2.12.0‑1, zigbee‑herdsman 10.4.0
• Adapter driver: ember, tcp://<ip>:6638, baudrate: 115200, rtscts: false
• Coordinator A (original): SMLight SLZB‑06M (EFR32MG21) @ 192.168.50.198 — reports EmberZNet 8.0.2 [GA] build 397, EZSP 14
• Coordinator B (new): SMLight SLZB‑06Mg24U (EFR32MG24) @ 192.168.50.9 — SLZB‑OS Core v3.3.3.dev4, Zigbee FW rev 20250212 (SDK 8.0.2.0.397), EZSP 14
• Network: channel 11, PAN 0x61ee, ext‑PAN 0xc906bc61e7a7f814, coordinator IEEE 0x9035eafffeb08f1a; ~170 devices / ~110 routers (router‑dense)

Representative logs
This morning (Coordinator B / MG24, over TCP):

warning: zh:ember:uart:ash: Frame(s) in progress cancelled in [011ac2020682af7e]
error: zh:ember:uart:ash: Received ERROR from adapter, with code=RESET_ASSERT.
error: zh:ember:uart:ash: ASH disconnected | Adapter status: ASH_NCP_FATAL_ERROR
error: zh:ember:ezsp: Fatal error, status=ASH_NCP_FATAL_ERROR. Last Frame: [FRAME: ID=40:"GET_NETWORK_PARAMETERS" Seq=28 Len=30]
error: zh:ember:ezsp: ERROR Transaction failure; status=ASH_ERROR_TIMEOUTS. Last Frame: [FRAME: ID=40:"GET_NETWORK_PARAMETERS" ...]
error: z2m: Error: ASH_ERROR_TIMEOUTS
at Ezsp.ezspExportKey (.../ember/ezsp/ezsp.ts:6244)
at EmberAdapter.initTrustCenter (.../ember/adapter/emberAdapter.ts:890)

Earlier (Coordinator B), version + a different crash point:

info: zh:ember: Adapter version info: {"ezsp":14,"revision":"8.0.2 [GA]","build":397,...}
info: zh:ember: [STACK STATUS] Network up.
info: zh:ember:ezsp: Received network/route error ROUTE_ERROR_ADDRESS_CONFLICT for "0".
info: zh:ember: [INIT TC] Adapter network matches config.
error: zh:ember:ezsp: Fatal error, status=ASH_NCP_FATAL_ERROR. Last Frame: [FRAME: ID=90:"SET_SOURCE_ROUTE_DISCOVERY_MODE" ...]

Coordinator A (original MG21, over TCP):** same Frame(s) in progress cancelled [011ac2020682af7e] → RESET_ASSERT → ASH_NCP_FATAL_ERROR at NETWORK_INIT / GET_NETWORK_PARAMETERS.

Note the recurring cancelled‑frame buffer […1ac2020682af7e] across many runs and both radios.

Timeline / what triggered it
Migrating from the MG21 to the MG24 via ember backup/restore. The MG24's native IEEE differs (0xe07291fffeb9d765), so herdsman's custom‑EUI64 write was needed to present the network's coordinator IEEE (0x9035…). That write asserted at GET_EUI64 on 8.0.2. We then set the IEEE directly in SLZB‑OS (got past GET_EUI64). In the process the MG24 booted ~8× as coordinator (addr 0) with its native IEEE before the IEEE was corrected, which appears to have left address‑conflict state in the mesh. Since then neither coordinator completes startup.

What we've tried (and results)

• Set MG24 adapter IEEE → 0x9035eafffeb08f1a via SLZB‑OS → passed GET_EUI64, then asserts later.
• Cleared NVM on the MG24 (firmware flash dialog) → still asserts; ADDRESS_CONFLICT persists (so it's not the radio's own NVM).
• ~8 h "settle" with all coordinators off → no change (devices never relearn while coordinator is down).
• Repeated restart cycles → no convergence.
• USB connection (to isolate TCP): the MG24 in USB mode did not enumerate as a serial device on HA OS (no /dev/serial/by-id, no ttyUSB/ttyACM) — couldn't run the USB test.
• Confirmed only one coordinator on air at a time (no dual‑coordinator clash).

What this rules out

• Not coordinator hardware/NVM/IEEE — two different radios (one freshly wiped, correct IEEE) fail identically.
• Common factors that remain: TCP transport (Frame(s) in progress cancelled + ASH_ERROR_TIMEOUTS, variable crash point), the mesh (persistent ADDRESS_CONFLICT for "0"), and EmberZNet 8.0.2 (only firmware available for these SLZB radios).

Questions

1. Is the ROUTE_ERROR_ADDRESS_CONFLICT for "0" storm + NCP RESET_ASSERT on 8.0.2 a known firmware issue, and is there a firmware/version that tolerates it?
2. How do we clear mesh‑side address‑conflict pollution when the coordinator asserts before devices can relearn? (Any way to start the stack "quietly" / without source‑route broadcasts?)
3. Could the Frame(s) in progress cancelled / ASH_ERROR_TIMEOUTS indicate the TCP serial bridge is the real culprit (vs. the address conflict)? Recommended way to isolate transport if USB won't enumerate?
4. Is there a clean recovery short of re‑forming + re‑pairing ~170 devices (e.g., via ember-zli)?

Related issues seen
Koenkk/zigbee2mqtt #28842 and #23198 (ADDRESS_CONFLICT for "0" on SLZB ember); #25860 (Frame(s) in progress cancelled); hassio‑zigbee2mqtt #849 / Koenkk #25920 / #30455 (SLZB‑06M TCP drops / ECONNRESET / ASH timeouts).

What did you expect to happen?

No response

How to reproduce it (minimal and precise)

No response

Zigbee2MQTT version

2.12.0

Adapter firmware version

20250220

Adapter

SMLight MG24U

Setup

Home Assistant Green.

Device database.db entry

No response

Debug log

No response

Notes

No response

[devincornell7]

Update with a lot more detail now that I've narrowed it down.

Short version: Z2M resumes the network fine — it loads all 165 devices and connects to MQTT — and then the NCP hard-asserts at GET_EUI64 during the post-resume init, every single time. I've reproduced the identical crash on two completely different coordinators:

• SLZB-06M (EFR32MG21), EmberZNet 8.0.2 [GA] build 397, EZSP 14
• SLZB-06Mg24U (EFR32MG24), EmberZNet 8.0.2 [GA] build 397, EZSP 14

Both over TCP. Z2M 2.12.0-1, zigbee-herdsman 10.4.0. Network is around 165 devices, ~110 of them routers, channel 11.

Things I've tested and ruled out:

• Not the coordinator hardware — two different chips (MG21 and MG24) fail exactly the same way.
• Not the transport. ASH counters are clean every run: Retry frames 0, Retry dupes 0, CRC errors 0, Comm errors 0, ACK timeouts 0.
• Not an address-conflict/mesh thing — recent runs show no ROUTE_ERROR_ADDRESS_CONFLICT at all.
• Not device count or power — same crash whether the whole mesh is powered or roughly half is off.
• Not a version change — a backup from when it last worked has the same Z2M 2.12.0-1.

Tail of a representative run (it's the same every time):

[STACK STATUS] Network up.
[INIT TC] Adapter network matches config.
[CONCENTRATOR] Started source route discovery. 1247ms until next broadcast.
z2m: zigbee-herdsman started (resumed)
 ... (lists all 165 devices) ...
Currently 165 devices are joined.
Connecting to MQTT server at mqtt://core-mosquitto:1883
warning: zh:ember:uart:ash: Frame(s) in progress cancelled in [001ac2020682af7e]
error:   zh:ember:uart:ash: Received ERROR from adapter, with code=RESET_ASSERT.
error:   zh:ember:uart:ash: ASH disconnected | Adapter status: ASH_NCP_FATAL_ERROR
error:   zh:ember:ezsp: Fatal error, status=ASH_NCP_FATAL_ERROR. Last Frame: [FRAME: ID=38:"GET_EUI64" Seq=33 Len=13]

In a couple of runs the fatal frame was ID=276 "EXPORT_KEY" inside initTrustCenter instead of GET_EUI64, so it looks like it's dying in the post-resume EUI64 / link-key export that herdsman does for the coordinator backup. My guess is that export is choking the 8.0.2 NCP on a network this size (165 devices / ~110 routers). Might be the same underlying issue as #31754 (MG24 + 8.0.2 + RESET_ASSERT).

A few questions:

• Is a RESET_ASSERT during the post-resume GET_EUI64 / EXPORT_KEY a known 8.0.2 problem on larger networks?
• Is there a recommended EmberZNet version for big meshes — 8.1.x / 8.2.x, or should I drop back to 7.4.x?
• Is there any way to disable or throttle that post-resume key export as a workaround? I'd rather not re-form the network and re-pair 165 devices if I can avoid it.

Can attach full herdsman debug logs if that helps.

[Nerivec]

Is a RESET_ASSERT during the post-resume GET_EUI64 / EXPORT_KEY a known 8.0.2 problem on larger networks?

That error comes from the firmware entering an unhandled state, having no choice but to crash (likely around NVM corruption considering the scenario).

Is there a recommended EmberZNet version for big meshes — 8.1.x / 8.2.x, or should I drop back to 7.4.x?

Any should work fine, recent versions add quite a lot of bugfixes compared to the older 7.4/8.0 though. You can try 8.2.x (been stable for quite a while now, it's a year old), or the pre-release 9.0.x:
https://github.com/Nerivec/silabs-firmware-builder/releases

Is there any way to disable or throttle that post-resume key export as a workaround? I'd rather not re-form the network and re-pair 165 devices if I can avoid it.

There are only 2 export key happening on backup, TC & network. Nothing the number of devices on the network would affect (per-device link key is disabled, so, no export happening there).

There are devices known for causing ROUTE_ERROR_ADDRESS_CONFLICT; usually, re-pairing them clears it.

---

Try this:

1. Clear NVM3 by flashing appropriate GBL (or APP for a more radical approach): https://github.com/Nerivec/silabs-firmware-recovery?tab=readme-ov-file#nvm3-clear
2. Flash desired firmware
3. Write IEEE
4. Start Z2M
5. Permit join on "All", let the countdown expire
6. Let things settle for a couple hours

[devincornell7]

Thanks, that got me a good bit further.

I flashed smlight_slzb06Mg24_zigbee_ncp_8.2.2.0_115200_sw_flow.gbl (from v2025.6.2-update1 — the model special-cases note says slzb06Mg24u uses the slzb06Mg24 build, so that's what I used). It's now on EmberZNet 8.2.2 [GA] build 436, EZSP 18. For the NVM3 clear I used the SLZB web UI "Clear NVM" checkbox during the flash, then wrote the IEEE back.

On start it now forms from the backup (same PAN 0x61ee / channel 11) rather than resuming, so the NVM clearly did get wiped. But it still crashes during init, before I can get anywhere near permit join:

zh:ember: Adapter version info: {"ezsp":18,"revision":"8.2.2 [GA]","build":436,"major":8,"minor":2,"patch":2,"type":170}
zh:ember: [INIT TC] Forming from backup.
zh:ember: [INIT FORM] Forming new network with: {"panId":25070,"radioChannel":11, ...}
zh:ember: [STACK STATUS] Network up.
zh:ember: [INIT FORM] New network formed!
zh:ember:ezsp: Received network/route error ROUTE_ERROR_ADDRESS_CONFLICT for "0".
zh:ember: [CONCENTRATOR] Started source route discovery. 1247ms until next broadcast.
zh:ember:ezsp: Received network/route error ROUTE_ERROR_ADDRESS_CONFLICT for "0".
warning: zh:ember:uart:ash: Frame(s) in progress cancelled in [001ac2020682af7e]
error:   zh:ember:uart:ash: Received ERROR from adapter, with code=RESET_ASSERT.
error:   zh:ember:uart:ash: ASH disconnected | Adapter status: ASH_NCP_FATAL_ERROR
error:   zh:ember:ezsp: Fatal error, status=ASH_NCP_FATAL_ERROR. Last Frame: [FRAME: ID=90:"SET_SOURCE_ROUTE_DISCOVERY_MODE" Seq=33 Len=9]

So it's still the ROUTE_ERROR_ADDRESS_CONFLICT for "0" followed by a RESET_ASSERT, just at SET_SOURCE_ROUTE_DISCOVERY_MODE now. Since it re-forms with the same PAN/IEEE from the backup, I assume the device(s) causing the conflict are still out there and still treat it as the same network.

Two questions:

• Was the web UI "Clear NVM" checkbox too light for this? Should I be doing the dedicated NVM3-clear GBL from silabs-firmware-recovery (or a full erase via universal-silabs-flasher) instead?
• Since it asserts during init, before permit join, I can't get far enough to find or re-pair the device that's causing the conflict. Is there a way to bring it up and keep it up despite the conflict — e.g. disabling source-route discovery / the concentrator — so I can locate and re-pair the culprit? Or is the realistic move to form a brand-new network (new PAN) and re-pair everything?

Can attach full debug logs if useful.

[Nerivec]

Was the web UI "Clear NVM" checkbox too light for this? Should I be doing the dedicated NVM3-clear GBL from silabs-firmware-recovery (or a full erase via universal-silabs-flasher) instead?

The clear NVM in the SMLight UI uses the same file as the one I pointed to (silabs-firmware-recovery).
The APP clear GBL (section below NVM3 clear) is one step further, it erases the entire firmware (takes lot longer), not just the NVM. It can help recover soft-bricked devices.
The NVM clear should be fine in most cases though. But might as well try another clean start with APP clear before going with the nuclear option (re-pair).

From the behavior you describe, I'd say some device is indeed triggering the ASSERT. There is no way to prevent this though (ASSERT is the "last-resort" fallback), could be any number of messages sent by the device (or devices!) that's doing it (can't say more precisely without a debugger on the adapter).
Some devices appear to handle IEEE change very well (some technically don't even need the coordinator to rewrite its IEEE; they will just follow), and some appear to handle it very poorly. First guess would be, something went wrong with one or more devices during the very first post-migration startup, and now these devices are "stuck".

You could try a startup without the antenna on the coordinator (or even with some aluminum foil wrapped around it). That will effectively kill its range, and prevent all/most devices from reaching it and vice-versa. It might allow you to start Z2M properly, and then you can put it back on, see if things settle better.
Whether that works all depends on what the offending devices actually do...

Last resort will indeed be a clean start with brand new network config and re-pair everything (routers first). Be sure to clear NVM beforehand, and pick a good channel, if not already, as well as generating new IDs/key.
It's a pain, but in some cases, after something like this, it's much cleaner than spending days trying to find a workaround, that may or may not hold up.

[devincornell7]

I ended up doing a clean install and everything is up and running as expected. Thanks for your help - much appreciated.