feat: add scene handling (move from ZHC)

[Nerivec]

Move scene handling to ZH (spec).
Main goal is to better sync state/scenes so we don't get as many mismatches in e.g. frontend.

• Adds missing barrierControl & hvacThermostat clusters with respective scene-capable attributes (per R8 spec)
• Use more advanced sync mechanisms (depending on request)
  • scene from Zigbee
  • scene from state (clusters cache)
  • state (clusters cache) from scene
• Sync all clusters/attributes (ZHC did only a subset)
• Unicast sync done using view command to save "as it is on device" [subject to change, may not be necessary, needs actual tests]
• Groupcast sync done locally on "best-effort" (no feedback on request, many view requests required otherwise)
• Remove scenes on group(s) removal according to spec
• Add migration from .meta.scenes to .scenes for database
• Fix a somewhat dangerous shallow endpoint.clusters clone in Device (no cloning needed at all).

TODO:

• Z2M API endpoints and converter bypass for old scene stuff
• ZHC cleanup (remove all converters, remove auto-added converters on devices)
• State publishing on recall (what old recall converter did)
  • Investigate possible friction with devices using recall off-spec
• Handle state (sync/publish) when transition time is involved (either via recall override or in underlying state)

CC: @MfCrizz

[Nerivec]

@Koenkk any initial thoughts on this?
For the Z2M part, could add a dedicated extension or just do pass-through in bridge.

[Koenkk]

@Nerivec similar to groups, I agree this also deserves to be in zigbee-herdman indeed, code looks good to me!

[Nerivec]

@Koenkk can you take a look at the code before I start working on missing coverage? Cleaned up quite a bit, improved typing. I think it's about right?

For the Z2M/ZHC update, we're going to need new MQTT API endpoints (bridge/request/scene/*).
We also add tempory bypasses (deprecated for 3.0) in Z2M to call ZH API directly instead of going to converter when the old API is used (e.g. old frontend). This would allow proper ZHC cleanup (remove per-device converters for scene stuff).

[Koenkk]

LGTM!

Only question; how are we going to publish an updated state? Basically what is now done through the recalledState here.

[Nerivec]

Haven't gotten that far yet. Still need to figure out the details for Z2M/ZHC... 😅

Also have to handle the transition time better I think (should not publish scene state immediately when it won't be like that for possibly several seconds). I figure some devices will already report parts of the state on their own (stuff that's configured in reporting). Have to check what they do when transition time is involved (do they publish at start or end of timer, or several times in-between). Need to de-conflict this.

[github-actions]

This pull request is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

[hildebrau]

@Nerivec are you still working on this?

[burmistrzak]

Not stale 🍲

[Nerivec]

Need to go back to this when I have some time. It's rather time-expensive since we need to do a lot of testing with actual devices. There are also some pretty annoying barriers to overcome due to how the codebase is designed.

[Nerivec]

@Koenkk

State publishing on recall (what old recall converter did)

Any ideas on this one? As far as I can tell:

• We can't hijack the converter to be a dummy (just retrieve scene from ZH and remap) since it could trigger before ZH code executes which might cause misalignment. Plus there is the trouble of having to remap from raw cluster state back to ZHC-level state (values being converter-dependent).
• We can't emit a dummy Controller event since we're way down in Endpoint for the relevant logic.
• Adding event listening on every device endpoints at Z2M-level would create many event listeners.

[Koenkk]

I see no other way than z2m storing the state on a storeScene and setting that state back when it calls a recallScene. It will be a bit more tricky with the addScene, basically have to do the same as what is now done in zhc.

[Nerivec]

Not sure I get what you mean.
ZH already has the scenes stored (and fully synced) with this PR, but they are not key/value like ZHC has, it's raw cluster attribute name/value.
The only element that's missing, is the way to remap the ZH stored scenes to have the ZHC per-device expected state payload published on scene recall.

I suppose we could add a recallGroupScene and recallEndpointScene in ZH Controller, which would allow to emit to Z2M, but it does leave having to "do the jump to ZHC" somehow.
Could we maybe do some kind of tiny remapper in ZHC that takes the raw cluster attribute name/value and trigger the relevant "attr report" fz? The list of possibilities is known since the scene attrs are limited.
I'm thinking this would allow to trigger the state update as intended, without have to do the full frame building/parsing if we mock entire message logic all the way from ZH.

// in controller.ts `recallEndpointScene`
const sceneState = endpoint.recallScene(...);
// ...

this.selfAndDeviceEmit(device, "scene_recalled", {sceneId, groupId, transTime, sceneState});

// in Z2M on `scene_recalled` event
zhc.onSceneRecalled(...);

// in ZHC index.ts
export async function onSceneRecalled(...) {
    // trigger fz for defined clusters/attrs
}

[burmistrzak]

My two cents:
I think the remapper is an adequate solution if we truly want to keep the current payload format. ☺️

We should however use this opportunity to make sure that only one combined state payload per device/group is published on scene recall.

[Koenkk]

What about emitting a attributeReport message on scene recall? This will trigger the ZHC converters.

[Nerivec]

Would work, but that's what I mentioned above, would need to recreate the whole frame for each attribute of each cluster (meta requires rawData, and lqi would be a problem).

zigbee-herdsman/src/controller/controller.ts

Lines 1129 to 1138 in a62a4d5

	this.selfAndDeviceEmit(device, "message", {
	type,
	device,
	endpoint,
	data,
	linkquality,
	groupID,
	cluster: clusterName,
	meta,
	});

[Koenkk]

That's true, for LQI just put the last reported LQI

[Nerivec]

Mocking attributeReport is not doable though. Or shouldn't be done.
Let's say a group with 20 devices, recall a scene, that's 20*6 frames to build & emit.
Sure way to kill lower-end setups 😅

[burmistrzak]

Mocking attributeReport is not doable though. Or shouldn't be done. Let's say a group with 20 devices, recall a scene, that's 20*6 frames to build & emit. Sure way to kill lower-end setups 😅

I would agree with this assessment.
But what about pre-processing scenes, i.e. passing them through ZHC on startup and caching the payload for use during runtime?