Looking at the screenshot above, I found the following:
The highlighted Zigbee packages (aside form the first one) are present only when the IKEA device (0x1662) is close to the HE Hub (0x0000) - which is also the Trust Center.
When pairing the IKEA remote far away from the HE hub (even if really close to a Zigbee repeating-capable device), these packages do not appear in the logs (aside from the first, the Request Key)
I need to read a lot more on the Zigbee security layer, but my basic understanding is this:
Request Key command: When it joins the Zigbee mesh, the IKEA remote is asking the Trust Center for the Trust Center Link Key (0x04).
Transport Key command: Wild guess: the Trust Center is indeed replying to the Request Key command BUT only using whisper / low-power mode; when you are not physically close enough, you don't "hear" the message. Some other mechanism might be at play here; what is certain is that this message is not delivered from 0x0000 if the device is not in close proximity. Then how did this message got to my sniffer, which was 3 meters away?
Verify Key and Confirm Key commands: The IKEA remote then verifies the received key with the Trust Center (0x0000, the HE Hub) to make sure it "heard" it well (whispering can distort sounds ).
Permit Join Request command: Using this acquired key, the IKEA remote is temporarily accepting new network joins (usually only the coordinator can do this) - I believe that these IKEA remotes were/are used to guide/herd other IKEA devices (bulbs?) into the Zigbee mesh.
One of the biggest changes to HE C-8 is the use of Zigbee 3.0 and one of the most significant changes to Zigbee 3.0 has something to do with the Trust Center functionality.
My guess is that C-8 is not able/willing to transport the Trust Center Link Key to IKEA devices joining the mesh, and these devices then get upset and go into "jerk mode" by refusing to honor their bindings (buttons don't do anything; don't know if periodic attribute reporting, like battery reports, work correctly).
I've ordered the SNZB-01P (the newer round version of SNZB-01) which is also Zigbee 3.0 so I should be able to try that on the C-8 at the weekend.
Has anyone from Hubitat acknowledged that this is an issue yet? I noticed a thread posted acknowledging issues when migrating from C-5 to C-8 but I don't know if that is related.
Zigbee doesn't use a 'whisper mode' during join; that's strictly a Z-Wave thing. When a Zigbee mesh is in 'add device' mode, all routers (with end device capacity) can accept join requests from a new device within their range ; once the inclusion is underway, all message frames actually pass directly between the end device and the parent router accepting the join; the hub needn't be in range of the new device at all-- it will route frames to the parent (which then are retrieved by the end device during subsequent 'data request' transfers).
Wireshark sometimes tends to obscure the actual source and destination at the link level; the capture lines may show only the Zigbee network layer source and destination (so a message coming from, say, end device 0x52ea to coordinator will show Source = 0x52ea and destination= 0x0000... even though that capture line represents the first hop of the message exchange between 0x52ea and its parent router 0xb835.
To see this you need to double click on the capture line and look at the 802.15.4 Data section where the actual source and destination for this first hop are displayed.
For example here's part of a capture of an end device (Tuya 4-button) joining a C-8 through an Iris V2 plug (0xb835); I made sure the join would be via the plug by cranking the C-8's power down to the minimum and locating it about 25' away from the room where the Iris plug and Tuya button were located.
Prior to line 65 the message transfers shown are between the joining end device and parent router (it's already been assigned short ID 0x52ea by its parent router 0xb835). At line 65 there's an exchange between the end device and coordinator (destination 0x0000). But expanding the details for that line shows both the Zigbee Network Layer S/D and 802.15.4 Data S/D and there you see where the actual first hop destination of this frame appears as that of the parent router 0xb835.
Prior to the C-8, the short ID's that a router would issue to a joining device would be randomized and change each time a device was reset and joined (they'd be broadcast as a device announcement to check for potential collisions). The C-8 has a different join behavior (evidently to help migration scenarios) and keeps a prior short ID assignment the same (by correlating the device's MAC address to a previous shortID assignment in its database). At least that's what appears to be happening...
That means that when a device previously joined to a C-8 is reset, it will retain its old short ID (deleting the device from the database prior to a rejoin will result in a new randomized short ID assignment),
The SNZB-01P paired fine. I've tried the IKEA E2123 again whilst already paired and now get powerSource showing, maybe that was due to the updated driver? Still not working though.
Tony, thank you for taking the time to look over this!
This is all very interesting that you can see how the message is relayed by jumping from router to router! Very cool stuff indeed! Thanks!
With this new unlocked knowledge, I found the following:
When I pair the IKEA Rodret Dimmer (E2201) via a router device the following happens:
The pairing process is starting.
The Coordinator (HE) immediately issues the "Transport Key" command towards the IKEA Rodret Dimmer without previously receiving the "Request Key" command. By following the routing chain, I can see that this command reaches the IKEA Rodret Dimmer.
Bindings and Attribute Reporting configuration happen ...
IKEA Rodret Dimmer issues the "Request key" command towards the Coordinator. After going through the routers chain, this command reaches the Coordinator.
The Coordinator ignores this command and does not respond with the "Transport Key" command as expected by the IKEA Rodret Dimmer device. Why?
When I pair the IKEA Rodret Dimmer (E2201) directly to the Coordinator (by keeping it very close):
The pairing process is starting.
The Coordinator (HE) immediately issues the "Transport Key" command towards the IKEA Rodret Dimmer without previously receiving the "Request Key" command.
Bindings and Attribute Reporting configuration happen ...
IKEA Rodret Dimmer issues the "Request key" command towards the Coordinator (directly, no routers/hops).
The Coordinator responds to this command by sending (again) the "Transport Key" command towards the IKEA Rodret Dimmer.
IKEA Rodret Dimmer issues the "Confirm Key" and "Verify Key" commands.
IKEA Rodret Dimmer issues the "Permit Join Request" command.
Seems like the Coordinator is refusing to respond to the "Request Key" command if the requesting device is not speaking to it directly, but via a router (the "Request Key" package reaching the Coordinator has wpan.src16 != zbee_nwk.src because wpan.src16 is the address of the last hop, and zbee_nwk.src is the address of the requesting device).
Is this behavior normal or is there a bug/quirk of HE?
Unfortunately I don't know enough about the key exchange protocol to hazard a guess... I think the initial unsolicited 'Transport Key' is normal on initial joins when the device gets sent the mesh's network key; that's the unique key shared among all devices on a given Zigbee mesh (so they can all understand the same broadcast messages), as opposed to link keys which get shared between two individual devices communicating securely (actually aside from transporting the network key on joins, I don't know if those types of secure exchanges using unique link keys happen in hone automation networks since HA devices always talk to the coordinator rather than each other).
When a router is handling the join, it assigns a random short ID to the new device and then exchanges several messages with the coordinator-- which includes the route to the new device and the short ID it has assigned; that's evidently when the new join behavior of the C-8 can come into play and override the short ID assignment with a different one; at least that's my theory. In any case, at that point the coordinator (as trust center) would start the network key exchange, either directly if it's a child device, or via routed messages.
The unique network key for a given mesh is itself encrypted during the key exchange with a link key when it gets sent to a joining device; prior to Zigbee 3.0, for home automation devices the network key was encrypted with the ''well known" link key (the 16-byte one that starts with "5A:69..." ).
If you haven't configured your Wireshark setup with the 'well known key', doing that might make some of your traces easier to understand and might give another clue as to what's going wrong. There's an explanation on how to do that here: Nordic Semiconductor Infocenter ; see the section on 'Configuring Wireshark for Zigbee'
There also might be some info in this link Zigbee Network Key Sniffing | digiblurDIY but I haven't tried doing that myself... I'm still trying to understand how the basic join process works..
Hi @mike.maxwell, can you please check if this is the desired behavior of HE (C-7 and C-8)?
This would explain why IKEA remotes only work when paired really close to the hub (to chose it as parent). Well, I believe that IKEA remotes are at fault here for bailing out the pairing process if they don't receive the "Transport Key" message when they want and ask for it, although it was previously offered to them.
I haven't seen Hubitat (any hardware version) refuse to issue keys (any type) when requested by a router on behalf of another device.
I have seen routers not forward received key requests to the hub though...
If this is reproducible behavior let me know the specific device(s) that are subject to this issue and I can see if i can replicate it.
Huh, tested with a C8 hub running 2.3.7.126 (in beta, but there aren't any zigbee plarform changes in it), using an E1524 TRADFRI remote control running 117C-11C1-24040005 firmware, and joined it through a Third Reality 3RSP019BZ outlet without any issue.
I've be playing with frame capture for 2 hours using E1810 (same as your E1524) and E2201 (new Rodret remote) pairing close to the hub and via a router and came up to this conclusions:
When paired directly to the hub, Hubitat responds to the "Request Key" command with a "Transport Key" command. Upon receiving the Trust Center Link Key, the device verifies the key then broadcasts "Permit Join Request". Physically, the device blinks twice, then the light turns off. Pairing time is very short.
When paired using a router, Hubitat does not respond to the "Request Key" command. The last 4 lines in the screenshot above do not appear in the logs. Physically, the devices keep the little light on for a much longer time, no blinking twice at the end of the pairing process.
The E1810 ignores the failure to receive the Trust Center Link Key and goes on like nothing happened (this is nice). Newer IKEA models respond to the lack of the "Transport Key" command from the hub by broadcasting the "Leave" command. HE ignores this "Leave" command and thinks the device paired successfully (I have a lot of "Unknown" devices in my Zigbee Graph because of this)
In the screenshot above, we can see that the "Request Key" issued by the Rodret reaches the Coordinator via the router (IKEA Askvader On/Off Switch), Coordinator ignores it like a boss, and this makes the Rodret broadcast the "Leave" command (Elvis has left the building). Moments later, the Coordinator is asking the Rodret for some attribute, like nothing happened but the device is no longer paired to the mesh.
).
