Interesting results from the test: I powered off every Zigbee device except for the centralite keypad and the iris motion sensor, neither of which are repeaters.
I brought a new set of Gardenspots into the house and paired them with the hub. Changed one of the MirrorMe buttons to use the new device. Sat them down at my feet.
Picked up the RGBgenie. I sat in the same room as the hub, with the rgbgenie in my hand and about 6 maybe 7 feet of empty air between it and the hub.
The user experience with the RGBgenie was significantly worse without the lights. I'd say one in every three or four taps made a change on the gardenspot.
The only thing that occurs to me is that the Gardenspots at my feet were likely the repeater for the RGBgenie in this situation. So I moved the garden spots to another room with a wall and a bunch of electronics between. Then I stood about 2 feet away from the hub but where I could see the lights. No real change in behavior.
I know that Z-Wave mesh won't re-route instantly. What could be done to ensure that the RGBgenie goes a more direct route?
Number one suspect. Note that @mike.maxwell keeps these on a hub by themselves so they do not interfere with the other Zigbee devices on his main hub. Version does not negate the issue. Only Sengled that do not repeat at all can be excluded as a cause of Zigbee mesh issues, since they are all end devices, unlike all other Zigbee bulbs that I am aware of.
I completely understand the suspicion, but also recall that Mike uses unreleased code to sync his hubs that I have no access to, and they will ask me to shut down the HubConnect code before reporting any other problem. That might be fine for unrelated/external things like say water sensors, but I can't have routine outages to the main lights in our house. Running two hubs is not WAF-safe.
Also, when powered off completely at the breaker they don't impact the result. In fact, it's worsened. See the test results above.
I just redid this test where I powered off the RGBgenie by pulling the battery, then powered off the lights by pulling the plug. Then I put two walls between us and no line of vision, and used my wife to relay light changes to me.
First I powered on the RGBgenie, and confirmed its actions were seen in the logs. Then the lights were powered on, and testing through the Hubitat UI confirmed it was communicating.
Response? No change. Every 4th or 5th button press on RGBgenie made a change on the lights, and after 5 button presses nothing worked until I pulled the battery again. Then it want back to intermittent changes.
I'm not saying the BR30s aren't problematic, but I think we've ruled them out of this particular problem.
@bobbyD I've beginning to wonder if I've been assuming it was my own ignorance about Zigbee... but perhaps all my problems with Zigbee might be a bad zigbee interface on the C5?
Are there any diagnostics which can be used to get more information about what's visible to the hub?
I doubt it.. From what you have said.. You are in a high RF noise area.. and you swapped with a friend that has no zigbee issues and reported the same problems in your location and not in his.. You ruled out a hub issue
Here's another interesting one that defies all logic. I powered back on all the lights, and suddenly the controller is working perfectly. I'm watching the debug logs and every touch I make on the RGBgenie is seen and communicated. I've been playing with it for 5 minutes now and it hasn't locked up even once.
Just for clarity: I have ZERO objection to buying another hub. These hubs don't even rate 1% of the total cost of my smart house expenditures. But whatever mechanism I use to sync devices needs to be something that @bravenel and @bobbyD won't tell me to turn off any time I'm debugging a different problem. If Hubitat had an officially support hub sync, I'd be all over that -- especially because it could allow more testing with less house operation impacts.
Z-Wave is also RF. One of the reasons that I have a deep comfort with Z-Wave has been that IME isolation testing has yielded rational, deterministic results.
After spending most of today trying to get smarter about Zigbee, I'm even more convinced it's effing magic not technology.
Okay, I'm revving myself up for a huge facepalm here I guess
In tracking down some other hub internals I saw repeated comments by Bruce and Mike that they were still using an older, internal hublink they'd created pre-public release. And I've seen the super-active HubConnect channel. If there is a supported hublink then I've managed to completely overlook it.
And if it's supported mechanism, why don't Bruce and Mike trust it?
However, HubConnect is a lot more powerful, and it has a couple other things that I like: it supports custom drivers (I think HubLink has only RGBW bulbs but no plain CT, for example) and it doesn't clutter the "host"/server hub with virtual/shadow devices (not sure why it was designed that way). While it is open reported that you will be asked to disable third-party code for the purposes of troubleshooting, I have never seen staff suggest this for HubConnect; in fact, I think Mike said it's what he'd be using had it come along before he wrote his own custom code. I guess I'm saying: your caution is warranted, but I wouldn't be afraid.
Every time I've asked for help with the Zigbee problems they've told me to disable all custom code. Even back before I had any custom code, and was running only Hubitat-provided things.
And these lights... they are a WAF problem. For most other things we have physical backups but for these recessed RGB lights there's no way to put them on a physical switch and they are 100% of the lighting in the two main rooms in the house.
We are incredibly happy with how it worked out, but it does leave these things in the MUST-WORK-AT-ALL-TIMES category