Duplicate z-wave events

So I’ve had trouble with my zwave network slowing down for some time, and I think I’ve identified the issue but not the cause. I seem to be getting duplicated z-wave events delivered to devices.

This makes sense for the issue I am seeing.
Z-wave event trigger node fires off a z-wave event.
Hub responds and another devices activates fairly quickly.

Immediate subsequent activities on the zwave network are delayed with an increasing time per event triggered. Motion sensor activated, contact sensor activated, etc etc.

What made me start to suspect duplicate events is when I use the aeotec chimes the device will quickly and subsequently fire off the “chime” at increasingly delayed intervals the more things I trigger. The weird part is the chime will fire multiple times in a row with increasing number of times chimed the more subsequent events are triggered. i.e. 1 chime, then 5 in a row, then 7 in a row, all stepping on each other, ad infinitum.

If I stop all activity for a minute or two things calm down and go back to quick response starting the whole slowdown over again.

Also, in the logs on the Hubitat I see many repeat z-wave events, and they are not device or manufacturer specific. (See attached photos).

Is there some way to track down where this is happening? I have a large number of zwave devices and I’ve tried randomly unplugging one or two devices from the network with no positive results. I’m rather frustrated at this and not sure how to troubleshoot this, sans starting over and adding things slowly. Which I really don’t want to do, and at this point would leave me in an “unsafe” environment what with the goings on in Los Angeles at the moment.

Hoping for some suggestions!
Thanks!

With a brief glance, I'd suggest you could be hit by two issues...

You have Power Reporting devices. Those are notorious for clogging a network. It's mostly because the terminology is so opaque.. you can set when the power messages occur BUT there's at least two overlapping options. You can have it report on change AND you can have it report every X minutes. You really only need one. I lean towards change.. and Large Change at that. I do not choose 5 watt I choose 50 or 100 watt. So.. to determine if that's an issue, for a day, or an hour, set the reporting to be as close to never as possible. See if the reports and especially the double reports vanish. THEN adjust the change option down from never to half of never :smiley: See if you can live with that.

The other problem appears to be a mesh issue between that Leak Detector and the Hub. The battery device is trying and trying and trying to inform you but the responses (ACK) from the hub to the device might not be getting there... so the battery device sends again, and again, and again.

Pop it off the wall for an hour or 6 and move it 10ft closer to a repeater. Monitor at the end of the hour and see if it's gotten better. Depending on just how many repeaters are in the area, you probably want to run a ZWave Repair.

3 Likes

Ahh I can do that, but you seem to be hung up specifically on the devices I posted,which were more of an overall reference than specific devices.
Even just a door sensor exhibits this behavior, 10 contact door opening events for only opening the door once.

Reboot the hub from the settings menu and make the changes @csteele suggested and it should solve your problem.

I had that same issue, duplicate zwave events.

Cause: Hidden phantom nodes that are in your zwave mesh, but not shown in your list of zwave devices.
How did this happen: I removed a zwave device by using the "force remove button", but the device is still there, in reality. (For example, a zwave outlet).
How to fix: Physically remove the device (that was force removed). Use PC Controller software to get rid of the phantom node.

Could this be your problem?

This is what I’m thinking is going on, there’s enough repeaters and wired zwave plus devices I should not have issues with network reach. I’m thinking packets are getting routed through nonexistent devices and causing repeat packets, then they finally find a working route and they all arrive at the intended device.

I have a coupe of zwave sticks but I’ve had issues trying to get it to include in with the Hubitat. Also not entirely sure which software to use.

I’ve tried most of these, and the power reporting devices are set to never as I don’t care about reporting, I had just hit config/refresh for a visual in the log. There’s also an aeotec repeater 6 path from the basement and several ge plugin light switch modules all within 10 feet for a path to the hub, which should be 2 hops, possibly 3 if it takes the scenic route.

For shits and giggles though I will try unplugging that device do a repair, leave it unplugged for a day, plug it back in and try another repair. Also that device is powered with a battery, according to aeotec that device even when powered does not forward.

ZWave doesn't work exactly as you're describing...

The Hub has all the routing info. The devices have highly limited route info, slices of the full table.

For traffic originating on the hub, the routing info is included in the packet. Fixed, unchanging.

The hub (node 1) will send a routable packet to the next device (node N) with info in the packet telling that node where to route and telling the device further along, where to send the packet, and then finally on to the intended destination. 4 hops max.

So you'll get a packet trace (zniffer) that looks like:

The first 2 (of 3) are an unsolicited message from a device (Node 80) to the hub via #25 and #201.
The 2nd 3 are the hub acknowledging the message, also via #201, 25 to #80.
The 9th column is displaying the info, including 'arrows' showing which hop the packet is for.

The final 4 packets are non-routed (directly between the hub and device) called Singlecast, for comparison. Only routed packets contain the fields needed to route and they don't deviate. They don't self heal, middle of the packet flow. Instead, a routed error packet can be generated indicating the packet never reached the destination.

This capture shows that...
The Hub sends a command to device #190 via #137. Node 137 tries 3 times to forward the packet, fails and sends a Routed Error response back to the hub. The hub thinks "ok, that's a Lock, FLiRS, so ask node 190 to be a beamer for it." In other cases the Routed Error doesn't recover. The message is just lost.

Node 50 tries to send an unsolicited message... and there's a failure. One bit got garbled. The Home network DF 92 9C 98 was received by the Hub as DF 92 9C 90 and the hub doesn't converse with that network, so it discards it. Node 50 tries again and this time it gets through.

You have to be careful with CRC_ERROR because it might just be the Zniffer itself that didn't get the packet. It's critical to observe the next packets to see if the intended recipient responded correctly. IF yes, then the Zniffer's physical position is probably the cause. IF NO, as in this case, then the normal recovery (retry) solves it.

4 Likes

Thanks for all the suggestions.

I have turned off all reporting for power monitoring switches, moved things around. I’m still seeing duplicate events for >EVERY< single zwave device, not isolated to just the ones in my screenshots.

I just ordered a c7 hub and I plan on migrating devices over manually instead of doing the newly featured import. This seems like the easiest and most thorough route to take. There’s over 200+ zwave devices on my network currently, troubleshooting this just seems far too complicated. There doesn’t seem to be an easy way to have any view into the zwave network, so I’ll migrate one or two devices a day and create the rules around them. Not ideal but hopefully will identify where the issue is.

Thanks all.

Duplicate z-wave messages has been the bane of my z-wave network for my whole HE life. I so wish there was a better way to debug the mesh and messages. I have redone my network too many times and I'm so tired of starting over, not knowing if that will fix it anyway.

Understatement.

Even SiLab's PC Controller & Zniffer only get you half way.

Other systems have had issues with message duplication and it was solved on the Controller software by changing timing code:

I'm still wondering if routing tables, ghost nodes, or buggy HE Hub code is the issue. A low level Hub Z-Wave log with send and receive messages would help...

I think that it's usually phantom nodes, but as they say "your mileage may vary".

When you say phantom nodes, what do you mean here? Z-Wave nodes that are part of the Z-Wave mesh but not visible on the HE Hub Z-Wave page, or something else?

Yes

Correct.
The way to fix that is relatively simple. Go to settings, Z-Wave details, and there you will see a listing of all your Z-Wave devices in node order.
In PC controller you will also see total listing of all your nodes in node order. Compare the two lists if there are any in PC controller that are not in Hubitat that means of that there are Phantom nodes.
To get rid of a phantom node, in PC controller mark that node as failed and then remove failed button.

3 Likes

So... the c-7 lets you see the zwave messages log. Independent of the main log, it’s under the zwave settings section.

1 Like

Only the C-7?

yes

Confirmed no phantom nodes using PC Controller. Started new thread showing that HE Hub is initiating the Basic Get commands.