Unable to remove ghost node with either C7 or PC Controller

🛎️ Get Help
1

I have ghost node I cannot remove. I have have been using PC Controller to remove ghosts for a while now, so it is not new to me. In fact, I removed a couple of failed GE switches and subsequent nodes this weekend.

I removed a GE outlet and was attempting to add it back into my mesh via z-wave inclusion when the inclusion failed and never completed the initialization. That created a ghost node... I thought no big deal, I'll use PC controller to remove it...

No dice... On PC Controller, it either says failed or just hangs without throwing an error...

I'm using an Aeotec Gen5 Z-stick... Like I said, it was working great before... However, those were all physically failed devices that were physically removed...

But this was an incomplete join, and hence, my issue... Pinging the z-wave god @bcopeland...

Anyone else have any ideas? I was thinking about rolling back to my cloud backup from last night... I'll have to go through and rejoin a few devices I replaced today, but at least I won't have the partial failed inclusion node...

Sometimes PC Controller will allow me to mark the node as failed... Sometimes it won't... But when I do get it to turn red, and then hit remove failed, I get this or nothing and it just hangs...

01:32:03.129 Remove Failed Node failed in 00:00:04.021

By the way, in the C7, I was getting this message... Again, I thought no biggie, I'll do it from PC Controller...

Failed node 7B remove status: 0 SDK failure node is no longer in failed node list

2

Have you tried cutting power to it and then removing it?

2 Likes
3

Says the device responded to the SDK ping. You’ll need to remove power to the the device before trying a refresh and remove. Doing a shutdown and power off of the hub (unplug from the wall) before and after wouldn’t hurt either.

3 Likes
4

If the recommendations from @aaiyar and @thebearmay don’t work in this case (they should) then restoring the cloud backup from before the issue occurred and including the Z-Wave database should work.

1 Like
5

Update: The power was removed from the device when I first posted. I was super tired so I went to bed planning on restoring from the backup database in the morning.

Got up and decided to give it one more try. PC Controller worked this time and the ghost node was removed. I wonder if the network needs time to settle or maybe it had something to do with the internal hub backup process that happens in the middle of the night?

Anyway, feeling pretty good, and with everything clean, I decided to add that GE outlet back into my mesh... During pairing, same dang thing happened again! It created a ghost node and a regular paired node!

Needless to say, I removed that outlet completely... Of course it wouldn't go nicely, so now I have two ghost nodes again that PC Controller can't seem to remove... And the outlet is completely disconnected...

In case you were wondering, I had another new outlet that I was able to add seamlessly in its place... That outlet added fine and no ghost nodes came with it...

I just have to remove the two created by that problematic outlet. I think I'll wait a while to see if time helps the situation.

Thanks for everyone's help.

1 Like
6

UPDATE: For anyone using PC Controller and having an issue getting rid of ghost nodes created during the inclusion process, I have some info...

I had to wait until the next day for PC Controller to be able to mark the nodes as failed and then remove them. I'm guessing it has to do with how the hub cleans and backs up the database in the middle of the night. I remember having inclusion issues with my C5 a few years ago and @bobbyD telling me that database action happens every night... This seems to be specific to ghost nodes created with a failed inclusion process. Other ghost nodes left behind when a device is removed seem to react fine to PC Controller...

@bcopeland

7

These are the types of problems that pushed me away from Z-wave. Zigbee just does not have these issues.

8

Fair enough. But how does this observation and your preference help @SuperDupe with his existing z-wave mesh?

1 Like
9

If waiting overnight seems to fix things after the scheduled hub cleanup, curious if support has a way of running that cleanup on demand.

1 Like
10

That's a great question and something I also wondered about... That's why I tagged @bobbyD and @bcopeland. If that could be done with a push of a button... But I also might be wrong... I just know I had a failed inclusion ghost node on Saturday and couldn't removed it with PC Controller until Sunday. Then I got two inclusion ghost nodes on Sunday and couldn't remove them with PC Controller until this morning...

I'm hoping maybe either Bobby or Brian can weigh in as to if its in my head or there really is something that happens overnight.

1 Like
11

There is no nightly clean-up task running on C7 hubs, only on older models.

2 Likes
12

@SuperDupe

If there's no nightly clean-up, it might be that ghost removal works better when the z-wave network has quietened down (or there's no backlog of transmissions for the radio) ..... @bobbyD - any thoughts on this?

13

There has to be something going on... The quieted network is a good theory also. All I know is that I tried for hours on Saturday and eventually gave up and went to bed. I got up pretty early the next morning and it worked... There was probably about 6 hours in between.

Then yesterday, on the other ones, I gave up around noon. I tried again last night before bed, several hours later and it still didn't work. This morning, got up and it worked right a way...

Time and network activity might be the common factor...

1 Like
14

Could there be anything else that correlates? Or do you think its just network activity?

15

The other thing to try is shut down your hub and pull the power (from the plug side) and wait 20 secs or so then start up again. That should clear the Z-Wave radio..

1 Like
16

We have no control over when the radio reverses itself and decides to add the node back to failed node list so it can be removed safely.

1 Like
17

I pulled power on the hub the first day, but it had no effect... From what bobby just posted, apparently the radio just has to randomly allow it... I wonder if in the SDK, there is something tied to the date? Say if it hasn't been able to connect for a day or at least a date change, then put the node into the failed node list?

I almost want to create another inclusion ghost just to test that theory... Wait, no I don't... :joy: :rofl: :crazy_face:

3 Likes
18

I've watched PC Controller do its Is Failed? process in Zniffer and I can affirm that a lot of the time, no packets leave the radio. The ZWave Radio Chip is using Cache, is the only obvious answer. And of course what's cached? The last working info.

Is Failed? leads to a device being added to a Failed List.. Remove Failed looks in that list first. If the device isn't there, it fails. If it was in the list, it again tries to contact the device. Again, it probably gets it from Cache and fails.

A quiet mesh perhaps allows the garbage collection/cache refresh to run. Overnight, it might run enough times to reliably remove the ghost cached entries. Possibly slamming the Radio will force the old cached entries to be purged to make room for new devices.

Lots of maybe's to possibly explain it :slight_smile:

1 Like
19

In PCS would hitting "update" - update the stick (and therefore the internal cache) with the HE info?

1 Like
20

Yes. It asks the primary controller to update the DB of the secondary, I can't say if it would affect the Cache.. If the two DB are too far apart, it can be unwilling to do the update.. it tells you that by saying update complete far faster than is possible :slight_smile:

2 Likes