Hubitat Hub Generating Network Multicast 'Storm' Using mDNS

This has seemed to work for the last 24 hours!

Great! I've had no issues with my ESP32 or ESP8266 with mDNS disabled either. Glad there's a work around while the Hubitat team is working a fix.

1 Like

Confirming that I also have a similar issue. Hubitat c-8 on 2.3.9.158 and WLED running in two dig2go kits, firmware 15.0 beta 2. With mDNS enabled on WLED, seeing consistent and frequent crashes on WLED causing the lights to reset. Disabling mDNS appears to β€œresolve” it as a workaround.

Is this something that must be fixed by Hubitat? I see that it’s under investigation but no official or recent response.

Yes, Hubitat needs to fix it. The hub is generating duplicate MDNS update entries and they grow over time.

Since WLED is all about visuals, it made it more obvious to notice when it reset. Any other IOT or embedded devices could manifest very differently, if they are affected.

They recently fixed an unrelated issue that was discovered with Basic Rules and thermostats, so hopefully this is still on their radar.

1 Like

What is a current status for this issue?
I have multiple (around 20) ESP32-based controllers and noticed they are randomly disconnecting from WiFi (connections is randomly up and down). I have no idea how to troubleshoot if this is related to the described MDNS or could be something else.
Unfortunately I am in a big apartment complex with well too many WiFi networks around.

I would say it's resolved.

There was an update, months ago, that reduced the MDNS traffic to "normal", in my opinion.

Understand that MDNS is chatty. For the benefit it brings, I don't actually understand why so many messages are produced per minute. However, looking at a tcpdump, I can't say I see more from a specific Hubitat hub than I see for, say my Homebridge instance, or my Asus router. Now... I do have 5 Hubitat Hubs alive online at once and so I see a larger total than I see for one instance of Homebridge.

I did NOT turn this answer into a 'project' where I'd capture some great number of packets and sort/bin them for actual counts. I just watched them roll by to get a feel for the traffic.

This topic was started with a report of a "storm" and that was found and fixed. I'm not seeing a storm today, or anytime in the past couple of months. That's not to say something else MDNS'ish isn't contributing to your issues. I looked at 4700 MDNS packets in roughly 8 minutes and don't consider that to be 'too much'. Remember, when the 'storm' occurred, there were 2400 packets in 1 second.

1 Like

Thank you for the update and explanation.
So, my case could be related to something else. WiFi in a building is very crowded. In general and to my big surprise my network looks stable except for the ESP32 devices. I am not sure how to trouble shoot this issue and make them happy on the network.

@gopher.ny

Both hub running 2.3.9.180, One is ethernet and the other is Wifi. Homekit / Airplay is not installed on either hub. I am using Hubmesh just to test some things, not sure if shutting it off would help?

I have been having intermittent networking issues for a while now and I thought I had it solved with some router tweaks but its back again. Issue is that my RDP to a local machine will just freeze for a second or two, and also if I am on a meeting using VOIP the audio cuts out for a few seconds. The audio is a major annoyance, the RDP freeze is how I just now correlated it exactly with the MDNS spikes.

Finally decided to just get Wireshark setup and a little chart going. Did not take long to catch two huge MDNS storms all coming from my C8 hubs.

Is there any way to mitigate this so it does not lag the entire network? Can I block MDNS on the ethernet port with a firewall rule or something? One of the hubs is on Wifi so no idea how to block that unless I do it by IP I suppose.

This is annoying to the point where if I could shut off MDNS on the hub I would.

Its is just thousands and thousands of this junk from each hub:

1 Like

@jtp10181 - By any chance, do you happen to have a SiliconDust HD HomeRun TV tuner on your home network? I was periodically having mDNS multicast storms on my home network. I also used Wireshark (rookie skills here!) and my issue appeared to be caused by the HD HomeRun device. I have had the HDHomeRun for many years without issue. I even tried a new HD HomeRun, and the problem persisted. Definitely a random occurrence, as far as I could tell. But, it makes me wonder if something else on my network might be the instigator and then the HD HomeRun goes into a mDNS storm death-spiral? I will say, that ever since I disconnected the HD HomeRun from my network, I have not noticed any mDNS multicast storms (at least nothing that caused the problems with my older UniFi APs and UniFi network switches - these were CPU constrained by all of the storm traffic!)

Just thought I'd share what I have seen with respect to mDNS storm issues. It definietly feels like something is not quite right.

1 Like

Nope, I do not have such a device. I might have to capture some more stuff and dig into a little deeper to see if there any any patterns that seem to trigger the hubs.

But the thing is, both hubs did it, at separate times independent of each other. So it is like the original cases of this being found where it seemed to be on a timer repeating every X minutes.

I can setup some more captures for a longer time period and see if it goes in a repeat cycle.

EDIT: I changed a setting on my routers avahi daemon (put it back to defaults and disabled the reflector setting I had added), just in case that was it. Seems to be going fine but then just had another RDP freeze and sure enough checked the chart and another huge mDNS spike. 3k packets in the matter of 500ms - 1 s

2 Likes

Just found this is probably a No, duh, since it is multicast the traffic does not necessarily route through the router. The only way to block it from hitting my other devices would be to isolate it on a separate vLAN.

Would be nice if we just had a way to shut off mDNS, if I dont use HubMesh or Homekit what do I need it for?

1 Like

Will investigate.
Is there a specific interval to these spikes?

Looks like its 1200s (20 minues) apart on both hubs.
My active hub is also sending more garbage out than the dev hub.

Actually now just realized I was missing some spikes on that chart after my RDP froze again. My very idle C7 hub seems to be the worst. Still right around 1200s apart consistently with all 3 hubs.

1 Like

Also looks like it builds up steam over time possibly. I shut down my C7 hub and rebooted the Dev C8. The Dev hub spikes are MUCH shorter now compared to the main hub which has been up for 5 days now.

2 Likes

It does, at least on mine. I stopped reporting this after the discussions a few weeks/months ago as it didn't seem to be a priority to fix - but it is still happening. Very easy to see in wireshark, and gets worse and worse over time for me.

I would completely turn mdns off on my hubs if I could (I don't need it for anything I know of). But I can't.

2 Likes

Initially it was a big problem. It got whacked down in one release, but over time, most has been put back, apparently. Wireshark, a few moments ago, confirmed that I'm seeing thousands of MDNS packets in a short time ago (2800 MDNS packets in ~2 seconds from a single Hub) which is consistent with the reports from April.

2 Likes

Yeah it would not bother me if it was just the spike with no consequences which is why I never bothered to look into it before. During meetings my audio will cut out for 1-2 seconds so then I cannot hear what people are saying. Then the meeting software will say I have poor network quality and work fine again for a bit. This is EXTREMELY annoying. I have 100% tied these mDNS spikes to my RDP freezes, which I have previously determined happen at the same time as my audio lag.

I was honestly not sure what I would find with Wireshark as I thought this issue was resolved, so I was expecting to find some other device causing issues. Ran it for a few minutes and saw some big spikes, clicked on it to check and low and behold, Hubitat mDNS spam.

So then I ran it some more and next time my RDP froze I checked the chart and sure enough, right when a mDNS storm happened.

Now I am just angry I did not check Wireshark a long time ago, this issue has been annoying me for months. Pretty much all summer I think.

2 Likes

It got 'fixed' in April and then with each new Platform release a sliver got put back, if my memory of the release notes is accurate. Now, I think it's back to it's pre-April code because it looks quite similar to what Wireshark was showing me then.

1 Like

MDNS is used (at a minimum) by the Homekit Integration and as we've seen, there's a scattering of people where it goes "Not Responding" way too often. I think code got put back to try and help those people. Whack-A-Mole is a frustrating game :smiley: But who better than Gopher to fix a mole problem? :slight_smile:

3 Likes

I agree that is how it went away, and came back. At least that matches my memory.

Still not a great answer for those of us NOT using HomeKit though. :wink:

But I agree, gopher can get to the bottom of it given time and energy and prioritization.

2 Likes