Hubitat Hub Generating Network Multicast 'Storm' Using mDNS

Not only is the flood exactly hourly, it's got some correlation to the hub's start time.

I looked at the hub event logs and saw:

systemStart System startup with build: 2.3.8.138 2.3.8.138 4/08/2024 4:55:37.746 pm

and thought I could predict that the flood would occur on minute 55 of any hour. I was able to capture the flood in that minute with another 2400 packets in a second: 07:55:17.212149 to 07:55:18.296927

1 Like

@gopher.ny Is anyone looking at this on Hubitat's side? I'm seeing the same thing.

Yes, he's aware. :smiley:

2 Likes

Good deal. Just checking since there were zero posts in here.

3 Likes

A temp workaround might be to disable mDNS on the ESP32. If you remove the name on the WiFi setup screen, I wonder if it would cause the network stack to ignore all the broadcast traffic from the Hubitat?

image

1 Like

Wondering if this is also tied to the issues some people are having with Homekit going "no response" randomly. Since it uses mDNS HomeKit “No Response”, any resolution?

At wake-up this morning, my connection to HomeKit was dead from my C8. No Response mode.
I did the normal restart of the C8 and it came back.

In order to debug, I created a rule where every minute I would set a virtual switch called switch A, and when HomeKit saw switch A turn on it would turn on another switch called switch B.
In the rule I would check if switch B did not come on within 10 seconds I would declare HomeKit dead.

Interestingly enough HomeKit died exactly 1 hour after I rebooted my C8.
I am still investigating.

Also the C8 was no longer in the _hap._tcp list.

It's ONE second every 3600 seconds. It's 2400 packets (of 100mb) on a gig network. I saw 16-20 mDNS packets scattered within the middle of the flood: apple products (Homepod, TV, computer), cameras, dd-wrt, and a couple other Hubitat Hubs. I can't see how this could lead to that. :smiley: Anything's possible, but it's mostly just 1 second of latency.

2 Likes

There seem to be some “quirks” in Hubitat’s mDNS implementation. Most of the time there’s not much operational impact for most people. Another example:

1 Like

The "quirks"..... that's the thing that's weird to me... Under the Hubitat environment is just a Linux system, and most Linux systems use a core suite of tools and libraries to handle all the networking services like DNS, DHCP, etc.

So I am always trying to figure out WHY does Hubitat have networking issues? WHY can't Hubitat exist on LANs with jumbo frames? WHY doesn't the DHCP service honor DNS servers in my DHCP scope (you have to change them manually)? WHY are there mDNS abnormalities? I just don't understand, these basic networking issues should not exist on any up-to-date Linux based system.

1 Like

I just noticed while looking at the Discovery App, that in addition to my Hubitats being listed in _hap._tcp,
They also appear here:


Anyone know what this is about?

Isn’t that the Hub Mesh? TCP

I have Hub Mesh sharing several devices, but this entry comes and goes. Mostly goes.

Just a quick check in on this. I scanned last night and the packet 'storms' are now around 800 per second every hour. Its definitely growing over time and my LED controllers have started rebooting or becoming unresponsive because of Hubitat doing this.

Is there a way to disable mDNS on the Hubitat? I only have one hub and I have its IP reserved in my DHCP server.

I'm willing to share my network logs with the devs if that helps solve this.

This issue is also affecting others and discovered in March:

Unexpected mDNS queries from Hubitat devices - :bellhop_bell: Get Help / Devices - Hubitat

@gopher.ny is probably the one to examine the network issue and was tagged in the other thread. You might want to document your network layout - model of router, network configuration (VLANs), and how your hub is connected - ethernet or WiFi. My guess is that there is some interaction with users' network setup as these issues are fairly uncommon. For example, the only "quirk" I have is that hubitat.local gets flipped to the IP address. It's the only .local device I have that does that. I'm using a fairly generic flat network on UniFi equipment.

1 Like

Correct - a post in the other thread indicates the mDNS issue is known and is being investigated.

I have the same problem as @antamiga . Rebooting the C7 works for three days and then my WLED controllers start crashing/rebooting constantly.

It's unfortunate. Maybe it's not well recognized but the storm is of informational/broadcast packets. They don't do anything but update an existing table of values. In the storm's case, it's repeating the updates and so they can all be discarded, and would be under normal circumstances. I think the ESP32 is just underpowered enough that it is unable to process one packet in the time it takes to receive a packet. That should mean that any packet burst would overwhelm that processor. By that I mean, the network is capable of overwhelming them. Hubitat hubs are only 100mb network devices and most of us have a 1gig network.

I am not trying to downplay the mDNS storm that Hubitat is producing, but instead point out the precarious position all of us are in with any processor that can't dispatch a packet in the time it takes to receive a packet. The buffer size is just too limited but you might want to research increasing the network buffer size on the ESP32 for that specific use: WLED. It might be that it isn't a memory hog on it's own and could live side by side with a bigger buffer.

This has seemed to work for the last 24 hours!

Great! I've had no issues with my ESP32 or ESP8266 with mDNS disabled either. Glad there's a work around while the Hubitat team is working a fix.

1 Like