I currently have the C8 hub running the latest (at the time of this post) platform version - 2.3.9.193. I use the Ethernet port and the IP address is assigned via DHCP / reservation (so the IP address is still fixed / static). I do have jumbo frames enabled as I utilized VMWare and other hypervisors where my guests / virtual machines are all mounted via NAS (QNAP) / NFS. This is a definite beneficial use case for jumbo frames. I started having issues with mine freezing up:
* Non-responsive from a network standpoint (couldn't ping the device, could
not access the diag page / port 8081)
* None of the rules were working which control mostly Z-Wave / Zigbee
devices
I did a bit of research and determined my issue is likely a result of the jumbo frames. While there are no direct communications that would likely utilize jumbo frames when connecting to Hubitat, services like mDNS can absolutely take advantage of jumbo frames. Long story short, my issue was simple to solve - I simply disabled jumbo frames on the device that was acting as my mDNS forwarder (forwards mDNS packets between different subnets) - that solved it. However, to be safe I also ended up putting my C8 into an isolated VLAN where I could better control these flows and ensure something down the road doesn't catch me by surprise. Considering this issue has been around for some time, I'm guessing implementing a fix is easier said than done - Hubitat has some amazing people on the support side (including the community) and if it was simple, they would have addressed it! Fortunately, its not hard to solve the issue. And frankly most residential / non-commercial users do not benefit from jumbo frames and could completely disable them for simplicity. If you are going to have some devices not enabled for jumbo frames and others enabled, you have to be careful and know what you are doing. In my case, I now use a dedicated storage network - jumbo frames only exist there and its the only place I benefit.
I did want to take a few to clarify a few points. While my expertise on Hubitat pales in comparison to the experts on this community, so I won't go into any hubitat specific troubleshooting which is ubiquitous on this forum from qualified experts. However, my area of expertise is networking - I have architected some of the largest enterprise networks for companies such as American Express and Bank of America. I will focus on the networking side in my following comments / information.
- First and foremost I can confirm that in the aforementioned platform version, the jumbo frame issue still exists. I was able to duplicate the issue repeatedly. A singly jumbo frame won't cause immediate issues but multiple over a period of time absolutely will!
- I want to clarify two points that get confused regarding MTU and what can be fragmented - frames versus packets. When referring to jumbo frames, we are referring to layer 2 frames (there are 7 layers in the networking OSI model - research as you please if you want more info). This is in contrast to packets, which are represented by layer 3+ of the OSI model (i.e. TCP/IP / UDP packets). Some L2/L3 switches will actually let you configure IP MTU and Ethernet / L2 MTU as separate components. Many switches require a global setting - L2 jumbo frames are enabled or disabled - all or nothing deal. Just depends on the platform.
a. layer 2 frames CANNOT be fragmented! Fragmentation occurs at layer 3+. Fragmentation occurs when the packet size exceeds the frame size - fragmentation is done by your router. Some packets enable the "do not fragment" bit, telling the router it can't fragment the packet. In this case, packet fragments are dropped. Generally, fragmentation is something you want to avoid as it impacts performance and causes out of sequence packets.
i, So what does this really mean? The MTU can be defined at two key areas (depending on the device of course). The IP / L3 MTU (which is back to packets) and the L2 MTU (typically Ethernet for most of us) which is configured on layer-2 switches (switches that have no ability to route packets and usually only have a management IP address).
ii. Lets say your router is set with an IP MTU of 9000. Lets also say your host / server is set with an MTU of 9000. But what if the server and/or the router are connected to a switch that does not support jumbo frames? If the router tries to send packets to the server with an MTU size of say 3000, the traffic simply drops at the switch. Remember, L2 frames cannot be fragmented - switches cannot fragment! And the router won't try to fragment - why should it? It has an IP MTU of 9000 - and it has no clue the switch its connected to doesn't support jumbo frames! Discrepancies in IP MTU can be handled by fragmentation (although undesirable) - so if the server and router have mismatched MTUs but the switch supports a jumbo frame, everything will still work assuming the "do not fragment" bit isn't set.
b. Layer 3 packets can be fragmented as described previously. Rule of thumb - L3 MTU mismatches can be handled via fragmentation, while L2 / frames cannot be fragmented! Also, frame sizes do not have to match - as long as all devices in the path can handle the relevant frame size, everything works. If the frame size exceeds any L2 device in the path, it breaks.
Hopefully that clarifies MTU a bit and differentiates the different components. My suggestion if you are having these issues are below. Keep in mind if you have jumbo frames disabled everywhere then you really have nothing to worry about. Even if the switch is configured for jumbo frames and cannot be configured (my 8P Netgear L2 switches support jumbo frames on all ports - and it cannot be changed!) this wont be an issue if you ensure all of your L3 devices (servers, routers, etc. - anything with an IP address) is configured with the standard IP MTU of 1500. If none of your devices are sending large packets in the first place, the L2 MTU is irrelevant as it won't be carrying any jumbo frames! In fact I generally keep my L2 MTUs at jumbo - they are simply the highway that carries the payload. My other devices define the payload!
-
Determine if you have jumbo packets - remember, your hub can freeze for other reasons as well. The most obvious is to check your devices (servers, streaming devices, router, etc.) and make sure the IP MTU is set to 1500 (or less in some cases especially if your ISP is DSL or a non-ethernet handoff). That being said its easy to miss a device especially when you have tons of them out there!
-
Remember, you can have large packet sizes enabled specifically where needed and use the standard 1500 elsewhere. As noted, my storage network has a dedicated subnet and those interfaces support an IP MTU of 9000. Everything else if 1500. In fact I don't even route my storage network - everything is directly connected to the same VLAN and there is no reason for that traffic to ever route beyond that.
-
If the switch you are connecting your hubitat device to does not support jumbo frames, then hubitat will never see them as they will drop when they attempt to ingress into the switch port. Thats why some people say you can simply connect hubitat to an older switch or one that is configurable and just disable jumbo frames. Keep in mind that if devices on your network are set with a large IP MTU, this could cause issues (i.e.mDNS repeaters running on a device with large IP MTU set - this traffic will be black-holed and never reach hubitat).
-
Put hubitat into an isolated VLAN - much better control. Unless you are routing multicast traffic, most general multicast packets will never reach Hubitat of its VLAN. An exception might be mDNS especially if your router or another device is re-broadcasting those packets to other subnets. Make sure that device has a standard 1500 MTU! Isolated VLAN or a VLAN with few devices makes sense and ensures you don't get blind-sighted later!
-
Connect a laptop or other device to the same VLAN (and ideally same switch) as Hubitat. Make sure your laptop supports a large IP MTU to match your L2 frame size (i.e. 9000). If you connect to a different switch, make sure it supports jumbo frames / the same frame sizes as the switch where Hubitat connects. And if the switch hubitat connects to doesn't support jumbo frames, don't even waste your time as hubitat won't ever see jumbo frames to begin with! Open wireshark and let it run for a while - ideally until Hubitat freezes. Now you can analyze the capture and identify any jumbo frames. Keep in mind large unicast packets (flow specific between two hosts) won't impact Hubitat. The only packets hubitat (or any network device) will process are unicast packets directed to it specifically or broadcasts / multicasts. Traffic such as ARP would never be close to 1500 bytes so you can ignore that. Once you identify the sources, you can begin to address the issue (i.e. configure those devices with a smaller MTU size, add filters at the source which can prevent forwarding to specific devices or VLANs, etc.).
-
Some switches also support port mirroring which is even better - you can mirror a port (sometimes called a span port) to another port. Now you connect your laptop / wireshark instance here and you will see EXACTLY what Hubitat sees. Some switches also allow for mirroring at the VLAN level as well. Frankly this is probably overkill - as long as your wireshark instance is in the same VLAN and ideally the same switch, you will see all relevant broadcasts / multicasts. Its unlikely there would be unicast flows targeting hubitat specifically that are large / jumbo packets. Most network protocols / general communication do not really use or benefit from larger packet sizes.
-
As previously mentioned if your goal is to simply eliminate large / jumbo packets / frames, just make sure you touch as many as possible and set them to 1500. Most devices default to 1500 and can't even be changed - i.e. most streaming devices, smart speakers (amazon, etc.) are set to 1500 and can't change. Would be no reason or benefit. In most cases its going to be your router or other network device or servers that act as mDNS repeaters that will be the issue. A laptop somewhere on the network with jumbo IP MTU enabled likely won't cause any issues at all even if missed! I would just leave frame size set to jumbo / 9k on your layer 2 switches. In many cases, you can't even change it! Focus on the devices producing the payload - if the car is set to drive in one lane it doesn't care that there are 9 other lanes next to it. If supported and you do change the frame size on all of your switches to standard / non-jumbo MTU, any rogue devices with a large IP MTU you missed might very well end up being broken - and they were likely not impacting your hubitat device in the first place!
I know this was long - I just saw a lot of frustration out there with this issue. I spent my share of time as well on it. On the network side, some of the information was limited and in some cases not specific when referring to jumbo frames and MTU. I am grateful I had this community to lean on when my hub was freezing up every couple of hours! There are some incredibly smart and helpful people out there that have my gratitude! Hopefully this pays it forward a little at least!
