I recently updated to my Hubitat to 220.127.116.11 and noticed that it's now hammering my DNS server with thousands of PTR queries almost every other hour for around 60 minutes at a time. It's basically a scheduled occurrence and seems to start right around the half hour, runs for about 60ish minutes and stops for an hour where it repeats doing it again.
Ex - it happens at 5:30pm, runs until 6:30pm then starts again at 7:30pm.
This coincided almost exactly after my hub updated to this version. Looking at my logs, everything was normal prior to the hub being updated.
It seems very similar to this issue that was previously fixed but I didn't see any newer topics on it:
(Can't link it but search "Many DNS Queries from new Hub")
Is this happening for anyone else on this update?
I am not seeing this on my DNS server. Are you blocking something it is trying to access? Some devices just keep trying over and over when they get invalid responses back.
Nothing that I'm aware of. It's just a bunch of PTR records to local IP addresses repeated thousands of times. It's basically doing exactly what was happening to the users in the old thread from 2019.
My blocklist didn't change over that period of time prior to the update to my hub. Not really sure what's going on.
This seems to be related to static IP devices on the hub (ex - TP Link Kasa, Hue Hub).
I went ahead and restarted the Hubitat Hub to see if it corrects anything. So far it's okay but I'll monitor to see if it starts again. It looks like it took about an hour or so until it started after the last hub update.
Adding that after about an hour post-reboot the issue reoccurred. It seems to be checking if the devices are there and then keeps retrying or something.
They're all local IPs assigned to devices that are connected to the Hubitat. The devices themselves are responsive to commands issued through Hubitat (ex - a light with a static IP that's being requested multiple times to the DNS server turns on/off immediately as usual).
It does not appear to be isolated to one specific app / set of devices - it's pretty much any device that has an IP and is connected to Hubitat.
As noted, this never happened prior to this latest update. I may try to rollback to a previous version if I can't figure something else out.
If anyone has an idea, I'd appreciate the help!
If your hub assigned a static IP on the hub itself or through the DHCP server? There are random issues that pop up when using the static IP on the hub itself, this could be one of them.
It's assigned on the DHCP server.
Seems to primarily be the Kasa devices, maybe something related to that. But I do see repeated PTR records for the Hue Bridge.
Seeing the same thing . Definitely a bug please look.into it. 109 is my hub.
Well good to know it's not just me!
I rolled back to 18.104.22.168 (along with a database backup from that release) and the issue appears to be gone. Seems like something with the latest update caused it.
I'm now seeing the normal requests from the hub and just a handful of PTR queries to local IPs over the course of an hour.
To summarize my posts:
- Significant uptick in PTR queries from the hub to my DNS server that occurred every other hour for around an hour at a time.
- DNS queries were to local IP devices attached to the hub
- Rebooting didn't work
- Rolling back to a prior release (22.214.171.124) and restoring a database backup solved the issue.
- No other changes were made to existing apps on the hub shortly prior to or at all after the update
- Appears to be similar to an older issue: Many DNS Queries from new Hub - #12 by storageanarchy
Hopefully this is something that can be looked into as I'd like to keep my hub up to date!
@gopher.ny might be able to look into it.
i believe the issue is it should not be trying to lookup private ip addresses only public ips.. not sure how that would be implemented.. i imagine the continual lookups are because it fails as normally you dont lookup ptr records for private ips.
i am thinking of switching back my kasa to the public drivers as then i have control of limiting the lookups myself.
i have verified on my other hub still using the public not built in kasa drivers it is NOT doing this so assume that is the issue.
more info.. definately in my case was the built in kasa driver trying and failing to do ptr lookups of local ips.. switched back to the user version of the driver after much reconfiguring and the requests all dissappeared.
I have Kasa, but never moved to the built in integration, still on the community version. Also the way my network is setup a reverse lookup internally should get a result for all my local IPs anyway. Are the repeated requests failing or returning results back to the hub? Looks like for @kahn-hubitat the DNS server is not responding at all which might be why it keeps hammering the queries.
Some LAN integrations trigger arp calls to look up MAC address of a local device.
Before 2.3.4, hub would run arp -a 126.96.36.199 for each IP address and cache the result for an hour. Starting with 2.3.4, hub runs arp -a without IP address and effectively caches its entire output for an hour. We found that each arp call would cause a small memory leak, and bundling them minimizes the leaks. These extra DNS calls a are a side effect of doing arp. And yeah, there may be a small flurry of them right after a reboot while arp's own cache doesn't have all local devices.
If these extra DNS lookups are an issue (and it needs to be more than "but it didn't used to work like this"), I could make MAC cache TTL tunable so it can be increased at will. And I'm open to other suggestions, too
If someone's DNS server was to not respond to the ARP query would that then possibly result in the OPs observation, where it keeps hammering for an hour straight?
I think mine should respond properly to the requests and I am not seeing excessive numbers queries.
All of the Hubitat requests on my system for local IPs received an NXDOMAIN response. I wouldn't mind an initial increase in calls upon reboot but this was happening every other hour resulting in 6k to 7k requests every two hours.
So I'm assuming what @jtp10181 stated might be what's happening - since it's getting NXDOMAIN back, it keeps retrying.
I wouldn't say its improper .. it.shouldnt be doing requests for private ips ie 192.168 or 10.0 etc
Especially when going to a public non.local server.
If the only DNS server you give the hub is outside of your local network where else is it supposed to send the requests? My setup I have a local caching DNS server, my router hands out the DHCP info with only that DNS. I have that DNS server configured to forward any local requests to the router, all other requests go upstream to some DOT servers I have selected.
Without the extra DNS server, most configurations would point DNS to the router (which usually runs its own caching server). The router would respond to local requests and forward everything else to the configured upstream servers. If you point the clients directly to an upstream DNS there is no chance to do any local resolution.
If names are not needed for this, perhaps -n could be added to the arp command to avoid the DNS lookups?
Thats not the point . The point is it shouldnt be trying to lookup.local names..why is.it?