Hub process slowdown after several days

I don't have any idea what you think might be an "underlying issue". It's called bad code. It comes in all stripes and flavors, but mostly it has to do with how apps connect to cloud or LAN resources. For example, using synchronous HTTP instead of asynch. @csteele's post discusses this in some detail.

Just as was and still is the case with PCs, there is all sorts of bad code out there. How do you as a consumer of a PC deal with this? You learn to only install code from trusted sources, not just anyone you can find with Google. Even then, sometimes bad code still causes horrible problems with PCs, allowing hacks and viruses to infect the PC. Are these problems something that Microsoft can deal with? On one level, yes, they constantly strive to improve the platform, but on the larger scale, no, they can't police what people put on their PCs. They can throw up a warning, are you sure you want to install this? But then, when you do, it's on you, not them. How is that any different that what is happening with Hubitat?

We encourage you and all users to critique the code you are using, to report publicly that some app is causing problems. Usually, the app author will address these problems, but not always.

To selectively disable an app, there is a grayed out X in the upper right hand corner of the Apps page. Click on it and a list of check boxes will appear just left of the apps. A similar tool exists on the Devices page.

Also, apply basic common sense: If your hub was running fine, and suddenly starts slowing down, what did you most recently install? Perhaps start with that app or driver. Check things out after you install them.

And it turned out that this app had serious problems, and brought hubs to their knees. We looked at the code, and it was a hot mess.

This isn't a wise thing to do. What can I say? If you install 5 bad apps in quick succession, and things break, what should you do?

At this time, yes it is. We don't have a magic wand here to wave over an app and know that it's good or bad.

The vast majority of our customers have not installed many custom apps. It's not hubris, it's our attempt to provide a complete set of apps to cover most automation scenarios. The very nature of a customizable hub means that there are infinite number of things that one might dream up to do with it, and obviously we can't implement an infinite number of apps. But, the 90/10 rule applies, and the vast majority of our customers don't need exotic apps.

This is the risk of providing a tool with power. We can't teach everyone how to write good rules. The community is constantly coming to help people figure things out. We actually don't expect the "average" consumer to use Rule Machine --> it is intended to provide power for more advanced users. Most basic automations can be accomplished with other built-in apps: Simple Lighting, Motion Lighting, Notifier, etc. Certainly, we can and will do a better job providing guidance and help for beginners.

But, when you are past being a beginner, and you take off the training wheels, is it realistic to expect that someone can't write a bad rule or write bad Groovy code? You can write an infinite loop in any programming tool, and in every programming environment that is a bad thing. I could bring SmartThings to its knees, at least my connection to it, with an infinite loop. Sure, in a multi-user cloud system an individual user process can be isolated and the rest protected from it. But Hubitat is a single user system and its single user can make mistakes that bring the hub to its knees. How can you protect against that?

OK, we've supported hundreds of devices and brought forward dozens of apps in the 20 months since Hubitat launched. We don't plan to stop. There will always be more devices, more integrations, etc. that can be supported. We've been running hard at it, and arguably have been more responsive and more productive than any of our competitors. That's the best we can do, and we aren't ashamed of the state of affairs.

4 Likes

Yup. I agree - I've been given suggestions by Hubitat staff both in the community forum and by email from support@hubitat.com

I think the slowdown in my hub can be attributed, in part, to badly written rules. I turned on logging for each rule, ran it, and changed it until there were no errors while it ran. I also removed a bunch of rules and virtual devices that I wasn't using.

At ~48 hours now, my hub has not had any slow downs.

But there is one rule that is still throwing errors - I don't know why that is, and I am hoping you or someone else can tell me what I've done wrong so I can fix it.

Thanks!

1 Like

I am fully aware that my rules might be the issue, but I have a difficult time finding examples to match what I would like to accomplish. It would be a really nice option to have a wiki or list or resource of example rules and how to structure them, that are "good" rules, for users to create rules. I think RM4 is great, but it is easy to create a problem rule, and not know it. Maybe even a best practices for RM. Here is an example of a motion lighting rule I have made (about 9 similar running on the hub) -

1 Like

That looks like a good rule. Does it give you any problems?

Correct - it definitely wasn't a wise thing to do in retrospect. I actually installed two of the worst offenders - Echo Speaks AND the influxDB integration at once. Things went from "oh yeah this is cool" to "what the hell" real quick and I wrote an irate E-mail to Bobby. But my point was.. it does happen. If it happened to me, it could happen to someone else with similar needs. I have limited time, so I like to get things done all at once. It's poor practice ofcourse, but it is what it is. People like me are then more likely to raise a stink about it on the forums or Amazon reviews and blame the hub, which sways the opinion of others into thinking Hubitat sucks.

I completely understand your outlook on this whole issue - you guys built a solid platform and are trying your best to achieve a delicate balance between satisfying the enthusiasts (usually the loud ones) to the average Joe (the guys that bring in the money).

My only real point is that if you guys intend on keeping the moderately savvy/enthusiasts on your platform in the long run - there needs to be a fundamental shift in your thinking on how to cater to us.

Might I suggest reconsidering a "Pro" hub with a beefier CPU/Memory etc? I did a disassembly of the C5 hub, and it looks like the SoC is an Amlogic chipset meant for use on Smart devices. Not really sure where the performance stands vs the S905 that was used on the C4, but I'm sure it's not all that better than a Raspberry Pi 3.

Personally, I would LOVE to see Hubitat available as a software package like homeseer where the Pro crowd could just install on their own hardware with near unlimited resources. Now whether that's a sound business decision or something that aligns with your future plans for the platform.. I don't know.

edit: I suppose this hinges on the assumption that beefier hardware would solve slowdown issues by brute forcing against poorly designed code, which I'm not entirely sure of.

FWIW, the CPU in the C-5 is the same CPU as the C-4, clocked a bit faster (different SOC).

We did look at a faster CPU option. Here's the problem: It didn't make very much difference, and I for one would be p.o'd to pay more for something that didn't perform a lot faster. There's a reason for that: the hub is not, in general, CPU bound. It is I/O bound. You can throw a much faster CPU at it but that does not speed up the mesh networks, or the LAN. Given that most of the reported hub slowdowns are from badly done LAN implementations, a faster CPU would not make any real difference.

We had a batch of bad C-4 hubs, actually a small percentage of one batch of hubs. These hubs were about 1/3 the speed of a normal C-4 hub. As an experiment I moved my home system to one of these to see what impact it had. Guess what, there was no discernible difference in how my Hubitat system functioned. All of my automations were still snappy fast, and I could still do development on it at the same time.

It's just as easy to bring a super computer to its knees as it is a small microprocessor.

7 Likes

Thank you for taking the time to respond. This really doesn't seem like it's a simple issue to solve - if anyone else runs into this thread in the future with slowdown issues I hope they see your response to this.

I will be shifting all the heavy lifting to an external box, while keeping Hubitat online as a conduit to deal with anything Zigbee. From what I can gather, your priority seems to be getting solid device support first, which I wholeheartedly agree with.

I will be keeping a close eye on your developments as I hope to bring it all back under one umbrella again soon.

Technically, that isn't true. But I see your point.

No, that isn't the only option. I think there are fundamental architecture changes that could be made - even staying within java/js - to provide more segregation between subsystems and base OS behavior that would help on many of these issues.

I also think that there is much more that can be done to help users see issues, such as meaningful performance statistics. We can (and do) roll our own with things like using Node-Red for device response measurements. Extending those types of performance measurements on a more granular level (app/driver) would give data for users to target drivers and apps more effectively.

But, anyway. Those are my opinions - I am OK if they differ from others'.

I like Hubitat a lot for what it is, I just think it could be much more, thus the passion around my argument. Keep up the good work.

4 Likes

I'm sorry but how else do you think they can do this? You disable the app it fixes the problem to them great it's not a issue that will effect the rest of us it's your issue that you need to then contact the developer to get sorted. They are a small but extremely efficient team and I don't want them to waist their time on constantly fixing other people's issue that they can't fix because they know nothing about.

Follow there advice is best let them fix and improve add to the things that are connected to them!

Most hubs that are coming out now don't allow any other manufacturers kit on let alone random bits of doggy code! I have seen and heard of loads of times that the team has helped the developers out with there issues even though they don't have to. But you asking them to go above the developer and fix there issues is not fair or realistic.

2 Likes

Do what I do uninstall it and find a better one you don't leave it on and say I'll just reboot my phone every day to get over it :joy::man_facepalming:t2:

My point was that it’s not always as easy as disabling an app to figure out if it was troublesome. Sometimes issues don’t show up for days. It looks like the general consensus is network/IO issues that may or may not pop up depending on how the device you’re taking to decides to respond. It’s time consuming and not a very effective troubleshooting method.

I’m probably not explaining this right, but try installing a few cloud based and LAN based smart apps and you’ll see what I’m taking about. It’s hard to pin down which one is problematic without spending a lot of time troubleshooting. And even then, it’s very easy to come to the wrong conclusion, which has happened to me multiple times.

Really the only solution I can think of is @JasonJoel suggestion - more granular metrics into what each sub-process is doing on your system. I don't need to know that the hub is taxed - I need to know what specifically is causing this. But I'm sure this has its own set of challenges - I'm not a programmer.

Also Uhhhh... I’m not asking hubitat to step in and fix third party code, my only suggestion was better tools to diagnose issues. Third party developers on Hubitat work for free, I can’t expect them to fully vet their code works fine in all use cases. But I’d like to be able to confidently say that their smart app is the one causing issues and help in anyway I can.

1 Like

Exactly @bravenel I admit my comment may of seemed like I was suggesting the Hubitat team was hiding something. Of course that is not how I meant it to come off. You brought up Microsoft and I could not agree more. I'm sure they had the same challenges in the beginning when this was a new product. Now if you need to know how to accomplish something in Windows it is just a search away. Microsoft is probably one of the most documented things on the internet these days. Do you think they did not have the same challenges in the beginning?

I'm sure when this was all new it was the same case and maybe all we need is just more time for Hubitat to get to that point. Anyway my point is I'm sure Microsoft had to do a lot of damage control in the beginning as well as a lot of end user education to get to the point they are now. I'm aware of the tools to disable apps etc. Why do we not have better tools to look at the performance of the hub? Windows has task manager it is real easy for me to see what processes are are using the most resources. I was surprised to find out the hub does not have a gigabit nic so from my perspective there is certainly room for improvement and performance upgrades.

I'm all about performance and would be happy to pay more money for more features to protect me from failures. I recently purchased a Unifi Gen 2 controller that has a battery back-up and will shut itself down in the event of a power failure to keep from corrupting the controller.

The odd thing is my performance issues did not happen until we lost power several times in one day. I had not implemented any new drivers or apps prior.

Maybe this is just a coincidence but I don't think we will ever know because the only solution for my issue was to disable custom apps and drivers and now my automated reboots as a fix.

I get the fact that is easy to lock yourself into a one solutions fits all fix but I'm afraid Hubitat will never develop and realize it's full potential with that type of insight.

I just think it is bad business to come off as close minded and so defensive to anyone who questions the performance no matter the root cause. I'm no programmer so for all I know the Hub itself may be full of faulty code causing the performance issues.

Because there are not easy ways to do this. We are looking at some basic things, but this is not our highest priority.

Anyone who's been around here for some time knows that I'm not close minded. As to being defensive, I'm just attempting to clear up a lot of misconceptions that have been trotted out as fact. I suppose I could go back to other things and leave this topic to spread false information, but once in a while it seems appropriate to just explain what is actually going on for those who are interested.

12 Likes

@bravenel

So what are your thoughts on the DB getting corrupted or something similar going on in my case? I just find it odd my hub was performing great until we had several power outages in a day.

I know I need to invest in a UPS have for sometime to not only protect my hub but the rest of my equipment.

Also I'm sure some of my comments come off as attacking Hubitat just as yours comes off as being defensive. I'm in no way trying to put down the team or the community as I'm sure your not trying to come off as defensive.

We all know peoples perception is there reality!

I certainly don't have any idea about your case! It is possible for DB corruption to happen when power is abruptly pulled from the hub. Our reboot process usually discovers this, and reverts to a DB backup if needed. It's a good idea to download a backup of your DB periodically. As far as I know, restoring a backup of a good DB restores the hub to the way it was functioning when that backup was made.

Did you contact support?

I have never done this myself but have seen posts from many that say a soft reset fixes database corruption issues. The community can chime in on this:

https://docs.hubitat.com/index.php?title=Soft_Reset

1 Like

I did and they made changes to my nic which required a reboot and of course things ran better for a few days. I then came across the 1% who setup a reboot rule and have been using that for a few weeks now. I still notice some slow processing of my automation's on occasion but not really had time to dig into it when I see it. Being Friday and now that I'm taking Monday off I hope to dig into it a little more this weekend.

@stephack Thank you I will look into this in more detail.

Yes, I have had issues including slowness where a soft reset has resolve them in the past. In fact I measure the response time of the hub with NodeRed and noticed the time go from around 2 seconds down to 500ms after the reset.

My response time seems to sit around 400-500ms, which I consider good.

I have seen the exact same thing, using the exact same tools. I have three hubs, and twice now my development hub has experienced ~2sec response times as measured by the same Node-Red flow. Backing up the database, and then performing a soft-reset, and then immediately restoring the backup has resulted in performance being restored to the ~0.5sec 'normal range'.