Hub process slowdown after several days

I don't think you guys should have to support user code. I agree that isn't reasonable.

What I do think:

  1. I wish the user code and system code had some sandboxing/segregation and one couldn't kill the other as easily as it does.
  2. If #1 can't be done from a technical standpoint, I wish Hubitat would make a very seamless, integrated way to use multiple hubs and share device data and control between the hubs. Something with auto device discovery/sharing, and all commands and properties of a device being on all hubs. HubConnect is nice, very nice, but has distinct downsides as well. Easy/native/robust hub integration would make it easier to segregate user code, hopefully without some of the device limitations we have with HubConnect.
  3. I wish that there was a better way to clear up database issues other than a soft reset or restoring backups. The database can be goofed up a number of ways, and has been shown to cause slowdowns when it grows/gets corrupt data in it/bloats, as such it would good to see improvements in this area to remove the need for user cleanup to fix them.
  4. I wish there were a way to see what app, driver, or RM rule was bogging down the system. Maybe through some metrics such as # of instances, execution time, CPU time, Memory Size, and IOPS. The fact that a regular reboot helps the situation for most shows that there should be SOME metric that could be watched.... SOMETHING is getting cleared out/reset/freed up when we reboot our hubs. Why can't we see what it is?
  5. I wish the system was easier to integrate to/from other automation systems. Native MQTT client support, more data in eventlog websocket (like event type - physical/digital), improvements to MakerAPI in terms of the URL syntax and authentication.
  6. I wish hub backups would prefix or suffix the file name with something unique to the hub. It is a pain to make backups on multiple hubs, and have to rename each one before backing up the next.
  7. I wish you could do code library imports on drivers/apps to make it easier to re-use code without copy/paste. It would also make updates much easier for developers.
  8. I wish there were a text based / programmatic way to make rules/scripts for automation. Although RM is excellent in MANY ways (kudos!), there are many of us that would love something that we could copy/paste/type in instead of many, many clicks to make a rule.

Anyway. Back to Friday night fun stuff. Keep up the great work.

10 Likes

That's a bad idea... even though its a small packet running 24/7 doesn't help the bottleneck.

I have been thinking about this idea for a few months now. I think it's the best approach to get certified drivers/apps to the end users.

+1 for this.

Honestly, I hate to see the ability to use custom apps curtailed. I use some. I like them. I understand the risk. I don't think that asking Hubitat to become a walled garden or somehow support custom apps is either practical or desirable. That's just my humble opinion.

Heck, I can apparently cause enough problems with RM4, and that is a support app.

I've been debugging code of various kind for, oh, 40 years or so, and I don't mind doing it. But it would be really, really nice to have some tools to point in the right direction... Is it really a custom app? Or is it an RM4 rule that I screwed up? Or is it a rogue driver? Yes, I can disable all of my apps and rules and reboot. But sometimes it take days for a slow down to start happening again. Am I supposed to live without half of my automations for days in the interest of debugging?

5 Likes

I believe having a complete hands off approach cannot be good in the long run, I understand there's a cost/benefit analysis in this decision. If there are apps/drivers out there in this forum/github/etc. that are causing issues with your companies home automation platform, and nothing is being done to even identify (from a company approach) which ones they are, let alone finding an end solution caused by them, even if it means deleting the posts linking to problem apps/drivers. This issue will only expand and not ever get better if it is not even known which are/are not causing problems.

4 Likes

Why not just save the backup to a different folder for each hub? That's what I do. My C3 backups go to a "main" folder and my C4 backups go to a "dev" folder.

4 Likes

I hope you realize how utterly impossible what you are suggesting is. We have many thousands of users. We aren't privy to what they install, or how their hubs perform. The suggestion that we become sensors of what people post, and what they choose to use of their own free will is stunning in its absurdity.

I guess what we should do, is make it very clear that when you load any third party code on your hub, that you are taking a big risk. The risk is yours to take if you want. But for goodness sake, take some responsibility for what you do. Show me any company that offers a programmable environment that doesn't take this approach.

5 Likes

You are when issues are reported. Instead of saying "it's a custom app you're on your own" A different approach would be what apps do you have? A mere simple database of coincidence can be established, therefore process of elimination of "which" is the likely culprit. And when this is determined then if the code is publicly posted here pull it down or publicly identify it someway, with a post stating you suspect the code is causing hub problems.

But again NONE of that is possible it you say, well you are on your own we don't even know what custom code you have, just disable it and see if your problems go away. Again short term this may say wise, long term it establishes a growing trend of bad code never identified expanding to more hubs causing more problems and not all will even bother trying support.

I would think that some sort of resource usage indicator would make troubleshooting much easier. If there was a list, graph, or numerical value that correlated to each app, rule, driver, etc we could tell where the problem was and have a baseline to compare to. I don’t know if it’s possible, and I don’t need it overly detailed, just something that lists what is going on in the hub and how much of the resources a app, rule, or whatever is using. I will then narrow down on that troublemaker and fix the problem.

I also think a Auto database cleanup of some sort would be good, to flush out issues and keep it running smoothly.

I am saying all of this, yet I don’t know enough about the internal workings to say what is possible. I do love my HE’s, wouldn’t trade them as I’ve seen the alternatives.

3 Likes

We don't have the resources to explore the custom code people install. People make modifications to the code they get from third parties. They run complex mixes of apps and drivers. We aren't able to watch each hub in operation for the amount of time it takes to diagnose these problem. We see at best a momentary snapshot of the hub's condition. You make it sound as if this would be so easy to solve, but it's not easy. Imagine Microsoft undertaking to understand and evaluate every app that runs on each Windows machine that misbehaves. It's a non-starter.

This boils down to individual responsibility. If you install third party code, that's your responsibility. That is the only reasonable policy we can take. No one, or at least most, would want to give up the ability to use shared code. But, a few outspoken users want us to take responsibility for those choices, to undertake an unreasonable effort in lieu of them taking responsibility for their own choices. It would be great if there were easy solutions for this, but there are not.

6 Likes

I didn't infer anything about "solving" I plainly stated simple identification, At which can be as simple as I stated before by coincidence. That doesn't require "much" resources it would some to simply log (could even be done with rule machine and virtual devices to click each app/driver reported), a simple excel spreadsheet when someone has issues of what apps/drivers they are using. Again I'm not stating you should "solve" the problems with the code, I'm stating that the problem code NEEDS to be identified by your staff in some fashion. So that a logical response AFTER you've asked what apps/drivers they have would be.

We've seen a lot of problems with those running "X" app like you are, we don't mess with custom code but you should start there and troubleshooting the problem contact the app developer.

A problem that never gets fixed it only spreads and expands if it's never identified as a problem, and leaving the troubleshooting to the users without any interface for them to see if their troubleshooting of "disable and see" is actually accomplishing anything is shooting in the dark.

2 Likes

Nope, it needs to be identified by the person who installs it. You have this great forum to broadcast to all what you find. For example. several people discovered that Echo Speaks was causing problems, and once they disabled or removed it their problem went away. I learned of that issue here in the community from users, not from support.

When we do find a problematic app specifically, we do contact its author or maintainer about the problems we see. At one point quite some time ago we saw a huge flood of issues associated with webCoRE, and we published our concerns and issued a warning about it (which may now be outdated, but we don't really know). We aren't sitting on known problems keeping them secret. If users are not willing to help out by following the diagnostic steps we suggest, and then telling people what they find, what's to be done? We can't perform those diagnostic steps ourselves on their hubs. As I said, only the person running the hub is really in a position to figure out what is broken. We're glad to help, and Support works with people all the time on issues such as this.

So, do the diagnostics we suggest, and then shout to the rooftops when you find the culprit on your hub.

Since we can't run their hub, it means we are even more in the dark. If you can't diagnose your own hub, don't install third party code in the first place. On the day you first notice unexpected behavior on the hub, what is the most recent thing you installed? Remove it, see if the problem goes away. Or better yet, restore yesterday's db backup. Does the problem go away? Bingo. How hard is that?

4 Likes

Once upon a time there was an unofficial list of bad apps. WebCoRE, and APIXU were on the list to name but two.

I was using APIXU at the time and deduced some issues and worked with Bangali to start the process before work called him away, I presume. I then published my efforts.

@nh.schottfam tackled WebCoRE (which I'm going to imagine was 100x harder than APIXU) and he's saying it's better now. Certainly the level of noise is massively reduced.

There's a pattern to be explored. The Community needs to glean the bad apps (and drivers) and see if anyone will pursue fixing them... certainly the Email from APIXU a couple of weeks ago proves there is vibrant community of developers that care.

I've been following THIS thread in the hope of 1) gleaning the current list and 2) making sure HubConnect ain't one :smiley:

6 Likes

Because you're assuming most will just install one thing say every two weeks to see if it causes problems before you even contemplate installing something else.

Most of these "slow downs" are stating they take multiple days to appear.....yes that makes diagnosing a BREEZE!

Limitations apparently

I'm not a trusting person when it comes to software. I have exactly one custom app on my hub, and it comes from someone I really trust. From the moment I installed it I was prepared to remove it at the first sign of some problem -- still am. But, it seems reliable, now after over a year of using it.

If you want to rush into things, that's up to you. I'm much more cautious about it. I know from being a developer of apps that (a) it isn't easy to get right, and (b) there are always bugs and more bugs after you fix the first bugs.

Oh, and I really should point out, that when we do look at the source code of some misbehaving apps, we see real garbage. Be really wary...

3 Likes

Except you're not the "typical" user, and I would bet the reason you're not using others software is because you have the technical ability to write your own, which the majority using this platform do not.

I for one would rather see development into new features and device support than dedicating resources for helping bad coding or user error. All of the talk of adding tools for monitoring resources would be good I guess but how can you even see usage if your hub locks up anyways? Seems silly to have dedication of time for Hubitat to add CPU usage or other resources for a very low percentage of users and I can't see a benefit. No other smart home hub has tools for monitoring resources.

6 Likes

Maybe we should start a public list that everyone can contribute to of apps that are known to be solid. And another list of questionable apps. And a list of developers who should be trusted. I suspect that would be a pretty short list. Then, if you install code from someone not on the list....

And as @csteele would be the first to admit, even those who should be trusted (he's one of them) make mistakes. Heck, look at my own record for bugs in RM, sometimes pretty bad ones....

14 Likes

That would be a great start.

1 Like

Go for it.

2 Likes