Who’s not being honest?
Companies like apple and Hubitat offer public beta versions of their software, and people who love being on the bleeding edge happily agree to use it and report on their experiences (as you yourself pointed out).
3 Likes
I manage all the betas for my software company that a couple hundred thousand clinicians use every day. As you can imagine, pushing out any functionality to non technical users can be a challenge and we definitely wouldn't have made it this far if we were relying on end users to be guinea pigs.
I don't see anything different with Hubitat's beta than what we do except perception.
Clarity from the organization can reaffirm confidence in betas and the product overall by dispelling myths/perceptions about what is/isn't tested and how.
We have a dedicated QA team that does manual and automated smoke testing.
Solutions are Alpha tested in house by our own BAs and QAs. Unfortunately, depending on the solution, Alpha testing will only get you so far in fake environments with fake data and sometimes the only way to TRULY test it is on a live connection.
Solutions are presented for beta and we ask for volunteers (if we don't already have a list of volunteers clamoring for it for the last year)
We enable features in their environments and let them beat on it for up to 90 days.
If no showstopping issues, proceed to general availability as scheduled. Report bugs/fixes as needed. Not all bugs will be resolved at go live. Some will still be on the backlog or will be in the process of being worked or deemed too unimportant.
When we go GA there will ALWAYS be bugs that we didn't find in beta.
This is where end-users can feel like THEY are the guinea pigs. If you did all this testing, why are there bugs? Did you not test this in real world scenarios? Do you not have quality people?
The reality is every customer's environment is configured differently and has different workflows as a result. Because of these variances, it can be difficult to pick up on all the bugs in beta.
Who helps the most:
Our bread and butter beta partners that raise their hand on everything, give great feedback, and identify bugs before we do. Sometimes they are large enough to have their own IT resources. Without these people, our product would have WAY more bugs at go-live. To me, that sounds like the people in this thread who are the early adopters, gluttons for punishment, and want the new new all the time.
Larger volume customers: because of size, they can often detect problems before smaller customers because it impacts them faster. They also have organizational workflows/challenges that we simply can't account for in every testing scenario.
So perceptually, yes I can see how it looks from the casual end user, but I'm sure there's a lot more going on behind the scenes.
If hubitat could post even just a PPT slide with a basic flowchart (without getting into the weeds) of basic steps functionality and releases go through from a process perspective, it can help provide clarity and give confidence to users who see multiple updates or have apprehensions about a bad release (bad releases happen even for my company).
Once customers became actively engaged with our beta process and knew what to expect we suddenly had a greater volume of quality volunteers. We have to turn folks away on every beta because there's just so many volunteers it becomes unmanageable and unscalable for our lean and mean company. This visibility can help with customer sentiment, not just for beta partners but for those end users who find a bug 6 months from now and say "didn't you test this?"
Edit:
One thing I forgot to mention. We release every single month. THAT release is not beta tested. It is QA tested and certain features or functionality in the release may have been beta tested but we simply don't have the time to beta test every release with 4-5 week dev cycles.
The exception is to our mobile app. We DO grant a test flight version of our mobile application to users who want it up to 6 days prior to release. The reality is, most end users don't want it because they are afraid of new bugs. We force customers off anything older than 2 versions behind to keep QA overhead lower.
3 Likes
Sounds similar to what the hubitat team does with every platform update (from what they’ve mentioned here in the forum, that is).
They could always choose to share more information about internal processes, or not. There will also always be a vocal group of power users in the online forum with narrow interests that may never be happy with what comes next.
I think the idea that users are guinea pigs, particularly the people that volunteer to beta test, is an overstatement. Guinea pigs never sign up for anything, they are experimented upon.
5 Likes
So I ran into the problem with another device (zwave August smart lock Pro) regarding s2 security and wanted to roll back to a previous version of Hubitat which was the only way I was able to add the device. However, in doing my research I need to roll back to a version before 2.2.8.x which isn't available on the 8081 diagnostic tool. Is there any way to revert back to an earlier version?
I posted back on the August Smart Lock issue back in May:
Edit: After a lot of fiddling I was able to get it to work. Not sure if it was something I did with the zstick or just 2 hours of repeated inclusion/exclusion/hub shutdowns that finally made it magically work.
1 Like
