What was/is wrong with MQTT?

Get Help Integrations
1

I am experienced EE so I never digg in deeply into gassilions sw protocols. Recently I design a very complex IO Device based on ESP32 hw but for communication with HE I choose a MQTT Communication Protocol. The reason was - recently added native MQTT support. This unfortunately did not work. HE did picked up all the devices but I couldn't map any. The reason is - current HE MQTT implementation doesn't support multiple attributes from a single Device. So a custom driver needed to be created. But thanks to the AI (MS Copilot) that was not a big deal.

The result is far beyond my expectations. My DIY IO Controller has built-in Web UI, instantly integrated with HA and thanks to the relatively simple custom driver nicely integrated with HE.

Now my question is - Why MQTT is not widely adopted?

Zigbee, Zwave, BT and WiFi should be used as a transport layer but MQTT is a Communication Protocol. Instead a huge mess of incompatible protocols was created (specifically this latest Matter/Thread addition). I am really surprised. What I am missing?

2

It is in the industrial world. It's very common with "IOT" and "Smart Sensors" - The trick is the output format - Typically a JSON blob with multiple attributes/values in a single update or the more standardized way is to just use a payload format like "Sparkplug". - Industrial sensors and RTU's is what MQTT was developed for (to enable loose coupling between ends of the connection, and to be lower overhead on low end field devices).

Home automation isn't that. - So MQTT has be slowing moving into the Home Automation world, and most things follow the HA (Home Assistant) format for value mappings. - The reason that it's not widely used, is the obvious requirement of a broker - That has to run somewhere. - Most casual users aren't running a RPI to run a broker - and that's entirely another level of complexity, versus just pairing or SmartStart (which is where most new things are going). Until HE recently added a internal broker, this was a problem in the HE space as well, and MQTT wasn't heavily used.

The problem with MQTT is the payload isn't well defined nor formally standadized, hence it's hard to get two ends written by two different companies to agree on the payload format.

Don't get me wrong, MQTT is great thing, and it's used heavily in industrial automation and data collection from "smart sensorts" (as is OPC-UA). - But it's a bit more complexity for the average Smart Home users.

My $0.02

3

:backhand_index_pointing_up:

4

Are you saying Zigbee and Zwave (not sure about this Matter/Thread) well defined and standardized? Than why do we need a specific drivers for near every Device? And Devices from Tuya and Aqara are very special case. These companies are not following any standards at all and trying hard to make everything proprietary. My good guess, it will be much better to standardized MQTT instead of creating huge mess with gazillions of whatever should be standard but it is not in reality.

Yes, MQTT Broker is required but I don’t think it is a showstopper. HE just added one built-in. But if MQTT will be primary protocol it will exist from day #1.

Just my thoughts from the EE point of view.

PS.

I already built a sizable number of DIY controllers. All they are based on Hubduino sw project. But now all new ones will be MQTT.

5

I would say that, yes. Especially for ZWave.

Zigbee’s manufacturer specific clusters, and lack of a device certification process, means it can be gamed and made proprietary by manufacturers. But the standard clusters are indeed well defined.

For devices that use standard clusters (ZigBee) and command classes (Zwave), you really don’t have to have a device specific driver in most cases. Drivers are usually done individually though as people don’t like extra code and crap on screens/loaded in the background.

Supporting every function in a generic driver makes the drivers very large when they don’t need to be. For instance in both ZigBee and Zwave the code to support all of a LOCK’s functions is huge - why have that loaded sitting around in memory for a device that isn’t a lock?

6

How do you propose to go about achieving this goal?

This was a design goal for Zigbee and Z-Wave. It’s a design goal for Matter. None of them achieve the goal to perfection.

It doesn’t change the fact that MQTT is perfect for your use-case: after all, what are the chances that Vitaliy(the electrical engineer) won’t be able to agree with Vitaliy (the software engineer) on a payload format? :wink:

7

Really someone would need to do like they did with LORA and make an equivalent of a Cayenne LPP payload definition and get wide enough adoption for it to be a defacto standard.

Sparkplug was kind of an attempt at that I guess.

8

I have no idea, I am EE not a SW engineer.

The key point is - this was a design goal. However the result is a huge mess.

Again, I don’t see why MQTT payload cannot be somewhat standartisized. How - is a different story and I have no idea how this standardization works for the SW.

BTW, HA MQTT payload worked perfectly fine. Why it cannot become a standard for Home Automations? Now I really like it and all my next DIY Devices will be ESP32/MQTT based.

9

Then Drivers should/must be be user configurable. Just my opinion.

10

statements like this is how we get new protocols :rofl:

11

Kinda curious about this statement

12

See the bits about the "Built-In MQTT Service" here - It was added as an internal option (Pro Only?) a few revs back.

13

This doesn’t seem like a software or hardware problem to me. It’s a people problem.

What’s the mechanism to enforce the rigid implementation of a technical standard on individual developers and/or smart home tech companies worldwide?

14

It could…. You would just have to figure out how to make a great format/definition and then get all people to voluntarily use it without adding a bunch of customizations/additions/extensions every time they feel like it. All without a paid certification process.

Oh wait, that is EXACTLY what ZigBee and LoRaWAN already is and tried to do, and you see how that turned out… No compliance/certification process = no obligation to follow the rules/format/documentation/schema.

I’m not trying to be a huge smart arse… It could definitely be done if enough people put in the effort, and saw enough value in standardization. There isn’t anything inherently deficient in MQTT to prevent it from working. So far no one has done that though.

I use MQTT - a lot - both privately and commercially. I like it. I’m just not convinced that anyone cares enough to make a customized payload format. Hell, in the industrial space you can barely even get any manufacturer to use sparkplug, which was made for similar reasons - and they actually have financial incentive to do so.

15

There IS a MQTT Payload format - It's SparkPlug - It's just few vendors use it. And HA built their own approach, along with a Discovery Service, focused on buttons, blubs, switches, contacts, etc. - Basic Home Automation devices.

So the HA approach is sort of a "de-facto" standard, but again, most vendors just want plug and play, and Matter DOES define things in a much more perscriptive matter versus Zigbee and Zwave, and there ARE alot of "generic" Matter drivers now in HE. (Locks are getting there, but there is alot of the V1.4 Matter standard that HE doesn't address in terms of attributes for different devices (they are adding more support for Solar, EV's, Power Management, etc. into the latter Matter standards).

So MQTT is more like a "reliable transport", that folks build what they want on top of - And you have effectively done this yourself - There is not "compliance or testing service" that validates compliance (Zigbee is the wild west, but had no fee's so it's cheaper, and folks built what they wanted, which is why it has such diversity of devices - It does have a standard from CSA, but lots of folks ignore that - Zwave at least had a single vendor that enforced a standard, SiLabs, until it was more recently opened, and Matter has the most complex compliance testing, with DCL's and PKI to control what's "allowed" and things can be revoked.

MQTT is just a loose coupling and transport - There isn't even much agreement on how to properly build a topic namespace. -

Again, I'm all for MQTT, and routinely used it at work, but given its design goals (for IOT and low resource endpoints to not have to maintain connections, subscriptions, etc.), and given Matter's release and design goals, I don't see MQTT taking over the Smart Home space, besides interested DIY folks (like yourself), and others on this forum, which is a SMALL, SMALL percentage of the HA market.

16

No. MQTT is not a new protocol. As I mentioned, I am not a sw developer and as a result don’t know how all that standardization works for sw/protocols. but to my eyes MQTT does everything including easy integrations between multiple platforms. This is what Matter/Thread is invented for. But why not to reuse already very well working MQTT (with whatever standardization is required)? To address your statement - why this heavy wait Matter/Thread is popped up? And instead of solving a problems so far it created many new.

17

Joke Whoosh GIFs | Tenor

18

Thank you for all details you provided. I do appreciate all your explanations. However as of today this Matter/Thread is not a solid solution. So far I am trying to avoid using Matter and specifically Thread devices.

19

Matter DOES reuse a bunch of stuff - Specifically, local IPV6, mDNS for discovery, cert based PKI. revocation lists. BLE for provisioning, and they tried to make most of it IP based - Then they had to deal with low power mesh sorts of devices, and they basically reused ZB approaches for Thread (and the CSA control both the ZB and Matter specs). And most importantly, the "big boys" (Apple, Google, Amazon) all agreed to build around a single standard (Not that they did all that great of a job on the initial versions, for companies with more $$ than God).

I don't think they chose MQTT due to the work Apple had done on CHIP (Connected Home over IP - Which is the early versions of Matter). - I don't think MQTT solved any particular problem for them (loose coupling connectivity), as the Matter spec covers ALOT more ground than the MQTT protocol does. Matter was design to be secure, support Multi-Admin, simpler provisioning, and more standardization for the given clusters for various device types. Go back and read some of the early Project CHIP stuff from 2019 - The problems they were trying to solve, aren't really addressed by MQTT

20

I pretty much strongly disagree with that statement (though I do agree Thread is still a bit too much of a black box), but Matter is infinitely more transparent and easier to troubleshoot than ZB or ZW - it's a bit more complex, but being IP based makes it a long term and obvious choice (at least for me)

YMMV, of course, but given the market power and control of Google, Apple, Amazon (and to loosely borrow a phrase) "Matter Resistance is futile"