[BETA] MQTT app Home Assistant Discovery and statestream support

[kevin]

This is the discussion topic for the MQTT app and it's use in connection with the Home Assistant Discovery Protocol and statestream. Both OpenHAB and Domoticz also support this discovery protocol. General discussion (not HA related) should instead be directed to the main topic here.

[BETA] MQTT app

It's been a long wait but I shall, at last, be releasing my mqtt app as an open beta 1 this weekend. It's still evolving and although it won't win any coding awards it is pretty feature complete and stable I hope. It's been through 5 major alpha releases and numerous smaller updates with over a hundred alpha testers although feedback has been very limited saddeningly. For those in the smaller alpha testers groups - thank you so much for your feedback, it's been invaluable ..There maybe a …

The ability to link HE with HA bi-directionally is a powerful and popular feature and provides auto discovery , status and control in both directions. It is however not the primary purpose of the MQTT application for HE - more a side benefit. I am not a seasoned user of HA so can provide little support for HA and possibly on HA issues the HA forum itself might prove more appropriate and helpful.

The MQTT app will discover devices advertised by HA using the statestream protocol. Statestream itself offers status reporting but does not offer the reverse control but some scripts are provided for placement in HA that facilitates this. For further device support beyond switch / light and sensor devices further scripts may be needed and I look forward to contributions here with those scripts so that I might include them with the app.

The MQTT app advertises HE inbuilt devices using the HA MQTT Discovery Protocol. This allows HA to discover these HE devices and add them into HA's Lovelace front end for bothrealtime status and control.

The range of device types supported between HA and HE includes switches, lights (white, colour and also colour temperature). many sensors and input booleans along with locks and keypads. More devices will be added over time but the majority are already covered. Climate devices like thermostats are not currently supported.

Additionally using the MQTT app any MQTT exposed device can be imported as an aribitrary 'manual' device into HE and associated with one of HE's 24 inbuilt virtual device drivers.

[kevin]

placeholder

[hhdsvfootball77]

MQTT support? Awesome! I can't wait to try this out.

[kevin]

Just a heads up that beta 1 doesn't handle duplicate device names in different device categories from HA very well currently. Please try and ensure your names are unique in HA (as reported in HA statestream). Fixed for the next beta 2

In order to fix this I am going to have to relearn HA Discovered devices adding the 'device category' to the DNI. This will mean that existing devices will get re-created but any RM rules and dashboard tiles will need redoing.

I don't know if HA statestream will protect against using duplicate device names in the same device category like 'switch' or 'light'. If it doesn't then topics within MQTT could get conflicting payloads from two or more devices which will be a real mess. I can't do anything about that, it would need HA to fix it.

[kevin]

I have just published beta1a which has a fix for HA discovery into HE using statestream . Devices were discovered but weren't updating. Essential update if you're using HA > HE discovery.

[bill]

Finally trying to move my setup over from the old MQTT bridge to this app, but no devices show up in home assistant. I'm following option "C" in the readme. I'm just looking for auto discovery of HE devices in Home Assistant at this point but no luck.

[app:1418](http://192.168.223.244/logs#app1418)2020-02-15 10:51:13.399 am [info](http://192.168.223.244/installedapp/configure/1418)MQTT: ================== Startup complete ==================

[app:1418](http://192.168.223.244/logs#app1418)2020-02-15 10:51:13.200 am [info](http://192.168.223.244/installedapp/configure/1418)MQTT: [168] Total Hubitat endpoints enabled on MQTT

[app:1418](http://192.168.223.244/logs#app1418)2020-02-15 10:51:13.199 am [info](http://192.168.223.244/installedapp/configure/1418)MQTT: [167] Total Hubitat devices are enabled on MQTT

[app:1418](http://192.168.223.244/logs#app1418)2020-02-15 10:51:13.133 am [info](http://192.168.223.244/installedapp/configure/1418)MQTT: [167] Hubitat 'everything' all capability devices enabled on MQTT

using MQTT explorer I see my devices... but I don't see any evidence at all of my home assistant discovery topic being published anywhere, and no devices in home assistant.

[bill]

Oh well... ok some progress. This confused me for a couple of hours... because its confusing.

I don't really understand the configuration page in the MQTT app. It appears that under "Publish these device capabilities to MQTT"

You have to de-select all under Everything.
Then select all devices you want under the individual type
Then de-select all for List devices Capabilities and Attributes to Hubitat Topic
Then under IMPORTANT: Enable all your devices for app access
select all again.

Why so many screens and what is the difference between them? Why does the MQTT app log think I have 323 devices when infact I have 168? What does "Enable aoo your devices for app access" even mean? What app?

I don't get any of this but now I see MQTT devices in Home Assistant. They don't work. They say "unavailable".

[bill]

OK well, I've got all of my devices discovered and working in Home Assistant, but I'm not quite sure what I did

[bill]

hmm... bummer. A couple of aeotec multisensors report a lot of things but no illuminance like they used to. Doesn't seem to be a category in the MQTT app.

I still don't understand the difference between "Everything" under "Publish these device capabilities to MQTT" and the individual categories below Everything, but selecting "Everything" doesn't get my multisensors to publish illuminance. I don't know what Everything does. Nothing it seems.

Also, experience so far is that this is slower than the mqtt-bridge solution it replaced. It's about 2-4 seconds between toggling a switch in Home Assistant, and the light coming on.

I was hoping this would be both faster and more reliable than the old mqtt-bridge app. Something in that setup seemed to degrade over time in terms of speed and reliability for me, but when working well it was much faster than this. I hope there's some tuning room here to make this faster as it gets closer to release.

[kevin]

Bill, if you read also the main thread (where there are errant posted HA issues) then nearly all your questions are answered ....

but to link from this topic to the main topic see this and some preceeding posts

[BETA] MQTT app

OK an update. Rather than enable devices individually under the capability drop downs e.g. dimmers, switches etc. Stephen was relying on using the 'Everything, all capabilities and attributes' drop down only , which should have worked but actually manifested two issues. homeassistant doesn't populate. The devices are incomplete within homie. This is a a bug on my behalf that I will now look to fix (grateful to everyone who finds these) So FTTB please enable devices under the individu…

You have to de-select all under Everything.
Then select all devices you want under the individual type
Then de-select all for List devices Capabilities and Attributes to Hubitat Topic
Then under IMPORTANT: Enable all your devices for app access
select all again.

You don't need to disable all devices under everything

You don't have to deselect all under everything but I recommend you do as it's partially broken (fixed beta2)

You do have to select capabilities under the individual capabilities (fixed but still recommended under beta2)

You don't need to select anything under 'List Device Capabilities and Attributes to Hubitat' this is unrelated

You should enable all devices under everything*

You should enable 'remember devices in HA if you want them to persist over HA restarts. and not show 'unavailable'

• *This is to satisfy HE security where the user has to manually enable devices granting access to other apps like MQTT.

Why so many screens and what is the difference between them?

I had hoped the read me addressed all these. In beta2 I'm hoping the 'select Everything' necessity has been removed.

Why does the MQTT app log think I have 323 devices when infact I have 168?

It may be counting them twice as its counting capabilities so if a device is enabled under say motion and temperature it's being counted twice.

What does "Enable aoo your devices for app access" even mean? What app?

It means enable all your devices so this app 'MQTT' has access to them

They don't work. They say "unavailable".

HA shows them as this if their status is not retained on MQTT.
Enable this

but no illuminance like they used to. Doesn't seem to be a category in the MQTT app.

[BETA] MQTT app

@PiXEL8 will be in beta2 [image] [image] Anyone need any other capabilities / attributes that are not implemented yet ?

but selecting "Everything" doesn't get my multisensors to publish illuminance. I don't know what Everything does. Nothing it seems.

as above

I was hoping this would be both faster and more reliable than the old mqtt-bridge app.

[BETA] MQTT app

Just as a teaser - beta 2 will be a lot faster in reacting to /set commands and instructing the device to update. Over twice as fast - thanks Andy for testing. The state update back to MQTT will be correspondingly faster too but still happens only when the device reports it has physically changed state so varies based on the device involved.

[kevin]

It's about 2-4 seconds between toggling a switch in Home Assistant, and the light coming on

This is switch on in HA to bulb lighting in HE ? Try it with a virtual device - I am seeing about 600mS , sometimes much less (300ms) . A virtual device has no Z network to traverse. The update of the state back onto MQTT happens when the device confirms it's new state - that's typically just under a second.

Is the Z- Network adding 3.5 secs ? - if so I can't do anything about that - but nor could the other MQTT solution.

BTW HA implements 'predictive state' - it sets the state based on what the command asked for. If the state isn't reported back correctly in about 1.5 secs it will unset the new state but it will update it again when the confirmation arrives from MQTT. I do not use predictive state for HE devices .../set from MQTT.

My times quoted are for beta2.

[bill]

ah I see the other thread now. That's the same behavior that I got stuck on.

Other capabilities I'd like to see:

illuminance
climate

Right now I have a thermostat that's showing up as:
sensor.rheem_econet_communicating_thermostat_heatingsetpoint
(and a few other devices with a similar scheme)

what was working in the past was making them be a climate.X device in Home Assistant so I could set it with climate/set_temperature service.

As far as speed goes, I am seeing strange behavior in the Home Assistant interface. I'll toggle a switch from off to on, it'll go to on, stay there for a second, the light comes on, then the HA slider toggles back to off (with the light still on), then back to on. That sounds like it might be related to 'predictive state'.

I suppose I'm really noticing the slowness on motion events that set several lights, not toggling things by hand. It's just a beat slower. Sounds like beta2 might address this. Can't wait. Great work by the way.

[bill]

so here's an example of slowness:

Hubitat motion event log:

Motion Office is active	DEVICE		2020-02-15 08:47:43.886 PM PST

Appdaemon logs on Home Assistant:

20:47:50.012431 INFO office: Motion active
2020-02-15 20:47:50.014870 INFO office: Setting leader light.corner_light to 65
2020-02-15 20:47:50.024643 INFO office: Setting follower light.office_light to 65
2020-02-15 20:47:50.034198 INFO office: Setting follower light.desk_lamp to 65

Hubitat Actually turns the light on almost 10 seconds later:

switch	on		Office Light was turned on [physical]	DEVICE	physical	2020-02-15 08:48:02.645 PM PST
switch	on		Office Light was turned on [digital]	DEVICE	digital	2020-02-15 08:48:02.387 PM PST

Not sure what's causing this but I don't think it's z-wave since operating the same lights via the hubitat control panel, or alexa voice control is much quicker.

[kevin]

Separating the ‘logic’ such that motion sensor trigger device and lights to change are on one system but the automation script is on another is poor if you want snappy response. Why not do all this on HE ?

Let’s await beta2 and see what the speeds are as I can’t compare to my own system. This is an order of magnitude different. There’s something else wrong. What hardware is HA running on ?

Illuminance is in beta2 , climate isn’t but could be handled by manual payloads if you wanted. Is the climate implementation HE > HA or HA > HE ? i.e. on which platform is the physical thermostat device ?

I’m assuming the climate device is on HA - can you fully expand the statestream topic for the climate device and post a screenshot for me. Someone who knows HA will have to write the HA automation script for this device - would you be able to do that ? (Statestream is not inherently bidirectional). Then HE can be used to control your climate device. I will send changes to ../set topics as I do now for HA lights and switches.

[bill]

The physical thermostat device is on the hubitat side. I'm trying to control it from home assistant.

HA, appdaemon, and mosquitto are docker containers on a fairly modern mult-core pc.

Why not just do the automation logic on the hubitat? Heh.

I initially had a bunch of automations in WebCoRE on smartthings. That worked, about as well as smartthings works, but webcore was too slow and unreliable on hubitat. I was told at the time that webcore was unsupported (after they said it was supported initially) and I should rewrite all of my automations in rule machine or rule engine or whatever its called. It seems to me that its even harder to do basic programmatic and logical tasks in that than webcore. I'd rather stab my eyeballs out than use another point and click, browser based "programming" interface frankly.

I rewrote my stuff in actual python on appdaemon, using an actual text editor, in actual source control. I've been controlling hubitat from appdaemon via the old mqtt-bridge app for at least 2 years now. For the most part its been predictable, and at it's best its performed well. Much better than webcore on either smartthings or hubitat for example, though as I said, every few days things start to slow down, sometimes a lot. I chalked this up to the somewhat abandoned mqtt-bridge app, but I'm starting to suspect the problem all along has been hubitat.

Back when I had my initial problems with webcore I asked about profiling tools, or even just a simple ps app to see what the hell the box was doing. Hubitat told me that their box was more than powerful enough and that such tools weren't necessary.

That certainly hasn't been my experience with hubitat. I've had performance issues from DAY ONE with this thing. Response times slow down, even the interface gets slow, and because its literally a black box you have no recourse other than rebooting in hopes that it'll perform for another 2-3 day stretch. If anything is going to be abandoned in my setup it's not going to be appdaemon, its going to be hubitat, even if that means I have to re-register my 169 z-wave and zigbee devices, something that I desperately want to avoid. It's just too much of a black box. That's fine when it works, but its very difficult to debug.

[bill]

Is this normal? Every one of my hubitat devices seems to think its in use by 6 copies of the MQTT app.

[kevin]

If anything is going to be abandoned in my setup it's not going to be appdaemon, it’s going to be hubitat

Well let’s try and avoid that. Beta2 will I hope meet your speed expectations and your logic model. I have never experienced slowdowns at all on any of my three C4 hubs. Hubitat is great for your Z network needs.

Every one of my hubitat devices seems to think its in use by 6 copies of the MQTT app.

No - that’s not right I don’t know how you achieved that.. you only have on MQTT and most importantly only one device created using the MQTT Client driver ?

Can you restart the hub and then do these go ?

[kevin]

The physical thermostat device is on the hubitat side. I'm trying to control it from home assistant.

Well , you’re implementing this the easier way as half of this is complete. The HE > MQTT support for thermostats is working fine I believe and so you can get real-time status into HA and control back from HA to HE for all attributes using the .../set topics. It will require manual ‘MQTT’ setup within HA though. I don’t yet advertise the HE thermostat using HA MQTT Discovery as a climate device so it will not be auto discovered by HA.

[bill]

Hey Kevin-

So I deleted the mqtt app and rebooted the hub the re-installed it. I'm still seeing 6 copies of it in every device, plus now all of my devices are unavailable in Home Assistant. I think the app initialization is failing here:

app:14812020-02-16 10:24:00.646 am errorjava.lang.NumberFormatException: null on line 1693 (updated)
app:14812020-02-16 10:23:52.281 am warnMQTT: t Unknown category for type temperature reported by device Rheem Econet: Communicating Thermostat

I'm going to unshare the thermostat.

Only 1 device with the mqtt driver: yes
Only 1 MQTT app instance: yes

[kevin]

I’m away from home currently but back shortly - try disabling all the HE devices enabled under Everything - All capabilities and attributes ..