[Beta] MQTT beta 3d (released 5th July)

[kevin]

Here is the link to beta3d , a hotfix update posted 5th July.
Import should work or usual copy raw / paste
Please update both app and MQTT Client driver , plus MQTT Text device driver if you use that

xAPPO/MQTT

Any new version announcements I'll add here in the top post.

It is a significant update from beta1/beta2 , now implemented as a parent app / child device driver to reduce event usage which is proving troublesome for the upcoming 2.2.0 firmware and updates (as all events are stored and have to be later backed up).

When updating you MUST follow the update instructions in the next post.

It should all be a lot faster but there are a lot of improvements and code changes so I do anticipate initial issues.

Please file bug reports on GitHub if you can - they are far easier for me to track there.

This I hope is now near feature complete and so should enter release when any reported bugs are squished. I do still have to further expand device support to more unusual devices and add homie discovery for devices beyond switches, limited sensors and dimmers.

As ever if your questions relate to Home Assistant (HA Discovery / statestream) or OpenHAB / homie usage using this app there are separate threads for exactly that purpose.

[BETA] MQTT app Home Assistant Discovery and statestream support

This is the discussion topic for the MQTT app and it's use in connection with the Home Assistant Discovery Protocol and statestream. Both OpenHAB and Domoticz also support this discovery protocol. General discussion (not HA related) should instead be directed to the main topic here. The ability to link HE with HA bi-directionally is a powerful and popular feature and provides auto discovery , status and control in both directions. It is however not the primary purpose of the MQTT applicat…

[BETA] MQTT app homie3 protocol support... OpenHAB, Athom Homey etc

This is the discussion topic for the MQTT app and it's use in connection with the homie3 / homie4 protocol. General discussion on the MQTT app (not homie or OpenHAB / Athom Homey related) should instead be directed to the main topic here. The MQTT app uses the emerging homie3 topic structure as it's default MQTT presentation ,it provides advertising, status and control bi-directionally. Additionally the homie3 structure can be minimised to just provide essentially status reporting and co…

PLEASE NOTE FIRST INSTALLATION BUG in beta 3d

There is an installation bug in beta3d where the child driver is orphaned, you will see this in the log.
MQTT> Connecting as Hubitat_temporary to MQTT broker null
Easy fix... after first installation do the following..

Manually delete all the MQTT child devices 'MQTT: Child device driver' from HE (not the app). Make sure only one app is installed. Leave the code for both the app and the device driver as is. Reboot the hub . A new device driver will be created. Everything should now work. Do not manually create the driver the app will do this for you.

[kevin]

Here I wanted to detail the changes in moving from beta1 or beta2 to beta3. This is IMPORTANT and a MANDATORY step to get things working with the new architecture.

1. Backup your existing system

2. Take a look at the MQTT client driver and make a note of these settings
   
   In beta3 you will enter them within the app not the driver.

3. Delete your MQTT client device and any and all other MQTT devices using my client driver. There shouldn't be any but none must remain. If needs be you can force delete this via the HE 'Devices' menu, select the device and click 'Remove Device' and 'Remove' after the 'in use' warning.

4. Install the new beta3 MQTT client driver code. Do NOT create a device - the app will take care of that now for you.

5. Either overwrite your beta1/beta2 app code with the new code or remove/delete and recreate the MQTT application. If all has been working well I recommend the former as that will retain your existing child devices and settings.

6. Restart your hub IMPORTANT

7. Did you restart your hub - you must do ! The previous driver can be very persistent at retaining the MQTT connection, maybe even running concurrently. Restarting definitely removes it.

8. Enter the MQTT app and configure your MQTT broker according to the settings you noted in step 2 via the MQTT Broker menu. Proceed through 'Done'

9. Check the MQTT broker has connected in the logs - if not restart the app again. The log should show

   Connecting as Hubitat_ebay44 to MQTT broker
   Connected to MQTT broker tcp://192.168.1.78:1883

where ebay44 is the name of your hub and an IP address matching your broker.

If you see MQTT: FATAL: No MQTT Broker configured - run through the app once again from the HE 'Apps' menu through to 'Done' and it should then be OK. If still not then delete the child device named " MQTT: Child device driver" and run the app again through to 'Done' and all should then be OK there's something not quite consistent here (to look into)

From now on the app should behave as before in beta1/beta2.

Please note the the 'Everything' device selector for publishing HE devices to MQTT has a bug that can cause devices to be non controllable if selected there. I advise you to currently not use this input selection and instead select devices individually using the capability drop downs below (dimmer, switch etc).

FYI: You will have a new child MQTT client device created named

You may alter the Device Name but DO NOT alter the DNI.

... updating as we go based on testers feedback

[kevin]

If you don't use arbitrary virtual devices imported from MQTT into HE, and that's most people, you don't need to read this !

Virtual Device and JSON changes in beta3:

Virtual devices are created and handled differently in beta3. These are arbitrary topics/payloads on MQTT that you'd like to create a 'virtual' device for in HE.

NB: These MQTT topics must not be within a homie/.. or homeassistant statestream topic tree and must be created via page two ' Virtual MQTT Devices and Data' of this app and not directly within HE.

This second page is a bit quirky still but getting better . Double check the updates you make in the device's data itself. Currently (and changed from previous behaviour) any field updated will cause you to have to re-select the device. I will look to revert this later.

You must create a virtual device here (select the device type at the top and then ensure all devices in the bottom 'edit this device' are deselected) then choose a name, click outside of the field and click the 'create device' button. Then 'enable' the device in the middle drop down and select it below to edit.

Attribute Values

Gone are the _ON and _OFF values and instead the values for enumerated attributes are handled in one _MAP data value for each attribute.

Now in attribute_MAP 's you can enter values for each attribute this device has to map the values between a MQTT topic (payload) and the HE attribute value. Do double check them in the actual device and watch out for an already existing space in some fields (backspace it out). This is one of the 'quirks' to fix later.

Attribute _Topic and _Cmd

You include here the _Topic for an attribute which is the 'status' topic on MQTT that updates the matching attribute value in HE and the _Cmd value which is the topic payload that will be updated on MQTT whenever that attribute changes on HE. Either or both are optional. Often only _Topic is used , e.g. for sensors that can't be updated.

Attribute value mappings

HE expects predefined values for certain enumerated attributes , e.g. the switch attribute should be 'on' or 'off'. Your MQTT device might use different values or capitalisation and so you can 'map' any value between MQTT<>HE. To do this complete the _MAP data entry with your MQTT values positionally equivalent with the HE expected values that are shown e.g.
[Yes,No] to map to [on,off]

• capitalisation is important and make sure you complete all values and have no spaces. This mapping of values will be used both to update the attribute in HE and also when (if) sending a command back to MQTT.

JSON

The new version supports extraction of a single value from a JSON key. It does not (currently) support nested json. Additionally a json payload can be written back on update - with a single json key/value being returned.

Defining JSON keys:

When you define a _Topic to recover a payload as a value you can append {keyName:} to the end of the MQTT topic and that value will be used for the payload and attribute update.

e.g.
example/a/b contains payload
{"first": "one","second": "two"}

defining a _Topic of `example/a/b{second:}` returns value two
defining a _Topic of `example/a/b` returns value {"first": "one","second": "two"} or  a two element map

the exact same for the _Cmd topic
'example/a/b{second:}'
when updated with value 'three'
would send a json payload with a single key value {"second": "three"}

I may support multi level nested json in the future.

[gcaley]

Hi Kevin,

Ive followed your instructions and i now have the new beta installed and working. I did have to delete the child driver and re-run app configuration as the broker gave the fatal error you desribed and then reported
app:6432020-04-20 22:14:08.713 errorMQTT: Problem creating child device java.lang.IllegalArgumentException: A device with the same device network ID exists, Please use a different DNI

The only part not mentioned in your guide was i updated the text driver in-place.

The new beta has picked up all my MQTT defined virtual devices, so all good.

I'll go and create some new ones and also have a look at the JSON payload.

Thanks for the hard work! Loving it so far!

[gcaley]

forgot to say I have this one error in the logs:

dev:9082020-04-20 22:17:38.234 infoMQTTStamfordKitchen was set to 17.3°C

app:6432020-04-20 22:17:35.704 infoMQTT: ================== Startup complete ==================

dev:11632020-04-20 22:17:35.684 infoMQTT> MQTT unsubscribing from: null/cover/+/current_position

dev:11632020-04-20 22:17:35.594 infoMQTT> MQTT unsubscribing from: null/cover/+/state

dev:11632020-04-20 22:17:35.525 infoMQTT> MQTT unsubscribing from: null/person/+/state

dev:11632020-04-20 22:17:35.413 infoMQTT> MQTT unsubscribing from: null/input_boolean/+/state

dev:11632020-04-20 22:17:35.320 infoMQTT> MQTT unsubscribing from: null/group/+/state

dev:11632020-04-20 22:17:35.261 infoMQTT> MQTT unsubscribing from: null/binary_sensor/+/state

dev:11632020-04-20 22:17:35.144 infoMQTT> MQTT unsubscribing from: null/sensor/+/state

dev:11632020-04-20 22:17:35.056 infoMQTT> MQTT unsubscribing from: null/light/+/brightness

dev:11632020-04-20 22:17:34.971 infoMQTT> MQTT unsubscribing from: null/light/+/state

dev:11632020-04-20 22:17:34.832 infoMQTT> MQTT unsubscribing from: null/switch/+/state

app:6432020-04-20 22:17:34.782 infoMQTT: Unsubscribing HA events

app:6432020-04-20 22:17:34.780 infoMQTT: [96] Total Hubitat endpoints enabled on MQTT

app:6432020-04-20 22:17:34.778 infoMQTT: [95] Total Hubitat devices are enabled on MQTT

app:6432020-04-20 22:17:34.739 infoMQTT: [95] Hubitat 'everything' all capability devices enabled on MQTT

app:6432020-04-20 22:17:34.372 infoMQTT: ==================================================

app:6432020-04-20 22:17:34.366 infoMQTT: Initialize exit

app:6432020-04-20 22:17:34.359 errorjava.lang.NullPointerException: Cannot invoke method getAt() on null object on line 5084 (lookupP)

dev:11632020-04-20 22:17:34.352 infoMQTT> MQTT subscribing to: garagepi/chestfreezer/temperature

[MRobi]

I've googled this and searched the boards but haven't come up with a solid answer.

I had some old NUC's laying around so I setup HA the other day just to play around with it and get a feel. I used the custom component to bring my HE devices into HA using Maker API.

Is there a benefit to this method over that one? Will it be faster, expose more attributes, etc?

[kevin]

I suppose this integration has three advantages and one disadvantage with respect to Home Assistant integration.

1. It is bidirectional allowing devices from HA to be created and controlled in HE. The alternate only pulls HE devices into HA.

2. MQTT is a great way to integrate more than two controllers, it’s a central exchange for status and control. It is a means to link multiple systems and devices. It is also operates without causing dependency on clients which is useful. It’s very flexible.

3. MQTT is a stand-alone protocol which is open , whilst the alternative is a custom protocol creating a one to one link. If you might expand later MQTT is a great basis.

4. The alternative is easier to set up and doesn’t require an intermediary (MQTT). It works well.

Another aspect is that this is an HE app whereas the alternative is an HA integration. Your preference..

You can use both should you wish...

[kevin]

I have updated the top post with the link to beta 3, enjoy..

As I've said it's a major update - please read the top few posts in this thread and let me know any issues or suggestions you have. Do make sure you install/update correctly particularly regarding the removal and deletion of the old driver and the restart. Any bugs I'll try and issue a hot fix promptly.

There are a few things still to expand out like Homie Discovery into HE, some of the more unusual devices to add and the virtual device editor is still quirky but almost there now. Almost feature complete. so I'm hopeful there wont be a need for a beta 4, just a few beta3 updates.

A Hubitat Package Manager install is my next thing to look into.

[kevin]

One thing I must mention is about enabling HE devices to be published to MQTT.

There is still an 'Everything' drop down at the top of the device list. The difference is it now works !! This publishes all the capabilities and attributes for the selected devices to MQTT and is the way I expect most people will publish their devices.

However there are also drop downs for each capability a device has and you can instead use these. But note these rigidly enforce the capability and will only publish to MQTT for that capability and the attributes within so for example an RGB color control capability does not publish on/off or dim (level) - you would have to enable the RGB device in switches and dimmers for that. Likewise .../set is not allowed if the capability is disabled, unless the device is enabled in 'everything'.

[LosinIt]

Kevin, I've found a bug, or at least and unexpected (by me) change. The "motion" topic is no longer being updated:

but the "status" is:

I've tried it with the sensor in the "All Capabilities" list and (having removed it from there) the "Motion Sensors" list with the same result.

I've also noticed that it takes about 3 minutes for the app to finish restarting after clicking "Done". This is an order of magnitude longer than it was. Is this due to having everything selected in the "All Capabilities" list?

[LosinIt]

Here's another confusing thing:

Previously I was using "homie/steve-s-hubitat/front-porch-multisensor/measure-light/illuminance" as the source of lux but that doesn't exist now. This device is selected in the "All Capabilities" list.

Looking at the $properties list I see measure-illuminance is the first item but it doesn't show anywhere. I did reboot the hub and restarted MQTT Explorer while it was in the process and I can see the $properties list being built in the History:

[jwilliams]

@LosinIt

Kevin, I've found a bug, or at least and unexpected (by me) change. The "motion" topic is no longer being updated:

but the "status" is:

I've tried it with the sensor in the "All Capabilities" list and (having removed it from there) the "Motion Sensors" list with the same result.

I've also noticed that it takes about 3 minutes for the app to finish restarting after clicking "Done". This is an order of magnitude longer than it was. Is this due to having everything selected in the "All Capabilities" list?

"motion"->"motion" was updated to just the "motion" topic. There was no need to have the redundant one under it (that was kind of a bug). Is your higher level "motion" topic updating correctly? (of course when I go to test, no one is moving around in my house). If not, turn up the logging for MQTT and see if the updates are being blocked.

Yes there is a little bit more of a slowdown in the startup / "finish" time. Property scanning and blocking takes a little while to run. I know Kevin is working on some optimizations to decrease this delay.

[tsviper]

I just installed this for the first time and am having some trouble. I can connect using MQTT Explorer just fine but not with this app.

What am I doing wrong?

[jwilliams]

@tsviper

I just installed this for the first time and am having some trouble. I can connect using MQTT Explorer just fine but not with this app.

What am I doing wrong?

Did you install the Child driver?

If not, you need to do that as the "broker" is created using the child driver

[tsviper]

Thanks for the help. I installed the driver and it auto created this:
I'm trying to connect to my existing mosquitto mqtt broker on my raspberry pi. Is this not how it's supposed to work?

[jwilliams]

You do not need to do anything with the child driver. the "MQTT" app itself will create, configure, and manage the child.

THe "configure" portion pulls in the values you set/specify in the "broker" section of the application.

[tsviper]

Thats what I thought but its not connecting to my mosquito broker on the pi for some reason.

[jwilliams]

Did you manually create the child or did the app?

I'd delete the child device (device, not driver), reboot the hub, make sure the broker settings are correct then "finish" / next.

That should recreate it correctly.

[kevin]

@tsviper

I just installed this for the first time and am having some trouble. I can connect using MQTT Explorer just fine but not with this app.

Someone else has reported similar and I didn't track down why - this was on a new install too. Let me try this myself on a new hub.

See Jeffs post immediately above this one too. That should fix it.

see post #2 in this thread - If you see MQTT: FATAL: No MQTT Broker configured -

MQTT beta3 released

Here I wanted to detail the changes in moving from beta1 or beta2 to beta3. This is IMPORTANT and a MANDATORY step to get things working with the new architecture. Backup your existing system Take a look at the MQTT client driver and make a note of these settings [image] In beta3 you will enter them within the app not the driver. Delete your MQTT client device and any and all other MQTT devices using my client driver. There shouldn't be any but none must remain. If needs be you can …

[tsviper]

Yes, the app created the child automatically. Thank you both...I'll give that a try.