That is exactly what I meant. If said add hoc device isn't also subscribe to updates from Hubitat, then why would you ever create MQTT messages for virtual devices that the MQTT app creates for said add-hoc devices?
The device utilises its own defined topic hierarchy - one of which is a status topic that it posts updates too and one is a command topic that it subscribes too. My app allows configuration of those two topics and the expected payload content.
Okay...then back to my original question. What difference does it make where the attribute state change occurred? Wouldn't your app have to create an MQTT message for all device status changes, not just those not created by your app? Why is it not creating an outbound MQTT message for changes caused by the inbound MQTT messages?
But also, if you change the state of the virtual device in Hubitat, you get an MQTT message with the same topic as if you had changed the physical device? You would have to have a different base topic for the virtual device than you did for the physical one, right?
It must not create an MQTT message for these or a loop will happen.. other changes it should.
No if I change the physical device it reports it's new status on say
MyNetwork/Device53/state
which I am subscribed to and can update the HE matching virtual device from (but I must not send a cmd back to MQTT or it will loop)
If I send a message from HE, orginated by say Rule Machien or a click it goes to a different topic, say
MyNetwork/Device53/command
which the device subscribes to and executes the physical state change
and then almost immediately the device responds with an updated state on
MyNetwork/Device53/state
which I am subscribed to and update the device status if needed (but I must not send a cmd back to MQTT)
Oh...okay. Now I understand.
An alternative would be for the app to track what events it is expecting and not create an MQTT message for those. For example, if it receives a switch state of on from the physical device, when the Hubitat device creates an On event, it doesn't create an MQTT message for that.
Another option would be for your virtual devices to use different commands for changes initiated in Hubitat than changes made via MQTT. That way, when the Hubitat event is created, you could distinguish between the two via the event type (physical vs digital). So, in your app, an on MQTT message would issue on onMqtt command to the device which would result in a switch event with a value of on and a type of physical. But a change made by rule machine issuing a command of "on" would create a switch event with a value of on with the type of digital instead. That would allow you to distinguish between the two. Of course, that would mean a whole host of changes to virtual drivers and the MQTT app. So, probably would have been practical when you started vs changing it now.
The drivers are the stock HE virtual ones so I can't change them
As I mentioned I do not want to maintain a state model within the MQTT app or indeed any persistent storage as it involves atomicState continual writes and I already have established this is not dependable in very rapid update scenarios.
Even then I couldn't determine if two successive 'off' events came from my app and then say Rule Machine or vica versa Not that in that scenario it really matters.
(Let me preface by saying I'm just throwing everything against the wall hoping something might stick or inspire a brilliant idea from you or someone else. Not thinking that any of this would be a complete, "wrap-it-up-with-a-bow" solution. So, if it helps, great. If not, you're only out a couple of minutes of your time. So, I only ask you receive it as it was intended.)
I think you've answered your own question. You've eliminated the only two ways to do it. The event structure does not track what caused the device event to occur. It's not intended to be that "knowledgeable" about the event. The driver doesn't "know" what's issued the command or what passed the data off to it to parse (in the case of z-wave or zigbee events). It just executes the command.
I was thinking that if you created the virtual devices under a parent virtual device that could help since you could create the event via the parent for the child with the type of physical. Since all of the built in Virtual devices don't distinguish at all, that field would be null for "real" events (the ones you care about). Then you'd only have to create a single driver, the parent.
Another option would be to unsubscribe from events from the virtual device before issuing the command and then re-subscribing. Bruce has said modifying the subscriptions is very "lightweight" to the hub (but for the life of me I can't remember the exact phrasing he used). This would only drop events if they occurred at the exact same time as the event generated by MQTT.
You mean the delay in writing it immediately causing other changes to be missed? For an app as large as your MQTT app, yeah, when you scale up to something of that size that is handling that volume of device events, then I could see that being a huge problem. It makes me wonder if having all of that logic within Hubitat itself is really viable for a large-scale MQTT implementations.
I know that when it comes to things like integration of Hue, the only way to achieve a reliable solution to prevent loops like the one you're describing is with custom drivers. The only advantage there is that you're only talking 4 or 5 custom drivers and not the dozen (or more) you'd have to create to cover the devices types your app supports. But look at the Hubduino project and all of the custom drivers for that. While it is a nauseating prospect to have to write that many custom drivers, it's not completely outside the realm of possibility. Especially when each is simplified and everything is standardized. Another advantage is you can then offload the processing of the details of the command from the app to the driver. The app only determines what device needs the info and then all the detail parsing the specifics of the command is done by the device instead of the app. Also, you can then use the driver to call to the parent app to trigger the MQTT message rather than having the app subscribe to any device events. When the driver parses an MQTT message passed off from the parent app, it doesn't call back to the parent to create an outbound MQTT message. But when a command is issued to it from a Rule Machine rule for example, it would call to the parent app to create an outbound MQTT message. This might be more viable on the scale you're working with than a subscription model.
You raise some interesting thoughts there, thankyou , I'll digest.
My reasoning behind (only) supporting the inbuilt HE drivers for adhoc input to HE is that it helps define and restrict the devices I support and avoids me getting asked for xyz support. I really don't have the time or to be frank interest in authoring and then supporting a load of drivers for peoples individual needs. My code is available for people to either create pull requests for additions or edit for their own individual needs and use (only)
For outgoing devices HE -> MQTT I support effectively every driver.
Great input .. thanks for taking the time to understand the issues
This is a tough problem. We had a similar issue with the Lutron integration, because sending a command to Lutron generates what looks like an external event from Lutron -- indistinguishable from one. We do use state to resolve this, to avoid looping.
Here's a fairly easy example .
I have a HE virtual dimmer currently off at level 0 linked to an MQTT device
The remote device updates it's status via MQTT to off at 50%
I know I need to send a level change command to the HE virtual device but want the off status to remain.
I can't do this without thought because I remember the level change turns the light on in HE
This level change then generates a switchedDim 'on' event and a level change event
I didn't expect the on event of course but anyway I can turn it off again. Some devices have a presetDimLevel command I believe.
Conversely setting the level to 0 does not turn the light off and a light can be turned off an retain its dim level. (which I think is the right approach)
But in actioning this I have received both an on and an off event and a level change, and importantly I didn't send an 'on' command so I didn't expect it eventing . This actually happens even if the light is already on, it again sends another on event.
So.. I'm messed up by attribute interactions getting back events I didn't expect and really not having a way to know if these should be sent back to the remote device. Automatically knowing what I might expect back (by keeping a note of the commands I've sent) isn't easy. I never sent that 'on' command. Also action order of these commands for example off/55% or 55%/off is important the off must go last or the device is left on. Knowing which attributes when updated can internally effect other attribute values is key to knowing in which order to apply commands ... hhhmmm
It's amazing even a simple dimmable bulb can be so awkward !! I dread implementing an arbitrary MQTT thermostat.
@bravenel I assume when a device generates several near simultaneous events the various event handlers can run concurrently and are not deterministic in sequence e.g. switch before level ? I think I read somewhere events can get re-ordered before delivery.
The platform is multi-threaded, so yes, it is non-deterministic with respect to relative timing of near simultaneous events.
For anyone interested I've implemented a reasonable solution to this It partly uses tagging that @tcp77 suggested, state or atomicState and a very brief 100mS enforced 'deaf' period to events, in my app only and for the one specific device.
When an MQTT status message comes in I tag the device involved and first check if the current attribute value matches - if it does I don't need to update so I drop that command thus avoiding an event (although it might not get sent anyway)
If I do need to update a device then I get right to the point of issuing the command and set atomicState.block = device.displayName , then issue the command and kick off a runInMillis(100,"unblock") call. This will remove the block in 100mS marking atomicState.block='FREE'. Each command I have to send (there could be several in a json payload) I update the atomicState.block, and send another runInMillis(100,"unblock") which resets/prolongs the unblock timer.
Now in my event handlers I immediately get the events device.displayName and compare it against atomicState.block and if they match I ignore (drop) the event.
This has the effect that I am deaf to all events from that specific device for 100mS after the last command I sent to it for MQTT originated 'status' update messages. Other devices subscribing to the device will receive events as normal and any other commands I send to the device are actioned without blocking.
The only potential issue is that if another app e.g. Rule Machine sends a command to that device during that 100mS I will not see the event. I know how to avoid that situation by keeping a list of expected events and their reported value - and then RM's command would be 'unexpected' but that adds significant overhead. Anyone creating a RM rule that triggers off the device change and updates back to the same device just needs to delay the response by say 250mS and it will all work fine. (I hope)
P.S. 100mS was a best observation based on the time for an event to fire following the command being sent. On my system I could get away with 50mS but on a more sluggish system it might need more. The good thing is that on a sluggish system the unblock command gets delayed too so it sort of self adjusts .
Interesting discussion 
Seems you've got a reasonable workaround, but thought I'd throw this out anyway.
Does device.updateDataValue() work from an App?
It was mentioned here - Device routine returns null - that you can manipulate a device Data object (update / get) from an App. Not sure if that's just getDataValue() though or whether you can write to it ....
If you can then couldn't you set something in there each time the device is updated from your App to show what events you want to ignore afterwards. Then ignore any subsequent events until it's cleared?
MQTT Status Change ON, 50% ->
virtualdevice.updateDataValue("MQTTEvents", "ON, 50%")
virtualdevice.on()
virtualdevice.setLevel(50)
Then as the events come back from the Virtual Device compare them to what you've stored and remove them until there's none left. When there's none left, or a non-matching event comes through, you know that this was from somewhere else and you should send a CMD out.
(I've not played with HE Groovy for a while so this may be b0ll0x in which case just ignore it)
What you suggest is actually what I was meaning with 'significant overhead'. I do use data values within devices for storing the MQTT topic values and mappings but it is quite slow.
What also deflected me from this approach is that some unexpected commands come back - for example setLevel() returns a switchedDim 'on' event. Now I know that I can expect it, but there are other devices where this is going to happen and I will get unexpected events from those. What I like though is that there are no artificial delays imposed in this approach.
There is also the question of intelligence in a 'virtual device' Should a virtual device just effectively be a display / attribute store template or should it act like the real device ? After all in the MQTT situation the real device and it's intelligence is often remote.
Take for example a blind (shutter) or a garage door. In the virtual devices a 'close' command starts the mechanism working but the device does not return the closed event for some time - maybe 10 seconds (often settable). Being able to set this delay to 0 would be ideal of course.
Your approach actually caters for this and mine doesn't because my 100mS expires long before that event arrives. For me I shall probably handle such 'delayed response' devices specifically or hope that I can set the closed state directly without using the 'close' command. After all the remote device will report 'closed' itself eventually in a status update.
PS I did initially try an approach as you suggested, initially with a Map object and later device data and it worked fairly well (except for unexpected commands) but it did add about 800mS to the processing as several write/reads have to be made to the devices data value. It was quicker with the map object but more awkward to code.
Also with concurrency it's hard to protect the integrity of that data value without passing it via atomicState - hence the map object again helped. I haven't totally discounted that as in practicality the device has already been updated so it's not impacting speed there as such - more in the the tidying up afterwards.
You keeping well Martyn ?
You've been quiet, hope things are sorting themselves out ...
Yeah pros and cons to each approach really.
What would be better would be for Virtual Devices to perhaps have a method where you could update their attributes without triggering an event. They are a bit of an oddball I guess because as you say often the action is remote so the Ack that a "normal" device would get (to confirm its state has changed) never arrives.
You really need:
setLevel()
--- generates an even that you can react to
setAttribute("state", "on")
setAttribute("level", "50")
--- doesn't generate events but the device is now updated to show the current values
Not sure how much you'd need to petition Mike / Bruce to get that though!
All OK here thanks, mostly just lurking on forums nowadays unless something interesting catches my attention I rarely speak
Even that is problematic as other apps subscribed to the device do need the events or they wont realise it's updated - it's just me that doesn't want them !
I think both these approaches are viable - i hate using 'delays' as a solution in code so I prefer the other approach .. I'll see how this works out. I'm still concerned over concurrency if two devices are involved simultaneously as this solution won't handle that. But I believe the parse{} method in HE's MQTT driver is single threaded and I use parent method calls now (not events) so that might avoid this happening.
@bravenel Having something returned in events that identified 'what/who' originated the state change would be so useful though... This could even help identify which user turned the device on/off which was a request in another topic for a history log. Sorry to tag you Bruce but this may be missed otherwise. Maybe it's just too awkward as it works internally currently.
Yeah, could be App ID number perhaps then you could filter out events caused by your own App.
1 Like
or identify RM interactions specifically...
or log a specific users name
.. an alternative being pass a token with a command that gets included in any associated events but that probably requires all drivers updating and supporting it 
- non starter
Most events are generated by devices, not apps. The current architecture of the hub does not support this concept.
I don't follow what RM has to do with this. Aren't you concerned about the response from the external device from a command sent to it, distinguishing that response from one generated externally? Isn't this something that should be handled in the driver that sends the command and processes the response?
