[BETA] YAMA (Yet Another MQTT App)

I wouldn't think so, no. Watching my mqtt broker, I only see new messages on topics when the value updates.

In hubitat drivers it is an option on how to do the value updates. Typically the driver authors would not force an update if the value hasn't changed, and let the platform decide when to actually store a state change. When it is left up to the platform, it won't do a state change unless the value has changed from the previous value.

I'm not aware of any of the inbox drivers that are forcing State changes, but it certainly possible if you are using some user drivers that they might be doing so.

First the good news - I changed the IP address to a DNS name and it seems to be working just fine.

image

I suspect it's an issue with the Wiz driver. Perhaps @zranger1 can comment.

Hi!

The Wiz driver does poll the lights to get current status. There's no other way to do it.
By default, it's one datagram transaction of about 150 bytes every 10 seconds.

If you've got a lot of lights, that might be generating some traffic. If this really is causing trouble, you can change the polling rate from the device page, or via the API.

1 Like

OK, so with 60 lights (6/second) each producing 8 MQTT messages it adds up and matches what I see. I take it comparing against the previous status isn't simple.

It's likely possible, but it would require touching a lot of things, and I can't get to it for a while -- not 'till at least after Burning Man.

@zranger1 You really shouldn't have to do anything special. As long as on the event publishing it doesn't force a state change (isStateChange: true), then the platform will auto-dedup events.

Something like this (stolen from one of my drivers):

result << sendEvent([name: "pushed", value: 1, descriptionText: "$device.displayName had Up Pushed (button 1) [physical]", type: "physical", isStateChange: false]

if isStateChange: false or omitted (default=false), then the hub will dedup events automatically.

Better described here:

Of course the problem may actually be in my YAMA app... I looked, and I specified filterEvents:false for the subscriptions, which from the docs:

filterEvents - Used for device subscriptions. Set to false to receive all events, defaults to true and events that do not have a changed value will not be processed.

So simply changing/removing that may "fix" what you are seeing.... Unless I put it in there for a specific reason - that is the part I can't remember. I don't "usually" change defaults unless there is a reason, that is the only thing giving me pause.

Here is a test build that simply sets all the event filters to TRUE instead of FALSE, if you want to try it. Open YAMA app replace all the code with the code below, and save. Then open the YAMA App and click done (or even better, reboot the hub):

Test Code
/*
 *  Import URL: https://raw.githubusercontent.com/Botched1/Hubitat/master/Apps/YAMA/YAMA%20APP.groovy 
 *
 *  ****************  YAMA (Yet Another MQTT App) APP  ****************
 *
 *  Design Usage:
 *  Publish Hubitat devices, commands, and attributes to MQTT 
 *
 *  Original code, but inspiration and some ideas taken from the work of Kevin (xAPPO) and MQTT Link (mydevbox, jeubanks, et al)
 *  
 *-------------------------------------------------------------------------------------------------------------------
 *  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
 *  in compliance with the License. You may obtain a copy of the License at:
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
 *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License
 *  for the specific language governing permissions and limitations under the License.
 *
 * ------------------------------------------------------------------------------------------------------------------------------
 *
 *  Changes:
 *
 *  V0.0.1 - 01/12/21 - Test release 1
 *  V0.0.2 - 01/13/21 - Test release 2. fixed init after reboot, and a few other code cleanup items
 *  V0.0.3 - 01/13/21 - Added sendAll command support. Made init only do commands and subscriptions, but not re-publish all attributes.
 *  V0.0.4 - 01/13/21 - Added logic to remove mqttDriver from deviceList if it was selected
 *  V0.0.5 - 01/20/21 - Added check for driver connection status on init 
 *  V0.0.6 - 01/23/21 - Added check for driver connection status before sending events to MQTT
 *  V0.0.7 - 01/30/21 - Split sendAll and Initialize methods apart. Delayed initialization code by 1s to prevent issues if the driver happens to send, or app sees, multiple "init" events.
 *  V0.0.8 - 02/06/21 - Changed topic structure, putting attributes in hubitat/hubName/deviceName/attributes/* and commands in hubitat/hubName/deviceName/commands/*
 *  V0.0.9 - 08/08/22 - Removed all ["filterEvents": false] from subscriptions as a test.
 *
 */

import groovy.json.JsonSlurper

definition(
	name: "YAMA (Yet Another MQTT App)",
	namespace: "Botched1",
	author: "Jason Bottjen",
	description: "",
	iconUrl: "",
	iconX2Url: "",
	iconX3Url: "")

preferences 
{
	page(name: "pageConfig")
}

def pageConfig() 
{
	dynamicPage(name: "", title: "", install: true, uninstall: true, refreshInterval:0) 
	{
		section("Configuration") 
		{
			input "mqttDriver", "capability.notification", title: "MQTT Driver", description: "MQTT Driver", required: true, multiple: false
			input "deviceList", "capability.*", title: "Devices", description: "Devices", required: true, multiple: true
			input(name: "logEnable", type: "bool", defaultValue: "true", title: "Enable Debug Logging", description: "Enable extra logging for debugging.")
		}
	}
}

def installed() 
{
	log.debug "----- in installed -----"	
	if (logEnable) log.debug "installed"
	initialize()
}

def updated()
{
	log.debug "----- in updated -----"
	if (logEnable) log.debug "updated"
	initialize()
}

def logsOff(){
    log.warn "debug logging disabled..."
    app.updateSetting("logEnable",[value:"false",type:"bool"])
}

def initialize() 
{
	if (logEnable) {
		log.warn "Debug logging is enabled. Will be automatically turned off in 30 minutes."
		runIn(1800,logsOff)
	}

	subscribe(mqttDriver, "parentComplete", deviceEvent, ["filterEvents": true])
	
	log.warn "State: " + atomicState.initialized
	if (atomicState.initialized != "initializing") {
		if (logEnable) log.debug "initialize starting"	
		atomicState.initialized = "initializing"
		runInMillis(1000,initializeMqtt);
	}
}

def initializeMqtt() 
{
	log.debug "YAMA app initialization: Beginning"
	
	// Unsubscribe from all events
	unsubscribe()
	
	// Remove the MQTT Driver from the device list, if it was selected for publishing
	if (deviceList.find {it.name == mqttDriver.name}) {
		deviceList.remove(deviceList.findIndexOf {it.name == mqttDriver.name})
	}

	// See if driver is connected to MQTT broker
	if (mqttDriver.currentValue("connectionState") == "disconnected") {
		log.debug "Error: Driver could not connect to MQTT broker! Could not Initialize app."
		return;
	}
	
	// Walk through selected devices to get commands and attributes
	for(item in deviceList){
		// Publish Commands
		commandList = item.getSupportedCommands()

		for(commandItem in commandList){
			mqttDriver.publish("${item}/commands/${commandItem}/set","")
			pauseExecution(100)
		}

		// Publish sendAll command
		mqttDriver.publish("sendAll","")
		pauseExecution(100)
		
		// MQTT Subscribe to all command sets
		mqttDriver.subscribe("+/+/+/set")
		pauseExecution(100)
		
		// Subscribe to sendAll
		mqttDriver.subscribe("sendAll")
	}

	// Subscribe to events form selected devices, and MQTT driver
	subscribe(deviceList, deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "mqtt", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "init", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "sendAll", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "parentComplete", deviceEvent, ["filterEvents": true])
	
	log.debug "YAMA app initialization: Complete"
	
	// Set state so new events will get processed in that handler
	atomicState.initialized = "completed"
}

def sendAll() 
{
	if (atomicState.initialized == "completed") {
		if (logEnable) log.debug "sendAll starting"	
		atomicState.initialized = "sendAll"
		runInMillis(100,sendAllMqtt);
	} 
	else {
		log.debug "sendAll aborted since initialized flag != completed"	
	}
}

def sendAllMqtt() 
{
	log.debug "YAMA app initialization: Beginning"
	
	// Set state so events that happen during init are ignored in the event handler
	//atomicState.initialized = "re-sending"
	
	// Unsubscribe from all events
	unsubscribe()
	
	// Remove the MQTT Driver from the device list, if it was selected for publishing
	if (deviceList.find {it.name == mqttDriver.name}) {
		deviceList.remove(deviceList.findIndexOf {it.name == mqttDriver.name})
	}

	// See if driver is connected to MQTT broker
	if (mqttDriver.currentValue("connectionState") == "disconnected") {
		log.debug "Error: Driver could not connect to MQTT broker! Could not Initialize app."
		return;
	}
	
	// Walk through selected devices to get commands and attributes
	for(item in deviceList){
		// Publish Commands
		commandList = item.getSupportedCommands()

		for(commandItem in commandList){
			log.debug "Setting ${item}/commands/${commandItem}/set"
			mqttDriver.publish("${item}/commands/${commandItem}/set","")
			pauseExecution(100)
		}

		// Publish Attributes
		attributeList = item.getSupportedAttributes()
		
		for(attributeItem in attributeList){
			curVal = item.currentValue("${attributeItem}")
			
			if (!curVal) {
				curVal="null"
			} else {
				curVal = curVal.toString()
			}
			mqttDriver.publish("${item}/attributes/${attributeItem}/value",curVal)
			pauseExecution(100)
		}
	}

	// Publish sendAll command
	mqttDriver.publish("sendAll","")
	pauseExecution(100)
		
	// MQTT Subscribe to all command sets
	mqttDriver.subscribe("+/+/+/set")
	pauseExecution(100)
		
	// Subscribe to sendAll
	mqttDriver.subscribe("sendAll")
	pauseExecution(100)
	
	// Subscribe to events form selected devices, and MQTT driver
	subscribe(deviceList, deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "mqtt", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "init", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "sendAll", deviceEvent, ["filterEvents": true])
	subscribe(mqttDriver, "parentComplete", deviceEvent, ["filterEvents": true])
	
	log.debug "YAMA app initialization: Complete"
	
	// Set state so new events will get processed in that handler
	atomicState.initialized = "completed"
}

def uninstalled() 
{
	unschedule()
	unsubscribe()
	log.debug "uninstalled"
}

def deviceEvent(evt)
{
	/*
	log.debug "name: " + evt.name
	log.debug "descriptionText: " + evt.descriptionText
	log.debug "source: " + evt.source
	log.debug "value: " + evt.value
	log.debug "unit: " + evt.unit
	log.debug "description: " + evt.description
	log.debug "type: " + evt.type	
	log.debug "locationId: " + evt.locationId
	log.debug "hubId: " + evt.hubId
	log.debug "displayName: " + evt.getDisplayName()
	log.debug "device: " + evt.getDevice()
	log.debug "deviceId: " + evt.getDeviceId()
    */

	log.debug "atomicState.initialized: " + atomicState.initialized
	log.debug "atomicState.initialized != completed: " + (atomicState.initialized != "completed")

    // MQTT driverparentComplete
	if (evt.name == "parentComplete") {
		log.debug "Setting atomicState.initialized = completed"
		atomicState.initialized = "completed"
		return;
	}
	
	if (atomicState.initialized != "completed") {
		log.debug "Aborting since atomicState.initialized != completed"
		return;
	}
	
    // If MQTT driver is init, then re-initialize app
	if (evt.name == "init") {
		//log.debug "Received init from the driver."
		initialize()
		return;
	}

    // MQTT driver re-send all
	if (evt.name == "sendAll") {
		sendAll()
		return;
	}
		
	// Process incoming HUB DEVICE events, and publish to MQTT topics
	if (evt.name != "mqtt") {
		// See if driver is connected to MQTT broker
		if (mqttDriver.currentValue("connectionState") == "disconnected") {
			if (logEnable) log.debug "Error: App could not send MQTT event. Driver is not connected to MQTT broker!"
			return;
		}
		else {
			mqttDriver.publish("${evt.getDevice()}/attributes/${evt.name}/value","${evt.value}")
		}
	}
	// Process incoming MQTT events from subscibed topics
	else
	{
		def jsonSlurper = new JsonSlurper()
		def object = jsonSlurper.parseText(evt.value)
		
		if (logEnable) log.debug "event device: " + object.device
		if (logEnable) log.debug "event type: " + object.type
		if (logEnable) log.debug "event value: " + object.value
		
		eventDevice = deviceList.find {it.getDisplayName() == object.device}
		//log.debug "eventDevice: " + eventDevice
		eventDeviceCommands = eventDevice.getSupportedCommands()
		//log.debug "eventDeviceCommands: " + eventDeviceCommands
		eventDeviceCommand = eventDeviceCommands.find {it.name == object.type}
		//log.debug "eventDeviceCommand: " + eventDeviceCommand
		eventDeviceParams = eventDeviceCommand.getParameters()
		//log.debug "eventDeviceParams: " + eventDeviceParams
		
		// Determine if the updated command accepts parameters or not, and process
		if (eventDeviceParams) {
			if (object.value) {
				newstr = (object.value).split(',')
				
				switch(newstr.size()) {
				case 1:
					//log.debug "size 1";
					param1 = (eventDeviceParams[0].type == "NUMBER" ? newstr[0].toLong() : newstr[0])
					eventDevice."$object.type"(param1)
					break;
				case 2:
					//log.debug "size 2";
					param1 = (eventDeviceParams[0].type == "NUMBER" ? newstr[0].toLong() : newstr[0])
					param2 = (eventDeviceParams[1].type == "NUMBER" ? newstr[1].toLong() : newstr[1])
					eventDevice."$object.type"(param1, param2)
					break;
				case 3:
					//log.debug "size 3";
					param1 = (eventDeviceParams[0].type == "NUMBER" ? newstr[0].toLong() : newstr[0])
					param2 = (eventDeviceParams[1].type == "NUMBER" ? newstr[1].toLong() : newstr[1])
					param3 = (eventDeviceParams[2].type == "NUMBER" ? newstr[2].toLong() : newstr[2])
					eventDevice."$object.type"(param1, param2, param3)
					break;
				case 4:
					//log.debug "size 4";
					param1 = (eventDeviceParams[0].type == "NUMBER" ? newstr[0].toLong() : newstr[0])
					param2 = (eventDeviceParams[1].type == "NUMBER" ? newstr[1].toLong() : newstr[1])
					param3 = (eventDeviceParams[2].type == "NUMBER" ? newstr[2].toLong() : newstr[2])
					param4 = (eventDeviceParams[3].type == "NUMBER" ? newstr[3].toLong() : newstr[3])
					eventDevice."$object.type"(param1, param2, param3, param4)
					break;
				case 5:
					//log.debug "size 5";
					param1 = (eventDeviceParams[0].type == "NUMBER" ? newstr[0].toLong() : newstr[0])
					param2 = (eventDeviceParams[1].type == "NUMBER" ? newstr[1].toLong() : newstr[1])
					param3 = (eventDeviceParams[2].type == "NUMBER" ? newstr[2].toLong() : newstr[2])
					param4 = (eventDeviceParams[3].type == "NUMBER" ? newstr[3].toLong() : newstr[3])
					param5 = (eventDeviceParams[4].type == "NUMBER" ? newstr[4].toLong() : newstr[4])
					eventDevice."$object.type"(param1, param2, param3, param4, param5)
					break;
				} 
			}
			else
			{
				log.debug "Empty value received unexpectedly - abort"
			}
		}
		else
		{
			// No parameters needed for this command, so process it as-is
			eventDevice."$object.type"()
		}
	}
}

'''

@JasonJoel, you're right -- I just hadn't though about this in a while.

I think I'm getting back some values that are noisy, like rssi, that are probably causing more state changes than necessary. I can force those to false so they don't get picked up unless something important changes, and that'll cut down the event traffic.

Will still be a couple of days -- we're on the last march towards getting our 50 foot long, 150,000 LED project ready to haul off to Blackrock city.

1 Like

Thanks for all the effort.

I'm now focusing on working around bugs in the current node networking implementation.

Strange -- I stopped getting messages other than the heartbeat. All the settings are the same and the log showed nothing out of the ordinary.

And now it started working again ... mysterious ways and all that.

My YAMA app just started doing this as well.
I'm only getting messages for heartbeats.

1 Like

I tried rebooting the hub, restarting my mqtt broker, neither worked.
What DID work was removing the YAMA app and recreating it.

Apparently spoke too soon...

Everything worked well for a handful of minutes, then everything screeched to a crawl.
The heartbeat is stuck, and only a couple of my devices are updating regularly.
I'm not sure why some would work, while others don't...

Odd. Obviously there haven't been any app/driver changes. Have you done any hub upgrades since it stopped working (not saying it is a hub problem, but it may help identify what changed).

EDIT: I just checked my dev hub, and I'm only getting heartbeat updates as well. Clearly something in the last hub update broke the app. I'll try to find some time to troubleshoot, but may be a few days.

Thanks. Yeah, I did the most recent (two) updates.
I can wait patiently. Thanks for your work on this.

There is some weird initialization issue. I was able to claw out of it (which means you might too), which may help until I get to a "real" solution.

I did a bunch of stuff, not sure which fixed it.

I did:

  1. YAMA Driver Details
    • Clicked, Disconnect, Connect, ParentComplete (I think this is what may have unstuck it), and Initialize
  2. YAMA App
    • Re-selected my devices, turned on debug logging, and saved app. Just opening, turning on debug logging, and re-saving may have been enough once it was "unstuck" though. It is works you will see all three of these events. If it isn't unstuck you will only see the bottom one:
app:5042022-09-26 10:28:42.866 am debug YAMA app initialization: Complete
app:5042022-09-26 10:28:42.161 am debug YAMA app initialization: Beginning
app:5042022-09-26 10:28:41.127 am debug initialize starting

Now that I've done that, I can't get it back to a broken state again, which I guess is good, but will be hard to fix...

I think the "initialize" is the fix.
I was able to break it again, but Initialize fixed it.
I did:

  1. Disconnect
  2. Connect
  3. Initialize (everything appeared to un-stick)
  4. Then I turned off the debug logging on the driver. (most things got stuck again, but oddly not everything)
  5. Initialize (everything un-stuck again)

It is definitely an app/driver initialization issue.

It breaks every time I reboot the hub too, so the manual intervention is only temporary. I'll look into it some more.

The initial setup/init was always the wonkiest in my code Might be time to actually try and fix it.

YAMA App

  • V0.1.0 - 09/26/22 - Fixed a number of initialization issues.

YAMA Driver

  • V0.2.0 - 09/26/22 - Fixed a number of initialization issues.

@ancker Give these versions a try. Worked much better for me, although there may still be another init issue in the app code I need to chase down. You would need to update both the app and driver code.