[RELEASE] Zigbee2MQTT Routing Driver

⚙️ Custom Apps and Drivers Custom Drivers
1

There's no denying that IKEA and Xiaomi make some enticing devices, but for whatever reasons may exist, the fact remains that they're unsupported on our platform of choice and incredibly difficult to reliably integrate with Hubitat directly, even with custom drivers. Others and myself have tried and burned many hours in the process.

Backstory

Until fairly recently I had no idea what MQTT was, or why I'd want to use it. That changed with my Hildebrand Glow driver for UK smart metering, which relies on it entirely. Once I understood that MQTT is essentially just a very quick and lightweight network messaging system and that support is already built-in to Hubitat, what Zigbee2MQTT might mean became painfully clear. It's right there in the name. Take in Zigbee radio messages, send them out via MQTT to your network.

My previous solution was to run an extra Hubitat hub just to keep my Xiaomi buttons happy on their own Zigbee mesh. Sadly, though these pernickity devices liked the C5 and C7, they didn't like my new C8 and harmony in the household was diminishing rapidly.

Now that I knew what Zigbee2MQTT was for I searched for the community app which was sure to be out there, and was surprised that what is quite a simple integration didn't already exist. Necessity being the mother of invention, here's my take on a Zigbee2MQTT Routing Driver.

Preflight

Before installing the routing driver you will need a working Zigbee2MQTT system. I'm using a Raspberry Pi 3 and a Sonoff Zigbee 3.0 USB Dongle-P.

Even if you have a working system, check the "How Do I Set Up Zigbee2MQTT" section below. There's one very important setting required.

Installation

Search for the keyword "general" on Hubitat Package Manager and you should see "General Drivers from BirdsLikeWires". Requires HPM v1.8.7 or later. Or install manually using the links below.

If you have any HPM installation issues please go to:

Apps > Hubitat Package Manager > Package Manager Settings > Remove a Matched Package

Choose my "General Drivers" package, unmatch the package (which doesn't uninstall it) and then link it back up again:

Apps > Hubitat Package Manager > Match Up

Configuration

After installing the driver, on your Hubitat system:

  1. Click "Devices" on the left navigation bar.
  2. Choose "Add Device" in the top-right.
  3. Click "Virtual".
  4. Name the virtual device whatever you like, it'll get changed. :slight_smile:
  5. Choose "Zigbee2MQTT" in the "Type" dropdown.
  6. Click "Save".
  7. Click "Devices" on the left navigation bar again, scroll down until you find "Zigbee2MQTT" and give it a click.
  8. Enter your MQTT broker IP address and, optionally, the username and password for it.
  9. Click "Save Preferences".

Supported Devices

You will need to install the matching IKEA or Xiaomi device drivers, which are used by the child devices to interpret the MQTT messages and convert them into Hubitat events. Some of these are combination drivers, supporting a direct Zigbee connection (if that's working for you) and MQTT messaging in one.

IKEA

  • E1744 Symfonisk Sound Controller
  • E1766 Open/Close Remote
  • E1812 Shortcut Button

Xiaomi

  • WXKG06LM / WXKG07LM Wireless Remote Switch
  • WXKG11LM / WXKG12LM Wireless Mini Switch

How Does This Work?

The Zigbee2MQTT driver connects to the MQTT broker, to which your Zigbee2MQTT installation is publishing messages. It watches out for device models it recognises and automatically generates or updates a child device in Hubitat with the correct driver and passes on the MQTT message. You can use that child device just as you would any normal device.

How Do I Setup Zigbee2MQTT?

There are great instructions on the Zigbee2MQTT website. Essentially, you need some sort of computer you can leave on all of the time and a Zigbee dongle. Any Raspberry Pi, even the oldest, would be more than capable.

One important thing! Under Settings > MQTT you must tick the "Include device information" option near the bottom of the page. Otherwise... well, there's not enough device information for things to work. :slight_smile:

image
image960×916 72.4 KB

I've now been running this since March 2023 and it has been almost perfect (completely flawless since April). I even moved the installation from a Mac mini to a Raspberry Pi and it continued to work without a hitch.

It's honestly amazing getting these working with Hubitat and I hope this helps others out too.

13 Likes
2

Saved for the future.

3

Great idea, @birdslikewires !

Count me in.

3 Likes
4

First successful connection: :+1:

dev:302023-03-14 21:27:11.709debugZigbee2MQTT : Payload : {"action":"single_left","battery":100,"click_mode":null,"device_temperature":25,"linkquality":0,"operation_mode":null,"power_outage_count":107,"state":"OFF","voltage":3148}

dev:302023-03-14 21:27:11.705debugZigbee2MQTT : Topic : zigbee2mqtt/WXKG15LM

dev:302023-03-14 21:27:11.697debugparse: topic: emlnYmVlMm1xdHQvV1hLRzE1TE0=, payload: eyJhY3Rpb24iOiJzaW5nbGVfbGVmdCIsImJhdHRlcnkiOjEwMCwiY2xpY2tfbW9kZSI6bnVsbCwiZGV2aWNlX3RlbXBlcmF0dXJlIjoyNSwibGlua3F1YWxpdHkiOjAsIm9wZXJhdGlvbl9tb2RlIjpudWxsLCJwb3dlcl9vdXRhZ2VfY291bnQiOjEwNywic3RhdGUiOiJPRkYiLCJ2b2x0YWdlIjozMTQ4fQ==

dev:302023-03-14 21:27:11.690traceZigbee2MQTT : parse : Bridge message received.

dev:302023-03-14 21:27:11.686debugparse: topic: emlnYmVlMm1xdHQvYnJpZGdlL2xvZ2dpbmc=, payload: eyJsZXZlbCI6ImluZm8iLCJtZXNzYWdlIjoiTVFUVCBwdWJsaXNoOiB0b3BpYyAnemlnYmVlMm1xdHQvV1hLRzE1TE0nLCBwYXlsb2FkICd7XCJhY3Rpb25cIjpcInNpbmdsZV9sZWZ0XCIsXCJiYXR0ZXJ5XCI6MTAwLFwiY2xpY2tfbW9kZVwiOm51bGwsXCJkZXZpY2VfdGVtcGVyYXR1cmVcIjoyNSxcImxpbmtxdWFsaXR5XCI6MCxcIm9wZXJhdGlvbl9tb2RlXCI6bnVsbCxcInBvd2VyX291dGFnZV9jb3VudFwiOjEwNyxcInN0YXRlXCI6XCJPRkZcIixcInZvbHRhZ2VcIjozMTQ4fScifQ==

dev:302023-03-14 21:27:11.677traceZigbee2MQTT : parse : Bridge message received.

dev:302023-03-14 21:27:11.673debugparse: topic: emlnYmVlMm1xdHQvYnJpZGdlL2xvZ2dpbmc=, payload: eyJsZXZlbCI6ImRlYnVnIiwibWVzc2FnZSI6IlJlY2VpdmVkIFppZ2JlZSBtZXNzYWdlIGZyb20gJ1dYS0cxNUxNJywgdHlwZSAnYXR0cmlidXRlUmVwb3J0JywgY2x1c3RlciAnZ2VuTXVsdGlzdGF0ZUlucHV0JywgZGF0YSAne1wicHJlc2VudFZhbHVlXCI6MX0nIGZyb20gZW5kcG9pbnQgMSB3aXRoIGdyb3VwSUQgMCJ9

dev:302023-03-14 21:27:10.025traceZigbee2MQTT : mqttConnect : Connection to broker 192.168.0.159 (zigbee2mqtt/#) is live.

dev:302023-03-14 21:27:10.021tracemqttConnect: hubitat.helper.interfaces.Mqtt@1c30e87

dev:302023-03-14 21:27:09.016traceZigbee2MQTT : parse : Bridge message received.

dev:302023-03-14 21:27:09.011debugparse: topic: emlnYmVlMm1xdHQvYnJpZGdlL2xvZ2dpbmc=, payload: eyJsZXZlbCI6ImRlYnVnIiwibWVzc2FnZSI6IlNhdmluZyBzdGF0ZSB0byBmaWxlIC9jb25maWcvemlnYmVlMm1xdHQvc3RhdGUuanNvbiJ9

dev:302023-03-14 21:27:00.174traceZigbee2MQTT : mqttConnect : Connection to broker 192.168.0.159 (zigbee2mqtt/#) is live.

1 Like
5

Getting some interesting topic and payload data there! The first message is the one I'd expect, then it's all base64 encoded for some reason. Do you have something specific set to make it do that?

Oh, or is this... oh, ignore me, it's coming from hubitat.helper.interfaces.Mqtt in trace mode.

Ooh, a WXKG15LM! Posh. :wink:

If you can send me the payloads for each of the button presses I can probably add in support for it. It only knows '06 and '07 wireless remote switches at the moment.

1 Like
6

No.. except that I created a username and password for MQTT in my HA, but probably the Mosquitto broker is not configured properly.. I have not used it for more than an year and I have forgot everything :slight_smile:

7

Haha, I've not even tested this with a password protected MQTT feed.

Thanks for being the guinea pig! :sweat_smile:

1 Like
8

I couldn't make it connect without an username/password, I was receiving

[dev:30](http://192.168.0.183/logs#)2023-03-14 21:17:50.664[error](http://192.168.0.183/logs#)org.eclipse.paho.client.mqttv3.MqttSecurityException: Not authorized to connect (method mqttConnect)

Something is wrong with my MQTT broker, I will continue tomorrow.

Update: it was the Zigbee2MQTT Settings->MQTT->"Include device information" option that was not checked.

1 Like
9

Awesome, it was doing it right in that case. :+1:

1 Like
10

Great work!

Just to be clear to newer users and those unfamiliar with zigbee2mqtt, you will need additional hardware per @birdslikewires 2nd comment, not just a radio/dongle.

@birdslikewires is it possible to provide some instructions to non groovy programmers (like myself) as to how to add other devices? I haven't looked at the library bundle but can more attributes be added?

3 Likes
11

Aye, I'll tidy the first post up to clarify at some point. :+1:

I've not even explained how to add the device (create a virtual device and select Zigbee2MQTT as the 'Type' if anybody does get to this bit).

Non-programmers, or non-Groovy-programmers? :wink:

Non-Programmers

If you have an IKEA or Xiaomi device you'd like adding, once you've got everything set up and the device paired to the Zigbee2MQTT hardware:

  1. Go to the Zigbee2MQTT device and turn on "Enable debug logging".
  2. Click on "Logs" in the left panel.

Operate each action that the device can make and note the "Payload" messages being received. For each action, copy the payload message into a text editor, one line for each and when you have them all, paste them in a message here (using ``` tags at the start and end of the messages to keep the info readable).

Devices which can themselves be controlled are not something I'm working on right now, but it's possible in the future. And there's no way I can write drivers for every device Zigbee2MQTT can support - in that case you'll need...

Programmers

Create a driver in the usual manner with the correct attributes, commands and methods to support it. You'll then need a processMQTT method which maps the incoming actions to Hubitat events. That's really about it.

I'll throw together a template driver when I get chance.

Note!

Pairing directly to Hubitat without this should be the primary aim, so always try that first. This was really created for those well-known-to-be-troublesome devices from IKEA and Xiaomi. :slight_smile:

6 Likes
12

Not sure what this means:

image
image1054×862 30.7 KB

13

Did you install manually or via HPM? If manually, did you install the library bundle as shown in the first post?

15

@birdslikewires - all is working OK with my Button Aqara WXKG12LM - a big thank you! :slight_smile: :+1:

image
image1595×599 45.6 KB

1 Like
16

You first need to install the library under "Libraries code" on the navigation bar, then install the driver in the usual "Drivers code" page. The instructions for that need to be clearer in the first post.

1 Like
17

Can you walk us through this a bit more as I'm still missing the "painfully clear" part.

We have on one hand a piece of hardware (HE) with a zigbee radio and some software (HE OS?) to manage it. On the other hand, another piece of hardware (say, a Pi), a zigbee radio in the form of a dongle and some software (zigbee2mqtt). I fail to understand what are the material differences between the two up to this point (of course HE goes much further up the stack).

  • Why would zigbee2mqtt be somehow better at talking to IKEA and Xiaomi/Aqara devices than HE?
  • Why can't whatever zigbee2mqtt is doing to be better at talking to them be implemented in HE OS ? (I am guessing that whatever it is can't be done in user code, as you and others have tried)

These are probably stupid/obvious questions, apologies for that, but there is so much to learn in this space, it is difficult to fully grok all the subtleties.

I'm interested because I own several Aqara devices and would like to be able to get some of those cheap IKEA devices, but I am way less interested in having to manage yet another Tamagotchi in my house... (one of HE's main selling points for me, as it hits the sweet spot 99% of the time)

1 Like
18

@birdslikewires,
This is great stuff, really good to see the community providing new ways to support devices!

I have Zigbee2MQTT setup on Raspberry Pi running Home Assistant for a few devices, and I was able to set this up, but these drivers are not supported for the reasons as you've explained above.

I wanted to get rid of Home Assistant, but seems I can’t at present.

Anyway, just wanted to ask, what is the difference or benefits in using this method instead of the Home Assistant Device Bridge?

Thanks again.

Virtual device

image
image940×511 45 KB

Some logs

dev:17212023-03-15 11:47:25.771 AMwarnZigbee2MQTT : No known driver for the RTCGQ11LM from LUMI.

dev:17212023-03-15 11:47:25.767 AMdebugZigbee2MQTT : Payload : {"battery":100,"device":{"applicationVersion":5,"dateCode":"20170627","friendlyName":"Aqara-Kitchen sensor-t620","hardwareVersion":1,"ieeeAddr":"0x00158d0007f81e2f","manufacturerID":4151,"manufacturerName":"LUMI","model":"RTCGQ11LM","networkAddress":3059,"powerSource":"Battery","stackVersion":2,"type":"EndDevice","zclVersion":1},"device_temperature":25,"illuminance":78,"illuminance_lux":78,"linkquality":138,"occupancy":true,"power_outage_count":235,"voltage":3025}

dev:17212023-03-15 11:47:25.765 AMdebugZigbee2MQTT : Topic : zigbee2mqtt/Aqara-Kitchen sensor-t620

dev:17212023-03-15 11:49:58.402 AMwarnZigbee2MQTT : No known driver for the TS011F_plug_3 from _TZ3000_okaz9tjs.

dev:17212023-03-15 11:49:58.396 AMdebugZigbee2MQTT : Payload : {"child_lock":"UNLOCK","current":0.08,"device":{"applicationVersion":160,"dateCode":"20210625","friendlyName":"Ajax plug","hardwareVersion":3,"ieeeAddr":"0x70b3d52b60011814","manufacturerID":4660,"manufacturerName":"_TZ3000_okaz9tjs","model":"TS011F_plug_3","networkAddress":6301,"powerSource":"Mains (single phase)","softwareBuildID":"20B+TZSKT11BS105","stackVersion":1,"type":"Router","zclVersion":2},"energy":2.39,"indicator_mode":null,"linkquality":211,"power":7,"power_outage_memory":null,"state":"ON","update":{"installed_version":-1,"latest_version":-1,"state":null},"update_available":null,"voltage":244}

2 Likes
19

Most of the Ikea buttons aren't directly supported by HE, so alternate methods of using them with HE are required.

20

Here I have a very easy answer: zigbee2MQTT is an open-source community project with at least several hundred of active contributors, that reverse-engineer non-standard (and not documented anywhere!) Zigbee devices from Ikea, Aqara, Tuya, and many other brands. This is probably 100 times more than the developers who make drivers for Hubitat. It is practically not possible to catch up with the new device manufacturers that pop up every month from China.

4 Likes
21

I get that, but it begs my second question:

From your answer, I infer that "catching up" cannot be done in user code on HE. Correct ? I can accept that, just curious as to why.