I am using a LAN device that pushes XML data to HE. The device has an option to prepend the XML data with a header. Here's a WireShark screenshot of the data transmitted by the device with and without the header. Note there is also a "trailer" if the header is present...
I've tested with both scenarios. If the header is present, my device driver's parse() function is driven and the inbound message contains all the data (header, XML, and trailer). So before processing the XML doc, the header and trailer must be stripped so that the results are identical to the data transmitted with the header suppressed. Thanks to @tomw for providing the code to strip off the header/trailer.
If the header is suppressed, my driver's parse() function is not driven. I don't see any error messages in the log.
Is this the expected result in both scenarios? Is HE examining the header to determine what driver to use, or is it looking at some lower-level TCP data? I opened a couple of threads requesting help to strip off the header/trailer and got a solution, but I'm mostly wondering about what happens when the header is suppressed? Do all LAN devices post messages with a similar header?
Does the DNI of your device match the MAC address of the sending device?
My Sonos Advanced app gets a metric ton of XML sent to the device drivers, and it’s just the plain XML, not wrapped up in a header and trailer like yours.
If the MAC matches then the driver gets the traffic to its parse method. You can use IP too, but MAC is better.
You should see an entry in the HE logs page when the hub can't find the intended device, i.e. one with a Device Network Id that matches the MAC or IP. If you see the warning message in the log it indicates the DNI values it has searched for. E.g.:
Port number in the title has been corrected. The LAN device is always sending to port 39501.
I read the doc (several times).
@daniel.winks The DNI is set to the TCP/IP address in hex, without the dots.
@sburke781 For grins, I edited the DNI to make it incorrect. If the LAN device transmits the header with the XML, I get the error message about "no matching device found...".
I changed the DNI to the MAC address (without the ":"s), and tried again. If the header is transmitted with the XML, it works as expected. If the header is suppressed, my parse function isn't driven and there isn't anything in the log.
To summarize...
If the DNI is a correct IP address or correct MAC address, and the header is transmitted with the XML, the driver's parse function is driven.
If the DNI is not a correct IP address or correct MAC address and the header is transmitted, I get the "no matching device found..." log error message.
Regardless of the DNI value (correct IP address, correct MAC address, incorrect IP address, incorrect MAC address, or junk), if the header is not transmitted, the driver's parse function is not driven and there is no message on the log saying what happened.
When you are sending the data, is it being sent as part of a HTTP request, or some other method. I could be wrong, but my understanding is that port 39501 is setup to handle incoming HTTP requests. When you are sending your data, what protocol are you attempting to use?
@sburke781 The LAN device is a OW-Server 1-wire to Ethernet Server. There's already a community driver for it, but it requires a polling rule (to do httpGet requests to get the XML) instead of utilizing the device's PUSH capabilities. I'm trying to implement the PUSH support as a learning experience.
Here's some info from the device documentation...
POST Client This feature automatically, on a timed basis, sends the XML file “details.xml” to a HTTP server. Its primary purpose is to get data that’s behind a firewall to a server residing on the Internet. By using the standard HTTP POST originating from behind the firewall, firewalls and other Internet filters see the communication as a standard browser request and typically will not block it.
Also according to the device doc, the XML sent by the LAN device is the same for httpGet replies and POSTs to a client.
Then I'm just as baffled as you.... Might be one for @gopher.ny to comment on....?
The reason I was asking about the protocol was I assumed the xml was being sent in some kind of raw format that only made sense to HE when the header text was included, but from what they explain on the product page at least I would not expect it to be any different to the data sent by my EcoWitt weather station. The only other thing that comes to mind is there could be some requirement for a header element not present in the straight xml version, like a content-type, etc.
Am I right in thinking you are just wanting to understand why this is the case, as well as whether you actually need to include the header stripping code? So for now you could include the header and strip off the header in your driver?
@sburke781 It's mostly curiosity. I would like a better understanding of what goes on underneath the covers. @tomw gave me some code to strip the header/trailer off, and I'm getting the XML data (with header) in my driver's parse function and can process it, so this isn't holding up anything.
If receiving the data without the header/trailer is easy enough to implement, I will probably change my code to work that way just to simplify it a bit. The less the code has to do to the data, the less chance there is for a bug or future problem if the sender changes the header in a future release. The code to strip the header/trailer is looking for certain character strings in the header/trailer to determine what to strip, so a change in the header data could cause a problem.
My parse only needs to do this to handle it having the headers:
Helper function so @CompileStatic works:
It looks like your app is stuffing what should be the HTTP body into a header field, and when you turn that option off it's sending TCP instead of HTTP. I suspect that TCP might not work for port 39501. HTTP definitely does, and it works for ALL HTTP, since the messages my app gets are HTTP NOTIFY rather than POST.
The HTTP POST capability allows the OW-SERVER to transmit sensor data at a specified rate to a specified URL. This simplifies collecting sensor data when the OW-SERVER is deployed behind a firewall.
Have you tried up a mapping in your app, and give the device that URL? If you can specify a URL to get POSTs, use that, it's easier. Port 39501 is really for crap that doesn't let you do that, like Sonos speakers, which let you specify a URL to have it send its messages to, but it sends them via HTTP NOTIFY and not POST. App mappings allow GET, PUT, POST, and DELETE only. But since your device supports POST, I'd use an app mapping before jumping to port 39501.
This sounds something like what I was suspecting, TCP vs HTTP....
I would expect an app would still expect an HTTP request to map to an endpoint, so not sure an app would solve the issue if it is a TCP vs HTTP issue.....
Given @user2164 has a working solution, probably worth waiting for more info from the HE elves as to what may or may not be happening. Unless there is a need for alternative comm's, probably not worth digging deeper at this stage, imo...
Honestly, looking at the device online... it 'pushes on a schedule', so this whole exercise is pretty moot, IMO. Polling is every bit as effective. The device isn't sending based on events received. It's just a scheduled push which is not going to be any more timely than a scheduled pull.
I'd have to see exactly what parse() is getting, but it looks like @user2164 could just split by '<' and take the tail(), then split by '>' and take [0] to get the XML itself.
I wonder why it's sending multipart in the first place instead of just XML with headers.
Since it's not holding anything up, I'm going to wait to see what HE staff says.
@daniel.winks I realize I could use polling instead of letting the LAN device push. That's how the existing community driver works. I'm mostly working on this as a learning experience. The port 39501 and parse() stuff is one of the topics documented in the new "Building a LAN or Cloud driver" suggests that it is a recommended design approach.
I'm concerned that there's no indication that HE ignored/discarded the incoming data. It seems like there should be some indication of that occurring (or maybe there is but I can't find it, which is also a concern). Maybe they don't want to flood the logs if there are a lot of ignored data streams detected, but there are ways around that, e.g., counters of occurrences. I'm not a TCP/IP expert, but I'm guessing the identity of the source IP address somewhere other than in the header sent by this LAN device. Having ignored data streams documented would possibly help in debugging a problem, e.g., "Inbound TCP data stream from 192.168.x.x cannot be processed", similar to the log message previously mentioned.
Everything in TCP contains both the source and destination info:
That's a typical TCP packet.
I'd imagine it'd be possible to send unsolicited TCP sent to Hubitat to a specific device driver based on the source IP in the TCP packet. I think the the issue would be how to handle it, considering HTTP already does get set to the driver, and HTTP is just a layer on top of TCP and every HTTP request received is also TCP received (well, I guess it could be UDP or some other protocol, but I've never seen HTTP over anything other than TCP).
I guess they could just do port 39502 or whatever for raw TCP.
@gopher.ny To be clear, I'm not asking for HE to support the LAN device's "header-less" XML data stream. I'm just asking for insight on what is going on when this happens, and why there's no indication that a data stream was discarded.
