Is the .lzf file format documented anywhere? I download the hub backup files each day and have wondered whether I could read and parse the data to extract, for example, rules.
Believe the short answer to that is No.
3 Likes
Correct. And if you want a long answer, read this topic. 
7 Likes
Its a compressed file
LZF format, written by Tatu Saloranta ([tatu.saloranta@iki.fi]
See LZF Format Specification for full description.
lzo and lzf are 2 well known very simple compression algorithms. lzf goes for low memory usage during compression. lzo goes for maximum decoding speed. Both are fast, both have little memory requirements, both have comparable compression rates (which means very poor).
LZF algorithm itself is optimized for speed, with somewhat more modest compression. Compared to the standard Deflate (algorithm gzip uses) LZF can be 5-6 times as fast to compress, and twice as fast to decompress. Compression rate is lower since no Huffman-encoding is used after lempel-ziv substring elimination.
PS, google can be your friend sometimes….
Correct. But the H2 database itself is encrypted. So decompression is insufficient to gain access to the backup database ....
3 Likes
Just read the 'rants' in the link. I do understand the issue with allowing someone to edit a running system and corrupt it. I have done that with my own code and it becomes tiresome. My question was much more benign. I wanted to simply parse the code and format an output that would allow me to look at the system in a more unified manner.
I, a while ago, use a system called HAI. It was a company in New Orleans that had a very extensive system. They were subsequently purchased and destroyed bu Leviton. The code way written largely with their user interface, much as Hubitat is. They did have, however, a print function that would dump the system in a pdf file. It was complete but poorly formatted. I was able to parse that file and generate a very readable data file.
Perhaps Hubitat could generate something like that in the future.
1 Like
There is an export function for apps on the App Status page if that is of interest. These files are not intended to be edited, and the format is not documented (it's vaguely JSON but not always exactly), but it is plain-text and easily accessible and sounds like it's a closer match to how you were able to scrape data from your old system.
(Their real intent is to allow you to import them later, whether to a different hub or restoring them to the same hub/database. It's also the basis of how app cloning works.)
But depending on your actual needs, there could be other ways.
It should be noted that Hubitat's app interface lets you create automations (this is the role of apps on Hubitat, or at least the main role), but isn't writing code per se; the apps themselves are written in code and simply present an interface for you to configure the app and do whatever it was written to do. This is even true for Rule Machine, no matter how much the rule actions summary may look like code.
If you do want to write code, there is a development environment for apps and drivers: Developer Documentation | Hubitat Documentation. But this would only really get you a better idea of what the app is doing (you wrote it, after all), not how it's configured (that's still on the UI side).
2 Likes
FWIW - you can export and save your automations.
2 Likes
Here is what i found when trying to deflate the file using some LZF tooling
2.3.4.134.lzf is a PRJ (Clarion Project) or MIF (Quartus Memory Initialization File (with rem)) file. You could rename the file extension from LZF to PRJ or MIF and then doubleclick it (or click here to learn more about that file type)
The content should be mainly XML based on what i could fond so far
Using Windows 10, I’ve created HE7 backups (Settings>Backup and Restore) using Firefox and Chome, via PowerShell 5.1 Invoke-WebRequest –output and Invoke-WebRequest saving the iostream using file.Write. All four methods produce files than differ by 4KB-12KB. All methods run consecutively within seconds. The latter two are to automate the backups. Is there a method to verify the integrity of a backup file (.lzf) without restoring to a hub?
Not to my knowledge.
Some of that will depend on the log file size and database size which will vary over time as records are added and removed...
2 Likes
The files should be the same or very similar in size if the backup files are created and downloaded with ~20 seconds.
The difference casts doubt on whether the files valid. I’ll run the automated backups, but will first trust the files created via the web interface should I need to restore.
The files are valid. This method has been in use for years with no known failures having anything to do with the lzf file. It's not that complex, but the database is constantly changing, whether you are aware of it or not.
2 Likes
I have another perfectly valid reason for wanting to access my backup files: I have some Drivers and Apps code that I need to extract. These backups are for a hub I no longer have access to so I can't just log in to extract the code. Yes I can import this backup into a 2nd hub to extract the data, but I don't have any spare hubs I can sacrifice for that purpose and I shouldn't have to do that...
These backups are backups of my data, from a device I own. It seems very user hostile to me that we are locked out of accessing our own data. I cannot think of any other device or software product where users are prohibited from accessing the backups of their own data. The whole point of a backup is to give users the ability to retrieve important data in the event of some sort of failure.
If you just don't want people modifying the backups to tamper with a running system, there are better ways of accomplishing that. I've personally worked with software products that do exactly this. (I work in software) For example: Embed into the database a field with a hash of the database file itself (Salt the hash with your own secret key to ensure the user can't regenerate the hash themselves). When importing: Hash the database file and compare to the stored hash. if different: Someone tampered with the file so refuse to import.
Using encryption for this is just a really lazy hack and not a terribly good idea from a security practice standpoint. Someone who's motivated enough can crack the encryption. I'm guessing the key isn't unique to the hub, because then the backups wouldn't be portable between hubs. It's also not a key that can be easily rotated. Which means every database likely uses the same encryption key. If one motivated person cracks the key and it gets shared online, then everyone's keys are compromised.
So if you are going to use encryption it should at the very least be encrypted using a key provided by the user so that:
- If one key gets compromised, it doesn't affect anyone else.
- And secondly: Since the user provides their own key they would be able to decrypt and view their own database data. (Again, you can use a hash or other techniques to prevent users from modifying it and importing a modified database)
1 Like
While I can't speak to anything else for certain, it seems like the kind of thing that is unlikely to change any time soon. So, working with what we can do:
It should be noted that a soft reset is a safe operation to perform on your hub. So, you can create a backup on one hub, download it (just to be safe, even though it will also be stored onboard), restore this other backup you have, get what you need off of it, then restore the backup you just made back to this hub before doing all of this. Then, aside from a few minutes of downtime on whatever hub you do this with, all will be as before afterwards.
(Yeah, I said "soft reset," but you didn't actually have to perform one; it does one as part of the restore process automatically, but it's the same idea. Just don't reset any radios, etc. — those are not part of local backups, and you have to jump through some hoops to do so, so it's not something you'd do accidentally, but I should mention it just in case...)
2 Likes
What you're basically complaining about isn't that you are locked out of accessing your own data (since you aren't), but that we don't offer tools to allow you to extract portions of a backup other than restoring the backup to a hub. While in theory such tools could be created, it certainly would not be a priority for us, as there are numerous other means to accomplish having access to apps and drivers code that you create. I don't get that this is "user hostile" at all, but you're welcome to your opinion.
2 Likes
Given we're a closed source commercial enterprise here, why would we allow a user to export our database thus exposing hubitat's schema to the world at large?
1 Like
