If I think about it, without actually looking at facts because that would limit me, there could be a Z* message type like 'here is the new master, route to me now!'. I can imagine ways that could be misused.
SiLabs calls that "Shift"
On the current hub.... 
It is definitely technically possible (in some scenarios with some controllers) as you can backup/restore controllers in zigbee2mqtt without re-pairing anything (I tested it as part of disaster recovery testing). Maybe in the next hubitat hub we'll be able to? I genuinely don't know as I'm not in the hardware beta group (if there is one).
I do know that Hubitat confirmed that it can not be done with zigbee on the current hubs, though. Multiple times.
A discussion of "what if" often will become polarized where one specific fact or rumor dominates one pole while another fact or rumor is the center of the opposite pole.
The ZWave specs have nothing related to redundancy or failover. it seems to be a 1962 concept from my vantage point 
Nothing about the current state of Home Automation was predicted in that spec. Hubs were not even a distant concept. It centered upon handheld controllers... what we think of as button remotes today. The spec offered secondary controllers that seems to have been conceptualized a lot like we'd use wall mounted tablets today. That is, remote control of devices, zero automation beyond that thing called "association" ie. the ability of one device to tell another device it did something.
There's no surprise that no one offers redundancy. There is perhaps a way of creating such a thing with the commands/tools given by the spec, but it's very hard and I doubt it would be complete enough to be a product.
One example is Secondary Controllers. They get a copy of the Primary Controller's ZWave table/DB when joining the mesh. Never to do so again. If the secondary is used to include or exclude a device, then the secondary can update its copy along with sending to the primary. But the Primary will never update the secondary. The secondary can ask for an update, and a lot of the time it works. Clearly the functionality of the command/tool is far behind the need. Therefore, using ZWave commands to do it alone, is a path to errors and unsupportability. One could use the LAN connection between hubs to have the primary inform the secondary of a need to update, and the secondary could then make the request, Error handling would have to be robust, however. Just read the messages of those that have a Secondary tied to SiLab's PC Controller and trying to understand where some newly added devices went. You'll see: "I just Excluded and Included again and the DB is all correct again."
This is just one of the EASY things to write about, that pops to mind. The practical side is that the underlying constructs are not available but that the pieces are tantalizingly close. I think Vera had a mechanism allowing them to chain multiple hubs for distance. That particular problem has been solved completely differently here in Hubitat'verse with HubConnect and HubMesh, and its predecessors. Again, showing that Hubs weren't conceptualized in the initial specs, perhaps even the concept of multiple meshes in a single home wasn't imagined either. 
And that covers ZERO of the issues with Zigbee.
Agreed!
My last post was more around controller shifting/offline restore without re-pairing (which is doable for both zigbee and zwave in some scenarios), not online redundancy or hot failover (which is not doable, to your point).
I probably wasn't clear enough.
This is very much possible - and there are other zigbee coordinators that currently support it. So it would not surprise me if a future Hubitat used a zigbee radio whose IEEE address could be changed by "software".
Edit -
I see @JasonJoel posted a much better description of where this stands!
I will say I don’t know much about the protocols just saying that if Hubitat could have this it would be a huge feature. Not only as feature but also help their sales.
I mean I have a cold spare sitting in a box as a just in case device. If it could be live on the network as an idle spare or somehow complete the mesh network as a second endpoint that would be killer.
Then if primary goes secondary takes over. You replace the broken one and you are back to having a spare. No outage time and things keep working how they should. No setup to monkey around with and not spending hours on a reconfig. Just seamless replacement of one goes out.
Now these things are solid state and pretty reliable but at some point they will fail. Power supply could fail, software or memory fails. Just having that backup already online and ready gives piece of mind and stability.
I mean you could build a special dock that could hold two c7’s. Then this dock would sync and interface. But you would have to have the radio each one used not the dock because then what is the point.
Sorry lost my train of thought but with the special dock thing it could cut the power to the down hub completely when the cloned radio is in use. This would keep the end devices from seeing 2 hubs. So the broken hub drops off the networks. All devices loose the hub signal.
Dock could have a z wave and zigbee client in them to keep an eye on it. Then the cloned c7 comes up and everything thinks the main hub is back.
Or perhaps you always have a cloned hub and the real ID of the radio is never broadcast to the devices ever. This dock thing would tell the C7 to use a specific radio address right from the start. That would be your networks IDs and it would be forever no matter what hardware you are really using. It could just be cloned by software onto the radio chips?
IF this were allowed, I would think it would defeat the purpose of the key ID.
Kinda like the Apple "Secure Enclave". It's not secure the minute you let another machine "assume" its identity, now is it? IF it were available to do, I would NEVER EVER buy a Zigbee "end" device (or whatever the word they use for 'don't get into my house w/o security on the lock')
Apple solves this by when getting a new iPhone the old one must be in physical proximity to the new one, it transfers everything, BUT you must STILL redo fingerprints and re-validate all the CC #'s it transfers.
I don't know how Zigbee could make it any easier than by re-pairing and matching those devices to the old IDs.
That is what I was thinking if there was like a special dock that would link the two units. So they know they are physically connected in this special dock and can exchange some sort of AES key with the dock and then with each other to authenticate. Then they know and can create a secure pair of hubs. This secure pair of hubs would then be linked and if one goes the other steps in, in order to replace one you have to keep like so many parts.
Like you have to sign into your account with at least one hub to have a new dock, otherwise you can only have a new hub using the same dock as you had before. Something like that where there would be authentication between the hubs.
Because really if you can get the key out of the hub and cloned onto another hub you can takeover the network without the end devices knowing. So you currently authenticate with the one hub however you can migrate to a new hub if needed. So why can't we just always sort of be in the virtual state where either hub could be the primary hub?
Agreed.
You couldn't run two primary z-wave controllers or two zigbee coordinators at the same time, but it would be excellent if the end user could regain functionality by just restoring a cloud backup to a new hub.
This is certainly possible already if you have a z-wave-only Hubitat network.
It is already possible for Hubitat with z-wave (even S2-paired device). And for other zigbee coordinators.
It comes down to how secure the radio database back up is. And off course since both z-wave and zigbee are low power protocols, they are proximity limited.
That is what I am thinking that you would have to be in the building on the network and take the original hub off the air to make the devices flip over. So if there was a special hub that could forcibly take the broken unit off air then it could have a new hub placed on air. It could coordinate this handoff and how it would work our the handoff.
Or even having both hubs on the same LAN they could exchange the link connection but if you made them have to fit inside of a special piece of hardware or even register them with Hubitat that could help secure the radio handoff?
Again I am just bringing this up as a way to bring redundancy to these type of things...
I LOVE THIS!
Please make this work! I had it going with two macs running Indigodomo, so one unit's rules were ALWAYS predicated on the other unit being down (ie: not answering packet calls on the network) then it would take over.
Easy when you can pair the computers to a NETWORK zwave switch. Cos the network zawve just talks IP. But there aren't many of those out there.
That is interesting to think if they were to make it somehow act like a network zwave hub.
I was thinking if this is possible to make a virtual hub and then this virtual hub can be hosted by either C7 at any moment. Just something needs to sync both of them so that both aren't on air at the same time. Also something needs to keep them in sync with software and rules and all that. If one vendor owns all the hubs and has all the redundancy that is huge.
If there was like a multi-hub dock where you drop 2 C7's in and they fit in half way or something and that passes through power. You wouldn't want the same power to both because then you have a single point of failure again. Also you don't want a single network interface you want them to pass through each. But this dock could authenticate both.
Better yet make a dock that is the sync device that connects via z-wave, zigbee, and ethernet. Or better yet wifi/bluetooth and run on batteries. That way if there is a power surge or network surge the device is isolated. Can't get hit by lightning or anything like that too. Then it can verify that it local and can see and monitor all the radios. It can report status back to both units and be the controller of controllers.
All that engineering and (shudder) testing to sell more $150 boxes? I kind of remember block diagrams of Matter hubs coordinating so maybe it's builtin there. More likely to happen there too.
Cool Idea, but I wonder How did we get this far in the conversation and no one has brought up how the drivers and apps are going to maintain/synchronize state and other internal data in realtime?
Hot spare controller/radios is one thing, but a functional system in this setup probably needs the apps and driver code to work correctly too.... it maybe possible, but could be a can of worms, quickly, I think.
I do this with main wifi private ip router both configured identically. But found you also need to reboot all switches because they wont route packets correctly if the mac of the default gw changes.
I.have an at&t access pt with kasa switches behind both routers and any network switches and can swap over remotely.
I also have something similar with my primary public ip router .. i can turn off the primary one and enable secondary .
Ie
Yeah that is kinda what I am thinking of is if there is a second hub completely configured and running identical to the first hub and if the first one goes offline then disable the power to it and enable radio on hub 2.
There would need to be some sort of network monitor device that would sync with both hubs but it would just need a zwave and zigbee client. Then when that device sees something break it would signal the change to take place. If hub 1 is dead then just cut the power to it, if it is on but radios aren't working then tell it to radio down. Wait a minute or two and then tell hub 2 to radio up in cloned state.
If both hubs are running the same apps, same software, same everything then it could just take over seamlessly except for the 1 minute waiting period. While the hub 1 is trying to recover or factory reset then sync again. If it came radio up after factory reset it wouldn't matter because the entire network is running on the cloned network IDs.
After hub 1 either comes back or gets replaced then it can join the network again and be the secondary and just leave hub 2 as primary. Or alert the user and say hey this hub went down but it is now back up do you want to switch back over? If you do then there will be a 1 minute outage and let it do the reverse switch back.
Just an idea I don't know how possible this is but it would be very nice to have redundancy built into the system.
