If you assign a domain name to HE and access it via TLS, the icons in the dashboards app reference the non-TLS page at its regular IP address instead of the address and protocol you used to log in. Since this is a security violation, modern browsers won't allow you to click on the icons to open your dashboard. Workaround is to use cloud dashboards.
And...I don't know hubitat that well yet, nor do I know the app architecture as I haven't begun writing my own apps yet, but this is potentially a wider (but minor) security vulnerability, depending on the design philosophy of what the apps are supposed to be capable of doing: It may be possible for an app to load content (and fillable forms) from external websites, while fooling the user into believing that they're still only interacting with their local hubitat on their local network, complete with the browser navigation bar still showing the user's intended URL.
If that is working as intended, (i.e. we always work under the assumption that you only install apps that you have absolute trust in) then there's no problem
It is not recommended that you access the dashboards in that way but instead access them through the cloud system that is built into the Hubitat system. Exposing your hub to the internet in such a way could expose the entire system to intrusion. You are doing something that was intended to ever be done so I'm not surprised it doesn't work. I'd recommend trying to learn more about how Hubitat is designed to work through some of the tutorials on Hubitat.com or through a more thorough reading of the forum.
I too have A records in my DNS and I use a local DNS domain for my home (no .com .biz .org. etc.) for example:
hubitat.mydomain
It works every time with dashboards and the full UI. I have used DNSmasq many times and now I am using a Pi.hole for my DNS server and it's running a custom version of DNSmasq. Also unbound is a great DNS server.
I also use dnsmasq for dhcp and DNS and have no issues accessing the hub UI and dashboards. Everything else in your OP is above my head so I'm afraid I would be of little assistance but wanted to add that for the average user, DNS works fine with HE.
While I'm more than capable of managing internal https/certs/keys on my home network, I don't any time it isn't explicitly necessary.
Why?
I would rather spend time with my kids than hours of network management every year. I simply don't see that level of security on my home network as value added.
Hmm...either Safari considers excepted self-signed certificates to be secure, or you've added the site to some kind of trust zone (or you've figured out how to load your own CA certs to hubitat, and if so, mind sharing how?) That does leave me wondering if Safari won't automatically prevent loading mixed content (IE doesn't, but nobody really uses it anymore.) At any rate, this is what I'm seeing on my end in the Chrome debug console:
Firefox, which is my daily driver, exhibits the exact same behavior (I just use chrome solely to debug pages because I don't install any addons on it.)
Some people build model trains, some people grow an elaborate garden, and some people build mini enterprise grade IT infrastructures It's just for fun in my case. That's often the same reason why people write their own apps and publish them.
Odd, I wonder why mine is doing this. I initially set it up over non-TLS with just an IP, I wonder if it stores that information into a configuration file somewhere when you first set it up and then it keeps using that information to make self references?
EDIT: When you access yours over plain HTTP and click one of those, does it point back to the HTTPS version?
No. And to be clear - the first time I accessed the hub using https in Safari was today. I do routinely use https in curl - for nightly backups and other functions.
It's just for fun in my case. That's often the same reason why people write their own apps and publish them.
