Want to mention:
On the Android App a Tile representing a Leviton Fan controller, the fan blade icon is grey showing the fan is off. But the text of the on off state is always on. When the tile is open, the fan speed reports correctly that the fan is off when it is off and on when it is on. Seems to me this is a minor mistake in the coding for the Tile or perhaps it is in the driver.
This must be an EZ dashboard tile, as legacy fan tiles do not show text for state.
Still, I cannot reproduce what you are seeing. I made an EZ dashboard to test, and put a fan on it, but I never get an "on" state for the fan, it is either "off" or it displays the speed. There is no "on" text that I can make appear no matter what, and it always says off when it is off. I am using the GE Zwave fan controllers.
Check what the state in the device commands page is saying for speed when that happens, as the tile is most likely just putting on the dashboard what the device is reporting, which is a driver issue, not a dashboard tile issue.
Edit: My guess is the driver is leaving speed saying "on" when the fan switch is saying off. What are you currently using for the driver?
There's a bug on the ON/OFF for Android, it will be fixed on the next release.