Simple accurate multi cell presence sensing BASH script

This is my first significant post to the community having ported from ST just a couple weeks ago.

Below is a simple BASH script and setup for presence sensing using cell phones and home WiFi. Yes I know the cell phones WiFi sleeps but this script deals with that as well as event noise on signal boundaries... at least on Android devices.

I am big on privacy so I never implemented a third party presence sensing solution on ST and was planning to implement on HE once ported. I expected to run into the same issues as everyone else when trying to use cell phones as a presence sensing solution so I was planning to use a multi mode solution as others have done, but would avoid the third party options like Life360 and Amazon geo-fencing. However, to my surprise it was relatively easy to implement a rock solid cell phone presence sensing solution, the details follow.

My network topology and setup do provide me an advantage but this should work for other setups with some tweaking. I have four WiFi access points at my residence (all with 2.4Ghz and 5Ghz transmitters). Three for the main WiFi and one for guest use. The main WiFi reaches into the street on all sides of the house while the guest access point is located below grade and limits connectivity to within a few feet of the house (done on purpose for security). Cell Phones are Android Pixels and set to use use non random MAC address for the home SSID and therefore get a static DHCP IP address when connecting.

So after logging and mapping my WiFi response times at the signal boundaries and each cell phones particular WiFi sleep pattern (it differs between models), I determined I do not need a multi mode setup. However, I did setup and test two methods, each worked well with no third party solutions involved. I will outline one below and the second more complicated one in a different post.

With any sensor system you need to consider the noise that will be present with the sensing event. In the first case the sensing event is the connecting and disconnecting from a WiFi network. Both can be quite noisy at the edges of signal boundaries. Both need to be dealt with individually as their "noise" is different. Chaotic and random connections and disconnections at signal boundaries have very different noise when I am waiting for an arrival event vs a departure event. Arrival events are much easier to deal with as there can be no false connection events. An arrival event can be considered an actual arrival event IF I have a confirmed departure event in the past and I am currently not present. Departure events on the other hand have many false events due to connections dropping at boundaries and WiFi sleep on phones.

Solution #1) A simple BASH script (code below) running on an always on computer that fpings the static IP's of the cell phones every 5 seconds (this could be a raspberry pi). A ping response signifies present but a non response requires a specific time threshold be crossed before a departure is logged (to eliminate false departs due to WiFi sleep and dropped connections). Arrivals and departures are sent to the HE via Maker API.

There are two important properties of this setup. One is the time fping will wait for a response on signal boundaries. This should be set to a value small enough that it provides a solid WiFi connection to the cell phone, but not too small that it causes a phone to be logged as disconnected when it is not, and also not too long where there is chaotic connection dropping and re-connecting. This establishes a perimeter inside of where reliable cell connections are possible. The second is the time to wait to log a departure event. This needs to be long enough to accommodate all the WiFi sleeps on each phone. Technically there is no noise associated with making this limit very long until you begin to approach the limits of what you consider a valid departure event (so start off long 5min or more).

This code is working perfectly on my Android phones and has a fping time of 1 sec and a depart timer of 5min. I originally started with fping at 500ms and depart at 3min, those setting produced many false departures and Alexa kept greeting me when I took out the dog.

Enjoy:

#!/bin/bash

#Static Phone WiFi IP's
ip1='10.10.2.60'
ip2='10.10.2.61'
ip3='10.10.2.62'
ip1='10.10.2.63'
user1time=0
user2time=0
user3time=0
user4time=0
fpingtime=1000 #time in mS to wait for ping response
depart=300 #time in Sec that you must be gone to be considered departed

while :
do
#Read current local set presence
inputfile="/home/user/presence.txt"
i="0"
while IFS="," read a b; do
i=$((i + 1))
if [ $i = "1" ]; then
user1P=$b
elif [ $i = "2" ]; then
user2P=$b
elif [ $i = "3" ]; then
user3P=$b
elif [ $i = "4" ]; then
user4P=$b
fi
done < $inputfile

phone=$(fping -c 1 -t$fpingtime $ip1 $ip2 $ip3 $ip4)

#echo $phone

datetime=$(date "+%m-%d-%y_%H:%M:%S")

if [[ "$phone" == *"$ip1"* ]]; then user1="1"; else user1="0"; fi	
if [[ "$phone" == *"$ip2"* ]]; then user2="1"; else user2="0"; fi
if [[ "$phone" == *"$ip3"* ]]; then user3="1"; else user3="0"; fi
if [[ "$phone" == *"$ip4"* ]]; then user4="1"; else user4="0"; fi	

#echo
#echo $datetime
#echo "user1 current:"$user1",stored:"$user1P
#echo "user2 current:"$user2",stored:"$user2P
#echo "user3 current:"$user3",stored:"$user3P
#echo "user4 current:"$user4",stored:"$user4P
#echo

#if [ "$user1" = "0" ]; then #code to log all no response events on a phone
#	timenow=$(date +%s)
#	if [ $user1time = "0" ]; then user1time=$(date +%s); fi
#	timediff=$(("$timenow" - "$user1time"))
#	timeh=$(date -ud "@$timediff" +"%H:%M:%S")
#	line=$datetime" user1 current:"$user1",stored:"$user1P",time:"$timeh
#	echo $line >> /home/user/user1_ping.log
#fi

#user1 arrive and depart logic
if [ "$user1" = "1" ]; then #is present
	if [ "$user1P" = "0" ]; then #status is curently not present update and log arrival
		timenow=$(date +%s)
		timediff=$(("$timenow" - "$user1time"))
		if [[ "$timediff" -lt "86400" ]]; then
    			line=$(date -ud "@$timediff" +"%H:%M:%S")
		else #code to deal with multiday depatrues
    			days=$(("$diff" / "86400"))
    			timediff=$(("$timediff" - ("$days" * "86400")))
    			timeh=$(date -ud "@$timediff" +"%H:%M:%S")
    			line=$days":"$timeh
		fi
        echo "user1 arrived: "$(date +%m-%d-%y_%H:%M:%S)", was gone: "$line >> /home/user/user1_presence.log
		user1time=0 #reset departed timer	
		HEVariable=$(wget -qO- --timeout=5 --tries=1 http://10.10.2.90/apps/api/23/devices/20/arrived?access_token=xxxxxxxxxxxxxxxxxxxx)
		txt="user1,1"
		sed -i -e "1s/.*/$txt/" /home/user/presence.txt #update current status
		#echo $HEVariable #HE response
	else
		user1time=0 #reset timer for when not long enough to be departed
	fi
elif [ "$user1" = "0" ]; then #is NOT present
	if [ $user1time = "0" ]; then user1time=$(date +%s); fi #start depart timer if not started already
		timenow=$(date +%s)
		timediff=$(("$timenow" - "$user1time"))
		if [[ "$timediff" -lt "86400" ]]; then
			line=$(date -ud "@$timediff" +"%H:%M:%S")
		else
			days=$(("$timediff" / "86400"))
			echo "days:"$days
            timediff=$(("$timediff" - ("$days" * "86400")))
            timeh=$(date -ud "@$timediff" +"%H:%M:%S")
            line=$days":"$timeh
        fi
	echo "user1 current not present time "$line #how long been gone
	if [ "$user1P" = "1" ]; then #status is still present
		if [[ "$timediff" -gt "$depart" ]]; then #been gone long enough to be not present
			timenow=$(date +%s)
            timediff=$(("$timenow" - "$user1time"))
			if [[ "$timediff" -lt "86400" ]]; then
				line=$(date -ud "@$timediff" +"%H:%M:%S")
            else
                days=$(("$timediff" / "86400"))
                timediff=$(("$timediff" - ("$days" * "86400")))
                timeh=$(date -ud "@$timediff" +"%H:%M:%S")
                line=$days":"$timeh
            fi
            echo "user1 departed: "$(date +%m-%d-%y_%H:%M:%S)", timer reached: "$line >> /home/user/user1_presence.log
			HEVariable=$(wget -qO- --timeout=5 --tries=1 http://10.10.2.90/apps/api/23/devices/20/departed?access_token=xxxxxxxxxxxxxxxxx)
			txt="user1,0"
			sed -i -e "1s/.*/$txt/" /home/user/presence.txt
			echo "user1 has departed"
			#echo $HEVariable
		fi
	fi
fi

#additional users here

sleep 4

done

exit

2 Likes

Well done, Grasshopper.

Extending presence boundary via Tasker and near WiFi state profile.

My WiFi has enough range to allow reliable presence detection long before exiting a vehicle and opening a door to the house (so all welcome routines are primed and waiting for a door opening event or a reset). However, this may not work for everyone and you may want a larger perimeter around your house. To do that here is a more complicated process that I have also tested and it works extremely well for presence sensing with a very wide boundary.

Using the Tasker app for Android you setup a profile using the WiFi near state. To do this you need to select either your own WiFi or other ones that are farther from your home. Since you don't need to connect to the the WiFi to trigger this profile you can trigger at a much farther distance than from a distance that would allow you to connect. Once you are near the selected WiFi you have Tasker open a VPN connection to your firewall/router via Mobile Data, set the presence sensor via the Marker API, and then close the VPN. The opposite can be done when setting a departure event with a farther WiFi to ensure you are actually leaving.

This works well for me since I have only one way in or out of my plan. However, for others you may need two or three depart profiles to cover all possible routes. I also recommend using several WiFi's OR'ed together so if one is down the others will still trigger the actions. Google maintains access point maps or a quick war driving session can map access points along your various routes for more complex automation's.

You can open the VPN either with directly in Tasker or with the Tasker OpenVPN plugin. NOTE: I did not test the plugin. The down side of this is Tasker requires battery optimization to be turned off, BUT you can automate just about anything on your phone with Tasker so its possible to offset this with other optimizations. Also this would require your phone to have internet access. While the first BASH script does not require any internet access.

enjoy:

Link to setting up tasker to open a OpenVPN connection
https://openvpn.net/faq/how-do-i-use-tasker-with-openvpn-connect-for-android/
Link to Tasker OpenVPN plugin:
https://play.google.com/store/apps/details?id=com.ffrog8.openVpnTaskerPlugin&hl=en_US&gl=US

how about this driver, dosnt need a man in the middle
with thanks from @thebearmay - it would be good if you could ping the MAC that would save needing to assign a fixed IP
i have mine set to every min ping 1 packet and after 5 fails change to away, it also can check the router incase of power failure so the system dosnt trigger a fale away due to router being offline

/*
 * Hubitat presence Ping
 *
 *  Licensed Virtual the Apache License, Version 2.0 (the "License"); you may not use this file except
 *  in compliance with the License. You may obtain a copy of the License at:
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software distributed under the License is distributed
 *  on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License
 *  for the specific language governing permissions and limitations under the License.
 *
 *  Change History:
 *
 *        Date        Who            What
 *        ----        ---            ----

        2021-11-05  thebearmay     Add Text Logging option
        2022-01-03    mark-c-uk    striped down to work as presance sensor
 */

metadata {
    definition (
		name: "Hubitat presence Ping", 
		namespace: "mark-c-uk", 
		author: "mark-c-uk",
	        importUrl:"",
	    	singleThreaded: true
	) {
        capability "Actuator"
        capability "Configuration"
        capability "PresenceSensor"
        capability "Initialize"
       
        attribute "pingData", "string"
        
        
        command "sendPing", [[name:"ipAddress*", type:"STRING", description:"IP Address (IPv4) for the hub to ping"]]   
            
            
    }   
}

preferences {
    input("ipAdd", "string", title: "IP address of device", required:true, submitOnChange: true)
    input("ipAddRout", "string", title: "IP address of router, to test befor sending away", submitOnChange: true)
    input("numPings", "number", title: "Number of pings to issue", defaultValue:3, required:true, submitOnChange:true, range: "1..5")
    input("pingPeriod", "number", title: "Ping Repeat in Seconds\n Zero to disable", defaultValue: 0, required:true, submitOnChange: true)
    input("awayPeriod", "number", title: "number of missed pings to be away", defaultValue: 5, required:true, submitOnChange: true)
    input("debugEnable", "bool", title: "Enable debug logging?")
    input("textLoggingEnabled", "bool", title: "Enable Text Logging", defaultValue:false)
    input "sendPushMessageto", "capability.notification", title: "Send a Push notification to?", multiple: true, required: false
}

def installed() {
	log.trace "installed()"
}

def configure() {
    unschedule()
    if(debugEnable == true) log.debug "configure()"
    updateAttr("pingData"," ")
    if (device.currentValue("presence") == null) updateAttr("presence","not present")
    if (ipAdd == null){
        log.warn "no ip address"
        updateAttr("pingData", "no Device IP address")
        state.validIP = false
    }
    else if (validIP(ipAdd) == true){
        if(debugEnable == true) log.debug "Device IP address format valid"
        updateAttr("pingData", "Device IP address format valid")
        state.validIP = true
    }
    else{
        log.warn "Device IP address format invalid"
        updateAttr("pingData", "Device IP address format invalid")
        state.validIP = false
    }
             
    if (ipAddRout == null){
        log.warn "no ip addressvfor router"
        state.router = false
    }
    else if (validIP(ipAddRout) == true){
        if(debugEnable == true) log.debug "router IP address format valid"
        state.router = true
    }
    else{
        log.warn "router IP address format invalid"
        state.router = false
    }
   
    if (numPings == null) numPings = 3
    if (awayPeriod == null) awayPeriod = 5
             
    updated()
}

def initialize(){
    configure()
}
def updated(){
	if(debugEnable == true) log.trace "updated()"
	if(debugEnable) runIn(1800,logsOff)
    if(state.validIP == true) pinger()
}

def refresh() {
	unschedule(refresh)
}
             
def updateAttr(aKey, aValue){
    sendEvent(name:aKey, value:aValue)
}

def pinger(){ 
    if (settings.textLoggingEnabled == true) log.debug "Ping initiated"
    if (state.responseReady == false){
        log.warn "not ready"
        if(pingPeriod > 0) runIn(pingPeriod, "pinger")
        return
    }
    if (sendPing(settings.ipAdd) < 100){
        if (state.away != 0) state.away = 0
        if(settings.textLoggingEnabled == true) log.debug "Presence 'present' for $ipAdd"
        updateAttr("presence","present")
    }
    else {
        if (device.currentValue("presence") == "not present"){
            if(settings.textLoggingEnabled == true) log.debug "Presence not present already $ipAdd, ${state.away} number of times"
        }
        else{
            state.away += 1
            if(settings.textLoggingEnabled == true) log.debug "Presence not presentfor $ipAdd , NP ${state.away}"
            if (state.away > awayPeriod){
                if (state.router == true){
                    if (sendPing(ipAddRout) < 100){
                        if(settings.textLoggingEnabled == true) log.debug "Presence set to 'not present' for $ipAdd"
                        updateAttr("presence","not present")
                    }
                    else {
                        updateAttr("pingData","router not present")
                    }
                } //end use router option
                else {
                    updateAttr("presence","not present")
                }
                if(settings.textLoggingEnabled == true) log.debug "not present' for $ipAdd, ${state.away} number of times"
            }
        }
    }
    if(settings.pingPeriod > 0) runIn(settings.pingPeriod, "pinger")
    if(settings.textLoggingEnabled == true && settings.pingPeriod > 0) log.debug "Next ping in $pingPeriod seconds"
}
             
def sendPing(ipAddress){
            if(textLoggingEnabled == true) log.debug "Hub internal ping method selected"
            state.responseReady = false
            
            hubitat.helper.NetworkUtils.PingData pingData = hubitat.helper.NetworkUtils.ping(ipAddress, numPings.toInteger())
            int pTran = pingData.packetsTransmitted.toInteger()
            if (pTran == 0){ // 2.2.7.121 bug returns all zeroes on not found
                pingData.packetsTransmitted = numPings
                pingData.packetLoss = 100
            }
            
            String pingStats = "Transmitted: ${pingData.packetsTransmitted}, Received: ${pingData.packetsReceived}, %Lost: ${pingData.packetLoss}"
            if(textLoggingEnabled == true) log.debug "Ping Stats for $ipAddress: $pingStats"
            //updateAttr("pingData", "$ipAddress: $pingStats") 
    state.responseReady = true
    return pingData.packetLoss

    //if (sendPushMessageto != null){
   //     sendPushMessageto.deviceNotification("some error" )
   // }
}
       
def validIP(ipAddress){
    regxPattern =/^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/
    boolean match = ipAddress ==~ regxPattern
    return match
}


void logsOff(){
     device.updateSetting("debugEnable",[value:"false",type:"bool"])
}

One issue I've found is the newer phone os's are doing the whole mac obfuscation thing which is a real pain when trying to set static IP addresses / detecting MACs etc. You have to remember to turn it off for each phone's WiFi connection.

I am using Unifi to detect device connections to WiFi and set presence.

Thanks for this. However, I have business grade firewall router and my WiFi is on isolated subnet controlled by my firewall, and the firewall only grants access to known MAC's. HE is on a different subnet and has no access to the WiFi subnet, so this driver would not work for me unless I open a hole in the firewall. Also my entire network has redundant systems/routers and are on UPS's, so power outages do not effect my routers or servers.

A 1min interval would be far too long for me. I need to know when someone arrives in <10sec before they open a door to the house, as all my welcome home automation's are primed by an arrival event and triggered depending on what door you come in. To get that arrival timing accuracy I had to deal with the arrival event noise by using a higher sampling rate.

This also looks likes it can only ping a single device at a time? I have four phones that I flood ping simultaneously. My BASH script runs in ~1500ms and 1000ms of that is the fping response wait. So I could push up the sampling rate to ~1500ms if I wanted to get an even higher sampling rate.

My goal is very high arrival timing accuracy so that I can execute customized person and group specific automation's upon opening a door. The routines that run depend on whom or what group of people have just arrived home.

Thanks for the suggestion, I could see using something like this for devices that are on the HE subnet.... that could be interesting.

Yes ran into that as well. I think Android 10 is when Google started that..... and they keep changing how and where you turn it off. Just had to hunt down the setting on my Daughters Pixel 4 the other day as she upgraded to Android 12..... at least its SSID specific so only the Home WiFi uses the real MAC address every other SSID is still random MAC.

1 Like

The hub isn't really designed for ping ( it works fine) I had it ping 2 phones at 30 sec intervals and it pushed up my 5min cpu 1% to around 3%.

Yea create one device per phone

A minute seems OK for me by the time car pulled up and got out etc. Even just walking up drive and to back door it was firing as I just opened the door

Might think of an away ping interval and a present interval