erikng / cacher Goto Github PK

Hi,

Thanks for the work on this script. Just deployed and it works a treat.
Is there any chance you could add metrics on drive space used by the cache or overall used/free ?

Also there is a typo bandwidth is showing as bandwith

Shane

I noticed that when using --configureserver, while the values were set, the Caching server did not respect the values until the service was restarted.

Cacher should restart the service to ensure this is set.

I get an error with the script:

Traceback (most recent call last):
  File "./test.py", line 940, in <module>
    main()
  File "./test.py", line 923, in main
    cacherdata = cacher(rawLog.readlines(), targetDate, friendlyNames, site_name)
  File "./test.py", line 324, in cacher
    x).group(1).replace('OS X ', 'macOS/').split('/')[0]
AttributeError: 'NoneType' object has no attribute 'group'

Attached is the debug log that causes this error (the target date is 2017-02-18

Debug.zip

Running the script on a different target date of 2017-02-28 works fine however.
Attached is the debug log for that target date as well.
Debug_28-02.log.zip

I was wondering if it is possible that the script show me the IP addresses of clients who downloaded files from my server?

Answer: It can but it would need to be written.

I'll try to write it, but I don't know much about python if you want do this with me I'll appreciate.

I get the following errors if nothing was served yesterday, on say a Sunday.

(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error

Seems to be related to what is passed to bc:

sudo bash -x ./Cacher

• loglocation=/Library/Server/Caching/Logs
• tmplocation=/tmp/CacheLogs
  ++ date -j -v-1d +%Y.%m.%d
• yesterday=2015.06.28
• mkdir -p /tmp/CacheLogs
• cd /Library/Server/Caching/Logs
• for i in '*.bz2'
• bzip2 -d '*.bz2'
  bzip2: Can't open input file *.bz2: No such file or directory.
• grep -E '^s*2015.06.28' ./AssetCache.pid ./Debug-2015.06.23-15:20.log ./Debug-2015.06.24-15:36.log ./Debug-2015.06.25-15:49.log ./Debug-2015.06.26-16:11.log ./Debug-2015.06.27-16:56.log ./Debug.log ./LastState.plist
• sed /Registering/d /tmp/CacheLogs/MergedLog-2015.06.28.log
• sed /public/d
• sed /registration/d
• sed /peers/d
• sed /Opened/d
• sed /ECResponse/d
• sed /ECAssetHandler/d
• sed /ECAssetRequestor/d
• sed /ECCacheReader/d
• sed /ECCacheManager/d
• sed /Bad/d
• sed -n -e 's/^.*Request //p'
• sed -n -e 's/^.*from //p'
• sort
• awk '!x[$1]++' FS=:
• sed -n -e 's/^.*start: //p ' /tmp/CacheLogs/MergedLog-2015.06.28.log
• grep -q MB /tmp/CacheLogs/Bandwidth_served.txt
• head -n1
• tr -d '()'
• tail -n1
• tr -d '()'
  ++ grep '' /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $1}'
• bwclient1=
  ++ grep '' /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $6}'
• bwserver1=
  ++ grep '' /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $11}'
• bworigin1=
  ++ grep '' /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $15}'
• bwpeer1=
  ++ grep '' /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $1}'
• bwclient2=
  ++ grep '' /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $6}'
• bwserver2=
  ++ grep '' /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $11}'
• bworigin2=
  ++ grep '' /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $15}'
• bwpeer2=
  ++ echo -
  ++ bc
  (standard_in) 2: parse error
• clientbw1=
  ++ echo -
  ++ bc
  (standard_in) 2: parse error
• requestedbw1=
  ++ echo -
  ++ bc
  (standard_in) 2: parse error
• applebw1=
  ++ echo -
  ++ bc
  (standard_in) 2: parse error
• peerbw1=
  ++ echo '*1024-'
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• clientbw2=
  ++ echo '*1024-'
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• requestedbw2=
  ++ echo '*1024-'
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• applebw2=
  ++ echo '*1024-'
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• peerbw2=
  ++ echo /1024-
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• clientbw3=
  ++ echo /1024-
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• requestedbw3=
  ++ echo /1024-
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• applebw3=
  ++ echo /1024-
  ++ bc
  ++ awk '{print "scale=2;"$1"/1024"}'
  ++ bc
  (standard_in) 1: parse error
• peerbw3=
  ++ grep MB /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $2}'
• mbbf1=
  ++ grep GB /tmp/CacheLogs/Bandwidth_First.txt
  ++ awk '{print $2}'
• gbbf1=
  ++ grep MB /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $2}'
• mbbf2=
  ++ grep GB /tmp/CacheLogs/Bandwidth_Last.txt
  ++ awk '{print $2}'
• gbbf2=
• echo 'Cacher has retrieved the following stats for 2015.06.28:'
  Cacher has retrieved the following stats for 2015.06.28:
• echo
• [[ '' == \M\B ]]
• [[ '' == \G\B ]]
• [[ '' == \M\B ]]
• [[ '' == \G\B ]]
• echo

++ ifconfig en0
++ grep 'inet '
++ grep -v 127.0.0.1
++ awk '{print $2}'
++ sed 's/./ /g'
++ awk '{print $1}'

• serverip=192
• egrep --only-matching -E '192.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' /tmp/CacheLogs/MergedLog-2015.06.28.log
• cat /tmp/CacheLogs/Total_IPs.txt
• sort
• uniq
  ++ wc -l /tmp/CacheLogs/Total_IPs.txt
  ++ awk '{print $1}'
• totalipnumber=78
  ++ wc -l /tmp/CacheLogs/Unique_IPs.txt
  ++ awk '{print $1}'
• uniqueipnumber=3
• echo 78 IP Addresses hit the Caching Server yesterday consisting of:
  78 IP Addresses hit the Caching Server yesterday consisting of:
• echo ' 3 Unique IP Addresses.'
  3 Unique IP Addresses.
• echo
• egrep -o '(\s(iOS\S+))' /tmp/CacheLogs/URL_Log-2015.06.28.log
• cut -d ' ' -f 2
• egrep --only-matching -E '8.(0)$' /tmp/CacheLogs/Total_iOS.txt
• egrep -o '(8.0.2)' /tmp/CacheLogs/Total_iOS.txt
• egrep --only-matching -E '8.(1)$' /tmp/CacheLogs/Total_iOS.txt
• egrep -o '(8.1.1)' /tmp/CacheLogs/Total_iOS.txt
• egrep -o '(8.1.2)' /tmp/CacheLogs/Total_iOS.txt
• egrep -o '(8.1.3)' /tmp/CacheLogs/Total_iOS.txt
• egrep --only-matching -E '8.(2)$' /tmp/CacheLogs/Total_iOS.txt
• egrep --only-matching -E '8.(3)$' /tmp/CacheLogs/Total_iOS.txt
  ++ wc -l /tmp/CacheLogs/Total_iOS.txt
  ++ awk '{print $1}'
• totaliosnumber=0
  ++ wc -l /tmp/CacheLogs/iOS_8.0.txt
  ++ awk '{print $1}'
• totalios800number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.0.2.txt
  ++ awk '{print $1}'
• totalios802number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.1.txt
  ++ awk '{print $1}'
• totalios810number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.1.1.txt
  ++ awk '{print $1}'
• totalios811number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.1.2.txt
  ++ awk '{print $1}'
• totalios812number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.1.3.txt
  ++ awk '{print $1}'
• totalios813number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.2.txt
  ++ awk '{print $1}'
• totalios820number=0
  ++ wc -l /tmp/CacheLogs/iOS_8.3.txt
  ++ awk '{print $1}'
• totalios830number=0
• egrep -o '(\s(model\S+))' /tmp/CacheLogs/URL_Log-2015.06.28.log
• cut -d ' ' -f 2
• egrep -o '(TV)' /tmp/CacheLogs/Total_models.txt
• egrep -o '(iPhone)' /tmp/CacheLogs/Total_models.txt
• egrep -o '(iPad)' /tmp/CacheLogs/Total_models.txt
• egrep -o '(iPod)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l /tmp/CacheLogs/Total_models.txt
  ++ awk '{print $1}'
• totalmodelnumber=0
  ++ wc -l /tmp/CacheLogs/Total_AppleTV.txt
  ++ awk '{print $1}'
• totalappletvnumber=0
  ++ wc -l /tmp/CacheLogs/Total_iPhone.txt
  ++ awk '{print $1}'
• totaliphonenumber=0
  ++ wc -l /tmp/CacheLogs/Total_iPad.txt
  ++ awk '{print $1}'
• totalipadnumber=0
  ++ wc -l /tmp/CacheLogs/Total_iPod.txt
  ++ awk '{print $1}'
• totalipodnumber=0
  ++ egrep -o '(AppleTV3,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalappletv31=0
  ++ egrep -o '(AppleTV3,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalappletv32=0
  ++ egrep -o '(iPhone3,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone31=0
  ++ egrep -o '(iPhone3,3)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone33=0
  ++ egrep -o '(iPhone4,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone41=0
  ++ egrep -o '(iPhone5,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone51=0
  ++ egrep -o '(iPhone5,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone52=0
  ++ egrep -o '(iPhone5,3)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone53=0
  ++ egrep -o '(iPhone6,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone61=0
  ++ egrep -o '(iPhone6,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone62=0
  ++ egrep -o '(iPhone7,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone72=0
  ++ egrep -o '(iPhone7,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totaliphone71=0
  ++ egrep -o '(iPad2,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad21=0
  ++ egrep -o '(iPad2,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad22=0
  ++ egrep -o '(iPad2,3)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad23=0
  ++ egrep -o '(iPad2,4)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad24=0
  ++ egrep -o '(iPad2,5)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad25=0
  ++ egrep -o '(iPad2,6)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad26=0
  ++ egrep -o '(iPad2,7)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad27=0
  ++ egrep -o '(iPad3,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad31=0
  ++ egrep -o '(iPad3,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad32=0
  ++ egrep -o '(iPad3,3)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad33=0
  ++ egrep -o '(iPad3,4)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad34=0
  ++ egrep -o '(iPad3,5)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad35=0
  ++ egrep -o '(iPad3,6)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad36=0
  ++ egrep -o '(iPad4,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad41=0
  ++ egrep -o '(iPad4,2)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad42=0
  ++ egrep -o '(iPad4,4)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad44=0
  ++ egrep -o '(iPad4,5)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad45=0
  ++ egrep -o '(iPad4,7)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad47=0
  ++ egrep -o '(iPad4,8)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad48=0
  ++ egrep -o '(iPad5,3)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad53=0
  ++ egrep -o '(iPad5,4)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipad54=0
  ++ egrep -o '(iPod5,1)' /tmp/CacheLogs/Total_models.txt
  ++ wc -l
  ++ awk '{print $1}'
• totalipod51=0
• echo A total of 0 iOS devices hit the Caching Server yesterday consisting of:
  A total of 0 iOS devices hit the Caching Server yesterday consisting of:
• echo ' 0 iOS 8.0 Devices'
  0 iOS 8.0 Devices
• echo ' 0 iOS 8.0.2 Devices'
  0 iOS 8.0.2 Devices
• echo ' 0 iOS 8.1 Devices'
  0 iOS 8.1 Devices
• echo ' 0 iOS 8.1.1 Devices'
  0 iOS 8.1.1 Devices
• echo ' 0 iOS 8.1.2 Devices'
  0 iOS 8.1.2 Devices
• echo ' 0 iOS 8.1.3 Devices'
  0 iOS 8.1.3 Devices
• echo ' 0 iOS 8.2 Devices'
  0 iOS 8.2 Devices
• echo ' 0 iOS 8.3 Devices'
  0 iOS 8.3 Devices
• echo
• echo Of these devices, here is the model breakdown:
  Of these devices, here is the model breakdown:
• echo ' 0 Total AppleTVs'
  0 Total AppleTVs
• echo ' 0 3rd Generation Apple TVs'
  0 3rd Generation Apple TVs
• echo ' 0 4th Generation Apple TVs'
  0 4th Generation Apple TVs
• echo ' 0 Total iPhones'
  0 Total iPhones
• echo ' 0 iPhone 4 [GSM]'
  0 iPhone 4 [GSM]
• echo ' 0 iPhone 4 [CDMA]'
  0 iPhone 4 [CDMA]
• echo ' 0 iPhone 4S'
  0 iPhone 4S
• echo ' 0 iPhone 5 [GSM]'
  0 iPhone 5 [GSM]
• echo ' 0 iPhone 5 [CDMA]'
  0 iPhone 5 [CDMA]
• echo ' 0 iPhone 5C'
  0 iPhone 5C
• echo ' 0 iPhone 5S'
  0 iPhone 5S
• echo ' 0 iPhone 5S [China Model]'
  0 iPhone 5S [China Model]
• echo ' 0 iPhone 6'
  0 iPhone 6
• echo ' 0 iPhone 6 Plus'
  0 iPhone 6 Plus
• echo ' 0 Total iPads'
  0 Total iPads
• echo ' 0 iPad 2nd Generation [Wifi]'
  0 iPad 2nd Generation [Wifi]
• echo ' 0 iPad 2nd Generation [Wifi + GSM]'
  0 iPad 2nd Generation [Wifi + GSM]
• echo ' 0 iPad 2nd Generation [Wifi + CDMA]'
  0 iPad 2nd Generation [Wifi + CDMA]
• echo ' 0 iPad 2nd Generation [M2012 Wifi Revision]'
  0 iPad 2nd Generation [M2012 Wifi Revision]
• echo ' 0 iPad 3rd Generation [Wifi]'
  0 iPad 3rd Generation [Wifi]
• echo ' 0 iPad 3rd Generation [Wifi + GSM]'
  0 iPad 3rd Generation [Wifi + GSM]
• echo ' 0 iPad 3rd Generation [Wifi + CDMA]'
  0 iPad 3rd Generation [Wifi + CDMA]
• echo ' 0 iPad 4th Generation [Wifi]'
  0 iPad 4th Generation [Wifi]
• echo ' 0 iPad 4th Generation [Wifi + GSM]'
  0 iPad 4th Generation [Wifi + GSM]
• echo ' 0 iPad 4th Generation [Wifi + CDMA]'
  0 iPad 4th Generation [Wifi + CDMA]
• echo ' 0 iPad Air 1st Generation [Wifi]'
  0 iPad Air 1st Generation [Wifi]
• echo ' 0 iPad Air 1st Generation [Wifi + Cellular]'
  0 iPad Air 1st Generation [Wifi + Cellular]
• echo ' 0 iPad Air 2nd Generation [Wifi]'
  0 iPad Air 2nd Generation [Wifi]
• echo ' 0 iPad Air 2nd Generation [Wifi + Cellular]'
  0 iPad Air 2nd Generation [Wifi + Cellular]
• echo
• echo ' 0 iPad Mini 1st Generation [Wifi]'
  0 iPad Mini 1st Generation [Wifi]
• echo ' 0 iPad Mini 1st Generation [Wifi + GSM]'
  0 iPad Mini 1st Generation [Wifi + GSM]
• echo ' 0 iPad Mini 1st Generation [Wifi + CDMA]'
  0 iPad Mini 1st Generation [Wifi + CDMA]
• echo ' 0 iPad Mini 2nd Generation [Wifi]'
  0 iPad Mini 2nd Generation [Wifi]
• echo ' 0 iPad Mini 2nd Generation [Wifi + Cellular]'
  0 iPad Mini 2nd Generation [Wifi + Cellular]
• echo ' 0 iPad Mini 3rd Generation [Wifi]'
  0 iPad Mini 3rd Generation [Wifi]
• echo ' 0 iPad Mini 3rd Generation [Wifi + Cellular]'
  0 iPad Mini 3rd Generation [Wifi + Cellular]
• echo ' 0 Total iPods'
  0 Total iPods
• echo ' 0 iPod Touch 5th Generation'
  0 iPod Touch 5th Generation
• echo
• egrep -o '(\s(Darwin\S+))' /tmp/CacheLogs/URL_Log-2015.06.28.log
• cut -d ' ' -f 2
• egrep -o '(/(14.0.0))' /tmp/CacheLogs/Total_OS_X.txt
• egrep -o '(/(14.1.0))' /tmp/CacheLogs/Total_OS_X.txt
• egrep -o '(/(14.3.0))' /tmp/CacheLogs/Total_OS_X.txt
  ++ wc -l /tmp/CacheLogs/Total_OS_X.txt
  ++ awk '{print $1}'
• totalosxnumber=0
  ++ wc -l /tmp/CacheLogs/Total_OS_X_10.10.txt
  ++ awk '{print $1}'
• total1010number=0
  ++ wc -l /tmp/CacheLogs/Total_OS_X_10.10.2.txt
  ++ awk '{print $1}'
• total10102number=0
  ++ wc -l /tmp/CacheLogs/Total_OS_X_10.10.3.txt
  ++ awk '{print $1}'
• total10103number=0
• echo A total of 0 OS X devices hit the Caching Server yesterday consisting of:
  A total of 0 OS X devices hit the Caching Server yesterday consisting of:
• echo ' 0 OS X Yosemite 10.10.0/10.10.1 Devices [Cannot distinguish between builds]'
  0 OS X Yosemite 10.10.0/10.10.1 Devices [Cannot distinguish between builds]
• echo ' 0 OS X Yosemite 10.10.2 Devices'
  0 OS X Yosemite 10.10.2 Devices
• echo ' 0 OS X Yosemite 10.10.3 Devices'
  0 OS X Yosemite 10.10.3 Devices
• echo
• egrep -o '(\s(http\S+))' /tmp/CacheLogs/MergedLog-2015.06.28.log
• cut -d ' ' -f 2
• egrep -o '(ipa)' /tmp/CacheLogs/Total_URLs.txt
• egrep -o '(zip)' /tmp/CacheLogs/Total_URLs.txt
• egrep -o '(pkg)' /tmp/CacheLogs/Total_URLs.txt
• egrep -o '(epub)' /tmp/CacheLogs/Total_URLs.txt
• egrep -o '(register)' /tmp/CacheLogs/Total_URLs.txt
  ++ wc -l /tmp/CacheLogs/Total_URLs.txt
  ++ awk '{print $1}'
• totalurlnumber=26
  ++ wc -l /tmp/CacheLogs/Total_IPAs.txt
  ++ awk '{print $1}'
• totalipanumber=0
  ++ wc -l /tmp/CacheLogs/Total_Zips.txt
  ++ awk '{print $1}'
• totalzipnumber=0
  ++ wc -l /tmp/CacheLogs/Total_PKGs.txt
  ++ awk '{print $1}'
• totalpkgnumber=0
  ++ wc -l /tmp/CacheLogs/Total_Books.txt
  ++ awk '{print $1}'
• totalepubnumber=0
  ++ wc -l /tmp/CacheLogs/Total_Server_Registers.txt
  ++ awk '{print $1}'
• totalsregistersnumber=26
• echo A total of 26 files were downloaded from the Caching Server yesterday consisting of:
  A total of 26 files were downloaded from the Caching Server yesterday consisting of:
• echo ' 0 Books'
  0 Books
• echo ' 0 iOS Apps'
  0 iOS Apps
• echo ' 0 Mac Apps'
  0 Mac Apps
• echo ' 0 Zip files'
  0 Zip files
• echo ' 26 Apple Server Registrations'
  26 Apple Server Registrations
• echo
• cat /tmp/CacheLogs/Total_URLs.txt
• sort
• uniq
• egrep -o '(epub)' /tmp/CacheLogs/Unique_URLs.txt
• egrep -o '(ipa)' /tmp/CacheLogs/Unique_URLs.txt
• egrep -o '(pkg)' /tmp/CacheLogs/Unique_URLs.txt
• egrep -o '(zip)' /tmp/CacheLogs/Unique_URLs.txt
  ++ wc -l /tmp/CacheLogs/Unique_URLs.txt
  ++ awk '{print $1}'
• uniqueurlnumber=1
  ++ wc -l /tmp/CacheLogs/Unique_Books.txt
  ++ awk '{print $1}'
• uniqueepubnumber=0
  ++ wc -l /tmp/CacheLogs/Unique_IPAs.txt
  ++ awk '{print $1}'
• uniqueipanumber=0
  ++ wc -l /tmp/CacheLogs/Unique_PKGs.txt
  ++ awk '{print $1}'
• uniquepkgnumber=0
  ++ wc -l /tmp/CacheLogs/Unique_Zips.txt
  ++ awk '{print $1}'
• uniquezipnumber=0
• echo 1 Unique files were downloaded from the Caching Server yesterday consisting of:
  1 Unique files were downloaded from the Caching Server yesterday consisting of:
• echo ' 0 Unique Books'
  0 Unique Books
• echo ' 0 Unique iOS Apps'
  0 Unique iOS Apps
• echo ' 0 Unique Mac Apps'
  0 Unique Mac Apps
• echo ' 0 Unique Zip files'
  0 Unique Zip files
• echo
• echo 'Cacher has retrieved the following stats for 2015.06.28:'
• echo
• [[ '' == \M\B ]]
• [[ '' == \G\B ]]
• [[ '' == \M\B ]]
• [[ '' == \G\B ]]
• echo
• echo 78 IP Addresses hit the Caching Server yesterday consisting of:
• echo ' 3 Unique IP Addresses.'
• echo
• echo A total of 0 iOS devices hit the Caching Server yesterday consisting of:
• echo ' 0 iOS 8.0 Devices'
• echo ' 0 iOS 8.0.2 Devices'
• echo ' 0 iOS 8.1 Devices'
• echo ' 0 iOS 8.1.1 Devices'
• echo ' 0 iOS 8.1.2 Devices'
• echo ' 0 iOS 8.1.3 Devices'
• echo ' 0 iOS 8.2 Devices'
• echo ' 0 iOS 8.3 Devices'
• echo
• echo Of these devices, here is the model breakdown:
• echo ' 0 Total AppleTVs'
• echo ' 0 3rd Generation Apple TVs'
• echo ' 0 4th Generation Apple TVs'
• echo ' 0 Total iPhones'
• echo ' 0 iPhone 4 [GSM]'
• echo ' 0 iPhone 4 [CDMA]'
• echo ' 0 iPhone 4S'
• echo ' 0 iPhone 5 [GSM]'
• echo ' 0 iPhone 5 [CDMA]'
• echo ' 0 iPhone 5C'
• echo ' 0 iPhone 5S'
• echo ' 0 iPhone 5S [China Model]'
• echo ' 0 iPhone 6'
• echo ' 0 iPhone 6 Plus'
• echo ' 0 Total iPads'
• echo ' 0 iPad 2nd Generation [Wifi]'
• echo ' 0 iPad 2nd Generation [Wifi + GSM]'
• echo ' 0 iPad 2nd Generation [Wifi + CDMA]'
• echo ' 0 iPad 2nd Generation [M2012 Wifi Revision]'
• echo ' 0 iPad 3rd Generation [Wifi]'
• echo ' 0 iPad 3rd Generation [Wifi + GSM]'
• echo ' 0 iPad 3rd Generation [Wifi + CDMA]'
• echo ' 0 iPad 4th Generation [Wifi]'
• echo ' 0 iPad 4th Generation [Wifi + GSM]'
• echo ' 0 iPad 4th Generation [Wifi + CDMA]'
• echo ' 0 iPad Air 1st Generation [Wifi]'
• echo ' 0 iPad Air 1st Generation [Wifi + Cellular]'
• echo ' 0 iPad Air 2nd Generation [Wifi]'
• echo ' 0 iPad Air 2nd Generation [Wifi + Cellular]'
• echo
• echo ' 0 iPad Mini 1st Generation [Wifi]'
• echo ' 0 iPad Mini 1st Generation [Wifi + GSM]'
• echo ' 0 iPad Mini 1st Generation [Wifi + CDMA]'
• echo ' 0 iPad Mini 2nd Generation [Wifi]'
• echo ' 0 iPad Mini 2nd Generation [Wifi + Cellular]'
• echo ' 0 iPad Mini 3rd Generation [Wifi]'
• echo ' 0 iPad Mini 3rd Generation [Wifi + Cellular]'
• echo ' 0 Total iPods'
• echo ' 0 iPod Touch 5th Generation'
• echo
• echo A total of 0 OS X devices hit the Caching Server yesterday consisting of:
• echo ' 0 OS X Yosemite 10.10.0/10.10.1 Devices [Cannot distinguish between builds]'
• echo ' 0 OS X Yosemite 10.10.2 Devices'
• echo ' 0 OS X Yosemite 10.10.3 Devices'
• echo
• echo A total of 26 files were downloaded from the Caching Server yesterday consisting of:
• echo ' 0 Books'
• echo ' 0 iOS Apps'
• echo ' 0 Mac Apps'
• echo ' 0 Zip files'
• echo ' 26 Apple Server Registrations'
• echo
• echo 1 Unique files were downloaded from the Caching Server yesterday consisting of:
• echo ' 0 Unique Books'
• echo ' 0 Unique iOS Apps'
• echo ' 0 Unique Mac Apps'
• echo ' 0 Unique Zip files'
  ++ echo
  ++ grep '' /tmp/CacheLogs/AlertInfo.txt
• finalalert='Cacher has retrieved the following stats for 2015.06.28:

78 IP Addresses hit the Caching Server yesterday consisting of:
3 Unique IP Addresses.

A total of 0 iOS devices hit the Caching Server yesterday consisting of:
0 iOS 8.0 Devices
0 iOS 8.0.2 Devices
0 iOS 8.1 Devices
0 iOS 8.1.1 Devices
0 iOS 8.1.2 Devices
0 iOS 8.1.3 Devices
0 iOS 8.2 Devices
0 iOS 8.3 Devices

Of these devices, here is the model breakdown:
0 Total AppleTVs
0 3rd Generation Apple TVs
0 4th Generation Apple TVs
0 Total iPhones
0 iPhone 4 [GSM]
0 iPhone 4 [CDMA]
0 iPhone 4S
0 iPhone 5 [GSM]
0 iPhone 5 [CDMA]
0 iPhone 5C
0 iPhone 5S
0 iPhone 5S [China Model]
0 iPhone 6
0 iPhone 6 Plus
0 Total iPads
0 iPad 2nd Generation [Wifi]
0 iPad 2nd Generation [Wifi + GSM]
0 iPad 2nd Generation [Wifi + CDMA]
0 iPad 2nd Generation [M2012 Wifi Revision]
0 iPad 3rd Generation [Wifi]
0 iPad 3rd Generation [Wifi + GSM]
0 iPad 3rd Generation [Wifi + CDMA]
0 iPad 4th Generation [Wifi]
0 iPad 4th Generation [Wifi + GSM]
0 iPad 4th Generation [Wifi + CDMA]
0 iPad Air 1st Generation [Wifi]
0 iPad Air 1st Generation [Wifi + Cellular]
0 iPad Air 2nd Generation [Wifi]
0 iPad Air 2nd Generation [Wifi + Cellular]

0 iPad Mini 1st Generation [Wifi]
0 iPad Mini 1st Generation [Wifi + GSM]
0 iPad Mini 1st Generation [Wifi + CDMA]
0 iPad Mini 2nd Generation [Wifi]
0 iPad Mini 2nd Generation [Wifi + Cellular]
0 iPad Mini 3rd Generation [Wifi]
0 iPad Mini 3rd Generation [Wifi + Cellular]

0 Total iPods
0 iPod Touch 5th Generation

A total of 0 OS X devices hit the Caching Server yesterday consisting of:
0 OS X Yosemite 10.10.0/10.10.1 Devices [Cannot distinguish between builds]
0 OS X Yosemite 10.10.2 Devices
0 OS X Yosemite 10.10.3 Devices

A total of 26 files were downloaded from the Caching Server yesterday consisting of:
0 Books
0 iOS Apps
0 Mac Apps
0 Zip files
26 Apple Server Registrations

1 Unique files were downloaded from the Caching Server yesterday consisting of:
0 Unique Books
0 Unique iOS Apps
0 Unique Mac Apps
0 Unique Zip files'

• /Applications/Server.app/Contents/ServerRoot/usr/sbin/server postAlert CustomAlert Common subject 'Caching Server Data: 2015.06.28' message 'Cacher has retrieved the following stats for 2015.06.28:

78 IP Addresses hit the Caching Server yesterday consisting of:
3 Unique IP Addresses.

A total of 0 iOS devices hit the Caching Server yesterday consisting of:
0 iOS 8.0 Devices
0 iOS 8.0.2 Devices
0 iOS 8.1 Devices
0 iOS 8.1.1 Devices
0 iOS 8.1.2 Devices
0 iOS 8.1.3 Devices
0 iOS 8.2 Devices
0 iOS 8.3 Devices

Of these devices, here is the model breakdown:
0 Total AppleTVs
0 3rd Generation Apple TVs
0 4th Generation Apple TVs
0 Total iPhones
0 iPhone 4 [GSM]
0 iPhone 4 [CDMA]
0 iPhone 4S
0 iPhone 5 [GSM]
0 iPhone 5 [CDMA]
0 iPhone 5C
0 iPhone 5S
0 iPhone 5S [China Model]
0 iPhone 6
0 iPhone 6 Plus
0 Total iPads
0 iPad 2nd Generation [Wifi]
0 iPad 2nd Generation [Wifi + GSM]
0 iPad 2nd Generation [Wifi + CDMA]
0 iPad 2nd Generation [M2012 Wifi Revision]
0 iPad 3rd Generation [Wifi]
0 iPad 3rd Generation [Wifi + GSM]
0 iPad 3rd Generation [Wifi + CDMA]
0 iPad 4th Generation [Wifi]
0 iPad 4th Generation [Wifi + GSM]
0 iPad 4th Generation [Wifi + CDMA]
0 iPad Air 1st Generation [Wifi]
0 iPad Air 1st Generation [Wifi + Cellular]
0 iPad Air 2nd Generation [Wifi]
0 iPad Air 2nd Generation [Wifi + Cellular]

0 iPad Mini 1st Generation [Wifi]
0 iPad Mini 1st Generation [Wifi + GSM]
0 iPad Mini 1st Generation [Wifi + CDMA]
0 iPad Mini 2nd Generation [Wifi]
0 iPad Mini 2nd Generation [Wifi + Cellular]
0 iPad Mini 3rd Generation [Wifi]
0 iPad Mini 3rd Generation [Wifi + Cellular]

0 Total iPods
0 iPod Touch 5th Generation

A total of 0 OS X devices hit the Caching Server yesterday consisting of:
0 OS X Yosemite 10.10.0/10.10.1 Devices [Cannot distinguish between builds]
0 OS X Yosemite 10.10.2 Devices
0 OS X Yosemite 10.10.3 Devices

A total of 26 files were downloaded from the Caching Server yesterday consisting of:
0 Books
0 iOS Apps
0 Mac Apps
0 Zip files
26 Apple Server Registrations

1 Unique files were downloaded from the Caching Server yesterday consisting of:
0 Unique Books
0 Unique iOS Apps
0 Unique Mac Apps
0 Unique Zip files'

• rm -rf /tmp/CacheLogs

print "Server version is %s and not compatible" % serverversion()

should be

print "Server version is %s and not compatible" % get_serverversion()

I think the device counts are calculated incorrectly.

It looks to be counting the device number based upon the get request numbers rather than actual devices.

For example $TOTAL_IOS_NUMBER is just a calculation of of how many requests were made $yesterday that included iOS user agents. A single device might make 20 requests a day.

That doesn't mean 20 devices hit the caching server.

We should look at unique IP addresses and then break down that list and look for iOS (or other) devices based on that. Still an assumption that a device has the same IP address for a 24 hour period. But possibly more accurate.

Does this work on Mojave with the Server App no longer managing it?

This will reduce logging errors.

I have some problems because I don't know how will I run the scrip in my Mac. I installed python but when I set the command "sudo cacher.py --configureserver" the command wasn't found.

I don't used python before. Please I hope you can help me with a some manual or something with the steps for installing the scrip.

Since the caching service was removed from Server in 10.13 log location has moved as well, breaking Cacher. Any plans to update Cacher for 10.13? Thanks!

When rebooting the server (say for an upgrade) the bandwidth statistics may be off the following day.

This will occur due when a small application is first downloaded followed by larger applications, ie MB content served first and GB content served later in the day.

I am getting an error, any help would be appreciated.

IOS-CCS:Cacher-master-2 macmini$ python cacher.py --targetdate "2019-09-19" --deviceids --teamsalert --teamswebhook "https://outlook.office.com/webhook/792f5196-ea16-45b3-a7e7-07708470b74b@e7119446-0f7e-4a91-a73a-7873a46c3354/IncomingWebhook/3b42557b61d54d91b7b3ca6c833b507d/f0031761-94c3-46db-xxxx"
Traceback (most recent call last):
File "cacher.py", line 891, in
main()
File "cacher.py", line 876, in main
cacherdata = cacher(rawLog.readlines(), targetDate, friendlyNames)
File "cacher.py", line 233, in cacher
totalbytesserved.append(bytes_served)
UnboundLocalError: local variable 'bytes_served' referenced before assignment

Cacher is not showing the types of devices that are connecting. It is only reporting the total amount of bandwidth and the total number of files.

This is with server 5.1 on OS X 10.11.4. I verified that LogClientIdentity is True in the /Library/Server/Caching/Config/Config.plist

output a message.

Getting on the following error on OS X Server 3.2.2 box, probably also occurs on OS X Server 4.0 since there are no bz2 files.
bzip2: Can't open input file *.bz2: No such file or directory.

Just installed the latest Cacher (2.0), and am getting this error. I'm getting 0 for all the other stats too.

Not sure if we are using cacher correctly, I just manually run the script everyday. Should I be using launchd or some other process?

macOS 10.12.1 (Build 16B2659)
Server 5.2 (Build 16S1195)

Cacher
Last Updated: 01-04-2017
version = '3.0'

With cache personal iCloud Data ON, cacher provides incorrect device counts. Mac OS device counts will be in the hundreds when there are less than 100 on the network.

3.0.4 has a great update to calculating the totals.
I'm just curious as to why I would be seeing such a large difference between what cahcer and my server logs say.

2017-03-06 12:53:20.673 Since server start: 7.09 GB returned to clients, 1.46 GB stored from Internet, 0 bytes from peers; 0 bytes imported.

Cacher has retrieved the following stats for 2017-03-06:
4.8 GB of bandwith served to client devices.
1.4 GB of bandwith requested from Apple
0.0 bytes of bandwith requested from other Caching Servers

My server only runs between 6am and 11pm each day, so there's no crossover between dates.

I get this error at the end of the script when it tries to create an alert.
2015-06-29 11:28:53.107 server[76024:507] Error Domain=AlertsDaemon Code=150 "Error: Insufficient entitlements." UserInfo=0x7ff268d01c60 {NSLocalizedDescription=Error: Insufficient entitlements.}

I have 3 servers I tried this on, all on 5.0.4, all 10.10.5, all with caching enabled and data in it.

The first one ran fine (copied the script file to each machine locally and ran from terminal, I also tried running through casper, also all had individually modified plist file and all hat the log client id command run).

The 2nd one throws back - Script result: bzip2: Can't open input file *.bz2: No such file or directory.

the 3rd one - bzip2: Can't open input file *.bz2: No such file or directory.
(standard_in) 2: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error

G'day eriking,
I have tried to use you cacher.py script on my caching server as I too need a nice way to to show this is saving us bandwidth.
Ever time i call cacher.py in terminal, i get the following output;

Traceback (most recent call last):
File "/usr/local/bin/cacher.py", line 913, in
main()
File "/usr/local/bin/cacher.py", line 896, in main
cacherdata = cacher(rawLog.readlines(), targetDate, friendlyNames)
File "/usr/local/bin/cacher.py", line 167, in cacher
frompeers_size = linesplit[17]
IndexError: list index out of range

python is not my strong suit and I hope this error is a simple misconfig on my behalf.
Any input from yourself would be greatly appreciated.

cheers Damien C

LogClientIdentity is not set
Please run sudo Cacher --configureserver and delete your log files.

sudo python ./Cacher --configureserver
LogClientIdentity is not set
Please run sudo Cacher --configureserver and delete your log files.

This is due to improper logic.

Upon Run the following shows:
(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 2: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error
(standard_in) 1: parse error

X IP Addresses hit the Caching Server yesterday consisting of:

When double clicking an alert generated by Cacher in the Server app - the view gets messed up, forcing to quit and restart the server app.

See GIF:

This is something I ran into recently while trying to get Cacher working with a new 5.2 server. I've always just manually ran sudo serveradmin settings caching:LogClientIdentity = 1 and had no issues. This time I tried to use sudo Cacher --configureserver. This reports a success, but Cacher never finds any stats. I think this is because --configureserver tries to set this setting with "yes", rather than "1". After manually running the appropriate serveradmin commands, Cacher is working without issue.

Looks like this line is what needs changed: https://github.com/erikng/Cacher/blob/master/Cacher#L636

I would like to pass in a different date than yesterday and get a report for that day. Appropriate errors should be issued for dates where logs do not exist anymore.

I see this error:

./cacher.py --targetdate=2017-06-20
Traceback (most recent call last):
  File "./cacher.py", line 950, in <module>
    main()
  File "./cacher.py", line 933, in main
    cacherdata = cacher(rawLog.readlines(), targetDate, friendlyNames)
  File "./cacher.py", line 177, in cacher
    frompeers_size = linesplit[17]
IndexError: list index out of range

It seems to be because the "Served all" line isn't using KB / MB / or GB:

bzgrep "Served all" /Library/Server/Caching/Logs/Debug-2017.06.20-07:16.log.bz2
2017-06-20 09:27:13.810 #ASDOW8CvyaXH Served all 530 of 530 bytes; 0 from cache, 530 stored from Internet, 0 from peers
2017-06-20 09:27:14.228 #Gv2SiVDKnjfx Served all 3061 of 3061 bytes; 0 from cache, 3061 stored from Internet, 0 from peers
2017-06-20 09:27:14.868 #0Bjo9F70zTHp Served all 530 of 530 bytes; 530 from cache, 0 stored from Internet, 0 from peers
2017-06-20 09:27:14.893 #Rv5z5LZVfqs/ Served all 3061 of 3061 bytes; 3061 from cache, 0 stored from Internet, 0 from peers
2017-06-20 09:27:17.353 #BYopVZ61tik+ Served all 3061 of 3061 bytes; 3061 from cache, 0 stored from Internet, 0 from peers
2017-06-20 19:08:24.110 #HRFS8tdWl/oo Served all 2671077 of 2671077 bytes; 0 from cache, 2671077 stored from Internet, 0 from peers
2017-06-20 22:09:51.306 #OYiGr6J5oTU2 Served all 3530118 of 3530118 bytes; 0 from cache, 3530118 stored from Internet, 0 from peers
2017-06-20 22:09:51.535 #Hi/jXmnizsOC Served all 2518340 of 2518340 bytes; 0 from cache, 2518340 stored from Internet, 0 from peers
2017-06-20 22:34:11.190 #3t7x/+Ft9289 Served all 5022 of 5022 bytes; 5022 from cache, 0 stored from Internet, 0 from peers
2017-06-20 22:34:11.190 #qB0Ld3yWIDf0 Served all 5023 of 5023 bytes; 5023 from cache, 0 stored from Internet, 0 from peers
2017-06-20 22:34:11.320 #HdH6mXxXQpq/ Served all 5639 of 5639 bytes; 5639 from cache, 0 stored from Internet, 0 from peers
2017-06-20 22:34:11.695 #MFU/uJqbMIOr Served all 5024 of 5024 bytes; 5024 from cache, 0 stored from Internet, 0 from peers
2017-06-20 22:34:13.309 #okBzTwaHA8D9 Served all 5979 of 5979 bytes; 5979 from cache, 0 stored from Internet, 0 from peers
2017-06-21 00:02:20.900 #FDN+/XnFoNtc Served all 2671077 of 2671077 bytes; 2671077 from cache, 0 stored from Internet, 0 from peers
2017-06-21 02:03:57.147 #dqwnWQVRnLIB Served all 35055540 of 35055540 bytes; 13572 from cache, 35041968 stored from Internet, 0 from peers

I'm using the latest version of El Capitan's Server app.

Will you add support Microsoft Teams?