jbrzusto / motusserver Goto Github PK

• need to use new tagAmbig table instead of batchAmbig
• include sensor metadata available via new API

Links like:

https://www.dropbox.com/l/scl/AAD...8k?text=1

are not currently handled by the dropbox API, so we can't script downloads.
Need to embroilHuman() for these...

2016-08-13 11:14:53 , post , http://motus.org/data/api/v1.0/deploytag.jsp , {"serno":"SG-0815BBBK1352","hash":"---","date":"20160813111453","format":"jsonp","login":"john","pword":"---","tagID":10695,"projectID":1,"status":"terminate","tsStart":1389398400.0,"tsEnd":1514764800.0,"deferTime":0.0,"speciesID":null,"markerType":null,"markerNumber":null,"lat":null,"lon":null,"elev":null,"comments":null,"ts":1471086893.0588}

ERROR: Error: Bad Request

This needs to work so that we can specify long-lived test tags, or known fixed-period deployments.
Alternatively, an API to update the tag lifespan would work.

We're checking whether new files are all more recent than the oldest existing files,
but "existing files" already includes the new files, given we've called sgMergeFiles.
So the test:

        canResume = min(f$ts) > lastTS

incorrectly returns FALSE always.

Since we no longer re-type a job from "email" to "newFiles", loading jobs for the email server
by job type doesn't make sense. We should instead load those jobs whose queue is null,
or better, add an E subqueue to /sgm/queue, and use that as the email queue.

To work around lack of setuid scripting, and factor out most functionality into the motus package.

• for each script X.R in /sgm/bin, create a shell script /SG/code/X that does

#!/bin/bash
printf -v A " %q" "$@"
sudo su -c "/sgm/bin/X.R $A" sg

Scripts to refactor:

• fixDOSfilenames.R (renamed from fix_8_3_filenames.R) Folded into parseFileNames()
• incomingEmail.sh
• mergeLotekFiles.R Folded into runNewFiles
• mergeSGFiles.R Folded into runNewFiles
• processIncomingFiles.R Folded into runNewFiles
• process_incoming_files.R Folded into runNewFiles
• processMessage.py: redone in R replacing python email package with /usr/bin/munpack -t
• runReceiver.R now rerunReceiver
• sg_remote
• sgwho.R
• updateAttachedSGs.R -> syncReceiver; spawned by sg_remote, and uses atjobs for recurrence

We should still plot new detections! To fix:

• grab receiver list from site .sqlite DB
• set time range to entire year

We have .DTA files from the same site where the serial number is constant but model is not.
So is serial number alone enough to identify the receiver?
Currently we use (model, serial number), which splits data in the case above.

this way, it's correct even if the GPS has not set the clock

Is claimJob() failing as an atomic get-and-set? Testing has failed to show any problem with how claimJob()
uses sqlite's transactional atomicity. And there are two newFiles jobs being created. How?

Answer

When handleFilesWrangled(j) is called, its last step is to set j$queue=0 and then move top job tj to
queue 0. At that point, we still have j$done=0, so there's a race: the processServer which claims tj might enqueue its subjob j before the emailServer has had a chance to mark j$done=1. If the processServer
wins, it again calls handleFilesWrangled(j) which sets j$queue=0, thereby creating another race in which another processServer which has obtained the inotify event for tj moving to queue 0, but not yet processed it, might be able to claim job j, since queue==0 is the condition for an unclaimed job.

Fix

Have handleFilesWrangled mark j$done=1 before moving tj to queue 0.

Seems like I've tripped over another R edge case; names(list(...)) is null when none
of the paramters in ... was named, but is a char vector (with "" for unnamed elements)
when at least one parameter has a name.

So safeSys() needs to take that into account.

R has more edges than a snowflake.

This will require fixing jbrzusto/find_tags#4

motusRegisterReceiver

• generates public/private keypair for a receiver that hasn't already self-registered with sensorgnome.org
• the SHA1 hash of the private key is used as the receiver's secret key for the motus API.
• users sg_remote and sg need to be merged as part of the re-working of how attached receivers
  connect
• for now, just create the keypair in a folder we own, rather than in
  /home/sg_remote/.ssh/generated_by_server

ltGetCodeset

• used to gain access to the RAM-locked copy of the Lotek codeset database
• but user 'sg' is already the owner of that, so don't use sudo if

Sys.getenv("USER")=='sg'` 

becaues there is currently no antenna status plot for these. Eventually, we'll add some indication of
antenna status based on all detections, real or otherwise.

e.g. single SG or Lotek files attached to an email

2016-08-13 11:15:02 , post , http://motus.org/data/api/v1.0/deploytag.jsp , {"serno":"SG-0815BBBK1352","hash":"38CFD4A404E36B6EA5A51E1430B3E4D8989F83F3","date":"20160813111502","format":"jsonp","login":"john","pword":"p@ndora's boxers","tagID":10695,"projectID":1,"status":"terminate","tsStart":1389398400.0,"tsEnd":1471046400.0,"deferTime":0.0,"speciesID":null,"markerType":null,"markerNumber":null,"lat":null,"lon":null,"elev":null,"comments":null,"ts":1471086902.9789}
=>
[1] "\r\n\r\n\r\n\r\n{\"importID\":5502,\"responseCode\":\"success-deploy\"}\r\n"

So it appears to work, but then the termination date stored is incorrect:

> subset(motusSearchTags(projectID=1), !is.na(tsEnd))[,c("tagID", "projectID", "status", "tsStart", "tsEnd")]
=>

tagID projectID status tsStart tsEnd
1 10695 1 pending 1388534400 1470919920

Note the change from 1471046400.0 in the request to 1470919920 in the query to see the results
of the request.

It's not unusual for a user to move a receiver from one fixed location to another, with or without
a site name change. We should detect these and generate a new receiver deployment record.
It is (fairly) safe to assume that the redeployment occurs between boot sessions.

Moreover, if a user records a new deployment with site name, but leaves the lat/lon/elev blank, these should be filled in automatically from the first new batch of data that includes files from after the
new deployment date.

Needs some thought...

Current column list of view returned by tagview:
hitID
runID
batchID
ts
sig
sigSD
noise
freq
freqSD
slop
burstSlop
runID:1
batchIDbegin
batchIDend
motusTagID
ant
len
batchID:1
monoBN
tsBegin
tsEnd
numHits
ts:1
motusDeviceID
tagID
projectID
mfgID
dateBin
type
codeSet
manufacturer
model
lifeSpan
nomFreq
offsetFreq
period
periodSD
pulseLen
param1
param2
param3
param4
param5
param6
param7
param8
tsSG
approved
deployID
status
tsStart
tsEnd:1
deferSec
speciesID
markerNumber
markerType
latitude
longitude
elevation
comments
id
bi
tsStartCode
tsEndCode
tagID:1
projectID:1
deployID:1
status:1
tsStart:1
tsEnd:2
deferSec:1
speciesID:1
markerNumber:1
markerType:1
latitude:1
longitude:1
elevation:1
comments:1
id:1
serno
receiverType
deviceID
macAddress
status:2
deployID:2
name
fixtureType
latitude:2
longitude:2
isMobile
tsStart:2
tsEnd:3
projectID:2
deployID:3
port
antennaType
bearing
heightMeters
cableLengthMeters
cableType
mountDistanceMeters
mountBearing
polarization2
polarization1
id:2
english
french
scientific
group
sort
id:3
name:1
tagsPermissions
sensorsPermissions
id:4
name:2
tagsPermissions:1
sensorsPermissions:1

tagview() leaves all the tag-based metadata blank, since there's nothing to join to the negative tag ID
that indicates an ambiguity.

Apparently, the problem is I don't understand the pch paramter to panel.xyplot()...
now I do: pch supplies the plotting characters by group, not by item.

need to use RECEIVERS.TXT rather than site .sqlite DB (which doesn't exist for Lotek receivers)

If motusRegisterReceiver() is called to register a receiver that has never connected to sensorgnome.org, we generate an openssh key pair for it. That key pair should be installed
on the SG if it ever does connect. This requires a change to the sgremote python script,
which is not currently on github

Files written before the GPS sets the SG clock can't be assigned to the correct (old) site using
only serial number and date; need to use boot number.

e.g. error from log:
Copying data to motus transfer tables...
Auto-disconnecting mysql connection (0, 1)
Error in value[3L] :
MotusQuery failed with error; Error in parseJSON(txt): parse error: premature EOF

                 (right here) ------^

• needs to incorporate private and normal data for (internal) comparison plot

maybe just quote non-named args when quoting?

Shortened filenames like POE9DA~6.GZ occasionally arrive in data dumps. These are perhaps
caused by people trying to copy raw SG files to the root directory in a VFAT partition, e.g.
on a memory stick. Reconstructing a usable original name can be done if the correct
receiver can be determined. When looking at other files in the same batch, even if there
are files from multiple receivers there, the first two characters of the shortened name will be
(ignoring case) the same as they were in the original names.

runReceiver.R uses R sink() to redirect R's stdout and stderr to the receiver logfile,
but these redirections are not inherited by child processes, so we use system(..., intern=TRUE )
to capture them. This captures a character vector of lines, which are then written to the
log file using cat(), which doesn't add newlines back in between items.

so we can verify emails are from legitimate users.

so that they match the codeset as shown in .DTA files
i.e. "Lotek-4" -> "Lotek4"

Once a job has completed (has no subjobs with done != 1), move it to /sgm/done,
so it won't have to be rescanned if/when process servers are restarted.

Eg. for project 77 (which has 2 SG and 10 Lotek receivers) I get only the SGs:

Query: (from my log file)

2016-08-11 16:51:56 , get , http://motus.org/data/api/v1.0/listsensors.jsp , {"serno":"SG-0815BBBK1352","hash":"8522163403A60D12317578B7A42B2936A30A92CD","date":"20160811165156","format":"jsonp","login":"john","pword":"XXXXXXXXXX","projectID":77,"year":null,"serialNo":null}

Reply:

{"data":[{"id":1554,"serno":"SG-0815BBBK1352","receiverType":"SENSORGNOME","deviceID":2,"macAddress":"0","year":2015,"status":"deployed"},{"id":1343,"serno":"SG-0815BBBK2114","receiverType":"SENSORGNOME","deviceID":3,"macAddress":"0","year":2015,"status":"deployed"}]}

• generate fullID factor with levels sorted by projLabel, as.numeric(mfgID)
• include motus tag ID in fullID, especially since otherwise there may be identical
  labels for different identical tags in different years

Write a small .sqlite extension to generate an .rds file directly from an sqlite query, to avoid the
high memory requirements of using dbGetQuery %>% saveRDS

features:

• all strings saved as factors
• little-endian, if .rds format supports it
• doubles as doubles
• integers as integers

compressed, of course.

we still want to get e.g. receiver serial number, codeset and antenna records, etc.

to speed lookup with tagview()

There's currently no motus.org API for retrieving GPS fixes from receiver data, so we don't
have these available for time-based lookup of a tag detection from a mobile site.
For now, we only have the lat/lon available in the recvDeps table.

Does this matter? The code ends up using the gain setting from A1 for this.

These tags are sought because they're present in the events table, but are not displayed
we join against the tagDeps table which has no record of them.
Fix this by having cleanTagRegistrations() add records to tagDeps to match the tsStart and tsEnd values generated for the events table.

What's missing:

• hourly indication of the receiver running but not generating pulses; mainly useful for diagnosing
  stalled antennas. (red vs. non-red on old-style operating_status plots)

• hourly indication of whether the GPS has stuck; mainly useful to indicate periods of time when the
  receiver clock might be drifting (yellow vs. green on old-style operating_status plots)

• perhaps harvest reboot indication from .DTA files (done in dd34358)

Currently, all files at an emailed link destination are downloaded recursively, and some users
are just resending links to the same location when new files are added, so we're wastefully re-downloading files we already have.

To fix, re-tool server(), handleEmail() and download.XXX() to use:

• FTP: rsync
• dropbox, googleDrive: check for existing files via respective API s