ucberkeleyseti / meerkat-backend-interface Goto Github PK

In the 2020-5-21 test session, the values for RA (floating point degrees) and RA_STR (sexagesimal hours in colon delimited string form) were not equivalent. Likewise, the values for DEC (floating point degrees) and DEC_STR (sexagesimal degrees in colon delimited form) were not equivalent.

Occasionally, the KATPortalClient's connection to the KATPortal server times out. When this happens, manual intervention is required to get the system back into an operational state. The reason for these timeouts is not understood and may be outside our code base, but regardless of the underlying cause, KATPortalClient should handle this situation more gracefully so that the backend remains in an operational state (to whatever extent that's possible).

The Coordinator should set DATADIR to /buf0/YYYYMMDD/NNNN, where YYYYMMDD and NNNN are from the YYYYMMDD-NNNN MeerKAT schedule block ID for an observation.

A mode to record only when the primary user observes sources that are present in a list we specify. This list could contain a selection of calibrators that we'd like recorded when they are observed. An external, editable yaml file could work.

Based on subarray sensor values, automatically calculate the size of the safety margin (to add to the packet start index) before scheduling the start of a recording.

For now I am treating the scan state in the same way as the track state: we wish to record data anytime the array is in either of these states. The scan state appears to refer to observations in which the array scans smoothly through changing RA and Dec values, as opposed to stop-and-stare observations at fixed RA and Dec (in the case of the track state).

As it stands, RA and DEC will update smoothly in the headers in the scan state. Are there any other considerations? Do we want to communicate to the processing nodes that the array is in the scan state or is the above sufficient?

Retrieve and write following sensors:
spmc.array_1_wide_0.telstate.telstate to acquire telstate Redis endpoint
Delay and phasecal information:
observation.script-last-phaseup
observation.script-last-delay-calibration

Some of the meerkat-backend-interface processes print a very aesthetically appealing banner on stdout when they start, complete with ASCII art. When run in as a daemon, these banners just go to log files where they consume disk space. These scripts should only output the banner when stdout is going to a tty.

On 2020-10-20 a scan was recorded even though triggermode was ostensibly idle. At that time, due to a bug (since fixed), the coordinator was resetting triggermode every time the array was deconfigured. However, it's not clear if this is the reason behind the unexpected recording.

When resetting DWELL to the prior value, use hashpipe-redis gateway groups - currently the log file has an entry for every processing node in the subarray as follows:

[2021-11-02 20:00:35,504 - INFO - coordinator.py:734] Published DWELL=0 to channel bluse://blpn62/0/set
[2021-11-02 20:00:35,504 - INFO - coordinator.py:734] Published PKTSTART=0 to channel bluse://blpn62/0/set
[2021-11-02 20:00:35,605 - INFO - coordinator.py:734] Published DWELL=60 to channel bluse://blpn62/0/set

Need to setup logrotate BLUSE logs on head node

This should make these messages a little more timeous and will also improve the legibility of the coordinator log files.

Diagnostic feature needed: ability to specify that recording is to take place only when certain receivers are being used (e.g. L, UHF, S0-S4).

Some history:array_1:target entries have no content. The latest example is 20210107T101143.000Z. Is this because CAM sends us empty values? We should avoid creating empty (i.e. useless) entries in Redis, so this may require an explicit check before adding a new entry.

A dry-run mode would store all the Hashpipe-Redis gateway messages in the Redis database but not actually publish them to the processing nodes. This would allow us to run meerkat-reconfig later on even if stream subscriptions were not made when the initial configure:[product_id] message was received.

On 2021-11-15 at 01:31:16 local time, we subscribed to m000_target but received array_2:m000_target (with the consequence that a target was set for array_2, which was not configured at the time. katcp_server logs show that m000 was indeed part of the currently active array_1 and not the inactive array_2. Further investigation is needed to determine the cause of this.

When performing a pair of ON/OFF scans, the location of the OFF scan is typically specified as the location of the ON scan plus an offset (typically 3 degrees in our recent MeerKAT test scans). The RA and Dec values published must include this offset so that the RA and Dec values the metadata will refer to the actual RA and Dec location (in J2000 epoch) that the array was pointing towards.

The ON/OFF scans from the 2020-05-21 test session had equal values for RA and Dec. The offset was not included in the RA/Dec values for the OFF scan.

Construct CBF sensor names from automatically determined prefixes on configure - perhaps using pool_resources or similar. Allows the coordinator to function when cbf_dev_ components are used.

How will we get pointing position info to the backend? This is needed for target selector logic as well as beamformer delay/phase offset calculations.

The coordinator needs to publish CHAN_BW to the processing nodes. CHAN_BW is the bandwidth of a single F engine frequency channel (aka "coarse channel"). If the spectrum is inverted in frequency (e.g. by sampling the second Nyquist zone or by an odd number of lower sideband mixes), then CHAN_BW is negative.

The following is always true:

F(chan N+1) = F(chan N) + CHAN_BW

The absolute value of CHAN_BW is ADC_SAMPLE_RATE / 2 / FENCHAN, but the sign is dependent on the IF tuning and ADC sampling regime.

The coordinator needs to publish CHAN_BW to the processing nodes. CHAN_BW is the bandwidth of a single F engine frequency channel (aka "coarse channel"). If the spectrum is inverted in frequency (e.g. by sampling the second Nyquist zone or by an odd number of lower sideband mixes), then CHAN_BW is negative.

The following is always true:

F(chan N+1) = F(chan N) + CHAN_BW

The Coordinator needs to publish some frequency reference point so that the backends will be able to calculate OBSFREQ from SCHAN, OBSNCHAN (== NANTSNSTRMHNCHAN), and CHAN_BW. Not sure what frequency info is available from CAM, but we should probably pass it through with minimal modifications/derivations. Probably best to come up with a single scheme that will work for all bands (if possible)

How to convey to backends? How to store in data?

Currently, this value is manually specified in config.yml. In future it would be useful to have a function that automatically determines it (by calculating data rates using sensor values pulled from the Redis database).

RA and Dec values that get published to the compute nodes' status buffers as RA, RA_STR, DEC, and DEC_STR should be in the J2000 epoch. If we are given RA/Dec in the current epoch, we will have to un-precess it back to J2000 epoch before publishing to the status buffers.

Recording of channelised antenna voltage streams can be synchronised across processing instances, for example beginning and ending when packets with specific predetermined IDs/indices/timestamps are received.

This is for delay calculations etc for beam forming etc.

Lately, the katcp_server has been unable to extract the CBF component prefix and has been using the defaults each time. This should be investigated to see if there has been a change in the ?configure message we receive from CAM via KATCP.

The current source name(s) should be published to the status buffers. For initial testing, the source name to be published should be derived from the MeerKAT metadata (probably via KATPortalCleint?), but as we start to form beams off-boresight, the source names will be our own.

MeerKAT metadata includes several "target" values that are actually a multi-field custom formatted string. The timing and accuracy of these values is not clear, nor is the format completely understood by us yet (nor is the reason for inconsistencies observed in the values), but presumably there is some way to find out in a timely and accurate way which target the array is going to observe. Alternatively there may be a "source" value that would fit our needs better if only we know about it.

Suffice it to say, getting the source name reliably is currently a but of a conundrum.

If an antenna that has been included in a subarray is manually marked as faulty by an operator, data-suspect for that antenna becomes true. As a result, the subarray_data_suspect sensor changes to true during the observation. This can be mitigated by considering the marked-faulty sensor as well when calculating the value of subarray_data_suspect.

Maintaining a history of key metadata values is useful for knowing which antennas have observed which sources and when. Two of the more critical metadata fields to memorialize this way are antennas and target. Unfortunately, there are several different ways to get at subtly different antennas and target metadata values. We need a way to canonicalize this.

An initial demonstration of how this historical data could be maintained has been setup to scrape the katcp and katportal log files on the head node to populate redis keys:

history:<array_name>:antennas
history:<array_name>:target

These redis keys refer to hashes whose fields are timestamps and whose values are strings. The timestamps represent UTC time and are formatted as:

YYYYmmddTHHMMSS.SSSZ

Where YYYY is the four digit year, mm is the month, dd is the day, T is literal 'T' to separate date and time, HH is the hour, MM is the minute, SS.SSS is the seconds to millisecond precision, and Z is a literal 'Z' to indicate UTC.

For strftime() implementations that support the %N conversion specification, the format string for this format this is %Y%m%dT%H%M%S.%3NZ. For strftime() implementations that lack %N support, the format string is %Y%m%dT%H%M%S.000Z. The latter form sets the milliseconds field to "000" for simplicity.

The values for the antennas fields are a JSON string that represents an ordered list of antenna names, such as:

["m001", "m002", "m003", "m004"]

Note that JSON require double quotes.

The values for the target fields are just the string taken directly from the sensor value.

One other point is that the keys for these history hashes need to be canonicalized. The current use of <array_name> (e.g. array_1) is derived from the name that our KATCP server gets from CAM. For the target name, however, the value has been taken from different sensors, some without the array name in their names. How to best canonicalize these is an open question. Additional details about available MeerKAT sensor values are needed to illuminate a path forward.