mwatelescope / mwalib Goto Github PK

View Code? Open in Web Editor NEW

10.0 6.0 2.0 57.1 MB

Library to read Murchison Widefield Array (MWA) raw visibilities, voltages and metadata into a common structure

License: Mozilla Public License 2.0

Rust 97.07% Python 2.93%

mwa radio-astronomy cfitsio rust

mwalib's People

Stargazers

Watchers

Forkers

d3v-null cjordan

mwalib's Issues

Improved FFI error handling

From Chris Jordan: "I've recently stumbled upon this guide for error handling: https://michael-f-bryan.github.io/rust-ffi-guide/errors/return_types.html
This makes things much cleaner by having mwalib track the last error and the caller allocates their own string to display it if they want to. Right now, mwalib FFI callers are allocating an error string length which could be overflowed (this is pedantic, but I think it's nice to have a loose end removed) and providing two extra arguments to every function. It's also possible to handle panics better (which aren't handled at all right now). I've been working on doing all this kinda thing through hyperbeam, let me know if you wanna chat about it."

Implement voltage data read functionality

The VCS team want a simple read method (for now) which will be for a coarse channel and a second of data (note: not a timestep necessarily for mwax as each sub file contains 8 seconds of data).

Do not keep fits files open permanently

Only open and close fits files as needed. This will also help make mwalib more compatible for pyo3.

Benchmarking of read methods

It would be great if MWALib could benchmark and track performance of read methods in order to easily detect performance regressions.
For example, Birli tracks benchmarks using Criterion to measure benchmarks: https://github.com/MWATelescope/Birli/tree/parallel_flagging

Implement external CI runner

Defining vcsorder for rfinputs >= 256

Because the legacy VCS format was only supported during a time when there were no more than 128 tiles (= 256 RF inputs) plugged in at any one time, the quantity called vcs_order which defined the ordering of the tiles in the legacy .dat files is, strictly speaking, not defined for RF input indices >= 256. One downside of not defining vcs_order for extra inputs is that without such a definition, one cannot convert from the MWAX format to the legacy format.

Assuming this is a desirable feature to have, I propose extending the definition of vcs_order for cases when the number of RF inputs >= 256. Since the number of RF inputs need not be a multiple of 256, simply extending the same pattern as exists for < 256 would not be desirable. Instead, I propose simply adding the extra tiles onto the end of the array sequentially. This would turn the existing definition

(input & 0xC0) | ((input & 0x30) >> 4) | ((input & 0x0F) << 2)

into

(input < 256 ? (input & 0xC0) | ((input & 0x30) >> 4) | ((input & 0x0F) << 2) : input)

which remains backwards compatible, and extends the definition of vcs_order to an arbitrary number of tiles.

When missing metafits keyword, provide better error message to users

The error should point them to the MWA wiki to download a fresh metafits for their observation, instead of just "key not found".

can't read gpuboxes when MILLITIM not set

there are several obsids where the MILLITIM header is not set correctly.

from the metafits:

INTTIME = 0.5s
NSCANS = 224

-> hdus.info():
Filename: /data/curtin_mwaeor/data/1059505936/raw/1059505936.metafits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      63   ()      
  1  TILEDATA      1 BinTableHDU     52   256R x 19C   [I, I, I, 8A, A, I, I, I, 14A, E, E, E, 24I, E, 16I, I, 10A, E, 24E]   

HEADER

 -> primary_hdu.header
SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                    8 / array data type                                
NAXIS   =                    0 / number of array dimensions                     
EXTEND  =                    T                                                  
GPSTIME =           1059505936 / [s] GPS time of observation start              
EXPOSURE=                  112 / [s] duration of observation                    
FILENAME= 'high_season1_2456507' / Name of observation                          
MJD     =              56506.8 / [days] MJD of observation                      
DATE-OBS= '2013-08-02T19:12:00' / [UT] Date and time of observation             
LST     =     356.212681700136 / [deg] LST                                      
DUT1    =           0.05084286 / UT1-UTC from astropy.time.Time.delta_ut1_utc   
HA      = ' 00:00:00.00'       / [hours] hour angle of pointing center          
AZIMUTH =                  0.0 / [deg] Azimuth of pointing center               
ALTITUDE=                 90.0 / [deg] Altitude of pointing center              
RA      =     356.033853326675 / [deg] RA of pointing center                    
DEC     =   -26.78033855169699 / [deg] Dec of pointing center                   
RAPHASE =                  0.0 / [deg] RA of desired phase center               
DECPHASE=                -27.0 / [deg] DEC of desired phase center              
DELAYMOD=  / MWAX correlator delay mode                                         
ATTEN_DB=                  1.0 / [dB] global analogue attenuation, in dB        
SUN-DIST=     139.231890839331 / [deg] Distance from pointing center to Sun     
MOONDIST=    98.80546109496019 / [deg] Distance from pointing center to Moon    
JUP-DIST=     111.036656502106 / [deg] Distance from pointing center to Jupiter 
GRIDNAME= 'EOR1    '           / Pointing grid name                             
GRIDNUM =                    0 / Pointing grid number                           
CREATOR = 'DJacobs '           / Observation creator                            
PROJECT = 'G0009   '           / Project ID                                     
MODE    = 'HW_LFILES'          / Observation mode                               
RECVRS  = '1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16' / Active receivers           
DELAYS  = '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0' / Beamformer delays                 
CALIBRAT=                    F / Not intended for calibration                   
CENTCHAN=                  143 / Center coarse channel                          
CHANNELS= '131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147&'
CONTINUE  ',148,149,150,151,152,153,154&'                                       
CONTINUE  '' / Coarse channels                                                  
CHANSEL = '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23' / Indi
SUN-ALT =    -49.2318908393306 / [deg] Altitude of Sun                          
FINECHAN=                 40.0 / [kHz] Fine channel width - correlator freq_res 
INTTIME =                  0.5 / [s] Individual integration time                
NAV_FREQ=                    4 / # of 10kHz chans avgd to make each fine chan   
NSCANS  =                  224 / Number of time instants in correlation products
NINPUTS =                  256 / Number of inputs into the correlation products 
NCHANS  =                  768 / Number of (averaged) fine channels in spectrum 
BANDWDTH=                30.72 / [MHz] Total bandwidth                          
FREQCENT=              182.415 / [MHz] Center frequency of observation          
TIMEOFF =                    0 / [s] Deprecated, use QUACKTIM or GOODTIME       
DATESTRT= '2013-08-02T19:12:00' / [UT] Date and time of correlations start      
RAWSCALE=                0.003 / Raw visibilities multiplied by this value      
VERSION =                  2.2 / METAFITS version number                        
TELESCOP= 'MWA     '                                                            
INSTRUME= '128T    '                                                            
QUACKTIM=                  0.5 / Seconds of bad data after observation starts   
GOODTIME=         1375470720.5 / OBSID+QUACKTIME as Unix timestamp              
DATE    = '2022-07-25T06:01:12' / UT Date of file creation                      
COMMENT After an observation starts, the receiver hardware changes take a       
COMMENT few seconds to stabilise.                                               
COMMENT QUACKTIM and GOODTIME represent the start of the first uncontaminated   
COMMENT data, rounded up to the next time-averaged data packet. Note that this  
COMMENT time may be before the first actual data in some or all gpubox files.   
COMMENT See metafits file documentation here:                                   
COMMENT https://wiki.mwatelescope.org/display/MP/Metafits+files                 
COMMENT https://wiki.mwatelescope.org/display/MP/MWAX+Metafits+Changes          
HISTORY Created by user apache                                                  
HISTORY Created on host mwa-django01p.curtin.edu.au                             
HISTORY Command: "mod_wsgi"

first few ImageHDUs in the gpubox file:

-> hdus.info():
Filename: 1059505936_20130802191158_gpubox01_00.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      11   ()      
  1                1 ImageHDU        12   (66048, 32)   float32   
  2                1 ImageHDU        10   (66048, 32)   float32   
  3                1 ImageHDU        12   (66048, 32)   float32   
  4                1 ImageHDU        10   (66048, 32)   float32   
  5                1 ImageHDU        12   (66048, 32)   float32   
  6                1 ImageHDU        10   (66048, 32)   float32   
  7                1 ImageHDU        12   (66048, 32)   float32   
  8                1 ImageHDU        10   (66048, 32)   float32   
  9                1 ImageHDU        12   (66048, 32)   float32   
 10                1 ImageHDU        10   (66048, 32)   float32   
 11                1 ImageHDU        12   (66048, 32)   float32   
 12                1 ImageHDU        10   (66048, 32)   float32   
 13                1 ImageHDU        12   (66048, 32)   float32   
 14                1 ImageHDU        10   (66048, 32)   float32   
 15                1 ImageHDU        12   (66048, 32)   float32   
 16                1 ImageHDU        10   (66048, 32)   float32   
 17                1 ImageHDU        12   (66048, 32)   float32   
 18                1 ImageHDU        10   (66048, 32)   float32   
 19                1 ImageHDU        12   (66048, 32)   float32   
 20                1 ImageHDU        10   (66048, 32)   float32   
 21                1 ImageHDU        12   (66048, 32)   float32   
 22                1 ImageHDU        10   (66048, 32)   float32   
 23                1 ImageHDU        12   (66048, 32)   float32   
 24                1 ImageHDU        10   (66048, 32)   float32   
 25                1 ImageHDU        12   (66048, 32)   float32   
 26                1 ImageHDU        10   (66048, 32)   float32   
 27                1 ImageHDU        12   (66048, 32)   float32   
 28                1 ImageHDU        10   (66048, 32)   float32   
 29                1 ImageHDU        12   (66048, 32)   float32   
 30                1 ImageHDU        10   (66048, 32)   float32   
 31                1 ImageHDU        12   (66048, 32)   float32   
 32                1 ImageHDU        10   (66048, 32)   float32   
 33                1 ImageHDU        12   (66048, 32)   float32   
 34                1 ImageHDU        10   (66048, 32)   float32   
 35                1 ImageHDU        12   (66048, 32)   float32   
 36                1 ImageHDU        10   (66048, 32)   float32   
 37                1 ImageHDU        12   (66048, 32)   float32   
 38                1 ImageHDU        10   (66048, 32)   float32   
 39                1 ImageHDU        12   (66048, 32)   float32   
 40                1 ImageHDU        10   (66048, 32)   float32   
 41                1 ImageHDU        12   (66048, 32)   float32   
 42                1 ImageHDU        10   (66048, 32)   float32   
 43                1 ImageHDU        12   (66048, 32)   float32   
 44                1 ImageHDU        10   (66048, 32)   float32   
 45                1 ImageHDU        12   (66048, 32)   float32   
 46                1 ImageHDU        10   (66048, 32)   float32   
 47                1 ImageHDU        12   (66048, 32)   float32   
 48                1 ImageHDU        10   (66048, 32)   float32   
 49                1 ImageHDU        12   (66048, 32)   float32   
 50                1 ImageHDU        10   (66048, 32)   float32   
 51                1 ImageHDU        12   (66048, 32)   float32   
 52                1 ImageHDU        10   (66048, 32)   float32   
 53                1 ImageHDU        12   (66048, 32)   float32   
 54                1 ImageHDU        10   (66048, 32)   float32   
 55                1 ImageHDU        12   (66048, 32)   float32   
 56                1 ImageHDU        10   (66048, 32)   float32   
 57                1 ImageHDU        12   (66048, 32)   float32   
 58                1 ImageHDU        10   (66048, 32)   float32   
 59                1 ImageHDU        12   (66048, 32)   float32   
 60                1 ImageHDU        10   (66048, 32)   float32   
 61                1 ImageHDU        12   (66048, 32)   float32   
 62                1 ImageHDU        10   (66048, 32)   float32   
 63                1 ImageHDU        12   (66048, 32)   float32   
 64                1 ImageHDU        10   (66048, 32)   float32   
 65                1 ImageHDU        12   (66048, 32)   float32   
 66                1 ImageHDU        10   (66048, 32)   float32   
 67                1 ImageHDU        12   (66048, 32)   float32   
 68                1 ImageHDU        10   (66048, 32)   float32   
 69                1 ImageHDU        12   (66048, 32)   float32   
 70                1 ImageHDU        10   (66048, 32)   float32   
 71                1 ImageHDU        12   (66048, 32)   float32   
 72                1 ImageHDU        10   (66048, 32)   float32   
 73                1 ImageHDU        12   (66048, 32)   float32   
 74                1 ImageHDU        10   (66048, 32)   float32   
 75                1 ImageHDU        12   (66048, 32)   float32   
 76                1 ImageHDU        10   (66048, 32)   float32   
 77                1 ImageHDU        12   (66048, 32)   float32   
 78                1 ImageHDU        10   (66048, 32)   float32   
 79                1 ImageHDU        12   (66048, 32)   float32   
 80                1 ImageHDU        10   (66048, 32)   float32   
 81                1 ImageHDU        12   (66048, 32)   float32   
 82                1 ImageHDU        10   (66048, 32)   float32   
 83                1 ImageHDU        12   (66048, 32)   float32   
 84                1 ImageHDU        10   (66048, 32)   float32   
 85                1 ImageHDU        12   (66048, 32)   float32   
 86                1 ImageHDU        10   (66048, 32)   float32   
 87                1 ImageHDU        12   (66048, 32)   float32   
 88                1 ImageHDU        10   (66048, 32)   float32   
 89                1 ImageHDU        12   (66048, 32)   float32   
 90                1 ImageHDU        10   (66048, 32)   float32   
 91                1 ImageHDU        12   (66048, 32)   float32   
 92                1 ImageHDU        10   (66048, 32)   float32   
 93                1 ImageHDU        12   (66048, 32)   float32   
 94                1 ImageHDU        10   (66048, 32)   float32   
 95                1 ImageHDU        12   (66048, 32)   float32   
 96                1 ImageHDU        10   (66048, 32)   float32   
 97                1 ImageHDU        12   (66048, 32)   float32   
 98                1 ImageHDU        10   (66048, 32)   float32   
 99                1 ImageHDU        12   (66048, 32)   float32   
100                1 ImageHDU        10   (66048, 32)   float32   
101                1 ImageHDU        12   (66048, 32)   float32   
102                1 ImageHDU        10   (66048, 32)   float32   
103                1 ImageHDU        12   (66048, 32)   float32   
104                1 ImageHDU        10   (66048, 32)   float32   
105                1 ImageHDU        12   (66048, 32)   float32   
106                1 ImageHDU        10   (66048, 32)   float32   
107                1 ImageHDU        12   (66048, 32)   float32   
108                1 ImageHDU        10   (66048, 32)   float32   
109                1 ImageHDU        12   (66048, 32)   float32   
110                1 ImageHDU        10   (66048, 32)   float32   
111                1 ImageHDU        12   (66048, 32)   float32   
112                1 ImageHDU        10   (66048, 32)   float32   
113                1 ImageHDU        12   (66048, 32)   float32   
114                1 ImageHDU        10   (66048, 32)   float32   
115                1 ImageHDU        12   (66048, 32)   float32   
116                1 ImageHDU        10   (66048, 32)   float32   
117                1 ImageHDU        12   (66048, 32)   float32   
118                1 ImageHDU        10   (66048, 32)   float32   
119                1 ImageHDU        12   (66048, 32)   float32   
120                1 ImageHDU        10   (66048, 32)   float32   

HEADER

SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    0 / number of data axes                            
EXTEND  =                    T / FITS dataset may contain extensions            
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
TIME    =           1375470717 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           
PROJID  = 'G0009' / MWA Project Id                                              
OBSID   = '1059505936' / MWA Observation Id                                     

VISIBILITIES (120):


 -> SCAN 0 IMAGE

XTENSION= 'IMAGE   '           /Image Extension created by MWA DataCapture      
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 /                                                
GCOUNT  =                    1 /                                                
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
TIME    =           1375470717 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 1 IMAGE

XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
TIME    =           1375470717 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 2 IMAGE

XTENSION= 'IMAGE   '           /Image Extension created by MWA DataCapture      
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 /                                                
GCOUNT  =                    1 /                                                
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
TIME    =           1375470718 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 3 IMAGE

XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
TIME    =           1375470718 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 4 IMAGE

XTENSION= 'IMAGE   '           /Image Extension created by MWA DataCapture      
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 /                                                
GCOUNT  =                    1 /                                                
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
TIME    =           1375470719 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 5 IMAGE

XTENSION= 'IMAGE   '           / IMAGE extension                                
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 / required keyword; must = 0                     
GCOUNT  =                    1 / required keyword; must = 1                     
TIME    =           1375470719 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)                           

 -> SCAN 6 IMAGE

XTENSION= 'IMAGE   '           /Image Extension created by MWA DataCapture      
BITPIX  =                  -32 / number of bits per data pixel                  
NAXIS   =                    2 / number of data axes                            
NAXIS1  =                66048 / length of data axis 1                          
NAXIS2  =                   32 / length of data axis 2                          
PCOUNT  =                    0 /                                                
GCOUNT  =                    1 /                                                
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 
TIME    =           1375470720 / Unix time (seconds)                            
MILLITIM=                    0 / Milliseconds since TIME                        
INTTIME =                  0.5 / Integration time (s)

I would expect that every second HDU should have the MILLITIM set to 500, but instead we have duplicate timestamps.

via birli, we can get the expected timesteps from mwalib according to the metafits. where there is a p next to the timestep, that timestep appears in provided timesteps.

[2022-07-25T06:49:30Z INFO  birli::cli] birli version 0.7.0
    Compiled on git commit hash: <no git info>
                Thu, 14 Jul 2022 12:46:49 +0000
             with compiler rustc 1.61.0 (fe5b13d68 2022-05-18)
    
    observation name:     high_season1_2456507
    Array position:       { longitude: 116.6708°, latitude: -26.7033°, height: 377.827m }
    Phase centre:         (0.0000°, -27.0000°) => (0h00m00.0000s, -27d00m00.0000s)
    Pointing centre:      (356.0339°, -26.7803°) => (23h44m08.1247s, -26d46m49.2187s)
    Scheduled start:      2013-08-02 19:12:00.000 UTC, unix=1375470720.000, gps=1059505936.000, mjd=4882187520.000, lmst=356.2094°, lmst2k=356.0303°, lat2k=-26.7803°
    Scheduled end:        2013-08-02 19:13:52.000 UTC, unix=1375470832.000, gps=1059506048.000, mjd=4882187632.000, lmst=356.6773°, lmst2k=356.4986°, lat2k=-26.7804°
    Scheduled duration:   112.000s = 224 * 0.500s
    Quack duration:       0.500s =   1 * 0.500s
    Output duration:      112.000s =  56 * 2.000s (4x)
    Scheduled Bandwidth:  30.720MHz =  24 *  32 * 40.000kHz
    Output Bandwidth:     30.720MHz =       768 * 40.000kHz
    Timestep details (all=230, provided=112, common=1, good=1, select=221, flag=230):
             2013-08-02 UTC +  unix [s]        gps [s]         p  c  g  s  f 
       ts0:      19:11:57.000  1375470717.000  1059505933.000  p           f 
       ts1:      19:11:57.500  1375470717.500  1059505933.500              f 
       ts2:      19:11:58.000  1375470718.000  1059505934.000  p  c     s  f 
       ts3:      19:11:58.500  1375470718.500  1059505934.500           s  f 
       ts4:      19:11:59.000  1375470719.000  1059505935.000  p        s  f 
       ts5:      19:11:59.500  1375470719.500  1059505935.500           s  f 
       ts6:      19:12:00.000  1375470720.000  1059505936.000  p        s  f 
       ts7:      19:12:00.500  1375470720.500  1059505936.500           s  f 
       ts8:      19:12:01.000  1375470721.000  1059505937.000  p     g  s  f 
       ts9:      19:12:01.500  1375470721.500  1059505937.500           s  f 
      ts10:      19:12:02.000  1375470722.000  1059505938.000  p        s  f 
      ts11:      19:12:02.500  1375470722.500  1059505938.500           s  f 
      ...

mwalib does not give any warning about these duplicate timesteps.

Notes from Andrew Williams' analysis of GPU box timing, written in 2020:

2012 to 2014(ish): The Dark Ages - chaos and uncertainty, before the civilised era. Code changed day-to-day, there were no dipole tests or flags, and the QUACKTIME and GOODTIME cards were a distant dream. Leap second offsets were hard-coded in dozens of places. Don't trust any timestamps to better than a few seconds. The data in a visibility file starts anywhere from -1 to 4 seconds after the obsid, and the first 0-4 seconds should be discarded, depending on correlator averaging time. This era is why Cotter discards the first four seconds of data, by default. Here be dragons in the data.

2014(ish) to mid 2017 : The Renaissance - the code is stabilising, dipole tests and flags exist. From here, visibility files should always start 2 seconds after the start of an observation. There is no QUACKTIM or GOODTIME card in original the metafits file, but newly created metafits files will have those cards.

mid 2017 to now : The Modern Era - stable correlator code, no more leap seconds. QUACKTIM and GOODTIME headers, and data files that always start exactly 2 seconds after the obsid. Because of that 2 seconds without data, only attenuation settings should ever contaminate recorded data, only if the initial and final attenuations are different, and only for one correlator dump time.

2021 onwards: The Future - new (already tested) receiver code drops attenuation change times to < 0.5 seconds. The new MWAX correlator solves all of the problems of the old correlator, and has no bugs of its own. The promise of new receivers heralds the dawn of a new age

Add new metafits key / value RAWSCALE

This is to let the consumer of mwalib know what scaling factor has been applied to visibilities to get them to be in "jansky-like" units

Change visibility pol to use enums

Change pol to be enum of XX,XY,YX,YY
Rename string pol to be pol_name
Add pol1 to be enum of X,Y
Add pol2 to be enum of X,Y
Also add attributes to help Biril convert to MS and uvfits by providing the appropriate order attributes.
So the struct will look like:
{
pol: XX,
pol_name: "XX"
pol1: X
pol2: X
},
{
pol: XY,
pol_name: "XY"
pol1: X
pol2: Y
},
...

Expose digital_gains, dipole_gains, dipole_deplays in rf_input to FFI interface

Digital gains read from metafits tiledata table should be divided by 64 and exposed as an f64

From cotter source:
// The digital pfb gains are multiplied with 64 to allow more careful
// fine tuning of the gains.
inputs[input].pfbGains[sb] = (double) gainValues[sb] / 64.0;

Just to be clear "pfb" refers to the reciever gains not the fine pfb gains.

This should be accounted for in mwalib. Divide the gains provided by the metafits by 64 then expose in an f64.

GPUBox numbers assigned 'incorrectly' for legacy VCS metafits

The GPU box numbers (CoarseChannel.gpubox_number) appear to be set to the receiver channel numbers (at least) for contexts created from legacy VCS ('VOLTAGE_START') metafits files. However, the ("true") GPU box numbers are needed, even for VOLTAGE_START metafits, in those cases where calibration solutions are actually built from VCS data via the offline correlator.

Visibility interface does not allow access to all timesteps, and coarse channel indices are ambiguous

TL;DR

Currently MWALib visibility interface only provides visibilities for timesteps which are common to all coarse channels. This is an issue because sometimes gpubox files start at different times, and the behaviour of cotter is to read these "incomplete" timesteps, however there is no way to do this in MWALib at the moment.

Background

There are multiple ways that MWALib currently looks at timesteps in different contexts:

MetafitsContext:
- sched_(start|end)_(gps_time_ms|unix_time_ms|utc|mjd) - scheduled
CorrelatorContext:
- gpubox_time_map - a map whose keys are each unix time from every gpubox, not necessarily contiguous. associates unix time with a map of gpubox_number to a tuple of batch number and hdu index.
- timesteps: a vector of TimeSteps which are common to the largest number of gpubox files. Contiguous
- (start|end)_(unix|gps)_time_ms can be before scheduled
TimeStep:
- unix_time_ms - UNIX time (in milliseconds to avoid floating point inaccuracy)
- gps_time_ms - gps time (in milliseconds)

There are multiple ways that MWALib currently looks at coarse channels in different contexts:

metafits file:
- CHANNELS and other headers on fits primary hdu, list of corr_chan_number, rec_chan_number, gpubox_number?, used by populate_coarse_channels?
- not sure if a gpubox is down, whether metafits file still says it's there.
MetafitsContext:
- no way to determine coarse channels from metafits context?
- use CoarseChannel.get_metafits_coarse_chan_array(metafits_coarse_chans_string) instead.
what about when we're missing a gpubox?
we don't care about handling a gpubox file which is not know about in the metafits.
need to know if it's legacy or mwax, and voltage or correlator to determine coarse channels from metafits
CoarseChannel:
- corr_chan_number - Correlator channel is 0 indexed (0..N-1)
- rec_chan_number - Receiver channel is 0-255 in the RRI recivers
- gpubox_number - ambiguous - only makes sense for legacy. in MWAX, this is the rec_chan_number.
coarse_chan_index - index of coarse channel according to CorrelatorContext.coarse_chans

The Issue

This is an issue, because the gpubox time map could look like this, for an observation with scheduled start time 100.

cc \ ts	99	100	101
0		x	x
1	x	x	x
2

note:

coarse channel 0 starts at scheduled starts
coarse channel 1 starts before scheduled start
coarse channel 2 has no data

Some users want to extract "all" visibilities (ts99 - ts101), where other users want to extract only "good" visibilities (ts100 - ts101, ignoring cc2). Currently MWALib only handles the "good" use case.

The changes

We would like to make the following changes:

MetafitsContext:
- timesteps - all timesteps that the metafits context knows about (from sched_start time to ...)
  - edit: it may not be necessary to include all known timesteps (sched_start etc.) since these are not necessarily going to be useful for reading visibilities. sched_start could be before or after the start of provided timesteps. The timesteps which are necessary to access all visibilities can't be known until the gpufits files are analysed, and so it may not be necessary to include this timestep array in metafits at all,
- coarse_chans - all possible coarse channels (based on metafits headers such as CHANNELS).
  - see: CoarseChannel::populate_coarse_channels
  - not necessarily contiguous, because of "picket fence"
- question: not sure if gpuboxes which are down are included? maybe document this.
CorrelatorContext
- gpubox_time_map - should be updated to include data for all timesteps that the correlator context knows about. This could be a superset or a subset of metafits, and is not necessarily contiguous.
  - this should be public accessible
  - accessible through FFI?
  - do we want to change the keys of the inner map from gpubox_number to some other key while we're at it? since gpubox_number is a vestigial concept for mwax? Probably not.
- timesteps - This should be updated to be a union of metafits and gpubox timesteps
  - this is the "master" timesteps, and is contiguous.
- common_timesteps - some interface for the user to determine the timesteps which are common to all provided coarse channels. Must be contiguous, and must all have data.
  - edge case: if a provided coarse channel is empty, then the common timesteps is also empty.
  - Edge case: if there is a gap in timesteps, everything after that gap is not included in common.
  - Possible implementation:
    - Slice - can this be empty? Yes. - probably the neatest.
    - Vector of TimeStep structs.
    - Vector of references? - might not be possible.
    - Option - Either Some or None of empty
- good_timesteps - same as common timesteps, but without timesteps before "good time".
- (start|end)_(unix|gps)_time_ms - this should now be determined from CorrelatorContext::timesteps
- duration_ms - this should now be determined from CorrelatorContext::timesteps
- provided_coarse_chans - just the provided coarse channels:
  - always a subset of MetafitsContext.coarse_chans
  - not necessarily contiguous, because of "picket fence"
  - edge case: do we care if any of these are empty? No, include these anyway.
- provided_timesteps - a vector of all timesteps which have any data associated with them, not necessarily contiguous.
all methods which take timestep or coarse channel indices as arguments:
- need a new error when there is no data for that coarse channel / time step idx combo.
  - implementation options:
    - new Rust error type, "no data for this combination"
    - FFI will need new return codes.
- must be modified so that the timestep indices are based on those from CorrelatorContext::timesteps.
- index of the coarse channel according to MetafitsContext.coarse_chans.
- e.g. CorrelatorContext::read_by_(frequency|baseline)

Nice to have:

remove any references to the word "actual", this includes comments, and impl fmt::Display for CorrelatorContext
add ability to specify timestep / coarse channel object for reading visibilities
methods which accept coarse channels could be duplicated to take either corr_chan_number or rec_chan_number instead of

Not necessary:

(start|end)_good_(unix|gps)_time_ms - not necessary, can be determine from good_timesteps.(first|last)
(start|end)_common_(unix|gps)_time_ms - not necessary, can be determine from common_timesteps.(first|last)
CorrelatorContext::provided_timesteps - e.g. if batch 00 and batch 02 were provided, but not batch 01, no known use cases.
CorrelatorContext::new, specify whether you want:
- all timesteps
- common timesteps

Potential issues / edge cases

correlator can provide visibilities from before metafits says data should start, e.g. obs_id 1065880128
- what does cotter do?
- negative indices? No.
- will anyone ever use this? Probably.
- everything prior to good_time is "useless" because receivers are changing in frequency and pointing
- good_time can never be before metafits start time because minimum quack_time is 0.5s
what if a leap second occurs during an observation? do unix_time_ms and gps_time_ms handle leap seconds the same way?
- M&C Guarantees that we never do an observation within ~12 hours of a leap second.
"picket fence" coarse channels
missing HDUs in middle of gpubox
missing batches (e.g. provide batch 00 and 02 but not 01)
provided gpubox with no data
what happens when a gpubox is down?

Provide mechanism to read and interpret MWA flag file format (.mwaf)

Need to check in cotter, what order the tiles/baselines are in.

Migrate to hifitime for time keeping

From Chirs Jordan: "I'm not sure how familiar you are with the rust library hifitime. We're using it in Marlu (and onwards) and it's wonderful; you can represent timestamps with nanosecond precision (hifitime::Epoch) and the struct methods allow you to convert to whatever (GPS, MJD, JD etc.). So, it might make sense to eventually use this as the single time-keeping method in the mwalib context structs, rather than having multiple time formats and things like seconds and milliseconds. Time resolutions etc. can be represented by hifitime::Duration, and this has the advantage of tracking units internally. I guess the trickiest part would be FFI; it's probably not worth exposing hifitime types to FFI callers, but you could convert to whatever you needed at the boundary. Just some food for thought."

Provide a polarisations array

This will just be a four element array so that callers don't need to guess the order of polarisations in the MWA data. e.g. "XX","XY","YX","YY"

Add extra attributes for times in mwalibContext

Add:

scheduled_start_unix_time_milliseconds
scheduled_end_unix_time_milliseconds
scheduled_end_utc
scheduled_end_mjd

metafits_fine_chan_freqs in FFI should be renamed to metafits_fine_chan_freqs_hz

Expose pointing table from metafits

Some metafits files will have a pointing table which would be necessary for some preprocessing.
See:

can't read obs with inconsistent timesteps

There is an issue in several obsids with an integration time of 2 or more seconds, where the TIME header in one of the gpuboxes is off by one second from the rest. Here are some examples, but the list is certainly not complete:

When you try and read these with mwalib, you get a NoCommonTimesteps error, because the the mapping from timestep to coarse channel/hdus to look like the example below. Basically, channel 9 has a different set of times to the other channels.

{
    1408127952000: {
        1: (0, 1), 2: (0, 1), 3: (0, 1), 4: (0, 1), 5: (0, 1), 6: (0, 1), 7: (0, 1), 8: (0, 1), 10: (0, 1), 11: (0, 1), 12: (0, 1), 13: (0, 1), 14: (0, 1), 15: (0, 1), 16: (0, 1), 17: (0, 1), 18: (0, 1), 19: (0, 1), 20: (0, 1), 21: (0, 1), 22: (0, 1), 23: (0, 1), 24: (0, 1)
    }, 
    1408127953000: {
        9: (0, 1)
    }, 
    1408127954000: {
        1: (0, 2), 2: (0, 2), 3: (0, 2), 4: (0, 2), 5: (0, 2), 6: (0, 2), 7: (0, 2), 8: (0, 2), 10: (0, 2), 11: (0, 2), 12: (0, 2), 13: (0, 2), 14: (0, 2), 15: (0, 2), 16: (0, 2), 17: (0, 2), 18: (0, 2), 19: (0, 2), 20: (0, 2), 21: (0, 2), 22: (0, 2), 23: (0, 2), 24: (0, 2)
    }, 
    1408127955000: {
        9: (0, 2)
    }, 
    1408127956000: {
        1: (0, 3), 2: (0, 3), 3: (0, 3), 4: (0, 3), 5: (0, 3), 6: (0, 3), 7: (0, 3), 8: (0, 3), 10: (0, 3), 11: (0, 3), 12: (0, 3), 13: (0, 3), 14: (0, 3), 15: (0, 3), 16: (0, 3), 17: (0, 3), 18: (0, 3), 19: (0, 3), 20: (0, 3), 21: (0, 3), 22: (0, 3), 23: (0, 3), 24: (0, 3)
    }, 
    1408127957000: {
        9: (0, 3)
    }, 
    1408127958000: {
        1: (0, 4), 2: (0, 4), 3: (0, 4), 4: (0, 4), 5: (0, 4), 6: (0, 4), 7: (0, 4), 8: (0, 4), 10: (0, 4), 11: (0, 4), 12: (0, 4), 13: (0, 4), 14: (0, 4), 15: (0, 4), 16: (0, 4), 17: (0, 4), 18: (0, 4), 19: (0, 4), 20: (0, 4), 21: (0, 4), 22: (0, 4), 23: (0, 4), 24: (0, 4)
    }, 
    1408127959000: {
        9: (0, 4)
    }, 
    1408127960000: {
        1: (0, 5), 2: (0, 5), 3: (0, 5), 4: (0, 5), 5: (0, 5), 6: (0, 5), 7: (0, 5), 8: (0, 5), 10: (0, 5), 11: (0, 5), 12: (0, 5), 13: (0, 5), 14: (0, 5), 15: (0, 5), 16: (0, 5), 17: (0, 5), 18: (0, 5), 19: (0, 5), 20: (0, 5), 21: (0, 5), 22: (0, 5), 23: (0, 5), 24: (0, 5)
    }, 
    1408127961000: {
        9: (0, 5)
    }, 
    1408127962000: {
        1: (0, 6), 2: (0, 6), 3: (0, 6), 4: (0, 6), 5: (0, 6), 6: (0, 6), 7: (0, 6), 8: (0, 6), 10: (0, 6), 11: (0, 6), 12: (0, 6), 13: (0, 6), 14: (0, 6), 15: (0, 6), 16: (0, 6), 17: (0, 6), 18: (0, 6), 19: (0, 6), 20: (0, 6), 21: (0, 6), 22: (0, 6), 23: (0, 6), 24: (0, 6)
    }, 
    1408127963000: {
        9: (0, 6)
    }, 
    1408127964000: {
        1: (0, 7), 2: (0, 7), 3: (0, 7), 4: (0, 7), 5: (0, 7), 6: (0, 7), 7: (0, 7), 8: (0, 7), 10: (0, 7), 11: (0, 7), 12: (0, 7), 13: (0, 7), 14: (0, 7), 15: (0, 7), 16: (0, 7), 17: (0, 7), 18: (0, 7), 19: (0, 7), 20: (0, 7), 21: (0, 7), 22: (0, 7), 23: (0, 7), 24: (0, 7)
    }, 
    1408127965000: {
        9: (0, 7)
    }, 
    1408127966000: {
        1: (0, 8), 2: (0, 8), 3: (0, 8), 4: (0, 8), 5: (0, 8), 6: (0, 8), 7: (0, 8), 8: (0, 8), 10: (0, 8), 11: (0, 8), 12: (0, 8), 13: (0, 8), 14: (0, 8), 15: (0, 8), 16: (0, 8), 17: (0, 8), 18: (0, 8), 19: (0, 8), 20: (0, 8), 21: (0, 8), 22: (0, 8), 23: (0, 8), 24: (0, 8)
    }, 
    1408127967000: {
        9: (0, 8)
    }, 
    1408127968000: {
        1: (0, 9), 2: (0, 9), 3: (0, 9), 4: (0, 9), 5: (0, 9), 6: (0, 9), 7: (0, 9), 8: (0, 9), 10: (0, 9), 11: (0, 9), 12: (0, 9), 13: (0, 9), 14: (0, 9), 15: (0, 9), 16: (0, 9), 17: (0, 9), 18: (0, 9), 19: (0, 9), 20: (0, 9), 21: (0, 9), 22: (0, 9), 23: (0, 9), 24: (0, 9)
    }, 
    1408127969000: {
        9: (0, 9)
    }, 
    1408127970000: {
        1: (0, 10), 2: (0, 10), 3: (0, 10), 4: (0, 10), 5: (0, 10), 6: (0, 10), 7: (0, 10), 8: (0, 10), 10: (0, 10), 11: (0, 10), 12: (0, 10), 13: (0, 10), 14: (0, 10), 15: (0, 10), 16: (0, 10), 17: (0, 10), 18: (0, 10), 19: (0, 10), 20: (0, 10), 21: (0, 10), 22: (0, 10), 23: (0, 10), 24: (0, 10)
    }, 
    1408127971000: {
        9: (0, 10)
    }, 
    1408127972000: {
        1: (0, 11), 2: (0, 11), 3: (0, 11), 4: (0, 11), 5: (0, 11), 6: (0, 11), 7: (0, 11), 8: (0, 11), 10: (0, 11), 11: (0, 11), 12: (0, 11), 13: (0, 11), 14: (0, 11), 15: (0, 11), 16: (0, 11), 17: (0, 11), 18: (0, 11), 19: (0, 11), 20: (0, 11), 21: (0, 11), 22: (0, 11), 23: (0, 11), 24: (0, 11)
    }, 
    1408127973000: {
        9: (0, 11)
    }, 
    1408127974000: {
        1: (0, 12), 2: (0, 12), 3: (0, 12), 4: (0, 12), 5: (0, 12), 6: (0, 12), 7: (0, 12), 8: (0, 12), 10: (0, 12), 11: (0, 12), 12: (0, 12), 13: (0, 12), 14: (0, 12), 15: (0, 12), 16: (0, 12), 17: (0, 12), 18: (0, 12), 19: (0, 12), 20: (0, 12), 21: (0, 12), 22: (0, 12), 23: (0, 12), 24: (0, 12)
    }, 
    1408127975000: {
        9: (0, 12)
    }, 
    1408127976000: {
        1: (0, 13), 2: (0, 13), 3: (0, 13), 4: (0, 13), 5: (0, 13), 6: (0, 13), 7: (0, 13), 8: (0, 13), 10: (0, 13), 11: (0, 13), 12: (0, 13), 13: (0, 13), 14: (0, 13), 15: (0, 13), 16: (0, 13), 17: (0, 13), 18: (0, 13), 19: (0, 13), 20: (0, 13), 21: (0, 13), 22: (0, 13), 23: (0, 13), 24: (0, 13)
    }, 
    1408127977000: {
        9: (0, 13)
    }, 
    1408127978000: {
        1: (0, 14), 2: (0, 14), 3: (0, 14), 4: (0, 14), 5: (0, 14), 6: (0, 14), 7: (0, 14), 8: (0, 14), 10: (0, 14), 11: (0, 14), 12: (0, 14), 13: (0, 14), 14: (0, 14), 15: (0, 14), 16: (0, 14), 17: (0, 14), 18: (0, 14), 19: (0, 14), 20: (0, 14), 21: (0, 14), 22: (0, 14), 23: (0, 14), 24: (0, 14)
    }, 
    1408127979000: {
        9: (0, 14)
    }, 
    1408127980000: {
        1: (0, 15), 2: (0, 15), 3: (0, 15), 4: (0, 15), 5: (0, 15), 6: (0, 15), 7: (0, 15), 8: (0, 15), 10: (0, 15), 11: (0, 15), 12: (0, 15), 13: (0, 15), 14: (0, 15), 15: (0, 15), 16: (0, 15), 17: (0, 15), 18: (0, 15), 19: (0, 15), 20: (0, 15), 21: (0, 15), 22: (0, 15), 23: (0, 15), 24: (0, 15)
    }, 
    1408127981000: {
        9: (0, 15)
    }, 
    1408127982000: {
        1: (0, 16), 2: (0, 16), 3: (0, 16), 4: (0, 16), 5: (0, 16), 6: (0, 16), 7: (0, 16), 8: (0, 16), 10: (0, 16), 11: (0, 16), 12: (0, 16), 13: (0, 16), 14: (0, 16), 15: (0, 16), 16: (0, 16), 17: (0, 16), 18: (0, 16), 19: (0, 16), 20: (0, 16), 21: (0, 16), 22: (0, 16), 23: (0, 16), 24: (0, 16)
    }, 
    1408127983000: {
        9: (0, 16)
    }, 
    1408127984000: {
        1: (0, 17), 2: (0, 17), 3: (0, 17), 4: (0, 17), 5: (0, 17), 6: (0, 17), 7: (0, 17), 8: (0, 17), 10: (0, 17), 11: (0, 17), 12: (0, 17), 13: (0, 17), 14: (0, 17), 15: (0, 17), 16: (0, 17), 17: (0, 17), 18: (0, 17), 19: (0, 17), 20: (0, 17), 21: (0, 17), 22: (0, 17), 23: (0, 17), 24: (0, 17)
    }, 
    1408127985000: {
        9: (0, 17)
    }, 
    1408127986000: {
        1: (0, 18), 2: (0, 18), 3: (0, 18), 4: (0, 18), 5: (0, 18), 6: (0, 18), 7: (0, 18), 8: (0, 18), 10: (0, 18), 11: (0, 18), 12: (0, 18), 13: (0, 18), 14: (0, 18), 15: (0, 18), 16: (0, 18), 17: (0, 18), 18: (0, 18), 19: (0, 18), 20: (0, 18), 21: (0, 18), 22: (0, 18), 23: (0, 18), 24: (0, 18)
    }, 
    1408127987000: {
        9: (0, 18)
    }, 
    1408127988000: {
        1: (0, 19), 2: (0, 19), 3: (0, 19), 4: (0, 19), 5: (0, 19), 6: (0, 19), 7: (0, 19), 8: (0, 19), 10: (0, 19), 11: (0, 19), 12: (0, 19), 13: (0, 19), 14: (0, 19), 15: (0, 19), 16: (0, 19), 17: (0, 19), 18: (0, 19), 19: (0, 19), 20: (0, 19), 21: (0, 19), 22: (0, 19), 23: (0, 19), 24: (0, 19)
    }, 
    1408127989000: {
        9: (0, 19)
    }, 
    1408127990000: {
        1: (0, 20), 2: (0, 20), 3: (0, 20), 4: (0, 20), 5: (0, 20), 6: (0, 20), 7: (0, 20), 8: (0, 20), 10: (0, 20), 11: (0, 20), 12: (0, 20), 13: (0, 20), 14: (0, 20), 15: (0, 20), 16: (0, 20), 17: (0, 20), 18: (0, 20), 19: (0, 20), 20: (0, 20), 21: (0, 20), 22: (0, 20), 23: (0, 20), 24: (0, 20)
    }, 
    1408127991000: {
        9: (0, 20)
    }, 
    1408127992000: {
        1: (0, 21), 2: (0, 21), 3: (0, 21), 4: (0, 21), 5: (0, 21), 6: (0, 21), 7: (0, 21), 8: (0, 21), 10: (0, 21), 11: (0, 21), 12: (0, 21), 13: (0, 21), 14: (0, 21), 15: (0, 21), 16: (0, 21), 17: (0, 21), 18: (0, 21), 19: (0, 21), 20: (0, 21), 21: (0, 21), 22: (0, 21), 23: (0, 21), 24: (0, 21)
    }, 
    1408127993000: {
        9: (0, 21)
    }, 
    1408127994000: {
        1: (0, 22), 2: (0, 22), 3: (0, 22), 4: (0, 22), 5: (0, 22), 6: (0, 22), 7: (0, 22), 8: (0, 22), 10: (0, 22), 11: (0, 22), 12: (0, 22), 13: (0, 22), 14: (0, 22), 15: (0, 22), 16: (0, 22), 17: (0, 22), 18: (0, 22), 19: (0, 22), 20: (0, 22), 21: (0, 22), 22: (0, 22), 23: (0, 22), 24: (0, 22)
    }, 
    1408127995000: {
        9: (0, 22)
    }, 
    1408127996000: {
        1: (0, 23), 2: (0, 23), 3: (0, 23), 4: (0, 23), 5: (0, 23), 6: (0, 23), 7: (0, 23), 8: (0, 23), 10: (0, 23), 11: (0, 23), 12: (0, 23), 13: (0, 23), 14: (0, 23), 15: (0, 23), 16: (0, 23), 17: (0, 23), 18: (0, 23), 19: (0, 23), 20: (0, 23), 21: (0, 23), 22: (0, 23), 23: (0, 23), 24: (0, 23)
    }, 
    1408127997000: {
        9: (0, 23)
    }, 
    1408127998000: {
        1: (0, 24), 2: (0, 24), 3: (0, 24), 4: (0, 24), 5: (0, 24), 6: (0, 24), 7: (0, 24), 8: (0, 24), 10: (0, 24), 11: (0, 24), 12: (0, 24), 13: (0, 24), 14: (0, 24), 15: (0, 24), 16: (0, 24), 17: (0, 24), 18: (0, 24), 19: (0, 24), 20: (0, 24), 21: (0, 24), 22: (0, 24), 23: (0, 24), 24: (0, 24)
    }, 
    1408127999000: {
        9: (0, 24)
    }, 
    1408128000000: {
        1: (0, 25), 2: (0, 25), 3: (0, 25), 4: (0, 25), 5: (0, 25), 6: (0, 25), 7: (0, 25), 8: (0, 25), 10: (0, 25), 11: (0, 25), 12: (0, 25), 13: (0, 25), 14: (0, 25), 15: (0, 25), 16: (0, 25), 17: (0, 25), 18: (0, 25), 19: (0, 25), 20: (0, 25), 21: (0, 25), 22: (0, 25), 23: (0, 25), 24: (0, 25)
    }, 
    1408128001000: {
        9: (0, 25)
    }, 
    1408128002000: {
        1: (0, 26), 2: (0, 26), 3: (0, 26), 4: (0, 26), 5: (0, 26), 6: (0, 26), 7: (0, 26), 8: (0, 26), 10: (0, 26), 11: (0, 26), 12: (0, 26), 13: (0, 26), 14: (0, 26), 15: (0, 26), 16: (0, 26), 17: (0, 26), 18: (0, 26), 19: (0, 26), 20: (0, 26), 21: (0, 26), 22: (0, 26), 23: (0, 26), 24: (0, 26)
    }, 
    1408128003000: {
        9: (0, 26)
    }, 
    1408128004000: {
        1: (0, 27), 2: (0, 27), 3: (0, 27), 4: (0, 27), 5: (0, 27), 6: (0, 27), 7: (0, 27), 8: (0, 27), 10: (0, 27), 11: (0, 27), 12: (0, 27), 13: (0, 27), 14: (0, 27), 15: (0, 27), 16: (0, 27), 17: (0, 27), 18: (0, 27), 19: (0, 27), 20: (0, 27), 21: (0, 27), 22: (0, 27), 23: (0, 27), 24: (0, 27)
    }, 
    1408128005000: {
        9: (0, 27)
    }, 
    1408128006000: {
        1: (0, 28), 2: (0, 28), 3: (0, 28), 4: (0, 28), 5: (0, 28), 6: (0, 28), 7: (0, 28), 8: (0, 28), 10: (0, 28), 11: (0, 28), 12: (0, 28), 13: (0, 28), 14: (0, 28), 15: (0, 28), 16: (0, 28), 17: (0, 28), 18: (0, 28), 19: (0, 28), 20: (0, 28), 21: (0, 28), 22: (0, 28), 23: (0, 28), 24: (0, 28)
    }, 
    1408128007000: {
        9: (0, 28)
    }, 
    1408128008000: {
        1: (0, 29), 2: (0, 29), 3: (0, 29), 4: (0, 29), 5: (0, 29), 6: (0, 29), 7: (0, 29), 8: (0, 29), 10: (0, 29), 11: (0, 29), 12: (0, 29), 13: (0, 29), 14: (0, 29), 15: (0, 29), 16: (0, 29), 17: (0, 29), 18: (0, 29), 19: (0, 29), 20: (0, 29), 21: (0, 29), 22: (0, 29), 23: (0, 29), 24: (0, 29)
    }, 
    1408128009000: {
        9: (0, 29)
    }, 
    1408128010000: {
        1: (0, 30), 2: (0, 30), 3: (0, 30), 4: (0, 30), 5: (0, 30), 6: (0, 30), 7: (0, 30), 8: (0, 30), 10: (0, 30), 11: (0, 30), 12: (0, 30), 13: (0, 30), 14: (0, 30), 15: (0, 30), 16: (0, 30), 17: (0, 30), 18: (0, 30), 19: (0, 30), 20: (0, 30), 21: (0, 30), 22: (0, 30), 23: (0, 30), 24: (0, 30)
    }, 
    1408128011000: {
        9: (0, 30)
    }, 
    1408128012000: {
        1: (0, 31), 2: (0, 31), 3: (0, 31), 4: (0, 31), 5: (0, 31), 6: (0, 31), 7: (0, 31), 8: (0, 31), 10: (0, 31), 11: (0, 31), 12: (0, 31), 13: (0, 31), 14: (0, 31), 15: (0, 31), 16: (0, 31), 17: (0, 31), 18: (0, 31), 19: (0, 31), 20: (0, 31), 21: (0, 31), 22: (0, 31), 23: (0, 31), 24: (0, 31)
    }, 
    1408128013000: {
        9: (0, 31)
    }, 
    1408128014000: {
        1: (0, 32), 2: (0, 32), 3: (0, 32), 4: (0, 32), 5: (0, 32), 6: (0, 32), 7: (0, 32), 8: (0, 32), 10: (0, 32), 11: (0, 32), 12: (0, 32), 13: (0, 32), 14: (0, 32), 15: (0, 32), 16: (0, 32), 17: (0, 32), 18: (0, 32), 19: (0, 32), 20: (0, 32), 21: (0, 32), 22: (0, 32), 23: (0, 32), 24: (0, 32)
    }, 
    1408128015000: {
        9: (0, 32)
    }, 
    1408128016000: {
        1: (0, 33), 2: (0, 33), 3: (0, 33), 4: (0, 33), 5: (0, 33), 6: (0, 33), 7: (0, 33), 8: (0, 33), 10: (0, 33), 11: (0, 33), 12: (0, 33), 13: (0, 33), 14: (0, 33), 15: (0, 33), 16: (0, 33), 17: (0, 33), 18: (0, 33), 19: (0, 33), 20: (0, 33), 21: (0, 33), 22: (0, 33), 23: (0, 33), 24: (0, 33)
    }, 
    1408128017000: {
        9: (0, 33)
    }, 
    1408128018000: {
        1: (0, 34), 2: (0, 34), 3: (0, 34), 4: (0, 34), 5: (0, 34), 6: (0, 34), 7: (0, 34), 8: (0, 34), 10: (0, 34), 11: (0, 34), 12: (0, 34), 13: (0, 34), 14: (0, 34), 15: (0, 34), 16: (0, 34), 17: (0, 34), 18: (0, 34), 19: (0, 34), 20: (0, 34), 21: (0, 34), 22: (0, 34), 23: (0, 34), 24: (0, 34)
    }, 
    1408128019000: {
        9: (0, 34)
    }, 
    1408128020000: {
        1: (0, 35), 2: (0, 35), 3: (0, 35), 4: (0, 35), 5: (0, 35), 6: (0, 35), 7: (0, 35), 8: (0, 35), 10: (0, 35), 11: (0, 35), 12: (0, 35), 13: (0, 35), 14: (0, 35), 15: (0, 35), 16: (0, 35), 17: (0, 35), 18: (0, 35), 19: (0, 35), 20: (0, 35), 21: (0, 35), 22: (0, 35), 23: (0, 35), 24: (0, 35)
    }, 
    1408128021000: {
        9: (0, 35)
    }, 
    1408128022000: {
        1: (0, 36), 2: (0, 36), 3: (0, 36), 4: (0, 36), 5: (0, 36), 6: (0, 36), 7: (0, 36), 8: (0, 36), 10: (0, 36), 11: (0, 36), 12: (0, 36), 13: (0, 36), 14: (0, 36), 15: (0, 36), 16: (0, 36), 17: (0, 36), 18: (0, 36), 19: (0, 36), 20: (0, 36), 21: (0, 36), 22: (0, 36), 23: (0, 36), 24: (0, 36)
    }, 
    1408128023000: {
        9: (0, 36)
    }, 
    1408128024000: {
        1: (0, 37), 2: (0, 37), 3: (0, 37), 4: (0, 37), 5: (0, 37), 6: (0, 37), 7: (0, 37), 8: (0, 37), 10: (0, 37), 11: (0, 37), 12: (0, 37), 13: (0, 37), 14: (0, 37), 15: (0, 37), 16: (0, 37), 17: (0, 37), 18: (0, 37), 19: (0, 37), 20: (0, 37), 21: (0, 37), 22: (0, 37), 23: (0, 37), 24: (0, 37)
    }, 
    1408128025000: {
        9: (0, 37)
    }, 
    1408128026000: {
        1: (0, 38), 2: (0, 38), 3: (0, 38), 4: (0, 38), 5: (0, 38), 6: (0, 38), 7: (0, 38), 8: (0, 38), 10: (0, 38), 11: (0, 38), 12: (0, 38), 13: (0, 38), 14: (0, 38), 15: (0, 38), 16: (0, 38), 17: (0, 38), 18: (0, 38), 19: (0, 38), 20: (0, 38), 21: (0, 38), 22: (0, 38), 23: (0, 38), 24: (0, 38)
    }, 
    1408128027000: {
        9: (0, 38)
    }, 
    1408128028000: {
        1: (0, 39), 2: (0, 39), 3: (0, 39), 4: (0, 39), 5: (0, 39), 6: (0, 39), 7: (0, 39), 8: (0, 39), 10: (0, 39), 11: (0, 39), 12: (0, 39), 13: (0, 39), 14: (0, 39), 15: (0, 39), 16: (0, 39), 17: (0, 39), 18: (0, 39), 19: (0, 39), 20: (0, 39), 21: (0, 39), 22: (0, 39), 23: (0, 39), 24: (0, 39)
    }, 
    1408128029000: {
        9: (0, 39)
    }, 
    1408128030000: {
        1: (0, 40), 2: (0, 40), 3: (0, 40), 4: (0, 40), 5: (0, 40), 6: (0, 40), 7: (0, 40), 8: (0, 40), 10: (0, 40), 11: (0, 40), 12: (0, 40), 13: (0, 40), 14: (0, 40), 15: (0, 40), 16: (0, 40), 17: (0, 40), 18: (0, 40), 19: (0, 40), 20: (0, 40), 21: (0, 40), 22: (0, 40), 23: (0, 40), 24: (0, 40)
    }, 
    1408128031000: {
        9: (0, 40)
    }, 
    1408128032000: {
        1: (0, 41), 2: (0, 41), 3: (0, 41), 4: (0, 41), 5: (0, 41), 6: (0, 41), 7: (0, 41), 8: (0, 41), 10: (0, 41), 11: (0, 41), 12: (0, 41), 13: (0, 41), 14: (0, 41), 15: (0, 41), 16: (0, 41), 17: (0, 41), 18: (0, 41), 19: (0, 41), 20: (0, 41), 21: (0, 41), 22: (0, 41), 23: (0, 41), 24: (0, 41)
    }, 
    1408128033000: {
        9: (0, 41)
    }, 
    1408128034000: {
        1: (0, 42), 2: (0, 42), 3: (0, 42), 4: (0, 42), 5: (0, 42), 6: (0, 42), 7: (0, 42), 8: (0, 42), 10: (0, 42), 11: (0, 42), 12: (0, 42), 13: (0, 42), 14: (0, 42), 15: (0, 42), 16: (0, 42), 17: (0, 42), 18: (0, 42), 19: (0, 42), 20: (0, 42), 21: (0, 42), 22: (0, 42), 23: (0, 42), 24: (0, 42)
    }, 
    1408128035000: {
        9: (0, 42)
    }, 
    1408128036000: {
        1: (0, 43), 2: (0, 43), 3: (0, 43), 4: (0, 43), 5: (0, 43), 6: (0, 43), 7: (0, 43), 8: (0, 43), 10: (0, 43), 11: (0, 43), 12: (0, 43), 13: (0, 43), 14: (0, 43), 15: (0, 43), 16: (0, 43), 17: (0, 43), 18: (0, 43), 19: (0, 43), 20: (0, 43), 21: (0, 43), 22: (0, 43), 23: (0, 43), 24: (0, 43)
    }, 
    1408128037000: {
        9: (0, 43)
    }, 
    1408128038000: {
        1: (0, 44), 2: (0, 44), 3: (0, 44), 4: (0, 44), 5: (0, 44), 6: (0, 44), 7: (0, 44), 8: (0, 44), 10: (0, 44), 11: (0, 44), 12: (0, 44), 13: (0, 44), 14: (0, 44), 15: (0, 44), 16: (0, 44), 17: (0, 44), 18: (0, 44), 19: (0, 44), 20: (0, 44), 21: (0, 44), 22: (0, 44), 23: (0, 44), 24: (0, 44)
    }, 
    1408128039000: {
        9: (0, 44)
    }, 
    1408128040000: {
        1: (0, 45), 2: (0, 45), 3: (0, 45), 4: (0, 45), 5: (0, 45), 6: (0, 45), 7: (0, 45), 8: (0, 45), 10: (0, 45), 11: (0, 45), 12: (0, 45), 13: (0, 45), 14: (0, 45), 15: (0, 45), 16: (0, 45), 17: (0, 45), 18: (0, 45), 19: (0, 45), 20: (0, 45), 21: (0, 45), 22: (0, 45), 23: (0, 45), 24: (0, 45)
    }, 
    1408128041000: {
        9: (0, 45)
    }, 
    1408128042000: {
        1: (0, 46), 2: (0, 46), 3: (0, 46), 4: (0, 46), 5: (0, 46), 6: (0, 46), 7: (0, 46), 8: (0, 46), 10: (0, 46), 11: (0, 46), 12: (0, 46), 13: (0, 46), 14: (0, 46), 15: (0, 46), 16: (0, 46), 17: (0, 46), 18: (0, 46), 19: (0, 46), 20: (0, 46), 21: (0, 46), 22: (0, 46), 23: (0, 46), 24: (0, 46)
    }, 
    1408128043000: {
        9: (0, 46)
    }, 
    1408128044000: {
        1: (0, 47), 2: (0, 47), 3: (0, 47), 4: (0, 47), 5: (0, 47), 6: (0, 47), 7: (0, 47), 8: (0, 47), 10: (0, 47), 11: (0, 47), 12: (0, 47), 13: (0, 47), 14: (0, 47), 15: (0, 47), 16: (0, 47), 17: (0, 47), 18: (0, 47), 19: (0, 47), 20: (0, 47), 21: (0, 47), 22: (0, 47), 23: (0, 47), 24: (0, 47)
    }, 
    1408128045000: {
        9: (0, 47)
    }, 
    1408128046000: {
        1: (0, 48), 2: (0, 48), 3: (0, 48), 4: (0, 48), 5: (0, 48), 6: (0, 48), 7: (0, 48), 8: (0, 48), 10: (0, 48), 11: (0, 48), 12: (0, 48), 13: (0, 48), 14: (0, 48), 15: (0, 48), 16: (0, 48), 17: (0, 48), 18: (0, 48), 19: (0, 48), 20: (0, 48), 21: (0, 48), 22: (0, 48), 23: (0, 48), 24: (0, 48)
    }, 
    1408128047000: {
        9: (0, 48)
    }, 
    1408128048000: {
        1: (0, 49), 2: (0, 49), 3: (0, 49), 4: (0, 49), 5: (0, 49), 6: (0, 49), 7: (0, 49), 8: (0, 49), 10: (0, 49), 11: (0, 49), 12: (0, 49), 13: (0, 49), 14: (0, 49), 15: (0, 49), 16: (0, 49), 17: (0, 49), 18: (0, 49), 19: (0, 49), 20: (0, 49), 21: (0, 49), 22: (0, 49), 23: (0, 49), 24: (0, 49)
    }, 
    1408128049000: {
        9: (0, 49)
    }, 
    1408128050000: {
        1: (0, 50), 2: (0, 50), 3: (0, 50), 4: (0, 50), 5: (0, 50), 6: (0, 50), 7: (0, 50), 8: (0, 50), 10: (0, 50), 11: (0, 50), 12: (0, 50), 13: (0, 50), 14: (0, 50), 15: (0, 50), 16: (0, 50), 17: (0, 50), 18: (0, 50), 19: (0, 50), 20: (0, 50), 21: (0, 50), 22: (0, 50), 23: (0, 50), 24: (0, 50)
    }, 
    1408128051000: {
        9: (0, 50)
    }, 
    1408128052000: {
        1: (0, 51), 2: (0, 51), 3: (0, 51), 4: (0, 51), 5: (0, 51), 6: (0, 51), 7: (0, 51), 8: (0, 51), 10: (0, 51), 11: (0, 51), 12: (0, 51), 13: (0, 51), 14: (0, 51), 15: (0, 51), 16: (0, 51), 17: (0, 51), 18: (0, 51), 19: (0, 51), 20: (0, 51), 21: (0, 51), 22: (0, 51), 23: (0, 51), 24: (0, 51)
    }, 
    1408128053000: {
        9: (0, 51)
    }, 
    1408128054000: {
        1: (0, 52), 2: (0, 52), 3: (0, 52), 4: (0, 52), 5: (0, 52), 6: (0, 52), 7: (0, 52), 8: (0, 52), 10: (0, 52), 11: (0, 52), 12: (0, 52), 13: (0, 52), 14: (0, 52), 15: (0, 52), 16: (0, 52), 17: (0, 52), 18: (0, 52), 19: (0, 52), 20: (0, 52), 21: (0, 52), 22: (0, 52), 23: (0, 52), 24: (0, 52)
    }, 
    1408128055000: {
        9: (0, 52)
    }, 
    1408128056000: {
        1: (0, 53), 2: (0, 53), 3: (0, 53), 4: (0, 53), 5: (0, 53), 6: (0, 53), 7: (0, 53), 8: (0, 53), 10: (0, 53), 11: (0, 53), 12: (0, 53), 13: (0, 53), 14: (0, 53), 15: (0, 53), 16: (0, 53), 17: (0, 53), 18: (0, 53), 19: (0, 53), 20: (0, 53), 21: (0, 53), 22: (0, 53), 23: (0, 53), 24: (0, 53)
    }, 
    1408128057000: {
        9: (0, 53)
    }, 
    1408128058000: {
        1: (0, 54), 3: (0, 54), 5: (0, 54), 6: (0, 54), 7: (0, 54), 8: (0, 54), 10: (0, 54), 11: (0, 54), 12: (0, 54), 13: (0, 54), 14: (0, 54), 15: (0, 54), 16: (0, 54), 17: (0, 54), 18: (0, 54), 19: (0, 54), 20: (0, 54), 21: (0, 54), 22: (0, 54), 23: (0, 54), 24: (0, 54)
    }
}

Ensure the correlator version and obsid's match for metafits and all gpubox files

Currently mwalib can theoretically let you use a metafits from 1 obs and data files from another obs assuming their dimensionality matches. This can be solved with a pretty simple check of keywords and filenames.

MWA constants to be exposed in C API

A change in v0.10.0 was to remove the constants MWA_LATITUDE_RADIANS, MWA_LONGITUDE_RADIANS, MWA_ALTITUDE_METRES, and SPEED_OF_LIGHT_IN_VACUUM_M_PER_S. However, this requires users of the C API to get these constants from elsewhere, but there is currently no C-exposed library that provides them.

Add MacOS support

Follow pattern implemented in Hyperbeam

Re-open GPUBox number issue for legacy VCS

The previous fix doesn't seem to have carried across to the C interface. After installing the latest version of mwalib and putting the library and header files in standard places, this program can be compiled and run for testing:
bug_report_gpubox.c:

#include <stdlib.h>
#include <stdio.h>
#include <mwalib.h>

#define ERROR_MESSAGE_LEN  1024

int main( int argc, char *argv[] )
{
    // Get the metafits file from command line argument
    if (argc < 3)
    {
        printf( "usage: %s  METAFITS_FILE  VOLTAGE_FILE\n", argv[0] );
        printf( "example: %s  1164110416_metafits.fits  1164110416_1164110725_ch109.dat\n" );
        exit(EXIT_FAILURE);
    }
    char *metafits_filename = argv[1];
    char *datfile           = argv[2];

    // Print out MWALIB version
    printf( "\nMWALIB v%u.%u.%u\n\n",
            mwalib_get_version_major(),
            mwalib_get_version_minor(),
            mwalib_get_version_patch()
           );

    // Load up metafits for a VCS observation
    char error_message[ERROR_MESSAGE_LEN];
    //error_message[0] = '\0'; // <-- Just to avoid a compiler warning about uninitialised variables

    MetafitsMetadata *metadata;
    MetafitsContext  *context;

    if (mwalib_metafits_context_new2( metafits_filename, &context, error_message, ERROR_MESSAGE_LEN) != MWALIB_SUCCESS)
    {
        fprintf( stderr, "error (mwalib): cannot create metafits context: %s\n", error_message );
        exit(EXIT_FAILURE);
    }

    if (mwalib_metafits_metadata_get( context, NULL, NULL, &metadata, error_message, ERROR_MESSAGE_LEN ) != MWALIB_SUCCESS)
    {
        fprintf( stderr, "error (mwalib): cannot create metafits metadata: %s\n", error_message );
        exit(EXIT_FAILURE);
    }

    VoltageMetadata *volt_metadata;
    VoltageContext  *volt_context;

    const char **voltage_filenames  = &datfile;
    size_t       voltage_file_count = 1;

    if (mwalib_voltage_context_new( metafits_filename, voltage_filenames, voltage_file_count, &volt_context, error_message, ERROR_MESSAGE_LEN ) != MWALIB_SUCCESS)
    {
        fprintf( stderr, "error (mwalib): cannot create voltage context: %s\n", error_message );
        exit(EXIT_FAILURE);
    }

    if (mwalib_voltage_metadata_get( volt_context, &volt_metadata, error_message, ERROR_MESSAGE_LEN ) != EXIT_SUCCESS)
    {
        fprintf( stderr, "error (mwalib): cannot get metadata: %s\n", error_message );
        exit(EXIT_FAILURE);
    }

    // Read off the coarse channels and their gpubox numbers
    uintptr_t nmc = metadata->num_metafits_coarse_chans;
    uintptr_t nvc = volt_metadata->num_coarse_chans;
    uintptr_t c;

    printf( "             METAFITS            \n"
            "Corr chan | Rec chan | GPUBox num\n"
            "----------+----------+-----------\n" );

    for (c = 0; c < nmc; c++)
    {
        printf( "%5lu %5lu %5lu\n",
                metadata->metafits_coarse_chans[c].corr_chan_number,
                metadata->metafits_coarse_chans[c].rec_chan_number,
                metadata->metafits_coarse_chans[c].gpubox_number
              );
    }
    printf( "\n" );

    printf( "             VOLTAGE             \n"
            "Corr chan | Rec chan | GPUBox num\n"
            "----------+----------+-----------\n" );
    for (c = 0; c < nmc; c++)
    {
        printf( "%5lu %5lu %5lu\n",
                volt_metadata->coarse_chans[c].corr_chan_number,
                volt_metadata->coarse_chans[c].rec_chan_number,
                volt_metadata->coarse_chans[c].gpubox_number
              );
    }

    // Clean up
    mwalib_metafits_metadata_free( metadata );
    mwalib_metafits_context_free( context );
    mwalib_voltage_metadata_free( volt_metadata );
    mwalib_voltage_context_free( volt_context );

    return EXIT_SUCCESS;
}

Makefile for compiling and testing:

CC = gcc
LDLIBS = -lmwalib

run_legacy_test: bug_report_gpubox 1164110416_metafits.fits 1164110416_1164110725_ch109.dat
    ./$^

run_mwax_test: bug_report_gpubox 1313388760_metafits.fits 1313388760_1313388760_144.sub
    ./$^

1164110416_metafits.fits:
    wget -O $@ http://ws.mwatelescope.org/metadata/fits?obs_id=1164110416

1313388760_metafits.fits:
    wget -O $@ http://ws.mwatelescope.org/metadata/fits?obs_id=1313388760

1164110416_1164110725_ch109.dat:
    dd if=/dev/zero of=$@ bs=32768 count=10000

1313388760_1313388760_144.sub:
    dd if=/dev/zero of=$@ bs=4096 count=1288001

...which translates to:

$ make -Bn run_legacy_test
gcc     bug_report_gpubox.c  -lmwalib -o bug_report_gpubox
wget -O 1164110416_metafits.fits http://ws.mwatelescope.org/metadata/fits?obs_id=1164110416
dd if=/dev/zero of=1164110416_1164110725_ch109.dat bs=32768 count=10000
./bug_report_gpubox 1164110416_metafits.fits 1164110416_1164110725_ch109.dat

and

$ make -Bn run_mwax_test 
gcc     bug_report_gpubox.c  -lmwalib -o bug_report_gpubox
wget -O 1313388760_metafits.fits http://ws.mwatelescope.org/metadata/fits?obs_id=1313388760
dd if=/dev/zero of=1313388760_1313388760_144.sub bs=4096 count=1288001
./bug_report_gpubox 1313388760_metafits.fits 1313388760_1313388760_144.sub

I get the output (e.g. for the legacy test):

$ ./bug_report_gpubox 1164110416_metafits.fits 1164110416_1164110725_ch109.dat

MWALIB v0.11.0

             METAFITS            
Corr chan | Rec chan | GPUBox num
----------+----------+-----------
    0   109   109
    1   110   110
    2   111   111
    3   112   112
    4   113   113
    5   114   114
    6   115   115
    7   116   116
    8   117   117
    9   118   118
   10   119   119
   11   120   120
   12   121   121
   13   122   122
   14   123   123
   15   124   124
   16   125   125
   17   126   126
   18   127   127
   19   128   128
   23   129   129
   22   130   130
   21   131   131
   20   132   132

             VOLTAGE             
Corr chan | Rec chan | GPUBox num
----------+----------+-----------
    0   109   109
    1   110   110
    2   111   111
    3   112   112
    4   113   113
    5   114   114
    6   115   115
    7   116   116
    8   117   117
    9   118   118
   10   119   119
   11   120   120
   12   121   121
   13   122   122
   14   123   123
   15   124   124
   16   125   125
   17   126   126
   18   127   127
   19   128   128
   23   129   129
   22   130   130
   21   131   131
   20   132   132

...and similarly incorrect numbers for the MWAX test. I believe the GPUBox numbers should always be in the range 01-24.

Originally posted by @robotopia in #41 (comment)

Provide array of baselines as member of mwalibContext object

This will mean callers do not need to guess the baseline ordering of the visibilities.
Simplest case would be an array with each element containing a struct with antenna1, antenna2 indices.

num_metafits_fine_chan_freqs_hz is not correct

Using 1260305032 obs, mwalib reports 24 fine channels but should be 3072.

Expose weights

Add function to read weights. For legacy MWA, this will just be a vector all 1's.

Change code coverage away from cargo tarpaulin

It breaks MACOS builds and isn't very reliable.

Provide a generate_expected_volt_filenames method on metafits context

This is so VCS software can open a MetafitsContext based on a metafits file, and then generate an array of filenames for a specific range of timesteps and generally 1 coarse chan (although the option is there for many), which is then used in a subsequent call to create a VoltageContext. This allows the VCS software to only require the user to provide a metafits file and optionally the start and end gps times.

Signature:
generate_expected_volt_filenames(path_prefix, gpstime_from, gpstime_to, coarse_chan_index_from, coarse_chan_index_to) -> Vec<&str>

Add electrical length to antenna

Also, the antenna position too

Expose calibration fit data from metafits file

Some metafits files will have a calibration fit table which would be necessary for some preprocessing.
See:

Antennas and rfinputs vectors not sorted consistently in VoltageContext

GRIDNUM should be optional

Culprit obsid: http://ws.mwatelescope.org/observation/obs/?obs_id=1100161368

mwalib-0.10.0/src/metafits_context/mod.rs:627

let grid_number = get_required_fits_key!(&mut metafits_fptr, &metafits_hdu, "GRIDNUM")?;

The error message is also poor:

Error: /home/chj/.cargo/registry/src/github.com-1ecc6299db9ec823/mwalib-0.10.0/src/metafits_context/mod.rs:627
1100161368.metafits HDU 1: Fits error: FitsError { status: 204, message: "keyword value is undefined" }

This happens because FitsError::Fitsio is being used (src/fits_read/mod.rs:353). This enum variant perhaps should not exist (almost certainly my fault, from a very green Rust developer's fingers), but at the very least, this should be replaced with FitsError::MissingKey. Other common fitsio error codes could also be handled here (see the code to know what I'm getting at).

Make mode an enum in metafits context

Update wiki "Installation via Github Release"

Some of the commands need updated to be able to install v0.10.0 release (not sure about earlier releases).

Here are the updated lines:
Download release from mwalib github releases. (Where X.Y.Z is the current release version)
wget "https://github.com/MWATelescope/mwalib/releases/download/vX.Y.Z/mwalib-vX.Y.Z-linux_x86_64.tar.gz" -O mwalib.tar.gz

Install
sudo cp mwalib/libmwalib.* /usr/local/lib
sudo cp mwalib/mwalib.h /usr/local/include

CoarseChannel.corr_chan_number not correct when only accessing subset of channels

Example:
I have the following gpubox files, which consist of 2 coarse channles:
"1297526432_mwax/1297526432_20210216160014_ch117_000.fits",
"1297526432_mwax/1297526432_20210216160014_ch117_001.fits",
"1297526432_mwax/1297526432_20210216160014_ch118_000.fits",
"1297526432_mwax/1297526432_20210216160014_ch118_001.fits"
These coarse channels represent the 9th and 10th (or 8th and 9th if zero indexed).

mwalib will set the corr_chan_number for 117 to be 0 and 118 to be 1. But this is not correct. The corr_chan_number should not change whether we have all or a subset of channels.

Expose version number to FFI

This will allow consumers, such as pymwalib, to check they are using a compatible version

Error when passing only a non 000 batch as the only file to a CorrelatorContext

In the code it checks the batch[0].gpuboxfile[0] which won't exist if you have only provided a 001 or 002 batched gpubox file!

Provide coarse channel metadata in the metafits context

Allow access to uncommon timesteps and coarse channels in voltages and visibilities

By default mwalib trims the start and end timesteps to only the common ones.

Provide a way for the caller to leave that functionality as is OR accommodate all the data that is available. For "missing" data this could mean raising an error if the caller tries to access it, or in the voltage case, return 0's. Will need to discuss how this should work with some users.

Implement python interface via pyo3

Relies on completing issue #7

Ensure legacy visibilities end up in cotter/CASA handedness

Cotter/CASA's imaginary values do not match mwalib for a legacy observation, as the visibilities needs to be conjugated one more time. This issue is to make that change and to also adjust the unit tests which compare to pyuvdata, which does not do this final conjugation of all visibilities. This would then make mwalib consistent between legacy and MWAX versions and also cotter/CASA.

Should ATTEN_DB be made optional?

Not all metafits contain this key. Is this because it was introduced at a particular time? @andreww5au

Include unmodified cotter results in coversion unit tests

Run a modified cotter (which disables cable-delay correction) plus all the other corrections turned off for a different observation and store the values in a CSV file in the test_data dir and compare converted mwalib visibilities. This is in addition to the pyuvdata csv data we check against.

Null termination not happening in C API

Some functions which populate char arrays do not place a null byte at the termination of the string. I believe it happens for error_messages in many functions, but below is code to reproduce the effect in mwalib_metafits_get_expected_volt_filename(). It counts as a bug because there's no way for the user of the C interface to know how many chars have actually been written, or where to place the null byte.

// Given voltage files and a metafits file, provide metadata about this VCS observation
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include "fitsio.h"
#include "mwalib.h"

#define ERROR_MESSAGE_LEN 1024
#define FILENAME_LEN 1024

int main(int argc, char *argv[])
{
    // Assume that the first file provided is the metafits file, and all others
    // are gpubox files or voltage files. Therefore, we need at least one file provided to main.
    int file_count = argc - 1;

    if (file_count < 1)
    {
        printf("At least one file is needed (if only one, it should be the metafits file).\n");
        // print_usage();
    }

    // Allocate buffer for any error messages
    char *error_message = malloc(ERROR_MESSAGE_LEN * sizeof(char));

    // Allocate buffer for the filename
    char *out_filename_ptr = malloc(FILENAME_LEN * sizeof(char));

    memset(out_filename_ptr, 'a', FILENAME_LEN); // Pre-fill array with 'a's

    // Create context pointers
    MetafitsContext *metafits_context = NULL;

    mwalib_metafits_context_new2(argv[1], &metafits_context, error_message, ERROR_MESSAGE_LEN);

    int retval = mwalib_metafits_get_expected_volt_filename(
        metafits_context,
        3,
        1,
        out_filename_ptr,
        FILENAME_LEN,
        error_message,
        ERROR_MESSAGE_LEN
    );

    // puts(error_message);

    puts(out_filename_ptr);

    return EXIT_SUCCESS;
}

Compile with

cargo build --release
gcc -O3 \
    volt-context-bug.c \
    -o volt-context-bug \
    -I ../include \
    -lm -lpthread -ldl \
    -L../target/release/ \
    -lmwalib

Provide mechanism for allowing caller to receive warnings from mwalib

For cases where there are no errors, but warnings, we need a way to provide these to the caller.

This could be done in a similar way to OpenSSL
"The OpenSSL is a cross-platform shared library that had much the same problem to solve. Their method has the library record detailed error information in an internal error queue, which the application can request when an error return value is seen and then present to the user in whichever way is appropriate." ... "There is a function provided for the application to request the "next error". This removes the earliest error from the internal queue, and returns it to the application".

So as warnings are generated as we read and validate the MWA files, we push a warning to the queue.

The caller / client app can pop them off until queue is empty and do whatever with them (log, print to stdout/stderr).

Test "test_get_fits_long_string()" fails in github CI, but not locally

See: Workflow output

Error is:

running 19 tests
test fits_read::tests::test_get_fits_long_string ... FAILED
test fits_read::tests::test_get_fits_key ... ok
test gpubox::tests::determine_gpubox_batches_invalid_count ... ok
test gpubox::tests::determine_gpubox_batches_invalid_count2 ... ok
test gpubox::tests::determine_gpubox_batches_invalid_filename ... ok
test gpubox::tests::determine_gpubox_batches_invalid_filename2 ... ok
test gpubox::tests::determine_gpubox_batches_invalid_filename3 ... ok
test gpubox::tests::determine_gpubox_batches_mix ... ok
test gpubox::tests::determine_gpubox_batches_old_format ... ok
test gpubox::tests::determine_gpubox_batches_new_format ... ok
test gpubox::tests::determine_gpubox_batches_proper_format ... ok
test gpubox::tests::determine_gpubox_batches_proper_format2 ... ok
test gpubox::tests::determine_gpubox_batches_proper_format4 ... ok
test gpubox::tests::determine_hdu_time_test1 ... ok
test gpubox::tests::determine_gpubox_batches_proper_format3 ... ok
test gpubox::tests::determine_hdu_time_test2 ... ok
error: test failed, to rerun pass '--lib'

Caused by:
  process didn't exit successfully: /home/runner/work/mwalib/mwalib/target/debug/deps/mwalib-d90059212587e9bb (signal: 11, SIGSEGV: invalid memory reference)
##[error]Process completed with exit code 101.

Handle missing metafits cards

The following cards could be missing from a metafits file and if so, cause mwalib to fail, unable to read the metafits file:

SUN-DIST, SUN-ALT, MOONDIST, JUP-DIST, GRIDNAME and GRIDNUM won't exist for old, already broken and unusable observations with no schedule_metadata row

When encountering no GRIDNAME and GRIDNUM use "NOGRID" and 0 respectively.

Provide contigous spw structures within provided, common and common_good for coarse channels

For each of: provided, common and common good timesteps/coarse channels; create multiple spws (spectral windows) which then have a vector of coarse channel indices. This will allow callers to have an easy way to get which channels are contiguous.

Structure could be:

struct spw {
coarse_chan_start_index: usize,
coarse_chan_end_index: usize,
metafits_fine_chan_start_index: usize,
metafits_fine_chan_end_index: usize,
bandwidth_hz: f64,
center_frequency_hz: f64, // this is literally the central frequency of the spw and may or may not line up with a fine channel bin
reference_metafits_fine_chan_index: usize, // "reference" is a real fine channel index from the metafits fine chan array and is the  (for odd number of fine chans). For even number of fine chans it is the n/2 bin.
}

There needs to be a public, stand-alone method to get the spw's for an arbitrary input of coarse channel indices. This is in addition to the provided, common and common_good spws.