cyriltasse / dynspecms Goto Github PK

Running ms2dynspec.py I get a segmentation fault.

As far as I can tell, everything is installed correctly but maybe I am doing something wrong in running it. I am running it within a singularity container for DDFacet. My data is calibrated using LINC and I have subtracted the skymodel from the visibilities to give the column “SUBTRACTED_DATA”. I can share a measurement set for debugging if useful.

This is my command and the script output:

Singularity> python3 ms2dynspec.py --ms ~/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms --data "SUBTRACTED_DATA" --srclist ~/GRB201006A/GRB201006A_src.csv
 ______             _____                ___  ___ _____  
 |  _  \           /  ___|               |  \/  |/  ___| 
 | | | |_   _ _ __ \ `--. _ __   ___  ___| .  . |\ `--.  
 | | | | | | | '_ \ `--. \ '_ \ / _ \/ __| |\/| | `--. \ 
 | |/ /| |_| | | | /\__/ / |_) |  __/ (__| |  | |/\__/ / 
 |___/  \__, |_| |_\____/| .__/ \___|\___\_|  |_/\____/  
         __/ |           | |                             
        |___/            |_|                             
                                 version v1.0-29-g0ad33b5
 - 08:10:14 - AsyncProcessPool             | Fixing parent process to vthread 0
 - 08:10:14 - AsyncProcessPool             | Worker affinities not specified, leaving unset
 - 08:10:14 - ClassMS                      | found 1 MSs matching /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: selecting DDID 0
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: selecting field 0
 - 08:10:14 - ClassMS                      | /home/rowlinson/GRB201006A/target/plotdata/L796450_123MHz_uv_pre-cal.ms: non-default column DATA
 - 08:10:14 - ClassMS                      | 1 MS section(s) selected
 - 08:10:14 - DynSpecMS                    | DynSpecMS version v1.0-29-g0ad33b5 starting up
Segmentation fault (core dumped)

I’m meant to be passing the SUBTRACTED_DATA column but this doesn’t seem to be passing through as I think the script is using “DATA”.

My source list is (file attached):

GRB201006A 61.89270 65.16462
BG1 61.542444 65.224421
BG2 62.172479 65.253684
BG3 62.120941 65.059329

The positions are in degrees and I give 3 background regions. My source (GRB201006A) is at the centre of the field.
GRB201006A_src.csv

Hi Cyril,

I prepared a target list of ~1700 sources and (unsurprisingly) DynspecMS is running slowly. I've also noticed that the pressure on HPC memory is pretty high. Will it help if I split the list into multiple (e.g. 10) sublists and run DynspecMS sequentially? (Or will that really make a difference?)

The target list includes grid points to cover the entire NenuFAR field, so I set --noff 0.

Cheers,
Xiang

Can we merge applybeam back into master?

Needs to be modified for the current ddf-pipeline to be able to use one big smoothed solution file not in the ms rather than individual files in the ms.

Because the code doesn't check that the download works it doesn't exit cleanly in various ways if it doesn't get picked up -- also this exposes a bug in the check for zero directions in the main code. I've fixed these simple things, give me push access and I'll upload.

Hi-

Ok - so I've been implementing various changed to DynSpecMS. Branch: MTilde_ASync

• First the bug of the off being disrtibuted it the top left quadrant is fixed
• the off are distributed in all facets with the same number per facet (and a minimum number of --nMinOffPerFacet (default 5))
• the Jones are not anymore applied at the inverse but as the Hermitian transpose as described in the DDFacet paper (this is the proper way to do). Just dealing with scalar case so far, but I'll have to extend it to full pol if we redo the calibration with the constrained solvers since we use diagonal solve in one step.
• DynSpecMS is now asyncronous, meaning it prepares the next datachunk as it processes the currect one. Make it more efficient by about 25%

The two first ones will highly improve the statistics robustness, and potential to detect and reject stuff. The last step should improve the results to get better spectra. In the few tests I did I see some improvement but nothing dramatic.

This is what the off/on (blue/green) now looks like. Only facets covered by the search radius and concerned with offs beeing scattered accross them.

This is the comparison between before (left) and now (right):

Noise is lower by 2% in the newer version with lower artifacts (in a few places it's higher, I'll experiment quickly with clipping the data with the high gains, but I think it's not too bad given we now will have good off statistics).

Joe - time to add you proper motion correction? Give me a module and I can update the code

The command line I've used was:

ms2dynspec.py --ms big-mslist.txt --data DATA --model DDF_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --DicoFacet image_full_ampphase_di_m.NS3.DicoFacet --noff 100

The facet layout is given by --DicoFacet argument

Hi, I bumped into the same issue for the second time after installing the ddf pipeline in a new machine (actually, a singularity).
The pipeline crashes at the dynspec step with the following error

Traceback (most recent call last):
  File "/opt/DynSpecMS/ms2dynspec.py", line 123, in <module>
    main(args)
  File "/opt/DynSpecMS/ms2dynspec.py", line 98, in main
    SaveMachine.PlotSpec()
  File "/opt/DynSpecMS/ClassSaveResults.py", line 116, in PlotSpec
    pdf.savefig(bbox_inches='tight')
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 2529, in savefig
    figure.savefig(self, format="pdf", **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/figure.py", line 2062, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 2212, in print_figure
    **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 2586, in print_pdf
    self.figure.draw(renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/figure.py", line 1493, in draw
    renderer, self, artists, self.suppressComposite)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/image.py", line 141, in _draw_list_compositing_images
    a.draw(renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 2635, in draw
    mimage._draw_list_compositing_images(renderer, self, artists)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/image.py", line 165, in _draw_list_compositing_images
    a.draw(renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/artist.py", line 55, in draw_wrapper
    return draw(artist, renderer, *args, **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/axis.py", line 1192, in draw
    renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/axis.py", line 1130, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/text.py", line 922, in get_window_extent
    bbox, info, descent = self._get_layout(self._renderer)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/text.py", line 300, in _get_layout
    ismath=False)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/backends/backend_pdf.py", line 2148, in get_text_width_height_descent
    renderer=self)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/texmanager.py", line 501, in get_text_width_height_descent
    dvifile = self.make_dvi(tex, fontsize)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/texmanager.py", line 365, in make_dvi
    texfile], tex)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/matplotlib/texmanager.py", line 335, in _run_checked_subprocess
    stderr=subprocess.STDOUT)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/subprocess32.py", line 334, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/subprocess32.py", line 614, in __init__
    restore_signals, start_new_session)
  File "/opt/matplotlib-2.2.3/lib/python2.7/site-packages/subprocess32.py", line 1393, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 2] No such file or directory: 'latex'
 - 14:59:52 - ClearSHM                  | Clear shared memory
 - 14:59:52 - ClearSHM                  | Clear Semaphores
 - 14:59:52 - ClearSHM                  | Clear shared dictionaries
FAILED to run ms2dynspec.py --ms mslist-L602890.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols DDS3_full_smoothed --rad 2. --image image_full_ampphase_di_m.NS.app.restored.fits --LogBoring 0 --SolsDir SOLSDIR: return value is 1
Traceback (most recent call last):
  File "/opt/ddf-pipeline/scripts/pipeline.py", line 1517, in <module>
    main(o)
  File "/opt/ddf-pipeline/scripts/pipeline.py", line 1421, in main
run(runcommand,dryrun=o['dryrun'],log=logfilename('ms2dynspec.log'),quiet=o['quiet'])
  File "/opt/ddf-pipeline/utils/auxcodes.py", line 54, in run
    die('FAILED to run '+s+': return value is '+str(retval))
  File "/opt/ddf-pipeline/utils/auxcodes.py", line 36, in die
    raise Exception(s)
Exception: FAILED to run ms2dynspec.py --ms mslist-L602890.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols DDS3_full_smoothed --rad 2. --image image_full_ampphase_di_m.NS.app.restored.fits --LogBoring 0 --SolsDir SOLSDIR: return value is 1

In both the machines pip install latex didn't fix the problem. In the previous installation I momentarily "fixed" the error by commenting this line of the script

Also Pepe had this problem, so I thought that it was the time to open an issue here.

For some reason in a few per cent of DR2 processing the size of freqs doesn't match the size of the relevant axis in the GOut structure. This causes the code to crash in plotting the frequency plots. I have fudged a fix which I'll put in a PR, basically to ensure that freqs is always the same size in the same way as is done for the times array.

I am trying to produce dynamic spectra for two directions from an AARTFAAC-12 data set having 24 channels and duration of 603.98 seconds. This results in:

...
Append MS 0................ 599/600 [================================================= ]  99%  - 0'53"
 - 11:44:29 - AsyncProcessPool          | [comp15] process comp15: exception raised processing job Stack_SingleTime:597: Traceback (most recent call last):
  File "/lofarsoft/DDFacet/DDFacet/Other/AsyncProcessPool.py", line 850, in _dispatch_job
    result = call(*args, **kwargs)
  File "/lofarsoft/DynSpecMS/ClassDynSpecMS.py", line 339, in Stack_SingleTime
    self.Stack_SingleTimeDir(iTime,iDir)
  File "/lofarsoft/DynSpecMS/ClassDynSpecMS.py", line 405, in Stack_SingleTimeDir
    self.DicoGrids["GridLinPol"][iDir,ich0:ich0+nch, iTime, :] = ds
ValueError: could not broadcast input array from shape (24,4) into shape (23,4)
    Append MS 0................ 600/600 [==================================================] 100%  - 0'53"
 - 11:44:29 - AsyncProcessPool          | Append MS 0: 600 jobs complete, average single-core time 2.12s per job
 - 11:44:29 - AsyncProcessPool          | Stack_SingleTime:*: 600 jobs returned an error. Aborting.

I'm getting the error below when running ddf-pipeline on a LoTSS field. Not sure if anyone else has seen this. Could it be something like there are no sources for it?

Reading metadata.............. 20/24 [========================================= ] 83% - 0'06 Reading metadata.............. 21/24 [=========================================== ] 87% - 0'07 Reading metadata.............. 22/24 [============================================= ] 91% - 0'07 Reading metadata.............. 23/24 [=============================================== ] 95% - 0'07 Reading metadata.............. 24/24 [==================================================] 100% - 0'08"
Traceback (most recent call last):
File "/net/para10/data1/shimwell/software/killmsddf/new-install/DynSpecMS/ms2dynspec.py", line 150, in
main(args)
File "/net/para10/data1/shimwell/software/killmsddf/new-install/DynSpecMS/ms2dynspec.py", line 104, in main
BeamNBand=args.BeamNBand)
File "/net/para10/data1/shimwell/software/killmsddf/new-install/DynSpecMS/ClassDynSpecMS.py", line 80, in init
self.InitFromCatalog()
File "/net/para10/data1/shimwell/software/killmsddf/new-install/DynSpecMS/ClassDynSpecMS.py", line 189, in InitFromCatalog
NOrig=self.PosArray.shape[0]
IndexError: tuple index out of range

• 18:18:51 - ClearSHM | Clear shared memory
• 18:18:51 - Multiprocessing | reaping 1 shared memory objects associated with 1 dead DDFacet processes
• 18:18:51 - ClearSHM | Clear Semaphores
• 18:18:51 - ClearSHM | Clear shared dictionaries
  FAILED to run ms2dynspec.py --ms mslist-L633681.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --imageI image_full_ampphase_di_m.NS.int.restored.fits --imageV image_full_low_stokesV.dirty.corr.fits --LogBoring 0 --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --srclist brightlist.csv: return value is 1
  Traceback (most recent call last):
  File "/net/para10/data1/shimwell/software/killmsddf/ddf-pipeline-DR2/ddf-pipeline/scripts/pipeline.py", line 1760, in
  main(o)
  File "/net/para10/data1/shimwell/software/killmsddf/ddf-pipeline-DR2/ddf-pipeline/scripts/pipeline.py", line 1724, in main
  run(runcommand,dryrun=o['dryrun'],log=logfilename('ms2dynspec.log'),quiet=o['quiet'])
  File "/net/para10/data1/shimwell/software/killmsddf/ddf-pipeline-DR2/ddf-pipeline/utils/auxcodes.py", line 54, in run
  die('FAILED to run '+s+': return value is '+str(retval))
  File "/net/para10/data1/shimwell/software/killmsddf/ddf-pipeline-DR2/ddf-pipeline/utils/auxcodes.py", line 36, in die
  raise Exception(s)
  Exception: FAILED to run ms2dynspec.py --ms mslist-L633681.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --imageI image_full_ampphase_di_m.NS.int.restored.fits --imageV image_full_low_stokesV.dirty.corr.fits --LogBoring 0 --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --srclist brightlist.csv: return value is 1

When I use DynspecMS on a source ~10 deg away from the phase centre, I get blank (all elements are zero) dynamic spec. The W and W2 images are blank, too. The same happens to the OFF pointings.

A few notes:

1. The elevation of the source is relatively low (compared to previous observations where DynSpecMS worked perfectly). Phase centre at ~40 deg elevation, source at ~30 deg.
2. We observed with 64 channels per SB with NenuFAR. That's different from our usual setting (12 averaged channels, edge channels flagged).
3. No error message popped up during DynSpecMS processing.
4. I've made images with columns KMS_SUB and DDF_PREDICT. Both images look fine.

Here is the script. Any suggestions?
dynspec.txt

Build a merged exoplanetary systems catalog from

https://exoplanet.eu/home/

https://exoplanetarchive.ipac.caltech.edu/

And a Gaia dr3 comparison stars catalog:

https://gea.esac.esa.int/archive/

The catalogs were not saved :((((( My bad. The tgz archives were made before the catalog was saved... I'm so pissed. It's fixed in the latest commit on ApplyBeam branch...

So far I've modified DDF to save the imaging weights, and also updated DynSpecMS to take them into account. Will post the results as they come.

I've found a bug in the quick look pdf files, the frequency axis is flipped upside down...

It seems that the WorkerProcessPool is no longer defined in the AsyncProcessPool in DDFacet.Other. Trying to start ms2dynspec errors out on line 6 ClassDynSpecMS.py:

from DDFacet.Other.AsyncProcessPool import APP, WorkerProcessError

with:

Traceback (most recent call last):
File "ms2dynspec.py", line 53, in
from ClassDynSpecMS import ClassDynSpecMS
File "/lofarsoft/DynSpecMS/ClassDynSpecMS.py", line 6, in
from DDFacet.Other.AsyncProcessPool import APP, WorkerProcessError
ImportError: cannot import name WorkerProcessError

Running ms2dynspec.py --ms mslist-L826950.txt --data DATA_DI_CORRECTED --model DD_PREDICT --sols [DDS3_full_smoothed,DDS3_full_slow_merged] --rad 2. --imageI image_full_ampphase_di_m.NS.int.restored.fits --imageV image_full_high_stokesV.dirty.corr.fits --LogBoring 0 --SolsDir SOLSDIR --BeamModel LOFAR --BeamNBand 1 --srclist brightlist.csv as in ddf-pipeline I get:

 - 14:56:24 - DynSpecMS                    | Using the surveys database
 - 14:56:25 - DynSpecMS                    | Created an array with 48643 records
 - 14:56:25 - DynSpecMS                    | Selected 7 target [out of the 48643 in the original list]
 - 14:56:25 - DynSpecMS                    | Including 14 off targets
 - 14:56:25 - DynSpecMS                    | Making random off catalog with 14 directions
Traceback (most recent call last):
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 195, in <module>
    main(args)
  File "/usr/local/src/DynSpecMS/ms2dynspec.py", line 113, in main
    D = ClassDynSpecMS(ListMSName=MSList, 
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 132, in __init__
    self.InitFromCatalog()
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 299, in InitFromCatalog
    self.PosArray=np.concatenate([self.PosArray,self.GiveOffPosArray(NOff)])
  File "/usr/local/src/DynSpecMS/ClassDynSpecMS.py", line 406, in GiveOffPosArray
    self.DFacet=DFacet=DDFacet.Other.MyPickle.Load(self.DicoFacet)
  File "/usr/local/lib/python3.9/dist-packages/DDFacet/Other/MyPickle.py", line 47, in Load
    G= cPickle.load( open( filein, "rb" ))
FileNotFoundError: [Errno 2] No such file or directory: ''

I could take a look at this but would need to know which branch is the current one.

Is it OK if I modify this to pull the target list out of the surveys database if it's not supplied?

That way we can add targets of interest without having to bother you. I'll just have to ingest the online file into the database.