teuben / dc2019

Improving Image Fidelity on Astronomical Data: Radio Interferometer and Single-Dish Data Combination (workshop material)

Improving Image Fidelity on Astronomical Data: Radio Interferometer and Single-Dish Data Combination (DC2019)

This was the material we used for the DC2019 workshop. We planned to have all scripts, code, documentation, and presentations here, but our data are typically large and we will supply URLs from where you can download them from. Some scripts were taking from other projects, and may not be maintained here so well.

A conference summary is being prepared, which is accompanied by script4paper (eventually those will be located in a different github repo).

Combination Techniques

We will probably try out a few techniques:

• feather talk by Ginsburg; see also the uvcombine package
• tp2vis [talk by Teuben]
• hybrid [talk by Kauffmann]
• sdint [talk by Rau]
• ...

Software needed

Since we are going to be CASA based this week, we only support Linux and Mac. WSL on Window10 is currently too slow to be useful (but hasn't been tested).

• CASA: https://casa.nrao.edu/casa_obtaining.shtml. The current release is 5.5, but 5.6 was just released. We have helper scripts in contrib.
  • [https://casa.nrao.edu/casadocs/casa-5.6.0/introduction/release-notes-560](5.6.0 Release notes)
  • [https://casa.nrao.edu/casadocs/casa-5.6.0/introduction/release-notes-560[(5.5.0 Release notes)
  • CASA 6 can also be installed via pip wheels. We also have a helper script.
• QAC (this will also install TP2VIS, SD2VIS, SDINT and AU)
• TP2VIS (comes with QAC)
• SD2VIS (optional, can come with QAC)
• AU (comes with QAC)
• WidebandSDINT (comes with QAC) - as of July 2020 CASA 5.7 has this included as sdintimaging()

and recommended software

• DS9: http://ds9.si.edu/site/Home.html (the XPA tools can also be very useful)
• vanilla python3 via miniconda or anaconda (an install script is available in dc2019, but you may also be able to install modules in CASA's python)
• a spectral cube fits viewer (ds9, carta, casaviewer, QFitsView, glue). See also https://fits.gsfc.nasa.gov/fits_viewer.html

Data needed

see README_DC2018_data for more details

We should have a USB and portable HDD during the meeting for copying large datasets, but we strongly recommend you come prepared with the data loaded on your laptop. We hope to have an estimate for the minimum amount of space you will need for the experiments, and/or bring your own spare external HDD. It will probably be closer to 100GB than 10GB.

Installation

See the INSTALL file for some current guidelines.

Paper

Starting at the tag "draft1" we have moved the "scripts4paper" to a fresh github repo called DataComb

dc2019's Issues

Small indent error in datacomb.py

I'm running with a version of python3only that I pulled a few days ago. When I run execfile('DC_script.py'), I got an error

...unindent does not match any outer indentation level ...

It pointed me to line 930 of datacomb.py, and just under def getFreqlist() (of line 920), I found that the comment block with three quotes may have one set of the quotes aligned with the wrong indentation.

Maybe someone can check on this.

autoconf based setup

We do need some kind of "setup". Be it some kind of "python setup.py" or an autoconf based "configure". This is likely related to the issue on hardcoded paths, for one.

colormap picking for paper, using skymodel

the fitscmap script can be used to quickly make a few example png for a beauty contest.
skymodel.fits may not be appropriate (it has no negatives).
Peter will use an observed version and try out a few (e.g. sky0 from QAC)

CASA6 has an exit error (for me)

I'm sure this is just me, a peculiarity, but just in case somebody has the same. Here's hopefully a solution soon.
Upon "quit" or "exit" there is an error, and the log window does not disappear.

This is the error message:
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/a7/astromake.git/opt/casa/casa-6.2.0-124/lib/py/lib/python3.6/site-packages/casashell/private/init_system.py", line 155, in stop_logger
p.kill( )
AttributeError: 'str' object has no attribute 'kill'

tclean: to concat or not to concat

Just ran into one of those notorious questions if to concat or not. In the past, with TPMS we found we should not concat, and the TPMS should be at the end of the list. With M100 there has to be a concat....

Now I'm running into the sky where e.g. in sky0.py the INT file gives identical maps if concat is used or now. But on sky4.py the qac_mac() , where no TPMS is used, there is a small (< 1%) flux difference if concat is used or not. Frustrating.

turbustat install for feather not working?

After pip installing the following packages:

image_registration radio_beam reproject spectral_cube turbustat sklearn statsmodels scikit-image

the line
import turbustat.simulator
will fail with "from .spectrum import generate_spectrum"

DC_script.py fails at step 6 and 7

This is the new "python3only" branch (although the casa5 version also does it). Steps 0-5 all work.

In step-6 I get:

SEVERE SynthesisImagerVi2::defineImage (file casa-source/code/synthesis/ImagerObjects/SynthesisImagerVi2.cc, line 594) Error in building Coordinate System and Image Shape : 2021-04-30 13:22:02 SEVERE MSTransformRegridder::calcChanFreqs calcChanFreqs failed, check input start and width parameters

In step-7 I get this:

Table .../scripts4paper/tmp/skymodel-c_120L.sd.SD_ro_TPfac1000000.0.ms does not exist

seed=123 in tp2vis random after all?

The seed is fixed for tp2vis, in order to check regression. And this seems to be ok.

However, in the logs, the line

UVW shape (3, 186300) 186300 [-2011.48609595 -2231.85289107 -1724.16830919]

always prints a different UVW[:,1]

CASA 6.2 - sdintimaging of continuum - RuntimeError: PSF, Residual, Model Image missing

Our DC_run script fails for continuum images in the middle of sdintimaging in CASA 6.2 (RuntimeError with the following message: PSF, Residual, Model Image (or sumwt) do not exist. Attached are the casalogger and terminal outputs in CASA 6.2 and 6.1 (no error - for comparison). Problem happens only for the continuum data - maybe related to our per-plane beams fix. Setup is given in our current GiTHub files DC_run, datacomb (executes SDint with n=1 when setting n=0 in DC_pars*), and (e.g.) DC_pars_GMC-b (holds the tclean parameters). I use only a single 12m dataset (not true concat with other data sets) for script testing.

CASA6.1-sdint-NOerror-logger.log
CASA6.1-sdint-NOerror-terminal.log
CASA6.2-sdint-error-logger.log
CASA6.2-sdint-error-terminal.log

CASA 6.1

Although casa6.1 is out, it will be a parallel effort to ensure the scripts work under CASA6. In this issue we plan to track those, but for the forseeable future casa 5.6 and 5.7 ( just came out) are going to be used.

IQA_script: get_aperture error: Invalid shape (1, 1, 1120, 1120) for image data

I tried the get_aperture-function on skymodel-b and I feel we have to adjust the function to the output-style of our DC_run script - or vice versa. If I am to fix it, I would need the an expert's comment on what is actually failing where and why.

M100 spectral gridding

This is a continuing frustrating story. The latest version of qac_bench5 , which contains the M100 re-gridded data, has the first or last channel masked, depending if you use SDINT or Feather.
In earlier versions the first channel would have not all data included, resulting in a different beam, and forcing us to clean with a beam_per_channel.

Unclear if this is a python rounding, or casa meta data (e.g. restfreq) issue.

Confusing Inputs for User

It is unclear to me what the user defined parameters are in DC_script.py.

In particular, I have deduced from the script and DC_script.md that I need to change datacombpath, TP2WISpath, pathtoconcat, and pathtoimage. But it is confusing where to put the names of the MSs that will be combined. In the current version of the script that I downloaded (on 9.4.21), there are a lot of commented out lines around where ‘thevis’ which is rather confusing. It seems to me that I need to edit thevis but that is not listed as a variable that needs input from the user in DC_script.md (though I found a reference to thevis at the end of DC_script.md but the text makes it seem as if this is an as needed basis but I think you need thevis no matter what?). I know I need to put the MSs somewhere and I am assuming thevis is it, is this true?

As an aside, as a new user I am confused between vis listed in the DC_script.md and thevis in the script DC_script.py currently above vis. They seem to be different things but are named as if they are the same.

I might suggest making a separate file that contains all the user inputs and a description of what each is to make it clear these particular variables are the user defined inputs. And thus a consequent .md file describing only and specifically the inputs in more detail. Right now the inputs are a bit buried in the DC_script.md file.

Additionally, an example for a particular dataset would be extremely useful.

execfile() vs. run

Here we like to contrast running scripts in two ways:
execfile('script.py")
and
run script.py --foo 1.0 -bar a.fits
which would make scripting a lot more flexible, allowing parameters (QAC does that by default).

The problem is that the run version does not know about casa anymore. In CASA6 this is less of a problem, but still needs to be set up before you can use casa tasks and tools.

A cop out for the paper is to have all scripts hard coded, but it would be nice to have an environment that can be parameterized more easily.

What exactly is the PB for the TP

When the TPMS is imaged, the pbcor map shows clear edge effects that the model for the PB was not so good. Especially for imaging a point source this is clearly visible.

casa5 and casa6 show up to 10% differences in tclean for INT, but not TPINT

When the INT (interferometry only) visibilities are imaged, there is an interesting polka dot like difference between the results from CASA5 and CASA6, with peaks to ~10%. However when the TPINT (TP vis + INT vis) are imaged, the difference is just numerical noise. Very similar pattern when the original skymodel is smoothed with imsmooth. those two maps also differ on the 1e-6 level with a hashed pattern and a slight imprint of the underlying signal.

DC_script.py fails at Step 4: Faridani short spacings combination (SSC)

When running the DC_script.py, the process fails at Step 4: Faridani short spacings combination (SSC) exiting with the following error in CASA ending with TypeError: string indicies must be integers.

The terminal output is:
###
Step 4 Faridani short spacings combination (SSC)
###
Regridding lowres image...
2021-04-15 16:27:31 WARN imregrid::::casa You've specified that you want to regrid the spectral axis without specifying the output shape. Normally the length chosen would be that of the corresponding template axis, however, the template spectral axis is degenerate and one cannot regrid an axis such that its output length is one. So, removing axis 3 from the axes list and just copying the input spectral information to the output image
2021-04-15 16:27:31 WARN ImageRegrid::regrid You are regridding an image whose beam is not well sampled by the pixel size. Total flux can be lost when regridding such images, especially when the new pixel size is larger than the old pixel size. It is recommended to check the total flux of your input and output image, and if necessary rebin the input to have smaller pixels.

SD from simanalyze and OTF differ too much

For the skymodel simulation at 115 GHz, the TP beam is nominally 57.06".
Simanalyze produces a gmc_120L.sd.image file, but this differs from the imsmooth produced OTF map in total flux by about 7% and more disturbingly the difference map has a 3 banded horizontal structure.

In skymodel-b.fits the region1 has 6644 Jy. (see also skymodel.md)
The OTF has around 6402 Jy (slighty dependent on the exact beam)
The SD map from simanalyze has 6035 Jy.

failed to find a viewertool...

command line scripts (e.g. called from the commandline) sometimes have fatal errors making plots, and issue a casa warning:

SEVERE imview:::: failed to find a viewertool...

Ticket to track redundancies in datacomb.py and DC_script.py

As of April 2021, the wrapper script datacomb.py can be tested, together with DC_script.py. The various combinations teams will run these scripts, with additional iterations, in order to test that the output combined images are as expected.

If anyone finds redundancies in the script (i.e. rms calculated several times in different methods), please document them here. Then we can coordinate in order to minimize unnecessary redundancies.

casa session crashed when doing the TCLEAN following tp2vis

I will try again with the copypointing=False in the concat of 12m and 7m visibilities, in case that makes a difference. Here is what I did when the error occured (using the Lupus 3 mms test data).

vis7m = datadir+'mst_07_nchan10_start0kms.ms'
vis12m = datadir+'mst_12_nchan10_start0kms.ms'
concat(vis=[vis12m,vis7m],concatvis=vis12m7m)

TPfits = datadir+'TP_12CO.fits'
TPim = 'TP.image' #You choose the name of your TP image
#importfits(fitsimage=TPfits,imagename=TPim)

tp2vis(TPim,'tp_winpix9.ms','12.ptg',rms=TPrms, winpix=9)

CASA <27>: tclean(vis=[vis12m7m,'tp_winpix9.ms'],imagename=lineimagename+'_clean',field=f
...: ield,spw='',phasecenter=phasecenter,mosweight=True,specmode='cube',niter=niter
...: ,cycleniter=cniter,gain=gain,threshold=threshold,usemask='pb',pbmask=0.3,**lin
...: eimage12m)
...:

0%....10....20....30....40....50....60....70....80....90....100%

0%....10....20....30....40....50....60....70....80....90....100%
libc++abi.dylib: terminating with uncaught exception of type casacore::AipsError: RegularFileIO: error in open or create of file /Users/aplunket/ResearchNRAO/Meetings/DataComb2019/Lup3mms/Post-workshop/tp2vis_clean.mask/logtable/table.dat: Too many open files
tar: 222AE4E9-8C5A-4C48-851E-421CA59B9352.dmp: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
rm: 222AE4E9-8C5A-4C48-851E-421CA59B9352.dmp: No such file or directory

---

CASA has crashed...

---

A crash report is being generated and submitted. The report
will contain information about CASA's environment when the
crash occurred (e.g. CASA log, call stack, CPU information).
This should not take too long...

Abort trap: 6
(base) pyrope:Post-workshop aplunket$