Hi,

I encountered some kind of "inconsistency" in DPPP when handling h5parm in DDEcal.
If I use DDEcal in the mode rotation+diagonal I just get a solution table, like phase000 with 120 times, 60 ants, 1484 freqs, 2 pol.
But If I just change to the mode diagonal in DDEcal I get a solution table, like phase000 with 120 times, 60ants , 1484, 1 dir, 2 pol. Where does this direction table come from and is it really needed if solving only for one direction. And why does it not appear if one solves for common rotation angle?

Alex

When trying to predict A-team models (in Prefactor), I get the error "std exception detected: Invalid argument" with the following command:

DPPP msin.datacolumn=DATA predict.sourcedb=Ateam_LBA_CC.make_sourcedb_ateam predict.type=predict predict.sources="[VirA_4_patch,CygAGG,CasA_4_patch,TauAGG]" numthreads=2 predict.usebeammodel=True steps="[predict]" msin=L658738_SB198_uv.ndppp_prep_target msout.datacolumn=MODEL_DATA predict.usechannelfreq=false predict.operation=replace msout=L658738_SB198_uv.ndppp_prep_target

Strangely, this only happens on certain MS files. Additionally, if I add "dummy=dummy" to the end of the above command, it works (after printing a warning about misspelled arguments)! This is with the current master -- I can supply the input files if they're useful...

ApplyCal now uses nearest neighbor interpolation. Linear interpolation should be better in many cases. Care should be taken when interpolating phases.

cmake can't find hdf5, but i have hdf5 installed (libhdf5-dev package).

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
HDF5_dl_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_hdf5_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_hdf5_cpp_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_hdf5_hl_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_m_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_pthread_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_sz_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
HDF5_z_LIBRARY_DEBUG (ADVANCED)
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "makesourcedb" in directory /packaging/kern/packaging/build/dp3
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP
    linked by target "DPPP" in directory /packaging/kern/packaging/build/dp3/DPPP

-- Configuring incomplete, errors occurred!
See also "/packaging/kern/packaging/build/dp3/obj-x86_64-linux-gnu/CMakeFiles/CMakeOutput.log".
See also "/packaging/kern/packaging/build/dp3/obj-x86_64-linux-gnu/CMakeFiles/CMakeError.log".

Having one step which can do everything will save quite a lot of processing time, also since DDECal can predict multi-threaded, while h5parmpredict predicts sequential.

Some combinations of solver+mode do not properly handle flags (i.e. the solver doesn't write any flag). So far I noticed it in DDE+diag and in DIE+fulljones. A check of all modes in both solvers should be done to be sure that flags are always properly saved in h5parms.

At the same time, it would be good to also check if applycal properly propagates all flags back to the MS, this was not the case for a few modes, but needs to be there for all of them.

It would be nice to make some tests on solving leakage. What is missing is a solver that can minimize a matrix like (1, a; b, 1).

Easy to fix by adding some methods to ThreadPool, but not a priority for now.

Currently, GainCal and DDECal assume that model visibilities are always good. However, when using 'pre-apply' (i.e. corrupting the model visibilities with solutions from h5parm), flagged solutions may introduce bad model data. This should be propagated into DDECal and GainCal.

We just encountered this while working on an NCP dataset. Shifiting from the pole to another position gives problems. After shifting, all the headers indicate the right position: msoverview reports the location given to the phaseshift and FITS images have the right coordinates in their headers. However, the data is (in this case) 180 degrees off in right ascension.

The data was phase-shifted from RA=00:00:00.000000 DEC=+90.00.00.00000 to RA=13:15:57.040099 DEC=+89.29.08.61202, however, physically the data is 180 degrees off and represents the field located at RA=1:15:57.040099 DEC=+89.29.08.61202.

This is the image of the phase shifted data (left), compared to an image of the source in the NCP field that should be there:

This is the image of the phase shifted data, with the RA in the FITS header modified by 180 degrees (right) compared to the corresponding location in the NCP field:

Pretty much everything except the bright source in the first image is more or less subtracted from the field, hence the low flux level, but it demonstrates the problem. I guess this has something to do with right ascension being undefined or degenerate at the pole?

Frits

On is too dangerous.

CMake disables DDECal when armadillo is not found, but this results in undefined references:

./CMakeFiles/DPPP_OBJ.dir/DPPP/DPRun.cc.o: In function DP3::DPPP::DPRun::makeSteps(DP3::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DP3::DPPP::DPInput*)': DPRun.cc:(.text+0x3228): undefined reference to DP3::DPPP::DDECal::DDECal(DP3::DPPP::DPInput*, DP3::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)'
collect2: error: ld returned 1 exit status
DPPP/CMakeFiles/DPPP.dir/build.make:343: recipe for target 'DPPP/DPPP' failed
make[2]: *** [DPPP/DPPP] Error 1
CMakeFiles/Makefile2:313: recipe for target 'DPPP/CMakeFiles/DPPP.dir/all' failed
make[1]: *** [DPPP/CMakeFiles/DPPP.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Since only the ionospheric screen fitting depends on armadillo, it's probably best to only skip building that instead of DDECal entirely.

I have just compiled the last master from github and might be doing something wrong here, but when I try to:

msin = xxx.MS
msin.datacolumn = DATA
msout = .
msout.datacolumn = CORRECTED_DATA
steps = [sol]
sol.type = ddecal
sol.mode = rotation+diagonal
sol.maxiter = 500
sol.tolerance = 1e-3
sol.propagatesolutions = True
sol.usemodelcolumn = True <- NOTE!

I get this error:
std exception detected: Key sol.sourcedb unknown

I don't have this problem with the DPPP on cep3/leiden, am I doing something wrong in compiling the git version?

thanks!
Fra

I am testing DDECal (latest master) within Factor (using a single direction), and am getting >99% solutions as flagged (weight = 0). I noticed that the number of iteration is very low. E.g.,

Total NDPPP time     106.76 real       84.77 user        0.42 system
   18.9% MSReader
   81.1% DDECal solvetec.
           64.9% of it spent in predict
           34.8% of it spent in estimating gains and computing residuals
                0.29% spent in constraints
            0.0% of it spent in writing gain solutions to disk
Iterations taken: [3|2,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|\
3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3\
,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,\
4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4\
|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|\
3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3\
,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3,4|3]
    0.0% MSUpdater msout.

I used the following parameters:

msin.datacolumn=DATA
msout=.
numthreads=1
solvetec.approximatetec=True
solvetec.detectstalling=False
solvetec.h5parm=h5parm
solvetec.maxapproxiter=50
solvetec.maxiter=75
solvetec.mode=tecandphase
solvetec.nchan=2
solvetec.propagatesolutions=True
solvetec.solint=1
solvetec.sourcedb=sourcedb
solvetec.stepsize=0.2
solvetec.tolerance=1e-05
solvetec.type=ddecal
solvetec.uvlambdamin=80.0
steps=[solvetec]

I tried changing the tolerance and stepsize but got similar results. The sourcedb and MS file are at ftp.hs.uni-hamburg.de/pub/outgoing/rafferty in case they're helpful.

Now that the threadpool is used, it turns out there was an extra #pragma omp parallel for in the Simulator. Since this used to be a nested omp loop, it had no effect, but now that the threadpool takes care of the parallelization, omp does parallelize this loop and starts a new group of threads in each threaad, giving rise to 48^2 threads on my 48 core machine.

I'm actually trying DPPP on MWA data, but it is a bit of an issue to compile the LOFAR repository on MWA nodes

When trying to do an averager step with freqstep=6, I get the error "6 is outside 4-byte uint range". I traced the message to line 385 of https://github.com/lofar-astron/DP3/blob/master/Common/StringUtil.cc. Changing the < to > fixed it, but I'm not sure this is really the correct fix.

Because the tests depend on the LOFAR CMake macros, they have been disabled for now. They should be changed to work within the new environment (possibly with some rewritten existing macros from LOFAR? Or using a new testing platform?)

How about making an initial release of the stand-alone distribution so that packaging can be tested (see #39)?

No install targets are currently given. Preferably they should be installed in such a way that the steps which are provided by a separate lib are automatically found as well.

MWA datasets are large and need to be split up by channel and processed by running DPPP several times. This should in principle be possible with the msin.nchan and msin.startchan parameters, however due to a bug MSUpdater does not actually use offset the channels at the proper index. This is because the MSReader never sets startchan in the MSInfo object, only nchan.

Reported face to face by @revoltek: even when explicitly setting the nthreads to 1, DP3 will still spawn a number of threads equal to the nr of cpus in the machine. These threads are not given tasks, so that indeed the %CPU usage is not above the requested number of threads, but it is nevertheless a bit confusing for debugging, and when launching e.g. 64 times a DP3 process with nthreads=1, the 4000 threads that are spawned do take some resources (even though it does not seem to be an issue right now).

This is probably a small change, like having to construct the thread pools after the nthreads is own instead of before.

Setting usebeammodel=true in a DDE solve results in an H5 solution file having NaN entries for some values. The h5parmpredict (subtract) then results in sources not being subtracted. Doing the subtraction using DI predict (subtract) seems to work however.
This is the case when working with AARTFAAC data sets, I have not verified whether it happens for LOFAR data.

It would be nice if the key applycal.correction takes just the effect name. Currently, for H5parm, it expects the name of the soltab. In the new behavior, applycal would look for any soltab with the given correction. If there is more than one soltab with the given effect, Applycal should look at a new key applycal.soltab or give an error if it is not specified.

Extra user service would that applycal.soltab can be specified instead of applycal.correction which will then look up the correction type in the soltab (as currently).

For complex (amplitude+phase) and full jones solutions, the soltab should be a list with the two soltabs (in arbitrary order, or first amplitude then phase).

Examples of new behavior:

applycal.correction = fulljones
applycal.soltab = [amplitude000, phase003]
// Full jones solution with amplitude000, phase000
---
applycal.correction = diagonal
// Diagonal correction with the unique two soltabs that contain amplitude and phase (error if there are more)
---
applycal.correction = diagonal
applycal.soltab = [amplitude042, phase001]
// Diagonal correction with amplitude042, phase001
---
applycal.correction = phaseonly
// Phase-only correction with unique phase soltab, otherwise error
---
applycal.correction = phaseonly
applycal.soltab = phase002
// Phase-only with soltab phase002
---
applycal.soltab = phase002
// Phase-only correction with soltab phase002 (type deduced from soltab)
---
applycal.soltab = [amplitude000, phase001]
// Diagonal or full-jones correction with soltabs amplitude000, phase001 (type deduced from soltab)

On Galaxy (one of the cluster of the Pawsey computer centre), there's no boost python. This is fine, but it seems that the python install scripts don't handle a missing boost python properly:

CMake Error at CMake/PythonInstall.cmake:120 (install):
  install FILES given unknown argument
  "/group/mwa/software/aoflagger/v2.12.1/galaxy/lib/python2.7/site-packages/lofar/dppp/".
Call Stack (most recent call first):
  DPPP/CMakeLists.txt:27 (python_install)


WARNING, Boost-Python not found; PythonDPPP will not be built.
-- Configuring incomplete, errors occurred!

Just removing the python install things from line DPPP/CMakeLists.txt:27 fixes this, but obviously doesn't install pythondppp anymore.

I phase shifted my data almost to the FWHM of the station beam. Then I want to take out the beam (array_factor since the original data was already corrected in the phase center with the element beam) like this:

NDPPP msin=phaseshifted.ms msout=out.ms steps=[ab] ab.type=applybeam ab.usechannelfreq=true ab.beammode=array_factor ab.updateweights=true

As I understand things, NDPPP now should take out the beam (array factor part) in the -current- phase center of the ms. However, almost nothing changes when I make an image. While I should see the my fluxes go up by a factor > 2. Seems like this step is broken and it does not correctly pick up the phase center??

For the lofar software I manually patched that for a while:

https://github.com/kernsuite-debian/lofar/blob/master/debian/patches/soversion

Would be nice if this finally lands upstream.

I am trying to copy a column to a new MS file with the following command:

DPPP msin=L658734_SB088_uv.ndppp_prep_cal msout=test.ms steps=[] msin.datacolumn=CORRECTED_DATA

but get the error:

 copying info and subtables ...

std exception detected: Table column CORRECTED_DATA is unknown

I verified with casacore.tables that the CORRECTED_DATA column is definitely there. gdb suggests the problem is in MSWriter::createMS:

Thread 1 "DPPP" hit Catchpoint 1 (exception thrown), 0x00007ffff1ea28bd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff1ea28bd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff5c2b00d in casa::SetupNewTableRep::setup() () from #2  0x00007ffff5c2b6a4 in casa::SetupNewTableRep::SetupNewTableRep(casa::String const&, casa::TableDesc const&, casa::Table::TableOption, casa::StorageOption const&) ()
   from /home/lofar/opt4/casacore/lib/libcasa_tables.so.2
#3  0x00007ffff5c2b8d0 in casa::SetupNewTable::SetupNewTable(casa::String const&, casa::TableDesc const&, casa::Table::TableOption, casa::StorageOption const&) ()
   from /home/lofar/opt4/casacore/lib/libcasa_tables.so.2
#4  0x00000000004bee5c in DP3::DPPP::MSWriter::createMS(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DP3::DPPP::DPInfo const&, unsigned int, unsigned int) ()
#5  0x00000000004bdd32 in DP3::DPPP::MSWriter::updateInfo(DP3::DPPP::DPInfo const&) ()
#6  0x000000000046830b in DP3::DPPP::DPStep::setInfo(DP3::DPPP::DPInfo const&) ()
#7  0x0000000000468351 in DP3::DPPP::DPStep::setInfo(DP3::DPPP::DPInfo const&) ()
#8  0x0000000000468351 in DP3::DPPP::DPStep::setInfo(DP3::DPPP::DPInfo const&) ()
#9  0x000000000045752c in DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) ()
#10 0x0000000000454ece in main ()

It works fine if I set msin.datacolumn=DATA, but not for any other existing column name.

It seems that on measurement sets that have been concatenated in frequency, the memory settings of DPPP_AOFlag are computed based on only one of the MSs. My suspicion is that in the following line, infoIn.nchan() is computed from one of the MSs, not on the total concatenated thing:

double timeSize = (sizeof(casacore::Complex) + sizeof(bool)) *
(infoIn.nbaselines() + 3*nthread) * infoIn.nchan() * infoIn.ncorr();

As reported by @adrabent.

I am getting a segfault when doing a DDECal solve with the following command:

DPPP msin.datacolumn=DATA solve.propagatesolutions=True solve.sourcedb=calibration_skymodel.make_sourcedb numthreads=6 msin.starttime="09Dec2016/00:09:21.076" solve.solint=1 solve.mode=tec solve.maxiter=50 solve.type=ddecal steps=[solve] solve.tolerance=0.0001 msin=chunk2.ms solve.nchan=1 solve.stepsize=0.2 solve.smoothnessconstraint=3e5 msin.ntimes=0 solve.approximatetec=True solve.maxapproxiter=25 solve.h5parm=fast_phase_35.h5parm solve.uvlambdamin=80.0 msout=.

Tammo Jan suggested using gdb to debug it, and as far as I can tell, it looks like it has something to do with a NULL pointer dereference in DP3::DPPP::Predict::process(). Changing line 364 of https://github.com/lofar-astron/DP3/blob/master/DPPP/Predict.cc to for (size_t thread=0; thread!=pool->NThreads(); ++thread) { fixed the segfault, but I don't really know if that's the correct fix (my knowledge of c++ is pretty limited!). I can provide the MS file, sky model, etc. if needed.

the value of the option "propagatesolutions" is not saved in the logs for the dde solver

I did a predict and this is the MODEL_DATA column, I am surprised too see all these spikes and a generally non-smooth behavior. This is the parset I used (note the predict was on all concatenated SBs):

MSReader selecting baselines ...
MSReader
input MS: /data2/fdg/bootes_sparse/tgts/mss/TC00.MS
baseline: [CR]S*&
band 0
startchan: 0 (0)
nchan: 976 (0)
ncorrelations: 4
nbaselines: 666
ntimes: 7190
time interval: 4.00556
DATA column: DATA
WEIGHT column: WEIGHT_SPECTRUM
autoweight: false
Predict pre.
sourcedb: mss/TC00.MS/Bootes_HBA.corr.skydb
number of patches: 1
number of sources: 1958
all unpolarized: false
apply beam: true
mode: array_factor
use channelfreq: true
one beam per patch:true
operation: replace
threads: 6
MSUpdater msout.
MS: /data2/fdg/bootes_sparse/tgts/mss/TC00.MS
datacolumn: MODEL_DATA (has been added to the MS)
weightcolumn WEIGHT_SPECTRUM
writing: data
Compressed: no

flush: 0

Processing 7190 time slots ...

0%ERROR - MeasTable::dUTC(Double) (file /net/lofar1/data1/oonk/rh7_lof_aug2017_trunk/casacore/src/measures/Measures/MeasTable.cc, line 4396): Leap second table TAI_UTC seems out-of-date.
Until the table is updated (see the CASA documentation or your system admin),
times and coordinates derived from UTC could be wrong by 1s or more.
....10....20....30....40....50....60....70....80....90....100%
Finishing processing ...

NaN/infinite data flagged in reader

Percentage of flagged visibilities detected per correlation:
[0,0,0,0] out of 4673615040 visibilities [0%, 0%, 0%, 0%]
0 missing time slots were inserted

Total NDPPP time 88656 real 494069 user 4699.93 system
0.9% MSReader
98.9% Predict pre.
0.2% MSUpdater msout.

when I solve diag+rot or diag in regions with noisy data some stations are set at 0, some others at very high values (the effect varies with different weights). Note that in many cases the first ~half of the stations are set to high values and the second half to 0.

See: https://www.dropbox.com/s/5qlfqi5bg9ol5pz/amp_polXX.png?dl=0

In the file H5Parm.cc line 18:

 H5Parm::H5Parm(const std::string& filename, bool forceNew,
                bool forceNewSolSet, const std::string& solSetName):
     H5::H5File(filename,
                !forceNew&&access(filename.c_str(),F_OK)!=-1?H5F_ACC_RDWR:H5F_ACC_TRUNC
               )

I think this can be fixed by replacing the whole line with the condition with
forceNew?H5F_ACC_TRUNC:H5F_ACC_RDONLY

some inconsistencies between gaincal and ddcal:

• the haparm option in ddecal is parmdb=*h5 in gaincal
• the caltype option in ddecal is called mode in gaincal
• the "caltype" and "mode" names are not called in the same way (e.g. complexgain vs fulljones)

As reported by Neal:

I've been writing and using hdf5 files with the new NDPPP - I run NDPPP via an os.system call from the script. It seems to work very nicely except for one thing: if I want to open the file for writing afterwards, it doesn't let me do this unless I exit Python and restart (thus presumably automatically flushing and closing the open file). Please could I check whether NDPPP closes the file explicitly, and if not if it's possible to do that? (I tried workarounds such as renaming the file, but the open-flag seems to follow the file around).

Even though the prediction inside h5parmpredict is done with the predict step, which understands the fulljones correction, h5parmpredict opens the given tab to read the directions etc. and doesn't take fulljones into account.

In the current definition of SourceDB, the position of Gaussians is defined as North over East. Here, North refers to the NCP. However, in both PyBDSF and DPPP, this is effectively interpreted as the 'up' direction in the image, which corresponds to the NCP direction in the phase center. This means that for sources far from the phase center, the position angle is wrong.

This could be fixed by computing, per Gaussian source, a position angle correction (given its position and the phase center of the observation, and the NCP). Projection effects should be taken into account here; perhaps it's necessary to declare in the definition of gaussians that an NCP-projection is used here.

Note that currently, PyBDSF and DPPP are consistent for a given pointing, so when not transferring models from one field to another there is no problem.

I have 2 AARTFAAC MS with slightly diffrerent times (subsecond difference). I tried to copy the data for every column containing time information in the MS, however I still get the exception that the metadata differs.

In DPPP MultiMSReader.cc we hit this exception:

      if (!(near(itsStartTime, rdinfo.startTime())  &&
                 near(itsLastTime, itsReaders[i]->lastTime())  &&
                 near(itsTimeInterval, rdinfo.timeInterval())  &&
                 itsNrCorr == rdinfo.ncorr()  &&
                 itsNrBl   == rdinfo.nbaselines()  &&
                 itsFullResNChanAvg == itsReaders[i]->nchanAvgFullRes()  &&
                 itsFullResNTimeAvg == itsReaders[i]->ntimeAvgFullRes()  &&
                 getInfo().antennaSet() == rdinfo.antennaSet()  &&
                 allEQ (getInfo().getAnt1(), rdinfo.getAnt1())  &&
                 allEQ (getInfo().getAnt2(), rdinfo.getAnt2())))
        throw Exception(
                 "Meta data of MS " + itsMSNames[i]
                 + " differs from " + itsMSNames[itsFirst]);

Could it be separated such that the exact mismatch is clear?

This is really necessary to avoid people apply incorrect beams (i.e. fuil instead of differential), which I unfortunately see happening a lot!

In certain cases there are several stations flagged and the solver tries anyway to find solutions, obtaining noisy results. Can you add a "minimum number of baselines" option that if not satisfied flag the solution? Alternatively the same result can be achieved with a flagger step. Not sure what is the best approach.

Not a major issue

package could avoid a useless dependency if debian/dp3/usr/bin/DPPP was not linked against libpython2.7.so.1.0 (it uses none of the library's symbols)          
package could avoid a useless dependency if debian/dp3/usr/bin/DPPP was not linked against libboost_date_time.so.1.65.1 (it uses none of the library's symbols) 
package could avoid a useless dependency if debian/dp3/usr/bin/DPPP was not linked against libgtkmm-3.0.so.1 (it uses none of the library's symbols)            

Predicting multiple directions is currently really slow because it uses a nested OpenMP for loop over i) direction; and ii) sources. When more than one direction is used, the second loop doesn't get multi-threaded anymore. In my case, I have 3 directions with ~100, 100 and 2000 sources, and because of this I have to wait forever. It's not hard to implement a parallel for loop with a thread pool, which would be the best approach with all combinations of directions/sources.

It could be useful to disable updating the data column (or flag, or weights). A use case for this would be to apply a rough bandpass before flagging. After that the flags would need to be stored, but not the data with this bandpass (which could be calibrated later).

Could be trivially implemented by adding a parsetkey msout.writedata which would override itsWriteData in MSUpdater.cc (similarly for weights, flags).

At the start of a GainCal step (from Prefactor), I get the error "std exception detected: vector::reserve". gdb gives the following info:

Processing 1798 time slots ...

0%[New Thread 0x15553c4da700 (LWP 31675)]
[New Thread 0x15553c2d9700 (LWP 31676)]
[New Thread 0x15553c0d8700 (LWP 31677)]
[New Thread 0x15553bed7700 (LWP 31678)]
[New Thread 0x15553bcd6700 (LWP 31679)]
[New Thread 0x15553bad5700 (LWP 31680)]

Thread 1 "DPPP" hit Catchpoint 1 (exception thrown), 0x000015554f3af8bd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x000015554f3af8bd in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x000015554f3d826f in std::__throw_length_error(char const*) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00000000004deba8 in std::vector<std::thread, std::allocator<std::thread> >::reserve(unsigned long) ()
#3  0x00000000005864d8 in DP3::ThreadPool::ThreadPool(unsigned long) ()
#4  0x00000000005846ea in DP3::DPPP::Predict::process(DP3::DPPP::DPBuffer const&) ()
#5  0x000000000055b3a6 in DP3::DPPP::GainCal::process(DP3::DPPP::DPBuffer const&) ()
#6  0x0000000000512e92 in DP3::DPPP::Filter::process(DP3::DPPP::DPBuffer const&) ()
#7  0x000000000049b369 in DP3::DPPP::MSReader::process(DP3::DPPP::DPBuffer const&) ()
#8  0x000000000045782f in DP3::DPPP::DPRun::execute(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, char**) ()
#9  0x0000000000454c1e in main ()

The command I used was:

DPPP gaincal.usebeammodel=True gaincal.parmdb=instrument_directionindependent msin.baseline="CS*&; RS*&; CS*&RS*" gaincal.caltype=phaseonly gaincal.usechannelfreq=True gaincal.type=gaincal gaincal.solint=1 gaincal.nchan=0 msin.datacolumn=DATA gaincal.sourcedb=/beegfs/stuf316/2A0335ours/prefactor/Pre-Facet-Target2/2A0335ours_field.make_sourcedb_target steps="[filter,gaincal]" msout=L658738_SB001_uv_12C377384t_164MHz.msdpppconcat gaincal.beammode=array_factor msout.datacolumn=CORRECTED_DATA gaincal.maxiter=500 numthreads=2 msin=L658738_SB001_uv_12C377384t_164MHz.msdpppconcat

Let me know if you need more info or the data...

I just tried to build it in Leiden and I get a segfault when running just DPPP. Other things like DPPP -v (returns DPPP 0.0), DPPP -h and a simple DPPP msin=ms1.ms msout=ms2.ms steps=[] all seem to work though (haven't used it on heavier tasks yet).

Opening this here for reference with regards to lofar-astron/prefactor#198.

A subselection of antennas is made using msin.baseline, to select only stations present in both the solutions and the MS they're applied to, but the subsequent applycal steps still read from the initial antenna table in the MS, causing a crash.

The code currently has many "assert(..)" statements, that in most cases seem to catch user errors (e.g. things like a missing parmkey value). assert statements should only be used for catching code bugs by the developer, since asserts are not enabled in a release build. When user errors happen, an exception should directly be thrown instead.

In our new cluster (AMD powered, not sure if relevant) I find this:

MSReader
input MS: /localwork/fdg/zwcl0634/nodysco/tgts/mss/TC00.MS
band 0
startchan: 0 (0)
nchan: 488 (0)
ncorrelations: 4
nbaselines: 703
ntimes: 899
time interval: 4.00556
DATA column: SMOOTHED_DATA
WEIGHT column: WEIGHT_SPECTRUM
autoweight: false
DDECal ddecal.
H5Parm: mss/TC00.MS/tec.h5
solint: 3
nchan: 1
directions: [[]]
use model column: true
tolerance: 0.0005
max iter: 500
propagatesolutions: true
detect stalling: true
step size: 0.2
mode (constraints): tec
coreconstraint: 0
smoothnessconstraint:0
approximate fitter: true
subtract model: false
UVWFlagger ddecal.
uvm: []
um: []
vm: []
wm: []
uvlambda: [-1e+15,250]
ulambda: []
vlambda: []
wlambda: []
phasecenter: []
MSUpdater msout.
MS: /localwork/fdg/zwcl0634/nodysco/tgts/mss/TC00.MS
datacolumn: CORRECTED_DATA (has been added to the MS)
weightcolumn WEIGHT_SPECTRUM
writing: data
Compressed: no

flush: 0

Processing 899 time slots ...

0%BLAS : Program is Terminated. Because you tried to allocate too many memory regions.

however the node has 256 GB of memory and I am using the same command I tried an similar (or even less powerful) systems.