scottransom / presto Goto Github PK

Hi @scottransom,

The filterbank header format with FREQUENCY_START <list of freqs> FREQUENCY_END as output by e.g. SIGPROC's splice cannot be read by PRESTO. Maybe support for this could be added?

I was trying to run a code with sifting. but there is a error. I have imported: sifting
File "./pipeline.py", line 278, in ACCEL_sift
cands.sort(sifting.cmp_sigma)
AttributeError: 'module' object has no attribute 'cmp_sigma'

Hi @scottransom,

I was wondering the reason why a dispersive delay time is subtracted from the start time of the observation when running prepdata with a non-zero DM. This only happens when the barycentering flag is enabled.

This seems to suggest that you would have different barycentric start times (given in the *.inf files) for different DMs. If this were true, then you should see a slope in the DM vs time plots for say a single pulse detected at multiple DMs (assuming dispersive effects are negligible).

Since you don't see this behavior, should the start time for observations de-dispersed at non-zero DMs really be the barycentric time output by TEMPO with DM = 0.0?

Thanks,
Aaron

I am trying to install PRESTO on Ubuntu. I believe I have installed all the necessary dependencies and set all environment variables as described. Make fails with the following error:

accel_utils.c:1648:71: error: ‘PROT_READ’ undeclared (first use in this function)
(fcomplex *) mmap(0, sizeof(fcomplex) * obs->numbins, PROT_READ,
^~~~~~~~~
accel_utils.c:1649:35: error: ‘MAP_SHARED’ undeclared (first use in this function)
MAP_SHARED, obs->mmap_file, 0);
^~~~~~~~~~
accel_utils.c:1650:29: error: ‘MAP_FAILED’ undeclared (first use in this function); did you mean ‘MAP_SHARED’?
if (obs->fft == MAP_FAILED) {
^~~~~~~~~~
MAP_SHARED

Hi,

I'm following the tutorial at:

https://www.cv.nrao.edu/~sransom/GBT_Lband_PSR_cmd_history.txt

This command:

rfifind -nocompute -time 1.0 -freqsig 6.0 -mask -o Lband_rfifind.mask lband GBT_Lband_PSR.fil

Results in:

rfifind: mandatory option `-o' missing

I assume it's supposed to be:

rfifind -nocompute -time 1.0 -freqsig 6.0 -mask Lband_rfifind.mask -o Lband GBT_Lband_PSR.fil

Then:

cp $PRESTO/python/dedisp.py .

dedisp.py doesn't exist in that path. There are two versions: $PRESTO/tests/dedisp.py $PRESTO/examplescripts/dedisp.py, I presume the former should be used.

Likewise:

cp $PRESTO/python/ACCEL_sift.py .

Path seems to be $PRESTO/examplescripts/ACCEL_sift.py

Thanks.

Starting from scipy 1.0.0 scipy.stats.histogram is now deprecated ad removed making it backwards incompatible. Running test_fftfit.py will result in ImportError:

python fftfit_src/test_fftfit.py
Traceback (most recent call last):
File "fftfit_src/test_fftfit.py", line 12, in
from psr_utils import gaussian_profile, TWOPI
File "/home/kat/software/presto/lib/python/psr_utils.py", line 4, in
from scipy.stats import histogram
ImportError: cannot import name histogram
make: *** [build] Error 1

The best is to use numpy.histogram:

diff --git a/lib/python/psr_utils.py b/lib/python/psr_utils.py
index 99225b8..daffb52 100644
--- a/lib/python/psr_utils.py
+++ b/lib/python/psr_utils.py
@@ -1,7 +1,6 @@
import numpy as Num
import numpy.fft as FFT
import Pgplot, ppgplot, bisect, sinc_interp, parfile
-from scipy.stats import histogram
from scipy.special import ndtr, ndtri, chdtrc, chdtri, fdtr, i0, kolmogorov
from scipy.optimize import leastsq
import scipy.optimize.zeros as zeros
@@ -94,7 +93,7 @@ def hist(data, bins, range=None, laby="Number", **kwargs):
data values are used to define the interval.
Note: This command also accepts all the keyword arge of plotbinned().
"""
- (ys, lox, dx, out) = histogram(data, bins, range)
+ (ys, lox, dx, out) = Num.histogram(data, bins, range)
xs = Num.arange(bins, dtype='d')dx + lox + 0.5dx
maxy = int(1.1*max(ys))
if maxy < max(ys):

Best,
Maciej

I tried to use mpiprepsubband instead of prepsubband, hoping that I get better performance.
I noticed that it doesn't support the -sub and -subdm!
Executing it throws an error.

Here is the output of calling mpirun -np 1 mpiprepsubband ...

readfile GBT_Lband_PSR.fil
Assuming the data is a SIGPROC filterbank file.


1: From the SIGPROC filterbank file 'GBT_Lband_PSR.fil':
                  Telescope = GBT
                Source Name = Mystery_PSR
            Obs Date String = 2004-01-06T11:38:09
             MJD start time = 53010.48482638889254
                   RA J2000 = 16:43:38.1000
             RA J2000 (deg) = 250.90875        
                  Dec J2000 = -12:24:58.7000
            Dec J2000 (deg) = -12.4163055555556
                  Tracking? = True
              Azimuth (deg) = 0
           Zenith Ang (deg) = 0
            Number of polns = 2 (summed)
           Sample time (us) = 72               
         Central freq (MHz) = 1400             
          Low channel (MHz) = 1352.5           
         High channel (MHz) = 1447.5           
        Channel width (MHz) = 1                
         Number of channels = 96
      Total Bandwidth (MHz) = 96               
                       Beam = 1 of 1
            Beam FWHM (deg) = 0.147
         Spectra per subint = 2400
           Spectra per file = 531000
      Time per subint (sec) = 0.1728
        Time per file (sec) = 38.232
            bits per sample = 4
          bytes per spectra = 48
        samples per spectra = 96
           bytes per subint = 115200
         samples per subint = 230400
                zero offset = 0                
           Invert the band? = False
       bytes in file header = 365



    ============Generate Dedispersion Plan===============

    
DDplan.py -d 80 -n 96 -b 96.0 -t 0.000072 -f 1400.000000 -s 32 -o DDplan.ps

Minimum total smearing     : 0.102 ms
--------------------------------------------
Minimum channel smearing   : 0 ms
Minimum smearing across BW : 0.00145 ms
Minimum sample time        : 0.072 ms

Setting the new 'best' resolution to : 0.072 ms
Best guess for optimal initial dDM is 0.496

  Low DM    High DM     dDM  DownSamp  dsubDM   #DMs  DMs/call  calls  WorkFract
    0.000     84.000    0.50       1   12.00     168      24       7    1





    ================Dedisperse Subbands==================

    
mpirun -np 1 mpiprepsubband -nsub 32 -downsamp 1 -o Sband ../GBT_Lband_PSR.fil

mpirun -np 1 mpiprepsubband -nsub 32 -lodm 0.000000 -dmstep 0.500000 -numdms 24 -numout 531000 -downsamp 1 -o Sband Sband_DM5.75.sub[0-9]*


      Parallel Pulsar Subband De-dispersion Routine
                 by Scott M. Ransom

Assuming the data are SIGPROC filterbank format...
Reading SIGPROC filterbank data from 1 file:
  '../GBT_Lband_PSR.fil'

    Number of files = 1
       Num of polns = 2 (summed)
  Center freq (MHz) = 1400
    Num of channels = 96
    Sample time (s) = 7.2e-05       
     Spectra/subint = 2400
   Total points (N) = 531000
     Total time (s) = 38.232        
     Clipping sigma = 6.000
   Invert the band? = False
          Byteswap? = False
     Remove zeroDM? = False

File  Start Spec   Samples     Padding        Start MJD
----  ----------  ----------  ----------  --------------------
1              0      531000           0  53010.48482638889254

[presto:08436] *** Process received signal ***
[presto:08436] Signal: Floating point exception (8)
[presto:08436] Signal code: Integer divide-by-zero (1)
[presto:08436] Failing at address: 0x5643979573b7
[presto:08436] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7fabe17f8980]
[presto:08436] [ 1] mpiprepsubband(+0x33b7)[0x5643979573b7]
[presto:08436] [ 2] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fabe06a4bf7]
[presto:08436] [ 3] mpiprepsubband(+0x625a)[0x56439795a25a]
[presto:08436] *** End of error message ***
[presto:08415] [[34117,0],0] usock_peer_send_blocking: send() to socket 30 failed: Broken pipe (32)
[warn] Epoll ADD(4) on fd 34 failed.  Old events were 0; read change was 0 (none); write change was 1 (add): Bad file descriptor
[presto:08415] [[34117,0],0] ORTE_ERROR_LOG: Unreachable in file oob_usock_connection.c at line 316
[presto:08415] [[34117,0],0]-[[34117,3],0] usock_peer_accept: usock_peer_send_connect_ack failed

prepdata crashes when the output number of bins in the .dat file exceeds the maximum value for an int32 of 2,147,483,647 bins. prepsubband may complete, but ends up writing a .inf file with a negative number of bins.

Also, the command-line interpreter can not take a -numout value that large either.

These two issues are independent.

I recommend updating the python code to be compatible with Python3 as more people are adopting Python3 and are having issues. Currently, setup.py just gives this misleading error message:

% python3 setup.py install --home=${PRESTO}
  File "setup.py", line 44
    raise Exception, "os not supported"
                   ^
SyntaxError: invalid syntax

Tested with two different installations of Anaconda python 2.7 on Ubuntu 16.04. Under the system's stock Python 2.7 everything works correctly instead. This might have to do with the C and/or Fortran libraries that are linked during the compilation of the dependencies (PGPLOT, FFTW etc.). I'm investigating this issue, if someone else has experienced similar issues please add to the discussion.

It would be nice to be able to run many instances of prepfold at the same time, e.g. using multiprocessing.Pool() in Python. However, this is not safe and ends up crashing because the polycos that are generated by each instance stomp on each other. It would be nice if unique tempfiles were used to prevent this from happening.

On many machines, this include causes an error, but it seems not to be necessary, so I recommend just deleting this line.

I'm getting a bunch of error messages when I run 'make' in $PRESTO/python, after successfully running 'make' in $PRESTO/src. In the attached text file, you'll find all the printed output that I get when I run 'make'.
presto_error.txt
I get a bunch of messages telling me to 'recompile with -fPIC'...I'm not sure what it wants me to recompile, I've tried recompiling pgplot with -fPIC in the flags but that does not solve the issue. Any help would be appreciated!

I am getting NULL values in all the rows in result set in presto client in using s3 partitions.

1. Data is stored in the S3 in the encripted form(client side encription) using AWS KMS master key.
2. EMR Hive is able to decrypt the S3 data on select query.
3. We have done below listed settings in the Presto.hive.properties but it is returning null values(screenshot is attached) against all the rows on making select [query.]

Your help will be really appreciable.
Gaurav

search_bin.o: In function `main':
/packaging/repos/presto/src/search_bin.c:206: undefined reference to `read_wisdom'
gcc -I/packaging/repos/presto/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include  -I  -DUSEFFTW -DUSEMMAP -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -Wall -W -fPIC -O3 -ffast-math -Wno-unused-result -Wno-unused-but-set-variable -Wno-unused-but-set-parameter -fopenmp -o /packaging/repos/presto/bin/prepsubband prepsubband.o prepsubband_cmd.o backend_common.o zerodm.o sigproc_fb.o psrfits.o -lcfitsio -lpthread -L/packaging/repos/presto/lib -lpresto -lfftw3f -lcfitsio -lm
collect2: error: ld returned 1 exit status
Makefile:296: recipe for target 'search_bin' failed

looks like read_wisdom is defined in include/ransomfft.h, but when I include that it fails even more. removing read_wisdom seems to solve the problem, not sure if that is the smartest thing to do.

Good afternoon,
I'm not sure this is an issue with the software, it may stem from my misunderstanding of it!

I've added a custom observatory in TEMPO's obsys.dat:

3573741.1 -269156.74 5258407.3 1 ACRE - AR

I would like to fold topocentric observations taken at this observatory.

In the .inf file corresponding to my observation, I have input my observatory name (I have tried both "AR" and "ACRE") and observing frequency. The following fold command still defaults to barycentre for polyco generation. In the produced polyco headers, the observatory code is -1 and the observing frequency 0.
prepfold -double -noxwin -par B0329+54.par timeseries.dat
Note, the par file also contains the observatory code "AR". I have tried without, too.

To try and tackle this, I went to generate polycos with TEMPO on its own, and then input them in PRESTO. The polycos are generated correctly with:
tempo -ZOBS=AR -ZFREQ=407.5 -ZTOBS=4.1 -ZSTART=57969.1 -ZNCOEFF=15 -f B0329+54.par

But when I try and input them into the fold, this is the result:

prepfold  -polycos polyco.dat -psr 0332+5434 -double -noxwin timeseries.dat  


        Pulsar Raw-Data Folding Search Routine
 Used for DM, Period, and P-dot tweaking of PSR candidates.
                 by Scott M. Ransom

Assuming the data are PRESTO time series of floats format...
Reading input data from 'timeseries.dat'.
Reading information from 'timeseries.inf'.

Folding a PSR_0332+5434 candidate.

Output data file is 'timeseries.pfd'.
Output plot file is 'timeseries.pfd.ps'.
Best profile is in  'timeseries.pfd.bestprof'.

Read 0 set(s) of polycos for PSR 0332+5434 at 57969.131833390893 (DM = 4.6704e-310)
MJD 57969.132 out of range (    0.000 to     0.000)
isets = 0

The output files are not created. The same happens with PINT-produced polycos.

Thanks for reading!
Best,
Emma

in tutorial,"makezaplist.py Lband.birds",this cmd needs a "birds" file ,but i can't find that in any directory,does it make by "accelsearch" or i need to make one by myself?

Some problems come up when you try to read a filterbank file into waterfaller.py. It does not seem as though there is support for the format, despite the description on the file preamble (e.g., it isn't even possible to get past the if statement from line 281 onwards ).

I can add support on my fork and then submit a pull request.
Should I wait for Chitrang's pull request to be finalized before working on this?

When I do python setup.py develop --user the install seems to work, but import presto fails. Not sure why. Doing python setup.py install --user works OK, but then I'll have to remember to re-install after any changes I make.

In single_pulse_search.py a call to scipy.stats.histogram is made, but this has been deprecated since scipy>=0.17 (see docs). Apparently the solution is to just use numpy.histogram instead.

This causes an AttributeError to be raised and thus there are no diagnostic plots created (all the .singlepulse files are still written though).

I have a dedispersed time series which I'm trying to run realfft on and it prints this message:

realfft() arrays lengths must be evenly divisible by 2.

and it then doesn't write anything to the output .fft file. Any idea what causes this? The .dat and .inf files were created by running prepdata on psrfits files.

Screenshot:

Hi
I have installed presto just like the INSTALL file said (I guess) but when installation finished (without error), presto's commands are not recogonized in terminal! here is a screenshot:
http://tinypic.com/r/161x34x/8

what's wrong?

We are currently packaging up and containerising tools like presto for easy setup on arbitrary systems, from laptop to HPC cluster. We are experiencing a problem with RFI find.

rfifind segfaults when ran as non existing user. This might sound weird, but we are building a pipeline where we encapsulate various packages like presto inside (docker) containers and run them as the UID matching the callers UID on the host.

To replicate the problem take a filter bank like GBT_Lband_PSR.fil, put that in a folder

put a Dockerfile in a same folder:

FROM kernsuite/base:3
RUN docker-apt-install presto python-presto

in that folder run

$ docker build -t kernsuite/presto .

Now run that container:

 docker run -v `pwd`:/data:ro  kernsuite/presto rfifind -o /tmp/bla /data/GBT_Lband_PSR.fil

which runs fine.

Setting the UID crashes rfifind:

docker run -u 5555:5555 -v `pwd`:/data:ro  kernsuite/presto rfifind -o /tmp/bla /data/GBT_Lband_PSR.fil

Hi, Scott. When I run the "make makewisdom" commond according to the INSTALL file. An error "Attempting to read the system wisdom file... failed." appears. I wonder why and what is the variable "fftwf_import_system_wisdom"? I have set the FFTW path correctly.

Hi @scottransom, is there a specific reason why MAXNUMONOFF is hardcoded to 40?

For X-ray data from some missions, it would be very helpful to increase it, so that we can exclude from the analysis all the frequent gaps due to telemetry and orbital occultation, SAA passages, etc, and get more consistent statistics.

(I'm tagging here also @ferrigno)

Hi Scott,

We are trying to use PRESTO to create dedispersed time series of some GC observations taken with Nenufar.

However, we are likely having problems with the memory allocation.
We suspect that this is due to the extremely low frequencies (45-82 MHz), which make the dedispersion challenging.

Below I am pasting the error I get with prepdata, when trying to create dedispersed time series from a filterbank file of a test observation, with the following characteristics:

aridolfi@nancep5:/data/aridolfi/B0950_test$ readfile B0950+08_digifilled.fil
Assuming the data is a SIGPROC filterbank file.

1: From the SIGPROC filterbank file 'B0950+08_digifilled.fil':
                  Telescope = Nancay
                Source Name = B0950+08
            Obs Date String = 2020-05-10T18:01:00.0282929
             MJD start time = 58979.75069477190846
                   RA J2000 = 09:53:09.3096
             RA J2000 (deg) = 148.28879        
                  Dec J2000 = 07:55:35.7499
            Dec J2000 (deg) = 7.9265972        
                  Tracking? = True
              Azimuth (deg) = 0
           Zenith Ang (deg) = 0
            Number of polns = 2 (summed)
           Sample time (us) = 5.12             
         Central freq (MHz) = 63.96484375      
          Low channel (MHz) = 45.3125          
         High channel (MHz) = 82.6171875       
        Channel width (MHz) = 0.1953125        
         Number of channels = 192
      Total Bandwidth (MHz) = 37.5             
                       Beam = 0 of 1
            Beam FWHM (deg) = 3.222
         Spectra per subint = 2400
           Spectra per file = 1799352
      Time per subint (sec) = 0.012288
        Time per file (sec) = 9.21268224
            bits per sample = 2
          bytes per spectra = 48
        samples per spectra = 192
           bytes per subint = 115200
         samples per subint = 460800
                zero offset = 0                
           Invert the band? = False
       bytes in file header = 349

We figured that this has probably something to do with the “worklen” variable in the “prepdata.c" code, or something related to the loading of the filterbank file in "sigproc_fb.c"

If you would like to give it a try, I copied the filterbank file that we used in our fortalesa server, for you to grab at:

/scratch1/dropfolder/nenufar_B0950+08_digifilled.fil

Note that the error occurs when we try to use DMs >= 40.6.
With any DMs lower than that value, prepdata works ok with the file that we are using in this example.

Would it be possible for you to fix this, so as to make PRESTO work with search-mode data taken at super-low frequencies, like those of NenuFar?

Thank you very much in advance!

Cheers,
Alessandro

=============================================================================
aridolfi@nancep5:/data/aridolfi/B0950_test$ prepdata -nobary -dm 40.6 -o  ts_B0950+08_digifilled B0950+08_digifilled.fil


          Pulsar Data Preparation Routine
   Type conversion, de-dispersion, barycentering.
                by Scott M. Ransom

Assuming the data are SIGPROC filterbank format...
Reading SIGPROC filterbank data from 1 file:
 'B0950+08_digifilled.fil'

   Number of files = 1
      Num of polns = 2 (summed)
 Center freq (MHz) = 63.964844
   Num of channels = 192
   Sample time (s) = 5.12e-06      
    Spectra/subint = 2400
  Total points (N) = 1799352
    Total time (s) = 9.21268224    
    Clipping sigma = 6.000
  Invert the band? = False
         Byteswap? = False
    Remove zeroDM? = False

File  Start Spec   Samples     Padding        Start MJD
----  ----------  ----------  ----------  --------------------
1              0     1799352           0  58979.75069477190846

Setting a 'good' output length of 1920000 samples
Writing output data to 'ts_B0950+08_digifilled.dat'.
Writing information to 'ts_B0950+08_digifilled.inf'.

Massaging the data ...


Error in gen_fvect(): Cannot allocate memory
Amount Complete = 0%
aridolfi@nancep5:/data/aridolfi/B0950_test$

Version of PRESTO used:

commit 853b985957597cfc364b69e1bf26dc447ca13876 (HEAD -> master, origin/master, origin/HEAD)
Author: Scott Ransom <[email protected]>
Date:   Sat Jan 9 16:06:37 2021 -0500

    Fixed bug when using -pfact/-ffact and have non zero fdot or fdotdot

Hi @scottransom,

I ran into the following compilation issue when running make while trying to install presto on the Compute Canada cedar server:

`gcc -I/home/efonseca/projects/rrg-istairs-ad/software/src/presto/include -I/cvmfs/soft.computecanada.ca/nix/store/z52y2009k7njxsww9bgkbiqipl195lnh-glib-2.48.2-dev/include/glib-2.0 -I/cvmfs/soft.computecanada.ca/nix/store/qqqzb2sg02m9b526ns49jszk6svz8px3-glib-2.48.2/lib/glib-2.0/include -I/cvmfs/soft.computecanada.ca/nix/store/ys5ah1dn6gz5iwmrh7fr9jyi41c8rx2d-cfitsio-3.41/include -I -I/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/intel2016.4/fftw/3.3.6/include -DUSEFFTW -DUSEMMAP -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -Wall -W -fPIC -O3 -ffast-math -Wno-unused-result -Wno-unused-but-set-variable -Wno-unused-but-set-parameter -fopenmp -I/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include -c -o dispersion.o dispersion.c

dispersion.c: In function ‘dedisp_subbands’:

dispersion.c:195:58: error: ‘dind’ is predetermined ‘shared’ for ‘shared’

#pragma omp parallel for default(none) private(jj) shared(sub,chan,numpts,dind)
^
make: *** [: dispersion.o] Error 1
`
This looks somewhat related to the latest commit that concerns shared-variable errors. Let me know if you'd like additional info.

• e.f.

If PGPLOT_DIR is not set, a bare -I eats -DUSEFFTW and you get a silent miscompilation of fastffts.c and a mystery failure on linking search_bin (missing symbol read_wisdom). On a system with pgplot actually installed, PGPLOT_DIR can be set to anything.

I suggest a change in the Makefile so that if PGPLOT_DIR isn't set the -I and -L don't get added to the command line. Patch follows and appears to work.

diff --git a/src/Makefile b/src/Makefile
index 277c2b0..78b0b4a 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -23,10 +23,15 @@ endif
 X11LINK := $(shell pkg-config --libs x11)
 PNGLINK := $(shell pkg-config --libs libpng)
 
-# Include and link information for PGPLOT v5.X (including shared libs!)
-# Typically you need to have your PGPLOT_DIR environment variable set
-PGPLOTINC = -I$(PGPLOT_DIR)
-PGPLOTLINK = -L$(PGPLOT_DIR) -lcpgplot -lpgplot $(X11LINK) $(PNGLINK)
+ifdef PGPLOT_DIR
+       # Include and link information for PGPLOT v5.X (including shared libs!)
+       # Typically you need to have your PGPLOT_DIR environment variable set
+       PGPLOTINC = -I$(PGPLOT_DIR)
+       PGPLOTLINK = -L$(PGPLOT_DIR) -lcpgplot -lpgplot $(X11LINK) $(PNGLINK)
+else
+       PGPLOTINC = 
+       PGPLOTLINK = -lcpgplot -lpgplot $(X11LINK) $(PNGLINK)
+endif
 
 # Include and link information for the FFTW 3.X single-precision library
 FFTINC := $(shell pkg-config --cflags fftw3f)

I'm performing a fairly standard prepfold command-
prepfold -nsub 128 -mask guppi_57955_P86Y1272_0003_rfifind.mask -p 0.00246068325 -dm 33.08 -searchpdd -o joecandsearchpdd *.fits
After it's done folding it searches over DMs, periods, pdots, and pdotdots as normal, but when "amount complete" reached 100% it didn't stop. It's currently at 145% and climbing...

I'll note I performed this exact same prepfold command sans the -searchpdd flag and it worked just fine.

Hi all,

I'd like to use this opportunity to introduce myself to everyone. I am Pablo Torne, and I am currently a Spanish PhD student at Michael Kramer's pulsar group in Bonn. I have a technical background + astrophysics, and I am involved in new receiver projects for Effelsberg (Ultra-Broad-Band and Phased Array Feed) and also on pulsar searching.

I am using PRESTO and I have found some bugs in the new version:

1. "explorefft" does not include new frequencies in the loaded zaplist file when using the "Z" option in the menu. The frequencies are marked in gray in the plot, but no numbers are added to the zaplist file.

2. The last version of "rfififind" ( 31Aug12 ) crashes with a segmentation fault when trying to recompute the mask with "rfifind -nocompute ...". Older version (14Oct11) works well and does not present this bug. I had a look in the code and compared the two versions but there is a big number of changes so I could not find the problem.

3. The code to read the header of a sigproc filterbank has a small bug that can lead to a wrong calculation of the number of bytes of the header and, consequently, also of the samples of the payload. This occurs when the filterbank file has a field in the header with errors or weird values. The new versions of sigproc already fix the problem, and I found the fix:
   -- In "$PRESTO/src/sigproc_fb.c" move line 31 to position 27, i.e.: move the "*nbytes = sizeof(int);" before the 'if' statement.

4. "prepsubband" throws an error "Error in chkfopen(): No such file or directory" at the end of processing if the "-o" option is not the last argument before the input file.

5. "prepsubband -sub" accepts a number of subbands (-nsub) which does not allow an even division of the number of channels of the input filterbank. The processing continues, and I get a WARNING (see attached file). I don't understand how prepsubband creates subbands with uneven number of channels... Wouldn't it be better that "prepsubband" stops with an error if the selected number of subbands does not fit well with the number of channels of the filterbank, like "prepfold" does?

6. In the help text of 'prepfold' there is a small typo at the very bottom: "infile: Input data file name. If the data is not in a regognized ..."

Numbers 3) and 6) are fixed in my fork of PRESTO. Scott, you can pull them if you like.

Cheers,
Pablo

I was trying to test PRESTO, and I ended up with this situation running:
DDplan.py -d 80 -n 96 -b 96.0 -t 0.000072 -f 1400.000000 -s 32 -o DDplan.ps
The result of it was "sh: line 1: 31437 Segmentation fault (core dumped) DDplan.py -d 80 -n 96 -b 96.0 -t 0.000072 -f 1400.000000 -s 32 -o DDplan.ps"

I have installed all the depended libraries that PRESTO need and have run make wisdom, make prep and make command successfully. But an error occurred when I run the python command which is "pip install .". This error message is:
'''
ERROR: Failed building wheel for presto
Failed to build presto
ERROR: Could not build wheels for presto which use PEP 517 and cannot be installed directly
'''

I have searched some solutions on the Internet, but none of them worked with this error. I am using Anaconda virtual environment and python version is 3.6. Does anyone know how to solve it?

When folding an artificial pulsar signal (created by injecting a pulse profile into a noise file), I get an error "MJD out of range" and in the next line "isets = 0". If I have tried changing the period and the MJD of these pulsars slightly in their .par files (since I am making pulsars with abnormally long periods for my project) and failed to correct the error. My pulsar periods (50, 70 , 100 seconds) are a significant amount of the observation time (600 seconds or 10 minutes) which I think may be responsible for the problem. Why does this error come up anyway? I could not find the source of this error in the source code for prepfold as well. Willing to provide any additional information as required.

I'm working on a radio astronomy software distribution based on debian/ubuntu named kern. Packaging up astronomy software should make the life of many astronomers much easier and will save a lot of time and agony all around the world. Since presto is a often used software package we decided to add it to kern. More info can be found here: http://kernsuite.info/. If the package is stable enough it may end up in Debian/ubuntu itself, could be already for the next Ubuntu release.

The presto package already works, but there is a problem with the python files. Ideally the presto python files would be located in a presto python module which would be then installed in the python. This requires a bit of restructuring of the python files.

Would it be an idea to do this? Forcing a user to set up a PYTHONPATH works, but it is not really the right way to do things. If we can restructure the python files in presto it would simplify the setup procedure a bit more, makes it possible to install it in a virtualenv and maybe even trough pypi using pip.

Folding a long scan of a short-orbital period pulsar, I used -searchpdd, then fixed the spin parameters and ran prepfold a final time with -nosearch in order to get TOAs. I've found that when -pdd is included in the prepfold command, get_TOAs throws an error saying the pfd file "...was made allowing prepfold to search," regardless of whether or not -nosearch is included.

In the help dialogue, -nosearch technically only turns of searching for p/pd -- but not pdd?

Trying to run the get_TOAs.py command on my machine gives the following error:

File "/usr/local/bin/get_TOAs.py", line 388
except ValueError, fftfit.error:
^
SyntaxError: invalid syntax

Could I get any guidance on what is to be done to be able to resolve this error?

Hey,

'make' gives an error just after compiling slalib, when trying to run

slalib/sla_test

slalib/sla_test: error while loading shared libraries: libsla.so: cannot open shared object file: No such file or directory

I reckon the problem is libsla.so hasn't been linked-in yet so the command fails.

I added an 'ldconfig' just before this command in the Makefile and this test finishes successfully.

Cheers.

Under certain conditions, mpiprepsubband terminates with a null pointer error on MPI_Bcast.

The MPI_Bcast call is in mpiprepsubband_utils.c line 164. It appears that s->start_subint and s->num_subint are not allocated and initialized for rank 0 under certain conditions, when s->datatype is SIGPROCFB.

Further inspection of the code suggests that in sigproc_fb.c, routine read_filterbank_files(..), s->start_subint and s->num_subint should be allocated and initialized, even if they are not actually used, for example with

    s->start_subint = gen_ivect(s->num_files);
    s->num_subint = gen_ivect(s->num_files);

Hi Scott and all,
I noticed some error in file names when I corrected for orbital motion by specifying the parameters with the -bin switch.
Digging a little bit with gdb and valgrind (--leak_check=full), I found out that every time there is orbital correction enabled, a series of memory leaks is produced. The number of such leaks is variable and valgrind messages are something like

==3007== Invalid read of size 8
==3007==    at 0x4281240: fold (fold.c:629)
==3007==    by 0x8050D28: main (prepfold.c:1298)
==3007==  Address 0x4b379c8 is 0 bytes after a block of size 319,496 alloc'd
==3007==    at 0x402BE00: memalign (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==3007==    by 0x4672B62: fftwf_kernel_malloc (in /usr/lib/i386-linux-gnu/libfftw3f.so.3.3.2)
==3007==    by 0x428A3EB: dorbint (orbint.c:24)
==3007==    by 0x804E26F: main (prepfold.c:824)

I'll investigate this more to look for possible fixes, but in the meantime maybe you guys have an idea of what is happening
Thanks in advance,
Matteo

How can I generate reduced chi2 vs period graph from the python code given in the repository?

I was trying to use prepfold on pulsar J0317+13 using the following commands:
psrcat -e J0317+13 > J0317+13.eph
prepfold -timing J0317+13.eph <etc>
and I get this error
Error: can not convert '+13:29' to RA or DEC in ra_dec_from_string()
: Success

This is a problem for two reasons. The first is that it should be able to parse this (and perhaps give a reasonable estimate for the arcsecond like 30) as this is the value from the latest psrcat version as shown on the website. The second is that it returns a 0 error code when in my mind it fails.

The function is in src/misc_utils on line 694 and I'll let someone more experienced in C to make the function more robust.

Cheers,
Nick

I tried to build the docker image by running

docker build .

in the docker directory, but it fails with

(...)
Step 8/28 : WORKDIR /code/presto/src
 ---> Running in 84532dfd21e5
Removing intermediate container 84532dfd21e5
 ---> 69090d2132a4
Step 9/28 : RUN make libpresto slalib
 ---> Running in 990c209e6eba
make: *** No rule to make target 'libpresto'.  Stop.

Hi @scottransom,
l'm installing presto on Centos. When l do "python tests/test_presto_python.py" , there is a problem:

python tests/test_presto_python.py
Testing FFT stuff... success
Testing FFTW call... success
Testing tablesixstepfft call... success
Testing reading infiles... success
Testing writing infiles... success
Testing allocation and freeing of memory... success
Testing psrparams and orbitparams stuff... success
Testing spectralpower and spectralphase... success
Testing vector shifting / rotation... success
Testing orbit integration stuff... success
Testing Fourier response generation... success
Testing angle functions... success
Testing get_baryv (barycenter)...
Error in chkfopen(): No such file or directory
path = 'resid2.tmp'

What's more, when l followed the PRESTO_search_tutorial, l met the similar problem when doing:

prepfold -p 1.0 GBT_Lband_PSR.fil
Error in chkfopen(): No such file or directory
path = 'resid2.tmp'

Running this command

prepsubband -sub -subdm 5.75 -nsub 32 -downsamp 1 -o Sband ../GBT_Lband_PSR.fil

There's an error with chkopen,
the output is the following:

          Pulsar Subband De-dispersion Routine
                 by Scott M. Ransom

Assuming the data are SIGPROC filterbank format...
Reading SIGPROC filterbank data from 1 file:
  '../TestData1/GBT_Lband_PSR.fil'

    Number of files = 1
       Num of polns = 2 (summed)
  Center freq (MHz) = 1400
    Num of channels = 96
    Sample time (s) = 7.2e-05
     Spectra/subint = 2400
   Total points (N) = 531000
     Total time (s) = 38.232
     Clipping sigma = 6.000
   Invert the band? = False
          Byteswap? = False
     Remove zeroDM? = False

File  Start Spec   Samples     Padding        Start MJD
----  ----------  ----------  ----------  --------------------
1              0      531000           0  53010.48482638889254


Warning:  You cannot (currently) barycenter subbands.
          Setting the '-nobary' flag automatically.
Writing subbands to:
   'Sband_DM5.75.sub00'
   'Sband_DM5.75.sub01'
   'Sband_DM5.75.sub02'
   'Sband_DM5.75.sub03'
   'Sband_DM5.75.sub04'
   'Sband_DM5.75.sub05'
   'Sband_DM5.75.sub06'
   'Sband_DM5.75.sub07'
   'Sband_DM5.75.sub08'
   'Sband_DM5.75.sub09'
   'Sband_DM5.75.sub10'
   'Sband_DM5.75.sub11'
   'Sband_DM5.75.sub12'
   'Sband_DM5.75.sub13'
   'Sband_DM5.75.sub14'
   'Sband_DM5.75.sub15'
   'Sband_DM5.75.sub16'
   'Sband_DM5.75.sub17'
   'Sband_DM5.75.sub18'
   'Sband_DM5.75.sub19'
   'Sband_DM5.75.sub20'
   'Sband_DM5.75.sub21'
   'Sband_DM5.75.sub22'
   'Sband_DM5.75.sub23'
   'Sband_DM5.75.sub24'
   'Sband_DM5.75.sub25'
   'Sband_DM5.75.sub26'
   'Sband_DM5.75.sub27'
   'Sband_DM5.75.sub28'
   'Sband_DM5.75.sub29'
   'Sband_DM5.75.sub30'
   'Sband_DM5.75.sub31'
Setting a 'good' output length of 537600 samples

De-dispersing using:
       Subbands = 32
     Average DM = 5.75

Amount complete =  97%
Error in chkfopen(): No such file or directory
   path = 'Sband_DM5.75.dat'

should I omit this?, since running the command that uses the generated subband files works properly, I'm talking about this one:

prepsubband -nsub 32 -lodm 0.000000 -dmstep 0.500000 -numdms 24 -numout 531000 -downsamp 1 -o Sband Sband_DM5.75.sub[0-9]*

parfile.py has this line
from slalib import sla_ecleq, sla_eqecl, sla_eqgal

This does not work for me. I have to change it to
from pyslalib.slalib import sla_ecleq, sla_eqecl, sla_eqgal

I think that is because I installed it from github:scottransom/pyslalib, where the package name is indeed pyslalib.

Also pyslalib isn't mentioned in the INSTALL file.

I wonder if it would be best just to pull pyslalib into PRESTO since slalib is already included?
Then it would be sure to be installed and the interface would be consistent.

Hi, Scott. When I run the "make makewisdom" commond according to the INSTALL file. An error "Attempting to read the system wisdom file... failed." appears. I wonder why and what is the variable "fftwf_import_system_wisdom"? I have set the FFTW path correctly.

I've noticed a bug when using prepfold on binary pulsars with psrcat and long dwell times. If I use the -psr option or give it a ephemeris file using psrcat -e. for example:
J2145-0750_default.eph.txt
then the resulting plot from prepfold does not time the pulsar correctly:

After some fiddling, I realised that if I remove the last line from the ephemeris as can be seen in the following file:
J2145-0750_with_no_units.eph.txt
Then the pulsar is timed correctly:

So prepfold doesn't seem to like the line "UNITS TDB "

We have noticed this on several different pulsar binaries.

Thank you for your time,
Nick Swainston

I tried your latest presto yesterday,but the problem still exists. my command is :python GBT_Lband_RSR.fil pipeline.py. Here are the error:
Data points written： 525600
Error in chkfopen(): No such file or directory，

Ran into a problem when running single_pulse_search.py on a .singlepulse file- I get a Python error on line 544:
AttributeError: 'module' object has no attribute 'stats'
I was able to fix it by just importing scipy.stats in addition to scipy at the beginning of the file, after which single_pulse_search.py runs as normal.

I have been running into this issue when running my pipeline.

  File "/mypath/presto/bin//DDplan.py", line 4, in <module>
    from builtins import zip
ImportError: No module named builtins

It builts with no errors and all test run correctly.

Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (3.3.4)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (1.19.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (1.5.4)
Requirement already satisfied: six in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (1.15.0)
Requirement already satisfied: astropy in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (4.1)
Requirement already satisfied: pyslalib in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (1.0.4)
Requirement already satisfied: future in /usr/local/lib/python3.6/site-packages (from presto==3.0.1) (0.18.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/site-packages (from matplotlib->presto==3.0.1) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/site-packages (from matplotlib->presto==3.0.1) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.6/site-packages (from matplotlib->presto==3.0.1) (2.4.7)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/site-packages (from matplotlib->presto==3.0.1) (2.8.1)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.6/site-packages (from matplotlib->presto==3.0.1) (8.2.0)
Building wheels for collected packages: presto
  Building wheel for presto (PEP 517): started
  Building wheel for presto (PEP 517): finished with status 'done'
  Created wheel for presto: filename=presto-3.0.1-cp36-cp36m-linux_x86_64.whl size=842979 sha256=a696a36afe22e095781148827d4f4875e1936fdee1b98acb92f98a59bf01dc54
  Stored in directory: /tmp/pip-ephem-wheel-cache-6_b368h1/wheels/62/75/87/478832a478ff1dabfc5c1f77378ef16e57f43300f40cd28d34
Successfully built presto
Installing collected packages: presto
  Attempting uninstall: presto
    Found existing installation: presto 3.0.1
    Uninstalling presto-3.0.1:
      Successfully uninstalled presto-3.0.1
Successfully installed presto-3.0.1
Requirement already satisfied: mpi4py in /usr/local/lib/python3.6/site-packages (3.0.3)
Testing FFT stuff... success
Testing FFTW call... success
Testing tablesixstepfft call... success
Testing reading infiles... success
Testing writing infiles... success
Testing allocation and freeing of memory... success
Testing psrparams and orbitparams stuff... success
Testing spectralpower and spectralphase... success
Testing vector shifting / rotation... success
Testing orbit integration stuff... success
Testing Fourier response generation... success
Testing angle functions... success
Testing get_baryv (barycenter)... success
Testing simple folding code... success

I know it has something to do with python version being incompatible or most likely pip version, but I can't figure out exactly what is causing it .

I tried building and running it with Python 3.8.5 and 3.6.13. I also tried installing python3.version -m pip install future but couldn't fix it.

Any idea what I'm doing wrong?