jdhenshaw / scousepy

ScousePy

About

Multi-component spectral line decomposition with scousepy. scousepy is a package for the analysis of spectral line data. For a comprehensive description of the algorithm and functionality please head to scousepy.readthedocs.io.

Note: scousepy has undergone some major updates in the latest release, namely:

• Workflow update -- scousepy now uses config files for set up. These control basic parameters for use throughout the workflow
• GUI for S1 - basic functionality is the same. Can also be run without using the config files
• GUI for S2 - added derivative spectroscopy for providing initial guesses
• Former S4 now merged into S3
• Former S5 and S6 now merged
• GUI for S3 - adaptive fit checker and fitting functionality

I am in the process of updating the documentation and tutorials. If you need assistance on running the new version of scousepy, please get in touch. For now I have included a simple example script in the tutorials directory.

Basic description

scousepy includes tools for the decomposition of both data cubes, individual spectra, and lists of specta.

Cube Fitting

Cube fitting with scousepy is divided into 4 main stages:

1. Defining the coverage. Here the use informs scousepy where to fit. scousepy will compute basic noise and moments, allowing the user to define a mask for fitting. Once defined, scousepy creates a grid of macropixels with a user defined size and extracts a spatially averaged spectrum from each. This can be run using the GUI or automatically using the configuration files.

2. Fitting the macropixels. scousepy uses a technique referred to as derivative spectroscopy to identify the number of components and their key properties. Fitting is performed via an interactive GUI.

3. Automated fitting. scousepy uses the best-fitting solutions from the macropixels defined in stage 2 as initial guesses for an automated fitting process that is controlled via user-defined tolerance levels.

4. Quality assessment. Here scousepy provides a GUI for quality assessment allowing the user to visually inspect their decomposition.

Single Spectra and lists of spectra

scousepy includes functionality for fitting individual or lists of spectra using the derivative spectroscopy technique. Further information and tutorials can be found at scousepy.readthedocs.io.

Installing scousepy

Requirements

scousepy requires the following packages:

• Python 3.x
• numpy
• matplotlib
• astropy
• lmfit
• tqdm
• pyspeckit >=0.1.21.dev2682
• spectral_cube >=0.4.4.dev1809

Please ensure that you are using the latest developer versions of both pyspeckit and spectral-cube (Github installation).

Note that for interactive fitting with pyspeckit you may need to customise your matplotlib configuration. Namely, if you're using scousepy on a Mac you will most likely need to change your backend from 'macosx' to 'Qt5Agg' (or equiv.). You can find some information about how to do this here

Installation

To install the latest version of scousepy, you can type:

git clone https://github.com/jdhenshaw/scousepy
cd scousepy
python setup.py install

You may need to add the --user option to the last line if you do not have root access.

Reporting issues and getting help

Please help to improve this package by reporting issues via GitHub. Alternatively, you can get in touch here.

Developers

This package was developed by:

• Jonathan Henshaw

Contributors include:

• Adam Ginsburg
• Manuel Riener

Citing scousepy

If you make use of this package in a publication, please consider the following acknowledgements...:

@ARTICLE{Henshaw19,
    author = {{Henshaw}, J.~D. and {Ginsburg}, A. and {Haworth}, T.~J. and
       {Longmore}, S.~N. and {Kruijssen}, J.~M.~D. and {Mills}, E.~A.~C. and
       {Sokolov}, V. and {Walker}, D.~L. and {Barnes}, A.~T. and {Contreras}, Y. and
       {Bally}, J. and {Battersby}, C. and {Beuther}, H. and {Butterfield}, N. and
       {Dale}, J.~E. and {Henning}, T. and {Jackson}, J.~M. and {Kauffmann}, J. and
       {Pillai}, T. and {Ragan}, S. and {Riener}, M. and {Zhang}, Q.},
    title = "{`The Brick' is not a brick: a comprehensive study of the structure and dynamics of the central molecular zone cloud G0.253+0.016}",
    journal = {\mnras},
    archivePrefix = "arXiv",
    eprint = {1902.02793},
    keywords = {turbulence, stars: formation, ISM: clouds, ISM: kinematics and dynamics, ISM: structure, galaxy: centre},
    year = 2019,
    month = may,
    volume = 485,
    pages = {2457-2485},
    doi = {10.1093/mnras/stz471},
    adsurl = {http://adsabs.harvard.edu/abs/2019MNRAS.485.2457H},
    adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@ARTICLE{Henshaw2016,
       author = {{Henshaw}, J.~D. and {Longmore}, S.~N. and {Kruijssen}, J.~M.~D. and {Davies}, B. and {Bally}, J. and {Barnes}, A. and {Battersby}, C. and {Burton}, M. and {Cunningham}, M.~R. and {Dale}, J.~E. and {Ginsburg}, A. and {Immer}, K. and {Jones}, P.~A. and {Kendrew}, S. and {Mills}, E.~A.~C. and {Molinari}, S. and {Moore}, T.~J.~T. and {Ott}, J. and {Pillai}, T. and {Rathborne}, J. and {Schilke}, P. and {Schmiedeke}, A. and {Testi}, L. and {Walker}, D. and {Walsh}, A. and {Zhang}, Q.},
        title = "{Molecular gas kinematics within the central 250 pc of the Milky Way}",
      journal = {\mnras},
     keywords = {stars: formation, ISM: clouds, ISM: kinematics and dynamics, ISM: structure, Galaxy: centre, galaxies: ISM, Astrophysics - Astrophysics of Galaxies},
         year = 2016,
        month = apr,
       volume = {457},
       number = {3},
        pages = {2675-2702},
          doi = {10.1093/mnras/stw121},
archivePrefix = {arXiv},
       eprint = {1601.03732},
 primaryClass = {astro-ph.GA},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2016MNRAS.457.2675H},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Citations courtesy of ADS.

Please also consider acknowledgements to the required packages in your work.

scousepy's Issues

interactive fitter breaks occassionaly with TypeError

Bug report:
Occassionally the interactive fitter in stage_2 breaks, displays a lengthy traceback and brings up a command prompt. After that the fitter window seems to continue to work but does not advances to the next spectrum. Instead "'enter' key acknowledged. Moving to next spectrum or next step..." is printed as often as "enter" is pressed. The fitting itself continues to work, i.e. the keys 1,2,3,f still work.

I have the feeling that this error occurs when pressing a key while scouse is calculating something in the background but I cannot actively reproduce it.
Edit: Now it happened when I definitely did not press any key.
Edit2: And it happened again on exactly the same spectrum as the first time.

The traceback that's printed to the terminal:

=============================================================
If you are happy with this fit, press Enter.  Otherwise, use the 'f' key to re-enter the interactive fitter.
'enter' key acknowledged.  Moving to next spectrum or next step...
Fitting 42 out of 54
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-13-31834ddaffa6> in <module>
      1 s = scouse.stage_2(s,
      2                    verbose     = True,
----> 3                    write_ascii = True
      4                   )

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/scouse.py in stage_2(self, verbose, write_ascii, autosave, bitesize, nspec, training_set)
    461                     bf = fitting(self, SAA, saa_dict, SAAid,
    462                                  training_set=self.training_set,
--> 463                                  init_guess=firstfit)
    464                 SAAid = SAA.index
    465                 firstfit=False

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/stage_2.py in fitting(self, scouseobject, SAA, saa_dict, count, training_set, init_guess, guesses)
    275                                             count, guesses=guesses,
    276                                             training_set=True,
--> 277                                             init_guess=init_guess)
    278
    279         return bf

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/stage_2.py in fitting(self, scouseobject, SAA, saa_dict, count, training_set, init_guess, guesses)
    244             self.spec = spec
    245             self.trainingset_fit(spec, init_guess=init_guess,
--> 246                                        guesses=guesses)
    247
    248             with warnings.catch_warnings():

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/stage_2.py in trainingset_fit(self, spec, init_guess, guesses)
    187                          guesses=guesses,
    188                          fittype=scouseobject.fittype,
--> 189                          use_lmfit=True)
    190             spec.specfit.plot_fit(show_components=True)
    191             spec.specfit.plotresiduals(axis=spec.plotter.axis,

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/config.py in decorator(self, *args, **kwargs)
    143             new_kwargs[arg] = kwargs[arg]
    144
--> 145         f(self, *args, **new_kwargs)
    146
    147     # documentation should be passed on, else sphinx doesn't work and the user can't access the docs

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/spectrum/fitters.py in __call__(self, interactive, usemoments, fittype, clear_all_connections, debug, guesses, parinfo, save, annotate, show_components, use_lmfit, verbose, clear, reset_selection, fit_plotted_area, use_window_limits, vheight, exclude, **kwargs)
    345                               debug=debug, use_lmfit=use_lmfit,
    346                               fittype=fittype,
--> 347                               guesses=guesses, annotate=annotate, **kwargs)
    348             else:
    349                 raise ValueError("Guess and parinfo were somehow invalid.")

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/spectrum/fitters.py in multifit(self, fittype, renormalize, annotate, show_components, verbose, color, guesses, parinfo, reset_fitspec, use_window_limits, use_lmfit, plot, **kwargs)
    810                           show_components=show_components,
    811                           use_window_limits=use_window_limits,
--> 812                           **kwargs)
    813
    814         # Re-organize modelerrs so that any parameters that are tied to others inherit the errors of the params they are tied to

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/spectrum/fitters.py in plot_fit(self, xarr, annotate, show_components, composite_fit_color, lw, composite_lw, pars, offset, use_window_limits, show_hyperfine_components, plotkwargs, **kwargs)
   1155
   1156         if annotate or ((annotate is None) and self.autoannotate):
-> 1157             self.annotate()
   1158             if self.vheight: self.Spectrum.baseline.annotate()
   1159

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/spectrum/fitters.py in annotate(self, loc, labelspacing, markerscale, borderpad, handlelength, handletextpad, fontsize, frameon, chi2, optimal_chi2_kwargs, **kwargs)
   1324         pl = matplotlib.collections.CircleCollection([0],edgecolors=['k'])
   1325         if hasattr(self.fitter,'annotations'):
-> 1326             self._annotation_labels = self.fitter.annotations()
   1327         else:
   1328             raise Exception("Fitter %s has no annotations." % self.fitter)

~/anaconda/envs/python3/lib/python3.6/site-packages/pyspeckit/spectrum/models/model.py in annotations(self, shortvarnames, debug)
    678             else:
    679                 label = ("$%s(%i)$=%8s $\\pm$ %8s" % (varname,varnumber,
--> 680                     Decimal("%g" % value).quantize( Decimal("%0.2g" % (min(np.abs([value,error])))) ),
    681                     Decimal("%g" % error).quantize(Decimal("%0.2g" % (error))),))
    682             label_list.append(label)

TypeError: bad operand type for abs(): 'NoneType'

AttributeError in scousefitter when ncomps>8 is detected

Hello,

I am using scousepy for the Galactic center data which has multiple components (some regions ncomps > 8). I noticed there is an error which recurs when I do the manual fit or when I use the interactive sliders to change the alpha or SNR. The total model, residuals and components are never drawn after this even with manual fits. I am attaching the screenshot below.

The manual fits are saved I think, but the plots don't get updated. The changes in alpha reduces the components to less than 8 but dspec doesn't work once the initial detected components were greater than 8.

I tried to find out what was the issue and tested the code with different datasets. This only happens when ncomps > 8. I did a temporary fix by changing the condition in scousefitter.py (line 1198) from:

if self.dsp.ncomps < 8:
    update_text(self.text_fitinformation,'pyspeckit fit information: ')

to

if self.dsp.ncomps < 15:
    update_text(self.text_fitinformation,'pyspeckit fit information: ')

I don't encounter the issue after this change. Everything works well (sample below).

Also, the manual fitting doesn't work with matplotlib versions > 3.5.3. Just a heads up in case someone encounters the same issue.

instances where AIC is nan

Results in scousepy defaulting to 0 component fits even if a reasonable model is available. Seems to be caused by nan in the spectra - sum of squared residuals therefore == nan as does the aic

ValueError: Too few parameters input. Need at least 6 for ammonia models

Hi
In stage 2, I select the ammonia model to fit the spectra, while in stage 3, this cannot provide the same length parameters to the fitter.
`Traceback (most recent call last):
File "/home/dell/scousepy_model/test_scousepy/test.py", line 15, in
s = scouse.stage_3(config=config_file)
File "/home/dell/.local/lib/python3.10/site-packages/scousepy-2.1.3.dev14+gd5a4b08-py3.10.egg/scousepy/scouse.py", line 779, in stage_3
indivspec_list_completed=autonomous_decomposition(self, indivspec_list)
File "/home/dell/.local/lib/python3.10/site-packages/scousepy-2.1.3.dev14+gd5a4b08-py3.10.egg/scousepy/stage_3.py", line 141, in autonomous_decomposition
results = parallel_map(decomposition_method, inputlist, numcores=scouseobject.njobs)
File "/home/dell/.local/lib/python3.10/site-packages/scousepy-2.1.3.dev14+gd5a4b08-py3.10.egg/scousepy/parallel_map.py", line 167, in parallel_map
return run_tasks(procs, err_q, out_q, numcores)
File "/home/dell/.local/lib/python3.10/site-packages/scousepy-2.1.3.dev14+gd5a4b08-py3.10.egg/scousepy/parallel_map.py", line 94, in run_tasks
raise err_q.get()
ValueError: Too few parameters input. Need at least 6 for ammonia models

I don't know how to solve it. I hope I can get your help.
Thank you

stage directory creation fails in interative run

When running scousepy iteratively as described in the tutorial run_iterative.py, re-running stage_3 on the full resolution dataset fails because dataset/stage_3/ cannot be created:

FileNotFoundError                         Traceback (most recent call last)
<ipython-input-42-8d5793f11b16> in <module>
      2                    [3.0, 1.0, 3.0, 3.0, 0.5],
      3                    njobs   = 4,
----> 4                    verbose = True
      5                   )

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/scouse.py in stage_3(self, tol, njobs, verbose, spatial, clear_cache, autosave)
    536         self.stagedirs.append(s3dir)
    537         # create the stage_3 directory
--> 538         mkdir_s3(self.outputdirectory, s3dir)
    539
    540         starttime = time.time()

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/io.py in mkdir_s3(outputdir, s3dir)
     52     """
     53     if not os.path.exists(s3dir):
---> 54         os.mkdir(s3dir)
     55     else:
     56         # TODO: error handling

FileNotFoundError: [Errno 2] No such file or directory: './dataset/stage_3'

The reason for this error is the missing dataset/ directory that was never created because stage_1 was not run.
Proposed fix: use os.makedirs instead of os.mkdir (the python equivalent of mkdir -p vs. mkdir) to create the parent directory dataset/ if needed.

remove multiple instances of noise calculation

stage_1.py
scousefitter.py

checking fits - modelstore doesn't include spectra that can't be fit

modelstore doesn't include spectra that can't be fit, which leads to problems (indexing mismatch) when checking the fits

Fix problems with negative reduced chi2

This may be a pyspeckit-related issue (e.g., pyspeckit/pyspeckit#258). However, we have yet to find a reproducible case.

installation: astropy_helper fails to install automatically

I just tried to install scousepy (thanks again for inviting me!) but it failed with the following message:

python setup.py install
git submodule command failed unexpectedly:
error: pathspec 'astropy_helpers' did not match any file(s) known to git.
Did you forget to 'git add'?
Downloading 'astropy-helpers'; run setup.py with the --offline option to force offline installation.
No source found for the 'astropy_helpers' package; astropy_helpers must be available and importable as a prerequisite to building or installing this package.

If the problem persists consider installing astropy_helpers manually using pip
(`pip install astropy_helpers`) or by manually downloading the source archive,
extracting it, and installing by running `python setup.py install` from the
root of the extracted source code.

Manually installing astropy_helpers (pip install astropy_helpers) solved the issue.

restarted from stage 5 error: type object 'fg' has no attribute '_endc_'

Beginning stage_6 analysis...

Traceback (most recent call last):
  File "/lustre/aginsbur/sgrb2/2013.1.00269.S/run_scouse_hc3n.py", line 95, in <module>
    s = scouse.stage_6(s, plot_neighbours=True, radius_pix = 2, figsize = [18,10], plot_residuals=True, write_ascii=True, verbose=verb)
  File "/lustre/aginsbur/repos/scousepy/scousepy/scouse.py", line 905, in stage_6
    "completed."+colors.fg._endc_)
AttributeError: type object 'fg' has no attribute '_endc_'

Haven't investigated, but we should do something about this.

AttributeError: 'NoneType' object has no attribute 'ncomps'

In[]: output_ascii_indiv(s, outputdir)

AttributeError Traceback (most recent call last)
Input In [19], in <cell line: 1>()
----> 1 output_ascii_indiv(s, outputdir)

File ~/anaconda3/lib/python3.9/site-packages/scousepy-2.1.3.dev6+ga1b6a6b-py3.9.egg/scousepy/io.py:354, in output_ascii_indiv(self, outputdir)
349 """
350 Outputs an ascii table containing the information for each fit.
351 """
353 saa_dict = self.saa_dict[0]
--> 354 table = make_table(self, saa_dict, indiv=True)
355 table.write(outputdir+'/best_fit_solutions.dat', format='ascii',
356 overwrite=True, delimiter='\t')
357 return

File ~/anaconda3/lib/python3.9/site-packages/scousepy-2.1.3.dev6+ga1b6a6b-py3.9.egg/scousepy/io.py:370, in make_table(self, saa_dict, saa, indiv)
368 solnlist = get_solnlist_saa(self, saa_dict)
369 elif indiv:
--> 370 solnlist = get_solnlist_indiv(self)
372 solnarr = np.asarray(solnlist).T
374 for j in range(len(solnarr[:,0])):

File ~/anaconda3/lib/python3.9/site-packages/scousepy-2.1.3.dev6+ga1b6a6b-py3.9.egg/scousepy/io.py:446, in get_solnlist_indiv(self)
444 spectrum = self.indiv_dict[key]
445 soln = spectrum.model
--> 446 if soln.ncomps==0.0:
447 solution_desc = get_soln_desc(0, soln,
448 spectrum.coordinates[0],
449 spectrum.coordinates[1])
450 solnlist.append(solution_desc)

AttributeError: 'NoneType' object has no attribute 'ncomps'

Hello,
I get the above error when I try to create the output ASCII files.
I get the same error for the statistics commands too (stats=scouse.compute_stats(s)). I am not sure if it is a mistake on my end. The previous commands till stage 4 work well (I am able to extract the spectra and the fits look good). The above error is for the data in the example directory.

Any pointers on how to fix this would be of great help.
Thank you!

Blank coverage.config can be produced, breaks next steps

If the coverage.config file is missing, the following error results:

Traceback (most recent call last):
  File "<ipython-input-4-e12c0a20cd7b>", line 1, in <module>
    s = scouse.stage_1(config=config_file, interactive=True) # GUI based
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scouse.py", line 349, in stage_1
    coverageobject=ScouseCoverage(scouseobject=self,verbose=self.verbose,
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scousecoverage.py", line 49, in __init__
    if (self.x_range[0] is None) or (self.x_range[0] < 0):
AttributeError: 'ScouseCoverage' object has no attribute 'x_range'

The blank coverage.config resulted from a failed run of example_script.py, though I don't understand the origin of the failure.

random nan channels

see #26

So @nicholcunningham has an example where a map with a dud channel in the datacube. The noise calculation doesn't like this very much - the way it currently runs is to remove any spectra where there are nans and then perform the noise computation on a random sample of the remaining pixels. This works for other cubes where, for example, the edge pixels are all nans. However, in this particular example there is a nan located at a specific channel in all pixels. Moreover, @nicholcunningham has another example where there is a dud channel basically at the peak in emission.

I've temporarily skirted around the issue by allowing the user to trim the spectral axis of the cube (see below). However, I didn't really want to do this as ideally you want to compute the noise from as many channels as possible. The previous use of ppv_vol just specified the fitting range (but the noise was computed from the full spectrum). So all I'm doing is effectively cutting out that dud channel. This doesn't work though if the user doesn't specify a velocity range in ppv_vol. Moreover it isn't very helpful if the dud channel is situated at the peak in emission. Any ideas on this one @keflavich?

Should this be something we make scouse do, or is this a case of having to pre-process the data?

lmfit converging to nan values

very occasionally scouse can fail as a result of the output values from lmfit converging to nan

No attribute 'plot_res'

During stage 2, I got the following error when hitting 'next'.

Traceback (most recent call last):
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/matplotlib/cbook/__init__.py", line 287, in process
    func(*args, **kwargs)
  File "/orange/adamginsburg/miniconda3/envs/python39/lib/python3.9/site-packages/matplotlib/widgets.py", line 585, in <lambda>
    return self._observers.connect('changed', lambda val: func(val))
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scousefitter.py", line 721, in update_alpha
    update_plot_model(self,update=True)
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scousefitter.py", line 1160, in update_plot_model
    self.plot_res.remove()
AttributeError: 'ScouseFitter' object has no attribute 'plot_res'

When I restarted, it worked, but this error can apparently happen intermittently.

stage 1: allow manual specification of fit area mask

allow users to specify the masked area to be fit with a mask file or array

Encountered an infinity

Fitting spectra...

divide by zero encountered in log10
Traceback (most recent call last):
  File "/blue/adamginsburg/abulatek/brick/first_results/thioformaldehyde/scousepy_template.py", line 14, in <module>
    s = scouse.stage_3(config=config_file)
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scouse.py", line 772, in stage_3
    indivspec_list_completed=autonomous_decomposition(self, indivspec_list)
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/stage_3.py", line 157, in autonomous_decomposition
    for i, result in enumerate(results):
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/stage_3.py", line 263, in decomposition_method
    Decomposer.fit_spectrum_from_parent(decomposer,guesses,guesses_parent,tol,res,fittype=fittype,)
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/SpectralDecomposer.py", line 152, in fit_spectrum_from_parent
    self.guesses = np.asarray([np.around(guess,decimals=int(rounding[i])) for i, guess in enumerate(guesses)])
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/SpectralDecomposer.py", line 152, in <listcomp>
    self.guesses = np.asarray([np.around(guess,decimals=int(rounding[i])) for i, guess in enumerate(guesses)])
OverflowError: cannot convert float infinity to integer

We should probably try to catch that case

Stop SCOUSE from overwriting everything

• add in checks to make sure people can't accidentally overwrite all of their work

Notice: You might be using enable_deprecations_as_exceptions that is removed in astropy v6

This is an automated issue from Astropy. You might be using code removed in astropy v6.0.dev from astropy/astropy#14670 in the following files:

* scousepy/tests/test_example.py

• scousepy/conftest.py

Please use https://docs.pytest.org/en/stable/how-to/capture-warnings.html instead.

If this issue is opened in error or irrelevant, feel free to close. We apologize for any inconvenience caused.

certain instances where the size of guesses is not a multiple of nparams

Leads to the following pyspeckit error: ValueError: Too few parameters input. Need at least 3 for gaussian models

model retained in memory when SNR or alpha leads to ncomps=0

Display should show a zero component fit and it doesn't - it shows the previous model in memory

Stage 2 manual fitting error (AttributeError: 'ScouseFitter' object has no attribute 'plot_res')

Hello,

I am using scousepy on an ALMA datacube. The spectra has multiple components so I have to use the manual fit in most cases. The manual fit interactive window becomes unresponsive if I go to the previous spectra. The fits are also not shown when I go back. Is going back not allowed during stage 2? As I understand the "refit=True" can be used to look at the plots after stage 2 is complete. Since I have a lot of spectra to fit (>500) I wanted to have the option to check the fitted spectra.

Also once I go to a previous spectrum and try manual fit again, the fit I had done before does not show up (ideally it should show the previous fit parameters and ask if I want to do a re-fit). I am not sure if it was erased.

Example:
I perform the fit manually shown below

I exit the code. I start stage 2 again, but the above fit does not show up when I go back to it. Shown below,

I get the following error in the terminal:

Stage 5 and 6 bug when using refine_grid

The figure in stage 5 closed and in the terminal stage 5 was said to be complete after only reviewing roughly a third of the spectra. A similar problem occurred in stage 6 where I refit 12 of the 45 selected spectra before the figure closed and I was told stage 6 was complete.
I've only ran into this issue when using refine_grid, though I'm not sure if it is coincidence or causation.

output directory not working

Setting the output directory different to the data directory isn't working - seems to default to data directory and then gets confused...

maps are not subscriptable

I get the error:

]Traceback (most recent call last):
  File "<ipython-input-20-e12c0a20cd7b>", line 1, in <module>
    s = scouse.stage_1(config=config_file, interactive=True) # GUI based
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/scouse.py", line 377, in stage_1
    generate_SAAs(self, coverageobject)
  File "/blue/adamginsburg/adamginsburg/repos/scousepy/scousepy/stage_1.py", line 169, in generate_SAAs
    SAA=results[j]
TypeError: 'map' object is not subscriptable

in stage 1. This most likely results from this part:

scousepy/scousepy/stage_1.py

Lines 155 to 169 in 5eb2028

	if scouseobject.njobs > 1:
	# if __name__ == 'scousepy.stage_1':
	# print(__name__)
	results = parallel_map(create_saa, inputlist, numcores=scouseobject.njobs)
	else:
	if scouseobject.verbose:
	results=[create_saa(input) for input in tqdm(inputlist)]
	else:
	results=[create_saa(input) for input in inputlist]
	if scouseobject.verbose:
	progress_bar = print_to_terminal(stage='s1', step='coverageend', length=len(coverage[:,0]),var=w)
	for j in range(len(coverage[:,0])):
	SAA=results[j]

in which parallel_map is used. Most likely, we need to do list(parallel_map...) or restructure the loop on line 168-169 to loop over the entries in the map rather than an index.

allow scousepy to register mouse clicks for fitting

currently uses event.key - expand to allow for mouse clicks as in pyspeckit

stage_6 runs into problems is check_spec_indices contains nans

This can happen when blank pixels are selected during stage 5

Update tutorial data set - crashes on load

Issue is the additional 4th axis in this fits file: https://github.com/jdhenshaw/scousepy_tutorials/blob/master/data/n2h%2B10_37.fits

need to trim this 4th axis and re run if there is an error similar to the one below...

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/opt/anaconda3/lib/python3.8/site-packages/scousepy-1.0.1.dev92+gad6f30b-py3.8.egg/scousepy/scouse.py", line 335, in stage_1
  self.load_cube(fitsfile=fitsfile)
 File "/opt/anaconda3/lib/python3.8/site-packages/scousepy-1.0.1.dev92+gad6f30b-py3.8.egg/scousepy/scouse.py", line 925, in load_cube
  _cube = SpectralCube.read(fitsfile).with_spectral_unit(u.km/u.s,
 File "/opt/anaconda3/lib/python3.8/site-packages/spectral_cube/io/core.py", line 118, in __call__
  return registry.read(BaseSpectralCube, filename, *args, **kwargs)
 File "/opt/anaconda3/lib/python3.8/site-packages/astropy/io/registry.py", line 520, in read
  data = reader(*args, **kwargs)
 File "/opt/anaconda3/lib/python3.8/site-packages/spectral_cube/io/fits.py", line 204, in load_fits_cube
  wcs = WCS(header)
 File "/opt/anaconda3/lib/python3.8/site-packages/astropy/wcs/wcs.py", line 530, in __init__
  self.wcs.set()
astropy.wcs._wcs.SingularMatrixError: ERROR 3 in wcsset() at line 2507 of file cextern/wcslib/C/wcs.c:
Linear transformation matrix is singular.
ERROR 3 in linset() at line 636 of file cextern/wcslib/C/lin.c:
PCi_ja matrix is singular.

finish ppv_vol

Turns out I forgot to implement selecting spatial regions of the data....I should get around to finishing this at some point...

newfile parameter does not follow structure of other path parameters

When re-running a stage with repeat=True and saving to a new file, the newfile parameter behaves different than the other path parameters in that it automatically adds outputdir and filename to the path.

set variables:
outputdir = './'
filename = 'dataset'

Expectation: newfile = outputdir+filename+'/stage_5/s5.repeat.scousepy results in ./dataset/stage_5/s5.repeat.scousepy
actual result: ./dataset./dataset/stage_5/s5.repeat.scousepy

Traceback:

s = scouse.stage_5(s,
    ...:                    blocksize = 6,
    ...:                    figsize   = [18,10],
    ...:                    plot_residuals = True,
    ...:                    verbose   = True,
    ...:                    repeat    = True,
    ...:                    newfile   = filename+'/stage_5/s5.repeat.scousepy'
    ...:                   )
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
|=================================================================================================| 3.4k/3.4k (100.00%)         0s
Showing map number 1: residstd
Checking block 35

Number of pixels examined interactively is now 36.0


A total of 2 spectra have been chosen for inspection
This includes 0 block(s) and 2 individual pixel(s).

scousepy stage 5 completed in: 0.5892878651618958 minutes

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-24-c7aaf917c9de> in <module>
      5                    verbose   = True,
      6                    repeat    = True,
----> 7                    newfile   = filename+'/stage_5/s5.repeat.scousepy'
      8                   )

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/scouse.py in stage_5(self, blocksize, plot_residuals, figsize, verbose, autosave, bitesize, repeat, newfile)
    744             if repeat:
    745                 if newfile is not None:
--> 746                     with open(self.outputdirectory+newfile, 'wb') as fh:
    747                         pickle.dump((self.check_spec_indices, self.check_block_indices, self.blocksize), fh)
    748                 else:

FileNotFoundError: [Errno 2] No such file or directory: './datasetdataset/stage_5/s5.repeat.scousepy'

stage_3 index outf of range on interative runs

When running the run_iterative.py tutorial, stage_3 fails with an index out of range error:

s = scouse.stage_3(s, [3.0, 1.0, 3.0, 3.0, 0.5], njobs   = 6, verbose = True)
Beginning stage_3 analysis...
Initialising spectra: wsaa = 20.0

|>------------------------------------------------------------------------------------------------|   1 /2.0k ( 0.05%) ETA     0s---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-14-d869f71710a5> in <module>
      2                    [3.0, 1.0, 3.0, 3.0, 0.5],       # SNR treshold, min. velocity width, velo. difference SAA - component (~twice), component separation
      3                    njobs   = 6,
----> 4                    verbose = True
      5                   )

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/scouse.py in stage_3(self, tol, njobs, verbose, spatial, clear_cache, autosave)
    549
    550         # Begin by preparing the spectra and adding them to the relevant SAA
--> 551         initialise_indiv_spectra(self, verbose=verbose, njobs=njobs)
    552
    553         key_set = []

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/stage_3.py in initialise_indiv_spectra(scouseobject, verbose, njobs)
     46         if verbose:
     47             for j in ProgressBar(range(len(saa_dict.keys()))):
---> 48                 prep_spec(j, saa_dict, njobs, scouseobject)
     49         else:
     50             for j in range(len(saa_dict.keys())):

~/anaconda/envs/python3/lib/python3.6/site-packages/scousepy-0.0.dev238-py3.6.egg/scousepy/stage_3.py in prep_spec(j, saa_dict, njobs, scouseobject)
     60
     61     # get the relavent SAA
---> 62     SAA = saa_dict[j]
     63     # Initialise indiv spectra
     64     indiv_spectra = {}

KeyError: 0

This is caused by initialise_indiv_spectra looping over range(len(saa_dict.keys())) instead of the actual keys in of .
The prep_spec function cannot access the SAA saa_dict[0] because they start at 642 for some reason.
If the ProgressBar prevents looping over the keys instead of indices, a method to rename the saa keys starting from 0 is needed.

Edit: To clarify: I am running the tutorial code on my own test dataset.

Fit checking not possible when ncomps predicted by dspec = 0

there is an issue during stage 2 where refit=True. There can be a case where dspec predicts a zero component fit, but the user has manually modified the fit to include ncomps>0. Because pyspeckit's specfit class is not initiated this can spit out an error and the model not displayed (we cannot recreate the model using spectrum.specfit.get_model_frompars.