iraf-community / pyraf Goto Github PK

The <skipped> must have been an accidental copy/paste on my part.

This would have never passed under Python 3.6 anyway, so we need to sanitize python ver = a little further under Python 3.

tests/test_describe.py:29: AssertionError
________________________________ test_dumpspecs ________________________________

    @pytest.mark.skipif(not HAS_IRAF, reason='Need IRAF to run')
    def test_dumpspecs():
        # get dumpspecs output
        out_f = six.StringIO()
        wutil.dumpspecs(outstream=out_f, skip_volatiles=True)
        out_str = out_f.getvalue()
        out_f.close()
    
        # modify out_str to remove a path that will always be changing
        out_str = '\n'.join([l for l in out_str.split('\n')
                             if 'python exec =' not in l])
        # modify out_str to handle old versions which printed Tkinter as camel-case
        out_str = out_str.replace('Tkinter', 'tkinter')
    
        # verify it (is version dependent)
        key = ('2' if IS_PY2 else '3', sys.platform.replace('2', ''))
        expected = REF[key]
>       assert expected.strip() == out_str.strip(), \
            'Unexpected output from wutil.dumpspecs: {}'.format(out_str)
E       AssertionError: Unexpected output from wutil.dumpspecs: python ver = 3.6
E         platform = linux
E         PY3K = True
E         c.OF_GRAPHICS = False
E         /dev/console owner = <skipped>
E         tkinter use unattempted.
E         
E       assert 'python ver =... unattempted.' == 'python ver = ... unattempted.'
E         - python ver = 3<skipped>
E         + python ver = 3.6
E           platform = linux
E           PY3K = True
E           c.OF_GRAPHICS = False
E           /dev/console owner = <skipped>
E           tkinter use unattempted.

tests/test_graphics.py:194: AssertionError```

1. I installed iraf from source, directory is ~/Downloads/iraf-2.16.1-2021.06.14
2. I installed pyraf from pip
3. mkiraf and then pyraf
4. I moved iraf to a new directory ~/.iraf-source
5. I deleted the old PATH and export PATH=/home/firestar/.iraf/bin/:$PATH
6. mkiraf and then pyraf, it said no directories found: ~/Downloads/iraf-2.16.1-2021.06.14/unix/hlib/zzsetenv.def

PS: epar fonts are still small

These are the some notes from @jhunkeler from his work on #45. Investigation is needed whether they are real bugs or just the tests need to be re-designed.

When importing pyraf from within pytest, I get the following error on Python 3:

==================================== ERRORS ====================================
_______________________ ERROR collecting debian/tests.py _______________________
tests.py:8: in <module>
    from pyraf import iraf
/usr/lib/python3/dist-packages/pyraf/__init__.py:126: in <module>
    iraf.Init(doprint=0, hush=1)
/usr/lib/python3/dist-packages/pyraf/iraffunctions.py:257: in Init
    userpkg.run(_doprint=0, _hush=hush, _save=1)
/usr/lib/python3/dist-packages/pyraf/iraftask.py:359: in run
    self._run(redirKW, specialKW)
/usr/lib/python3/dist-packages/pyraf/iraftask.py:1667: in _run
    ???
/usr/lib/python3/dist-packages/pyraf/iraftask.py:1470: in _runCode
    ???
<CL script clpackage.user>:28: in login
    Pipe1 = iraf.cl(Stdin=Pipe2, Stdout=1)
/usr/lib/python3/dist-packages/pyraf/iraftask.py:767: in __call__
    return self.run(*args, **kw)
/usr/lib/python3/dist-packages/pyraf/iraftask.py:359: in run
    self._run(redirKW, specialKW)
/usr/lib/python3/dist-packages/pyraf/iraftask.py:1202: in _run
    self._pyFunction(*pl)
/usr/lib/python3/dist-packages/pyraf/iraffunctions.py:2485: in _clProcedure
    clExecute(_sys.stdin.read(), locals=theLocals, Stdin=_sys.__stdin__)
/usr/lib/python3/dist-packages/pyraf/iraffunctions.py:3088: in clExecute
    exec(pycode.vars.proc_name+"(taskObj=iraf.cl)", locals)
<string>:1: in <module>
    ???
<CL script CL1>:12: in string_proc
    ???
/usr/lib/python3/dist-packages/pyraf/iraffunctions.py:2088: in clOscmd
    status = _subproc.subshellRedir(s, shell=shell)
/usr/lib/python3/dist-packages/pyraf/subproc.py:736: in subshellRedir
    return systemRedir((shell, "-c", cmd))
/usr/lib/python3/dist-packages/pyraf/subproc.py:718: in systemRedir
    process.run()
/usr/lib/python3/dist-packages/pyraf/subproc.py:790: in run
    sys.stderr.write(tostr(s))
/usr/lib/python3/dist-packages/stsci/tools/for2to3.py:82: in tostr
    return s.decode(encoding)
E   UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 56: ordinal not in range(128)
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 2.74 seconds ============================

On Python 2, it also does not work (#43) works well.

Hello everyone --

I've just installed the community iraf and pyraf on a Fedora-33 system to be sure that it will work correctly on newer linuces where 32-bit binaries are no longer supported.

One minor detail -- on older systems I have been setting the environment variable PYRAFGRAPHICS to 'matplotlib', which leads to somewhat nicer graphics rendering. This now fails with the message:

/usr/local/lib64/python3.9/site-packages/pyraf-2.1.15-py3.9-linux-x86_64.egg/pyraf/MplCanvasAdapter.py:21:
MatplotlibDeprecationWarning: 
  The 'resize_callback' parameter of __init__() was deprecated in Matplotlib 3.4 and will be removed two minor
  releases later. Use get_tk_widget().bind('<Configure>', ..., True) instead. If any parameter follows
  'resize_callback', they should be passed as keyword, not positionally.
  tkagg.FigureCanvasTkAgg.__init__(self, figure, master,
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
AttributeError: 'MplCanvasAdapter' object has no attribute 'show'

I'm running matplotlib version 3.4.2, with python 3.9.6.

Any ideas? If this is an easy fix, it'd be nice to have.

Thanks -- John

c/c @cdsontag

As reported by @olebole below:

When compiling for Python 3 (3.6), ... the C modules (sscanf and xutil) don't work "out of the box".
Pragmatically, I could resolve that by renaming them in setup.cfg:

-[extension=pyraf.sscanfmodule]
+[extension=pyraf.sscanf]
 sources = src/sscanfmodule.c
 optional = True
 fail_message = If this is Windows, it is ok.

-[extension=pyraf.xutilmodule]
+[extension=pyraf.xutil]
 ...

but I am not sure whether this is the best solution. Any idea?

Can't find the executable:

--> sobsolete
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
IrafError: Cannot find executable for task sobsolete

This seems to happen as the Python interpreter is shutting down, after being started without an existing cache. Apparently things are getting cleaned up in the wrong order or something. It's not a disaster for interactive use, but causes our regression tests to fail without producing any results. I started looking into this but hadn't got to the bottom of the problem and had better document it in the meantime.

> rm -fr pyraf
> pyraf
Created directory /home/jturner/test/pyraf for cache
setting terminal type to xterm...

   NOAO/IRAF PC-IRAF Revision 2.16 EXPORT Thu May 24 15:41:17 MST 2012
      This is the EXPORT version of IRAF V2.16 supporting PC systems.


  Welcome to IRAF.  To list the available commands, type ? or ??.  To get
  detailed information about a command, type `help <command>'.  To run  a
  command  or  load  a  package,  type  its name.   Type  `bye' to exit a
  package, or `logout' to get out  of the CL.    Type `news' to find  out
  what is new in the version of the system you are using.  

  Visit http://iraf.net if you have questions or to report problems.

  The following commands or packages are currently defined:

  (Updated on 2013-12-13)

clpackage/:
 adccdrom/      esowfi/         mem0/           rvsao/          user/
 cfh12k/        finder/         mscdb/          softools/       utilities/
 cirred/        fitsutil/       mscred/         sqiid/          vo/
 clpackage/     gemini/         mtools/         stecf/          xdimsum/
 ctio/          gmisc/          nfextern/       stsdas/         xray/
 cutoutpkg/     guiapps/        noao/           system/
 dataio/        images/         obsolete/       tables/
 dbms/          language/       plot/           ucsclris/
 deitab/        lists/          proto/          upsqiid/
PyRAF 2.2.dev Copyright (c) 2002 AURA
Python 3.6.2 Copyright (c) 2001-2017 Python Software Foundation.
Python/CL command line wrapper
  .help describes executive commands
--> .exit
Exception ignored in: <bound method _CodeCache.__del__ of <pyraf.clcache._CodeCache object at 0x7f8a7f730860>>
Traceback (most recent call last):
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/site-packages/pyraf/clcache.py", line 193, in __del__
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/site-packages/pyraf/clcache.py", line 184, in close
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/shelve.py", line 144, in close
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/shelve.py", line 172, in sync
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/dbm/dumb.py", line 121, in _commit
TypeError: 'NoneType' object is not callable
Exception ignored in: <bound method _Database.close of <dbm.dumb._Database object at 0x7f8a7f7309e8>>
Traceback (most recent call last):
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/dbm/dumb.py", line 279, in close
  File "/home/jturner/anaconda2/envs/test3/lib/python3.6/dbm/dumb.py", line 121, in _commit
TypeError: 'NoneType' object is not callable

I have installed iraf/pyraf with the astroconda python 2.7 conda environment.

Pyraf works well within this environment but I have been searching ways to run pyraf tasks with my own scripts in supported python versions.

I have been succesfull launching pyraf tasks in a python 3.7 using the subprocess library and activating the pyraf environement. I am using ubuntu 20.04 and the code looks something like:

bash_command = "bash -c 'source /home/user/anaconda3/bin/activate && conda activate iraf27 && python RunIraf_task.py task_conf_file.txt /home/user/data/folder/ && conda deactivate'"

p1 = subprocess.run(bash_command, shell=True, capture_output=True)

This works very well for most tasks and it makes possible to keep the pyraf installation isolated. However, I noticed an issue in the identify task: While trying to define a line (m key) the plot window becomes kind of frozen. This does not happen in the fit window (f key) which works as expected.

I guess this is because the task wants to display some information in the terminal or is waiting for some user input. I wonder if anyone has a workaround this issue or if they would suggest some strategy to run pyraf accross python environments.

Thank you for your work

If I use epar ccdproc in imred >> ccdred >> ccdproc, I can put [261:280,1:1032] in biassec, which is in a old iraf code by my teacher:

ccdproc @list.all o//@list.all ccdtype='' overscan+ biassec=[261:280,1:1032]

But if I use the terminal, it will say:

SyntaxError: Too many positional parameters for task ccdproc

And if I put (261:280,1:1032), (261:280;1:1032) or (261:280 1:1032), it will also pop out SyntaxError. It seems that pyraf syntax is slightly different from iraf. What is the right parameter syntax?

'conda install pyraf' installs stsci.tools-4.0.1 as well.
'from pyraf import iraf' , however, yields an error of
"ModuleNotFoundError: No module named 'stsci.tools.for2to3'"

Downgrading of stsci.tools to the version 3.6.0 by 'conda install stsci.tools==3.6.0' was needed to execute 'from pyraf import iraf' successfully.

conda version : 4.10.3
python version : 3.7.4.final.0
platform : osx-64 (OSX/10.15.7)

Thanks

Follow up of #45 -- Setup Jenkins nightly cron. We can skip cron if there has been a run on master in the past 24 hours (e.g., from push or merge commit).

First, I want to thank you for your incredible work on revitalizing IRAF. I was able to install IRAF (from Github) on a macOS Monterey machine (run_tests only showed 2 failed and 1 skipped, so remarkably good).

I installed XQuartz to run xterm and verify if most of the tasks I use (noao onedspec/twodspec, etc) work fine (they do but there's s a minor issue with some printed messages in APALL). I also installed X11iraf flawlessly (although I'm not using xgterm right now). Then, I installed the latest PyRAF version from this GitHub repository in order to run a Python code that I modified (not mine; it uses STSDAS, but I was not able to install that external package from their Github [by their words, it is nonfunctional]); imcopy, imarith and the tracing/extraction of the spectra in APALL (a test) work fine in PyRAF.

The only problem that I currently have is that the first frame of the interactive plot in APALL (the plot to define the apertures) doesn't behave correctly: I attach a short video (it is just an example) showing the problem: the first frame only allows certain keys to function: in this case, I used 'l', which worked; after that, I tried 'u', but the "cursor" got stuck at the center of the plot: it may "escape" from that position briefly, but returns almost immediately. The same happens for 'a', 'n', 'm' keys. The interesting part is that 'b' always works, so the plot to fit the background pops out and the fitting of the background ('f') runs OK. I can return to the first frame ('q'), but the problem still persists. 'b', 'f' and 'q' worked again in the second try.

So, I've been reading the Python codes of PyRAF, and I could figure out, for example, that to stop the blinking of the cursor, we can comment out line "hideTkCursor(self)" of the function "activateSWCursor" in "Ptkplot.py" file. About the major problem, I haven't solved it yet although I think I located the files related with that. This is my question: as I'm a macOS newbie, I don't know if someone ever faced this kind of behaviour in an older macOS or linux distributions (I remember I had a similar issue with CentOS and PyRAF many years ago but I don't remember the solution). Any help would be great!

Thanks.

PyRAF is packaged for Python 3 by running 2to3 at build time (confirmed by @cdsontag). This translation produces a clPrint function that does not emulate CL accurately, causing at least one of our Gemini IRAF CL scripts to fail. Specifically, the Python 3 clPrint adds one extra space at the end of the line and one space in between string arguments, which CL does not. This is not a cosmetic issue, because it's quite common to manipulate text data in CL by piping a string from print into commands like scan or translit or redirecting it to a file.

Strictly speaking, the PyRAF source is not at fault here, but 1) that's a bit academic when it results in a broken build and 2) solving the problem here would avoid maintaining patches downstream. I already have a candidate fix for this, but best add this issue with a bit of explanation...

@jhunkeler, @pllim

1. Link to results for master is available at https://ssbjenkins.stsci.edu/job/STScI/job/pyraf/job/master/ (this link is also reachable by clicking the "build status" badge on README).

2. On left menu, click on "Open Blue Ocean".

3. Select row pointing to "master" branch.

4. Click on the Python 2 build as shown

5. To make sure, you will now see a line like conda install -y -q python=2.7 iraf-all six on that page below the diagram.

6. Expand the line with with_env pytest tests --basetemp=tests_output --junitxml results.xml to see Python 2 test results. Example:

[STScI_pyraf_master-52EC45PPOTV4YGTS2XRZCI6UCVSWOPR23BCYCLQME64HJ7JJ445Q] Running shell script

+ with_env pytest tests --basetemp=tests_output --junitxml results.xml

============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/jenkins/workspace/STScI_pyraf_master-xxx, inifile: setup.cfg
collected 76 items

tests/test_basic.py ....                                                 [  5%]
tests/test_clcache.py .                                                  [  6%]
tests/test_cli.py ....                                                   [ 11%]
tests/test_core_nongraphics.py .......x........x...xx..                  [ 43%]
tests/test_describe.py .                                                 [ 44%]
tests/test_graphics.py ..........s                                       [ 59%]
tests/test_invocation.py ........................                        [ 90%]
tests/test_using_tasks.py .......                                        [100%]

 generated xml file: /home/jenkins/workspace/STScI_pyraf_master-xxx/results.xml 
=============== 71 passed, 1 skipped, 4 xfailed in 21.12 seconds ===============

7. To see Python 3 build, click on the bottom-most circle in the diagram. Then, repeat steps similar to Python 2 build above to see its results. For Python 3, it is currently (as of this writing) allowed to fail, so it will show a green status but you will see some failures. Unlike Travis, Jenkins cannot set individual build to red without failing the whole thing, even if that individual build is an allowed failure.

c/c @cdsontag

PY in the sky. Har har har...

Hi, could you please provide another link for the release notes and tutorials etc. ? because the original website (http://www.stsci.edu/resources/software_hardware/pyraf) is now under construction. That would be very helpful!

As requested in #51 (comment) but after #53 is merged.

• GitHub
• Astroconda
• PyPI

After the recent work to get all our Gemini IRAF tests running, I've had a "quick" look at the differences in results compared with PyRAF 2.1.15 on Python 2.7, suspecting that most of the 189/932 failures may be caused by just a couple of small issues, which does indeed seem to be the case.

The first thing is that iraffunctions.nint() follows the new behaviour of round() in Python 3, rounding half-integers to the nearest even number instead of up. So eg. CL and PyRAF on Python 2 both give nint(2.5) == 3, whereas PyRAF on Python 3 gives 2.

The other, less well-documented thing is that Python 3 float.__str__() produces 17-digit precision (like repr), rather than 12 digits in Python 2. Part of the issue there is just our tests being overly sensitive/brittle. However, it can lead to some odd consequences, such as comments getting truncated where there's less space or imexpr switching to double precision output when given extra digits. It can also produce somewhat larger differences where the outcome of pixel rejection differs here and there, as well as uglier user output in some cases.

So which is the "correct" behaviour? I think the baseline intention here is to emulate CL. For example, division has already been made to follow the CL & Python 2 behaviour on Python 3: 1/2 == 0. I think nint() is analogous to that and a lot of CL code has been written assuming that it always rounds upwards. At a glance, I can see some cases where breaking that assumption causes arcane bugs, eg. a large imaging WCS error due to the WCS getting copied from the wrong input FITS extension, because the reference extension number is now getting calculated incorrectly. "Fixing" nint() is a 2-line patch that I think would be worth including, for compatibility with CL (including the ability to write scripts that work with both cl and pyraf) and to avoid unnecessary work.

The argument for trying to emulate the precision of CL's real() is less obvious though. First, it's a bit of an uphill struggle to override a built-in type, with Python's native precision likely to crop up wherever any function fails to override it. Second, it seems to be coincidental that Python 2's float() behaves so much like IRAF's real(), printing 12 significant digits, switching to scientific notation at the same threshold etc. I do think IRAF has occasionally influenced Python and it may be that their considerations were similar here, but I don't see any evidence of PyRAF deliberately taking measures to reproduce the format shown by CL on Python 2; in fact, I think I found some corner case where it differs slightly, when looking into it. Third, an increase in precision should be numerically harmless. Fourth, there are other factors leading to (larger) floating-point differences anyway, such as what C library version IRAF was linked against. Fifth, CL scripts could explicitly format numeric output if they really care. So maybe it's better to accept this one and edit our tests and code to cope with it; I still have to finish looking into that.

Having said all of the above(!), since I had 189 failures to characterize, I actually did have a quick go at hacking PyRAF to reproduce Python 2's float.__str__() behaviour, successfully. Combining those changes with the nint() fix and a few more fixes in our tests (and disregarding 14 tests that still fail for reasons unrelated to Python 3), I was able to prove that all of the remaining failures are due to those language changes in Python 3, which I think is a useful point of reference and means that we're close to being able to say that Gemini IRAF works on Python 3.

Originated from a request by @olebole in #36 (comment)

p.s. Some of the tests in pandokia might require IRAF. I don't think those are going to be ported over.

p.p.s. This is more painful than it sounds. Given the resources we have, this might actually never happen...

Can't load this one due to another package not existing.

--> song
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
  File "<CL script clpackage.song>", line 27, in song
    iraf.rvx()
AttributeError: Undefined IRAF task `rvx'

User has HiDPI display, with 4K resolution, and has problems with PyRAF epar GUIs, with labels being too small. According to @cdsontag , there does not appear to be a way for user to configure font size.

As reported by @sosey , using the following quote (the acute character), which was from a copy-and-paste of instructions for an IRAF package:

’

which would be used to set a param like this:

dimension_in = ’+100,0,0,0’

gets translated to this:

dimension_in = '\xe2\x80\x99+100,0,0,0\xe2\x80\x99'

That is:

--> str('’')
'\xe2\x80\x99'

where PyRAF assumes the input is a string and adds the hex translation for the acute into the string during a strip() which is supplied and barfs later when there's string inspection.

This is unfortunate for users following documentation, but probably doesn't happen often.

We have gotten approval to do this (finally). So we need to work out the logistics of this. On our end I think Joe Hunkeler can handle it on our side, but I believe he would also need access to make changes in this repository. Let me know what needs to be done.

Trying to load this subpackage throws an IRAF error on my OSX and Linux boxes:
IrafError: Cannot find executable for task xobsolete

The online doc for the xray package seems to indicate that this package is a good candidate for removal, as it will "soon be removed from the system."

Setting parameters like the following works for , for example, apphot.
[IN]
iraf.apphot.datapars.itime = 10.
print(iraf.apphot.datapars.itime)
[OUT]
10.

It also works for immatch.geotran, however, it doesn't for immatch.wcsmap.
[IN]
iraf.immatch.wcsmap.fitgeom = 'rxyscale'
print(iraf.immatch.wcsmap.fitgeom)
[OUT]
general

Do I miss something?

Using
pyraf 2.1.15
iraf-community 2.16.1+2021.06.14

Thanks.

Some of our tests are failing on Python 3 with errors like this:

CL syntax error at `=' (file "/rtfproc/rtftests/gemini_iraf/gmosspec2/testgsreduction/test6/testgsreduction_test6.cl", line 111)

The problem can be reduced to the following script:

from pyraf import clscan

cltext = '''
char cenwavvalue[4]
cenwavvalue[1] = "580"
'''

scanner = clscan.CLScanner()
tokens = scanner.tokenize(cltext)

print(tokens)

With PyRAF 2.1.15 on Python 2.7.16 (where the error does not occur), this produces the following output, showing how the CL script is supposed to get tokenized:

[char, cenwavvalue, [, 4, ], NEWLINE, cenwavvalue, [, 1, ], =, '580', NEWLINE]

But on Python 3.7.9 (with either the same PyRAF 2.1.15 or a recent checkout of 2.2.1dev3), it produces the following, where the array subscript on the second line is treated as a single string, '[1]', rather than being split into opening & closing brackets and an integer:

[char, cenwavvalue, [, 4, ], NEWLINE, cenwavvalue, '[1]', =, '580', NEWLINE]

PyRAF matches different types of tokens using a regex for each type, defined in clscan.py. These get concatenated together into a long regex for a given "scanner" -- in this case _CommandScanner -- which is used to determine which of the patterns matches the next character(s). After quite a bit of digging, it seems that the problem is a change in the order of the individual regexes for _CommandScanner. Specifically, the t_string regex now appears before t_lbracket instead of afterwards, causing [1] to get matched as a string instead of matching the opening bracket.

The order in which the component regexes get concatenated (and hence their precedence) is determined by _namelist in generic.py. This iterates over the methods in the scanner class __dict__.keys(), which behaves differently in Python 2 & 3. In Python 2, it's just an ordinary dict, so I think its order is undefined but (fortunately) appears to be repeatable... In Python >=3.6, it's a mappingproxy object that preserves the order in which the methods are defined in the code. So t_string has moved before t_lbracket just because that's how they are written in the (parent) class _CommandScanner_1.

The error can therefore be solved simply by reversing the order of those 2 methods in in _CommandScanner_1, putting t_lbracket first. I confirm that this makes the error disappear and my test now runs to completion on Python 3 (with some small numerical differences that will be unrelated).

I would appreciate any input from @cdsontag here, because although I'm confident that this fixes the problem I was troubleshooting, I'm not sure what the consequences of reordering these regexes might be more generally. I think there can't be many other cases where the parsing is messed up in this way, since most of our tests are passing and I'm not spotting other errors that look similar but not identical. I suppose we could study the regex ordering that Python 2 actually produces and try to reproduce that, which should be safe, but I'm not sure whether it's possible to reproduce exactly, given that inheritance will force grouping of the methods within the __dict__ in Python 3. Thanks.

For example, see https://ssb.stsci.edu/pandokia/pandokia.cgi?query=summary&qid=3443

It is known that PyRAF will not support Python 3. So the question is:

• Do we keep the tests in a special Python 2 environment when everything else moves to Python 3? Or...
• Do we stop testing PyRAF altogether? Or...
• Do we have someone who can fix them?

The dither package wants pydrizzle, which it can't seem to find...

--> dither

      +------------------------------------------------------------+
      |           DITHER Version 2.3 (13 Nov 2009)                 |
      |                                                            |
      |  Deprecated tasks MultiDrizzle, PyDrizzle, xytosky,        |
      |  and tweakshifts have been removed from this package.      |
      |  Ureka 1.5.1 contains these deprecated tasks; found at     |
      |      http://ssb.stsci.edu/ureka/1.5.1/                     |
      |  The DrizzlePac Python package replaces those tasks.       |
      |  Use 'import drizzlepac'                                   |
      |  to load the new tasks under Python or pyraf.              |
      |  See http://drizzlepac.stsci.edu for details.              |
      |  No changes have been made to any IRAF-based tasks.        |
      +------------------------------------------------------------+
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
  File "<CL script analysis.dither>", line 64, in dither
    iraf.pyexecute('dither$tran_iraf.py', tasknames = 'tran', PkgName=PkgName,PkgBinary=PkgBinary)
  File "/Users/ely/anaconda/envs/iraf/variants/common//iraf/stsci_iraf//stsdas/pkg/analysis/dither/tran_iraf.py", line 9, in <module>
    import tran
  File "/Users/ely/anaconda/envs/iraf/variants/common/iraf/stsci_iraf/stsdas/python/tran.py", line 83, in <module>
    import pydrizzle
ImportError: No module named pydrizzle

dither can be loaded if you first import drizzlepac. But If this doesn't work or shouldn't be used anymore, It should probably be removed from future distributions.

Another one that can't startup on OSX or Linux

--> mscdb
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
IrafError: Cannot find executable for task mscdb

Update: this was the first of several related problems discussed here; see below.

It turns out that at least one of our CL scripts has a special character (non-breaking space) in a comment for some reason, which PyRAF happily treats as part of the comment under Python 2, while Python 3 gives the following error when trying to read the script (before PyRAF even gets as far as parsing it):

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 7931: invalid start byte

This can be reproduced trivially with a 2-line Python script:

fh = open('/home/jturner/repos/gemini/gmos/mostools/gmskcreate.cl')
text = fh.read()

I wonder whether utf-8 is actually the best assumption for CL scripts? This one is probably latin-1? Anyway, I'll remove the offending character from our script, but I just thought this might be worth mentioning in case other people report seeing the above error (especially when just running lpar) and I thought I'd put it in an issue so they can find it in a Web search. Feel free to close this without doing anything if you don't think there's a better default encoding specification.

The following snippet does not work with Python 2:

from io import StringIO
from pyraf import iraf

out = StringIO()
iraf.imhead('dev$pix', Stdout=out)
print out.getValue()

I get the following exception:

/usr/lib/python2.7/dist-packages/pyraf/iraftask.pyc in __call__(self, *args, **kw)
    765 
    766     def __call__(self,*args,**kw):
--> 767         return self.run(*args, **kw)
    768 
    769     def __repr__(self):

/usr/lib/python2.7/dist-packages/pyraf/iraftask.pyc in run(self, *args, **kw)
    357             if executionMonitor:
    358                 executionMonitor(self)
--> 359             self._run(redirKW, specialKW)
    360             self._updateParList(save)
    361             if Verbose>1:

/usr/lib/python2.7/dist-packages/pyraf/iraftask.pyc in _run(self, redirKW, specialKW)
    806         """
    807         try:
--> 808             irafexecute.IrafExecute(self, pyraf.iraf.getVarDict(), **redirKW)
    809         except irafexecute.IrafProcessError, value:
    810             raise IrafError("Error running IRAF task "+self._name+

/usr/lib/python2.7/dist-packages/pyraf/irafexecute.pyc in IrafExecute(task, envdict, stdin, stdout, stderr, stdgraph)
    337             gki.kernel.pushStdio(None,None,None)
    338         try:
--> 339             irafprocess.run(task, pstdin=stdin, pstdout=stdout, pstderr=stderr)
    340         finally:
    341             if stdgraph:

/usr/lib/python2.7/dist-packages/pyraf/irafexecute.pyc in run(self, task, pstdin, pstdout, pstderr)
    506         try:
    507             # begin slave mode
--> 508             self.slave()
    509         finally:
    510             self.running = 0

/usr/lib/python2.7/dist-packages/pyraf/irafexecute.pyc in slave(self)
    643                 xfer()
    644             elif msg5 == 'xmit(':
--> 645                 xmit()
    646             elif msg[:4] == 'bye\n':
    647                 return

/usr/lib/python2.7/dist-packages/pyraf/irafexecute.pyc in xmit(self)
    818                     return
    819 
--> 820             self.stdout.write(txdata)
    821             self.stdout.flush()
    822         elif chan == 5:

TypeError: unicode argument expected, got 'str'

With Python 3, this works nicely (after changing the print statement into a function).

Hello,

I was trying to create a quick notebook on processing some data that I could share with a colleague online via mybinder.org. However, I encountered the issue that pyraf wouldn't recognize the iraf installation. It comes down to setting the environment variables iraf and irafarch. I have opened this issue yesterday on the mybinder github site, where you can check the brief discussion we had:
jupyterhub/binderhub#575
They recommended that I get in touch with you to resolve this issue. The main suggestion was to use the os python module to set the environment variables at the first cell. I did that and unfortunately they still weren't set. I would appreciate if you have any thoughts/suggestions?

Many thanks,
Iskren

I have encountered this in tests that copy FITS files (using 8ea6169 under Python 3.7), but the following reproduces it without requiring any additional data:

>>> from pyraf import iraf
>>> iraf.copy('dev$pix.imh', './')

PANIC in `/rtfproc/anaconda3_2019.10/envs/pyraf3test/iraf/bin.linux/x_system.e': Out of memory
Killing IRAF task `copy'
Traceback (most recent call last):
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 880, in _run
    irafexecute.IrafExecute(self, iraf.getVarDict(), **redirKW)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 371, in IrafExecute
    raise exc
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 348, in IrafExecute
    irafprocess.run(task, pstdin=stdin, pstdout=stdout, pstderr=stderr)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 512, in run
    self.slave()
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 636, in slave
    self.msg = self.readString()
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 578, in readString
    return Iraf2AscString(self.read())
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/irafexecute.py", line 607, in read
    str(header[0:2]))
pyraf.irafexecute.IrafProcessError: Not a legal IRAF pipe record: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/rtfproc/rtftests/gemini_iraf/gmosspec2/iraf/test/test3.py", line 8, in <module>
    iraf.copy('dev$pix.imh', './')
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 836, in __call__
    return self.run(*args, **kw)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 375, in run
    self._run(redirKW, specialKW)
  File "/rtfproc/anaconda3_2019.10/envs/pyraf3test/lib/python3.7/site-packages/pyraf/iraftask.py", line 883, in _run
    str(value))
stsci.tools.irafglobals.IrafError: Error running IRAF task copy
Not a legal IRAF pipe record: b''
>>>

Hello @cdsontag, @stsci-ssb-ci (or @plim?),

since I am getting done with the Python 3 transition, I would also like to create a new version and push it to PyPI soon. However, the pyraf project there is still maintained by you (cdsontag, stsci-ssb). Could I ask to add me to the maintainer's list there so that I can upload new versions without bugging you?

Thank you very much!

Ole

More missing executables

--> xlocal
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
IrafError: Cannot find executable for task xlocal

This came up in a conversation, so just documenting it in case there is real interest.

As STScI winds down its support in the future, it would be more useful for this package to be in its own organization so that people or other institution(s) out there who still rely on this package may ramp up their support of it.

There is already a precedence with https://github.com/airspeed-velocity/asv .

Appeared today on pembry dev:
https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=26398
https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=26259

c/c @cdsontag , @jhunkeler

Sometimes resizing pyraf windows leads to the bug: the more you pull, the more it gets smaller, and it is not under your control.

pyraf 2.2.0

trying to load the kepler package hits an error as it's trying to load pyfits

--> kepler
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
    iraf.kepler(_save=1)
  File "<CL script clpackage.kepler>", line 33, in kepler
    iraf.pyexecute('kepler$kepbls.py', verbose = no, PkgName=PkgName,PkgBinary=PkgBinary)
  File "/Users/ely/anaconda/envs/iraf/variants/common//iraf/kepler/kepbls.py", line 1, in <module>
    import numpy, scipy, sys, time, pyfits, pylab, math, re
ImportError: No module named pyfits

I'm seeing quite a lot of Gemini IRAF tests failing with a message such as KeyError: "Error in parameter 'centerpars.cthreshold' for task phot\n'Key centerpars.cthreshold not found'", using a recent checkout of iraf-community PyRAF for Python 3. All but one are cases where the failing IRAF task gets passed parameters from an "additional" pset (can't recall whether that's the proper IRAF terminology) in the form centerpars.cthreshold=0..

For example, I see the error quoted above when doing the following with pyraf 2.2.1.dev3 and stsci.tools 4.0.1 on Python 3.7.9:

digiphot
apphot

phot dev$pix coords="" output="default" interactive- centerpars.calgorithm="none" centerpars.cthreshold=0.

This only happens some of the time, depending on the exact parameters used, due to the nature of the bug. With earlier PyRAF releases, the task instead continues to a prompt like the following in this particular case (which seems like dubious behaviour in itself, but that's a different problem!):

Centering algorithm (none) (CR or value): 

After a bit of troubleshooting, this seems to be a regression introduced by no longer running 2to3 on the source code (making it hard to spot, since there's no change to the offending line in the revision history!). Because kw.keys() no longer gets converted to a list by 2to3, the following block of code is now modifying a dict in place while iterating over it (rather than over an existing copy of the keys), leading to unpredictable behaviour:

I think the solution is just to put L912 back to this (or use copy or something):

        for key in list(kw.keys()):

Thanks!

James.

I want to run pyraf non-interactively; f.e. as a test during an automated build. But even trying to import it does not work. The follwing minimal example:

from stsci.tools import capable
capable.OF_GRAPHICS = False
from pyraf import iraf

When running directly from the command line, it works. However, when I redirect the input, it does not:

$ python foo.py </dev/null
stty: 'standard input': Inappropriate ioctl for device
i: Traceback (most recent call last):
  File "foo.py", line 3, in <module>
    from pyraf import iraf
  File "/usr/lib/python2.7/dist-packages/pyraf/__init__.py", line 126, in <module>
    iraf.Init(doprint=0, hush=1)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraffunctions.py", line 257, in Init
    userpkg.run(_doprint=0, _hush=hush, _save=1)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 359, in run
    self._run(redirKW, specialKW)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 1667, in _run
    self._runCode()
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 1470, in _runCode
    self._clFunction(*parList, **kw)
  File "<CL script clpackage.user>", line 32, in login
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 1220, in __getattr__
    return self._taskObj.getParam(paramname,native=1,mode="h",exact=1)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 564, in getParam
    exact=exact, prompt=prompt)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 616, in _getParValue
    field, native, mode="h", prompt=prompt)
  File "/usr/lib/python2.7/dist-packages/pyraf/iraftask.py", line 905, in _getParFromDict
    native=native,mode=pmode,prompt=prompt)
  File "/usr/lib/python2.7/dist-packages/stsci/tools/basicpar.py", line 367, in get
    if prompt: self._optionalPrompt(mode)
  File "/usr/lib/python2.7/dist-packages/stsci/tools/basicpar.py", line 643, in _optionalPrompt
    self.getWithPrompt()
  File "/usr/lib/python2.7/dist-packages/stsci/tools/basicpar.py", line 349, in getWithPrompt
    raise EOFError("EOF on parameter prompt")
EOFError: EOF on parameter prompt

This happens on Python 2.7 as well as on Python 3.6.

I just tagged a new release candidate, 2.2.0rc1, and uploaded it to PyPI:

• Release info on Github
• PyPI pyge

@pllim @cdsontag @jehturner @jhunkeler if you have a few moments to have a look there and check that the info shown there is correct. If you like, you can also try out the package 😁

Anyone else is warmly invited as well...

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - pyraf -> python=2.7

Your python: python=3.9

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.33=0
  - python=3.9 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.33

When I install pyRAF by conda install -c pkgw/label/superseded pyraf on here, this above is what it says.

These two tests failed with similar error on pembry public. @jhunkeler , do you know why?

https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=38273319
https://ssb.stsci.edu/pandokia/pandokia.cgi?query=detail&key_id=38273203

Traceback (most recent call last):
  File "/Users/shared/iraf_conda/bldtmp/porcelain.pkk9bc5SPq/porcelain/envs/public/lib/python2.7/site-packages/pandokia/helpers/runner_minipyt.py", line 583, in process_file
    module = imp.load_source( module_name, filename )
  File "site.py", line 703, in <module>
    main()
  File "site.py", line 694, in main
    execsitecustomize()
  File "site.py", line 548, in execsitecustomize
    import sitecustomize
  File "/Users/iraf/rtx/stsci_python/pyraf/.tmp/ssbvirt/ssbdev-osx/lib/python2.7/sitecustomize.py", line 13, in <module>
OSError: [Errno 2] No such file or directory: '/Users/shared/iraf_conda/bldtmp/porcelain.pkk9bc5SPq/porcelain/envs/public/python/lib/python2.7/site-packages'

https://pypi.org/project/pyraf/ still has old info from STScI. Info and metadata there needs to be updated to clarify that releases now come from iraf-community, not STScI.

I am having an issue using my cursor on the graphics window on a Mac. It seems to work variably, so sometimes it's fine, but sometimes it gets stuck at won't move once it's on the graphics window. I am using iraf in conjunction with the python3 version of this pipeline:
https://github.com/svalenti/pessto
and was directed here after posting the issue to the pipeline repository.

A video demonstrating the issue I am having can be found here:
https://cf-my.sharepoint.com/:f:/g/personal/evanscr8_cardiff_ac_uk/Etzklvf_5VZAvQbcDMKOgZEBY2UVQgzhUqhvObBS3PRulg?e=XywCdF

System details:

• OS: macOS Monterey Version 12.2
• IRAF Version 2.17
• PESSTO Pipeline Python3 branch

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

This one hits a strange error when you try to load it.

--> kpno
Traceback (innermost last):
  File "<CL script CL1>", line 1, in <module>
  File "<CL script obsutil.kpno>", line 18, in kpno
    if (iraf.access('spectimedb$')):
IrafError: Undefined variable `spectimedb' in string `spectimedb$'

Adding an issue for this as requested in #53 (comment). I'm currently not sure what these mean.

[...]
build/lib.linux-x86_64-3.6/pyraf/old_files/gki_sys_tests.py build/lib.linux-x86_64-3.6/pyraf/old_files/interactive_tests.py build/lib.linux-x86_64-3.6/pyraf/old_files/iraffunctions_tests.py build/lib.linux-x86_64-3.6/pyraf/old_files/irafpar_tests.py build/lib.linux-x86_64-3.6/pyraf/old_files/subproc_tests.py
Skipping optional fixer: buffer
Skipping optional fixer: idioms
Skipping optional fixer: set_literal
Skipping optional fixer: ws_comma
Can't parse build/lib.linux-x86_64-3.6/pyraf/ipython_api.py: ParseError: bad input: type=22, value='=', context=('', (384, 54))
Can't parse build/lib.linux-x86_64-3.6/pyraf/irafecl.py: ParseError: bad input: type=22, value='=', context=('', (191, 61))
Can't parse build/lib.linux-x86_64-3.6/pyraf/iraffunctions.py: ParseError: bad input: type=22, value='=', context=('', (2711, 22))
Can't parse build/lib.linux-x86_64-3.6/pyraf/iraftask.py: ParseError: bad input: type=22, value='=', context=('', (346, 67))
Can't parse build/lib.linux-x86_64-3.6/pyraf/cl2py.py: ParseError: bad input: type=22, value='=', context=('', (160, 56))
Can't parse build/lib.linux-x86_64-3.6/pyraf/clparse.py: ParseError: bad input: type=22, value='=', context=('', (438, 23))
Can't parse build/lib.linux-x86_64-3.6/pyraf/clscan.py: ParseError: bad input: type=22, value='=', context=('', (985, 32))
Can't parse build/lib.linux-x86_64-3.6/pyraf/tpar.py: ParseError: bad input: type=22, value='=', context=('', (1386, 96))
Fixing build/lib.linux-x86_64-3.6/pyraf/gkigcur.py build/lib.linux-x86_64-3.6/pyraf/gkiiraf.py build/lib.linux-x86_64-3.6/pyraf/gkitkbase.py build/lib.linux-x86_64-3.6/pyraf/gkitkplot.py build/lib.linux-x86_64-3.6/pyraf/graphcap.py build/lib.linux-x86_64-3.6/pyraf/gwm.py build/lib.linux-x86_64-3.6/pyraf/ipython_api.py 
[...]