C-RET gives AttributeError/NameError in emacs-29.1 about elpy

[tdhock]

by the way I would like to debug/fix this, but I'm not sure where to look in the elisp code, can you please help @jorgenschaefer ?
this is the code for C-RET:

(defun elpy-shell-send-statement-and-step ()
  "Send current or next statement to Python shell and step.

If the current line is part of a statement, sends this statement.
Otherwise, skips forward to the next code line and sends the
corresponding statement."
  (interactive)
  (elpy-shell--ensure-shell-running)
  (elpy-shell--nav-beginning-of-statement)
  ;; Make sure there is a statement to send
  (unless (looking-at "[[:space:]]*$")
    (unless elpy-shell-echo-input (elpy-shell--append-to-shell-output "\n"))
    (let ((beg (save-excursion (beginning-of-line) (point)))
          (end (progn (elpy-shell--nav-end-of-statement) (point))))
      (unless (eq beg end)
        (elpy-shell--flash-and-message-region beg end)
        (elpy-shell--add-to-shell-history (buffer-substring beg end))
        (elpy-shell--with-maybe-echo
         (python-shell-send-string
          (python-shell-buffer-substring beg end)))))
    (python-nav-forward-statement)))

[gopar]

Please do not ping Jorgen. He no longer works on this project (Project status is currently unmaintained).

Any who, I feel python-shell-send-string is a suspect here. Please try using python-shell-send-string and see if the problem persists. If so, then something changed in python.el core (emacs itself) and a bug needs to be filed there.
(Might be worth trying the python-shell-send-* family)

If it works, then this is a problem with elpy. Please update what you find, if anyone ever decides to tackle this they will have an idea of what broke.

Note for whoever works on this: I check the Emacs change logs for 28 and 29 but I didn't see anything that stood out

[tdhock]

Thanks for your response @gopar , sorry for pinging Jorgen, I did not know that he no loner works on this project.
I tried the same with emacs-28.2, and the problem happens, so this suggests the problem started happening with some change between emacs 27 and 28.
I tried using M-x python-shell-send-string x+1 RET at the (Pdb) prompt, and I get the same issue.
I also confirm that this problem happens when using emacs python-mode without elpy, so I will file an issue with emacs, thanks for the help!

[tdhock]

AFAICT the issue is related to the fact that python-shell-send-string is using a new method for sending code, as of emacs 28, which involves running the code through the python function below, which is defconst python-shell-eval-setup-code in python.el

def __PYTHON_EL_eval(source, filename):
    import ast, sys
    if sys.version_info[0] == 2:
        from __builtin__ import compile, eval, globals
    else:
        from builtins import compile, eval, globals
    try:
        p, e = ast.parse(source, filename), None
    except SyntaxError:
        t, v, tb = sys.exc_info()
        sys.excepthook(t, v, tb.tb_next)
        return
    if p.body and isinstance(p.body[-1], ast.Expr):
        e = p.body.pop()
    try:
        g = globals()
        exec(compile(p, filename, 'exec'), g, g)
        if e:
            return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)
    except Exception:
        t, v, tb = sys.exc_info()
        sys.excepthook(t, v, tb.tb_next)

I put that python code in /home/tdhock/__PYTHON_EL_eval.py, then loaded it into my python, and then I ran python-shell-send-string x+1 RET which gave me the following Traceback:

Traceback (most recent call last):
  File "/home/tdhock/.local/share/r-miniconda/lib/python3.7/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'Pdb' object has no attribute 'do___PYTHON_EL_eval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tdhock/__PYTHON_EL_eval.py", line 19, in __PYTHON_EL_eval
    return eval(compile(ast.Expression(e.value), filename, 'eval'), g, g)
  File "/home/tdhock/elpy_test.py", line 1, in <module>
    import pdb
NameError: name 'x' is not defined

The above suggests that on line 19 of __PYTHON_EL_eval the code eval(compile(ast.Expression(e.value), filename, 'eval'), g, g) is incorrect/buggy. g = globals() but in this context (Pdb) we want to evaluate the expression in the context of the Pdb breakpoint, not in the global environment. Not sure how to fix.

[tdhock]

hi @larsmagne and @astoff I saw that you were the last ones to edit __PYTHON_EL_eval, https://github.com/emacs-mirror/emacs/blame/master/lisp/progmodes/python.el
so I was wondering if you could please take a look at this issue, and see if it is possible/easy to fix?

To reproduce in python.el (not using elpy), put the following code in a python script file, for example test_pdb.py

import pdb
def f(x):
    pdb.set_trace()
    x+1
f(2)

Then open the file in emacs, C-c C-p to start a *Python* console, C-c C-c to send buffer, type x+1 RET at the (Pdb) prompt to observe that it works, then do M-x python-shell-send-string RET x+1 RET, which gives the following output on python console:

Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
> /home/tdhock/test_pdb.py(4)f()
-> x+1
(Pdb) x+1
3
(Pdb) 
Traceback (most recent call last):
  File "/home/tdhock/miniconda3/lib/python3.10/cmd.py", line 214, in onecmd
    func = getattr(self, 'do_' + cmd)
AttributeError: 'Pdb' object has no attribute 'do___PYTHON_EL_eval'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 19, in __PYTHON_EL_eval
  File "<string>", line 1, in <module>
NameError: name 'x' is not defined
(Pdb) 

The issue is that there is a AttributeError/NameError, whereas I expected that this should run fine and give the same result (3) as when I type x+1 RET at the (Pdb) prompt. After looking at the definition of python-shell-send-string, I think the issue is in __PYTHON_EL_eval which seems to always eval in the global environment, which is incorrect in this case (at Pdb prompt we may be inside of a function call, with other variables that are not present in global environment). This is a regression, as this works fine in emacs 27 (issue happens in emacs 28 and 29).
BTW I looked at the list of outstanding emacs bugs but I did not see any mention of this issue, https://debbugs.gnu.org/cgi/pkgreport.cgi?package=emacs;include=subject%3Apython
Also I tried filing this issue via M-x report-emacs-bug last week, but looking at the bug-gnu-emacs archive, I do not think my message went through https://lists.gnu.org/archive/html/bug-gnu-emacs/2023-09/index.html

[astoff]

That's right, I'm afraid the only way to interact with the debugger now is to type directly in the shell window. On the bright side, you can now use C-RET to evaluate a complex statement and see the result of the last line (if that was an expression), in Jupyter style.

Fixing this would probably require parsing prompt strings, which, while possible, wouldn't count as "easy to fix"...

[tdhock]

hi @astoff thanks for your response. Should I report this bug elsewhere, in order to get it fixed faster?
So this change was required in order to support the new feature about evaluating a complex statement and seeing the result of the last line? Can you please share an example of how that works?
Has this change been documented anywhere? I looked at the NEWS.28 file from emacs, and I see only the following, which does not mention this change:

** Python mode

*** New user option 'python-forward-sexp-function'.
This allows the user easier customization of whether to use block-based
navigation or not.

*** 'python-shell-interpreter' now defaults to python3 on systems with python3.

*** 'C-c C-r' can now be used on arbitrary regions.
The command previously extended the start of the region to the start
of the line, but will now actually send the marked region, as
documented.

is there any way to configure emacs to use the previous (emacs 27) behavior, where this was working?
something like (setq python-shell-send-string-method 'old) ?
Thanks again very much.