Unsatisfactory handling of errors in test expressions about doctest

[ag91]

Thanks for reporting this with examples!
I don't have much experience in reporting signals source, do you have a reference I can look up?
I guess at least not dropping the ball at the first failing test could be easily achievable.

A note to myself, the command to reproduce in my settings

emacs --batch --eval "(progn (package-initialize) (require 'doctest) (doctest \"test.el\"))"

where test

(defun divide (a b)
  "
>> (/ 1 0)
=> 2


>> (divide 1 2)
=> 3

>> (divide 1 0)
=> 3"
  (/ a b))

[ag91]

with my latest change this improves a little bit. How does that sound @doublep ?

An example of trace:

Warning (emacs): Test cases in comments can cause byte-compile warnings, please add '(not docstrings) to `byte-compile-warnings'
35
58
80
0 passed, 3 failed

test.el#4: (/ 1 0 => 2 but got Failure evaluating input:

(/ 1 0
which caused:
(end-of-file)

test.el#8: (divide 1 2) => 3 but got 0test.el#11: (divide 1 0) => 3 but got Failure evaluating input:

(divide 1 0)
which caused:
(arith-error)

[doublep]

I don't have much experience in reporting signals source, do you have a reference I can look up?

You could study either ERT or Buttercup source code. They use Edebug functionality to find backtraces. Buttercup is probably more readable.

The largest difference to your quick hack is that they print signal backtrace, which is useful if the error is signalled not from the function in question, but from something it calls (with 10 nesting levels). Though you could argue that doctest is aimed at simple functions.

+           (evaluated-input (condition-case err
+                                (format "%S" (eval (car (read-from-string input))))
+                              (error
+                               (format "Failure evaluating input:\n\n%s\nwhich caused:\n%s\n\n" input err))))

At the very least the error handler should use t instead of error. Otherwise the condition-case form doesn't catch non-error signals, just try this:

>> (signal 'wtf nil)
=> nil

Error output also confusingly repeats test code twice.

[ag91]

thanks I will give a look at buttercup source once I have time.
This change should do for a minimal functionality though.
Happy to receive a PR for more advanced stuff meanwhile I (find the time to) study.

[ag91]

closing this as I consider the more advanced version something nice to have but not strictly necessary yet.