Support errortrace about racket-mode

[greghendershott]

Note: This might not require literally doing racket -l errortrace run.rkt. It might suffice to (namespace-require errortrace) prior to the user's module. I don't yet understand exactly (a) what -l does and (b) what errortrace needs to work.

[florence]

Because racket-mode is pulling files from disk this is a little more tricky, as it might pull the already compiled versions off the disk.

https://github.com/florence/cover solves this by removing the compiled files then changing current-compile and the to load the errortrace compiler (when its in the correct namespace), and putting the compiled flies in a different location. If you want I could add this to racket-mode, but blowing away the compiled files might me more iffy if racket-mode than it is for cover. Thoughts?

[greghendershott]

1. https://github.com/florence/cover looks sweet!
2. I have no experience using errortrace directly or via DrRacket. As a result I'm looking forward to learning more about it -- and when I say "more", I mean "anything at all". :)
3. I wonder if it would be cool to have racket-mode highlight coverage directly in the buffer? (If that's already what you had in mind -- see 2. Or if this idea makes no sense at all -- also see 2.)
4. As for compiled files: Currently racket-mode is agnostic about them. Unlike DrRacket, racket-mode has no option to create/update them automatically. When you want to make (or delete) them, you have to pop out to shell and raco make or raco setup (or rm -rf compiled/). I've wondered about supporting this in racket-mode, but so far haven't.
   1. Maybe if racket-mode did support this, it would be easier? If it's responsible for creating them, it can recreate them for errortrace, and re-recreate them afterwards?
   2. Or maybe any errortrace support could just do this directly and independently? i.e. "Backup" compiled/, do errortrace stuff, then "restore" the originals??
   3. Again, if the above makes no sense, please see 2. :)

[florence]

2: Errortrace works by, essentially, taking some fully expanded code (i.e. code that is formed only of rackets primitives) and injecting instructions around these primitives. These instructions can do things like block tail call elimination (hence the better stack traces) or bang on a hash map saying "this code was run!" Cover does this by clobbering all compiled files then replacing the current-compile parameter with one that asks errortrace to instrument the expression prior to compilation.I think DrRacket and 'racket -l errortrace' does something similar.

3: Cover would completely support this! It has a racket API, and is actually designed (read mostly copied) from how DrRacket does its code coverage! The original inspiration for the project was, in fact, "it would be nice if things like racket-mode could use DrRackets code coverage tools."

4.I. Implementing errortrace would put you halfway to doing this, as errortrace would involve fiddling with use-compiled-file-paths and current-compile, which gives one control over: where compile files go and are loaded from, and if a compiled expression is allowed to be saved for future use.

4.II. I'm not sure about how do do this or what oddities one might run into. IIRC DrRacket handles this by by not handling it. (ie it its errortracing only happens on a (re)compile) but I'm not entirely sure... Its worth another look. I think replacing the current-load parameter to a function that says "if thats my file recompile" might do it... but I'm not sure. Maybe saying "racket-mode will clobber your compiled files if you turn this on" is good enough.

[florence]

Okay so for 4.II you can replace the current-load/use-compiled to skip finding compiled files for the current buffer. (Cover now does this :D)

[greghendershott]

Sounds great! If you want to submit a PR, that would be awesome.

(By the way, I pushed a lot of commits to master today. So probably easiest if you pull before starting this, vs. rebasing to master later.)

Otherwise let me know if you'd like me to take a crack at using Cover.

[florence]

I'll get started tonight. Could you point me at an example of how to wire emacs options to racket code?

[greghendershott]

On the Racket side, commands like ,run are implemented in cmds.rkt.

On the Emacs side, see racket-eval.el. There are a few flavors, depending on how you want to get information back from Racket, if at all.

racket--do-visit-def-or-mod is probably a good example. This Emacs function is a helper for the Emacs commands racket-visit-definition and racket-visit-module. It uses racket--eval/sexpr to send a ,def or ,mod command to the Racket side, because it wants to get back a list from Racket, containing file, line, col info. (And so long as the Racket side uses elisp-println to fix up some tiny differences like #t or #fvs. t or nil, Emacs should be able to read the Racket sexprs just fine.)

If you're thinking of an option like "errortrace on or off", then you probably want to add that as another optional argument for the ,run command. Existing examples of such options are a memory limit and whether to pretty print. So see this part of cmds.rkt on the Racket side, and racket-run on the Emacs side.

---

I probably left something out (sorry!) but hopefully that's enough to get you started and off course let me know if you have any questions.

[greghendershott]

By the way, so far I've avoided directly depending on or automatically installing optional or third-party Racket packages. Not even my own Racket packages. Why:

1. To be compatible with older Rackets (that didn't have raco pkg).
2. Not to complicate the Emacs package install.

Someday I'd like to rethink this. Because:

1. Some chunks of racket-mode would actually be helpful to split out as packages that could be used for other things.
2. So racket-mode can use third-party packages. Like, say, Cover.

But meanwhile, with the status quo, if you'd like to use code from your Cover package, there are a couple options:

1. copypasta. Maybe OK for a small amount of code, but otherwise non-ideal for you and me both.
2. One precedent is racket-mode's recent use of racket-find-collection. Basically the doc string tells the user they need to run raco install racket-find-collection for it to work. And the Racket code does a dynamic-require wrapped in a with-handlers to provide a graceful do-nothing fallback.

[florence]

I don't think copypasta is a good idea right now: Were in the process of fixing a ton of bugs, and there are a lot of enhancements to make (like supporting the config module like raco test does). I doubt the package will be fully stable for a few months.

The fallback do-nothing option is probably the best. Also, I should point out that cover hasn't been tested on anything other than 6.1.1. It should be find on older versions (Its core is mostly a copy/paste of how DrRacket does things, sans bugs and reliance on files existing), but its possible it just blows up completely.

[greghendershott]

Sounds good to me.

Also, I should point out that cover hasn't been tested on anything other than 6.1.1. It should be find on older versions (Its core is mostly a copy/paste of how DrRacket does things, sans bugs and reliance on files existing), but its possible it just blows up completely.

It sounds like you don't need more to-do items :), but, I suppose your info.rkt deps could change "base" to ("base" #:version "6.1.1"), which IIRC is the way to say "this package requires Racket 6.1.1 or newer".

[florence]

AHAH, I knew there was a flag for that somewhere. Graci!

How would that version flag effect a raco pkg install from racket-mode? Would the build just explode horribly and die?

[greghendershott]

How would that version flag effect a raco pkg install from racket-mode?

It wouldn't affect it. Currently, racket-mode doesn't try to install any packages programmatically (that's what I was trying to explain above). Instead, the docs suggest the user might want to install racket-find-collection, for example, to enable that feature. I was thinking similar for your package and this feature.

So if you updated deps this way, the user who ran raco pkg install cover would get a (good, helpful) error message.

[greghendershott]

And someday if racket-mode were to try to programmatically install Racket packages for the user, it would/should pay attention to the result and relay OK/fail to user.

[greghendershott]

Conversation with you and @rntz on #racket this afternoon nudged me to dig in and figure out the basic steps getting errortrace to work with racket-mode's model of evaluation. And, to do the basic "wiring" to prepare making this controllable from a defcustom or whatever up in Emacs Elisp.

Result: I just pushed a commit on a new errortrace branch.

In my initial testing with 6.1.1.6 this works, in terms of giving a better context stack. For example with it turned off:

; /: division by zero
; Context:
;  /tmp/foo.rkt:1:1 [running body]
;  /tmp/foo.rkt:15:0 f

With it turned on:

; /: division by zero
; Context (errortrace):
;    /tmp/foo.rkt:16:2: (/ x 0)
;    /tmp/foo.rkt:21:0: (g)

However. This does not deal with any of the other stuff we discussed:

1. Dealing smartly with compiled zos.
2. Getting and presenting coverage data.

So, I'm still looking forward to your help, if you still can!

[florence]

FWIW cover gave up on preserving compiled code. And as far as I can tell, DrRacket does too when code coverage or profiling is disabled (but not when debugging itself is on... I have no clue how that works).

Also I wonder: will racket-mode need to errortrace/test-cover multiple files at once? If not this is much easier.

[florence]

I've read thought the errortrace branch. It looks good to me, and how it deals with compiled zos is equivalent to how DrRacket does it.

I'm currently adding cover to it. I'll push something when I have a hook to get the coverage info, then leave it to you to handle presenting it?

[greghendershott]

Also I wonder: will racket-mode need to errortrace/test-cover multiple files at once? If not this is much easier.

I've hardly used coverage, so I don't have a good feel for this. What do you think?

I'm currently adding cover to it. I'll push something when I have a hook to get the coverage info, then leave it to you to handle presenting it?

Sounds good!

BTW I've pushed some more commits to that branch today. Assuming you forked and started with that branch, hopefully you can rebase to it without any conflicts? However if it's messy let me know and I can resolve.

[florence]

I've hardly used coverage, so I don't have a good feel for this. What do you think?

DrRacket only does single file coverage, so this might be a good starting point at least.

I've pushed some more commits to that branch today.

Rebased just fine!

[greghendershott]

I've continued to work on this and pushed more commits. I'm a little concerned they might be harder for you to rebase on, because I refactored some stuff that had been bothering me. Hopefully it will be OK. But if not, please don't hesitate to let me know and I'll try to help.

[florence]

The changes to add cover are actually very minor (about five lines). The bit I'm struggling with is how to get this to play nicely with the fact that cover may-or-may-not be installed. (Specifically how to parameterize over a parameter that may or may not exist, give that parameterize is a form.). I may have to make some larger structural changes to run, so I'll wait till your'e happy with the state of the errortrace stuff?

[greghendershott]

The bit I'm struggling with is how to get this to play nicely with the
fact that cover may-or-may-not be installed. (Specifically how to
parameterize over a parameter that may or may not exist, give that
parameterize is a form.).

One thing I do in run.rkt, which I originally saw in racket/sandbox, is
simply to make a dummy parameter for the other case (where the
dynamic-require fails, or isn't even attempted). That way there is
always some parameter that exists. I'm not sure if that idea would
help here?

I may have to make some larger structural changes to run, so I'll
wait till your'e happy with the state of the errortrace stuff?

Make sense. By the way, I've been going ahead and working on profiling
(but not coverage). As a result, I defined the idea of
"error-context-levels", which determine setting the various errortrace,
profile, and coverage parameters:

;; Definitions for the context-level member of rerun

(define profile/coverage-levels
  ;; "sibling" levels that all require errortrace
  '(profile         ;high with profiling-enabled
    cover/count     ;high with coverage-counts-enabled
    cover/execute)) ;high with execute-counts-enabled

(define errortrace-levels
  `(high     ;compile-context-preservation-enabled #t + errortrace
    ,@profile/coverage-levels))

(define context-levels
  `(low      ;compile-context-preservation-enabled #f
    medium   ;compile-context-preservation-enabled #t
    ,@errortrace-levels))

IOW the levels are:

'(low medium high profile cover/count cover/execute)

And that can be up in the Elisp UI.

Hopefully that's part of, or at least compatible with, the plumbing you
need.

So once this settles down, I'll stop poking it and let you take over.

[greghendershott]

Weird. GitHub doesn't recognize markdown in comments originating from email? I can't even edit the message here, to use markdown. It's like there's an ignore-markdown? bit stuck forever.

[florence]

What is the difference between cover/count and cover/execute? Do you mean "count the number of times something is executed"? Because cover does not (currently) support this, and probably wont for a while. Because cover will... cover... multiple threads having an execute count would require every expression to gain a mutex lock. I'm not happy with that solution. There are some fancier ways to do this (using thread cells) but those would require substantially more work, and are probably a version 3 feature. (supporting racket-mode is a version 2 thing).

[greghendershott]

I'm just trying to reflect what errortrace exposes, which seems to be these two favors?

[florence]

Aaahhh. I have no clue what those mean. But the code coverage stuff in errortrace is horribly broken[1]. DrRacket and cover bypasses that code and do their own instrumentation via the stacktrace@ unit. I think its probably better to do something more like DrRacket does and only check if an expression was run or not. On top of errortrace being broken I don't see any meaningful way to present the coverage information other than "this was run or not," at least not in the context of an IDE.

Basically I'm voting to collapse those two coverage levels into one "do the coverage" level.

[1] By horribly broken I mean "errors every time you attempt to actually use it." See http://bugs.racket-lang.org/query/?cmd=view&pr=14699

[greghendershott]

Yeah I'd encountered that undocumented values return.

Basically I'm voting to collapse those two coverage levels into one "do the coverage" level.

OK.

[greghendershott]

I just pushed commit 680b1b6 which implements support for errortrace, profiling, and coverage.

@florence I got reasonable results from the execute-counts flavor of errortrace's coverage, after consolidating the results. Admittedly I only tried relatively simple examples and cover may do better on more realistic ones.

[florence]

get-execution-counts aught to be fine. cover doesn't currently support execution counts anyway.

The big gain from cover would be make-covered?, which allows one to account for uncovered, covered, and irrelevant code (think comments, type annotations, etc).

Cover's PR 53 adds support for racket-mode with a very similar interface to errortrace if you do want to switch [when we get it merged in]. (Eventually cover and errotrace may be merged, but thats a ways away).

[greghendershott]

Cover's PR 53 adds support for racket-mode with a very similar interface to errortrace if you do want to switch

Definitely interested in trying that when it's ready, yes!

The big gain from cover would be make-covered?, which allows one to account for uncovered, covered, and irrelevant code (think comments, type annotations, etc).

Well as an early/naive user of coverage tools, I think I prefer the DrRacket approach (and status quo racket-mode approach) of highlighting only what's uncovered. i.e. Just show me what I need to do something about.

However I can appreciate it's reassuring to see the covered things affirmatively highlighted in green, too, and to know that is coming from actual this-was-covered data. Some people probably prefer that.

get-execution-counts aught to be fine. cover doesn't currently support execution counts anyway.

One thing I've noticed is that for a given file, the execution time seems to be significantly faster in racket-mode with the execution-counts approach, than with cover's annotation approach. (DrR also seems slower, but DrR is slower overall for me, so idk how much is due to the measurement approach.) Of course, there might be a trade-off here where slower means better results.

[florence]

I think I prefer the DrRacket approach (and status quo racket-mode approach) of highlighting only what's uncovered.

For a UI this probably doesn't matter. cover uses this to ensure that in-module tests are not counted for (or against) coverage. [This is behavior is also configurable. By default cover considers all submodules irrelevant.]

One thing I've noticed is that for a given file, the execution time seems to be significantly faster in racket-mode with the execution-counts approach, than with cover's annotation approach.

Yes. From my understanding get-execution-counts probabilistically samples the running program, whereas cover and DrR annotate every expression. This means that get-execution-counts is fast, but sometimes wrong. (The idea is that its close enough). DrR and cover are at best ~2x slower.