SBCL error: Heap exhausted during garbage collection about alive

[nobody-famous]

Does it do that immediately? Or does it take it a while?

[ab-10]

It does it after first or second "send to repl"

…

On Sun, Sep 24, 2023, 11:41 PM nobody-famous ***@***.***> wrote: Does it do that immediately? Or does it take it a while? — Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC54L5RDMHZCMGEQKSNBDWTX4CZLJANCNFSM6AAAAAA5FDRFUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nobody-famous]

OK, that's weird. I'll have to look into it and see if maybe there's some loop that's getting out of hand. Not sure what it would be filling up the heap with.

[ab-10]

Thanks, do let me know if there's any extra debugging info that would come handy

…

On Mon, Sep 25, 2023, 12:32 AM nobody-famous ***@***.***> wrote: OK, that's weird. I'll have to look into it and see if maybe there's some loop that's getting out of hand. Not sure what it would be filling up the heap with. — Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC54L5SZ3OIJM3O5QZGTVCTX4C7IZANCNFSM6AAAAAA5FDRFUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nobody-famous]

If you watch the sbcl process in top or something, when does the memory start to spike?

[ab-10]

Upon start the CPU utilisation goes to near 100% and memory to 1GB. First it outputs: Match: ["[9/27/2023 17:29:27][STARTING] Started on port 49326","9/27/2023 17:29:27","STARTING","49326"] Found port: 49326 Server port 49326 sbcl ERROR: INFO: Control stack guard page unprotected Then: sbcl ERROR: Heap exhausted during garbage collection: 48 bytes available, 80 requested. Immobile Object Counts Gen layout fdefn symbol code Boxed Cons Raw Code SmMix Mixed LgRaw LgCode LgMix Waste% Alloc Trig Dirty GCs Mem-age sbcl ERROR: 0 0 0 0 0 1171 727 0 0 0 1323 0 0 0 0.3 105243248 62241002 - 1 0.0000 sbcl ERROR: 1 4 0 0 3 10496 3193 19 0 1 826 0 0 0 0.2 475145504 287691386 2903 1 1.3849 sbcl ERROR: 2 416 5135 1288 3654 10003 3294 143 6 26 1010 0 0 0 0.3 473036112 2000000 154 0 0.0219 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0 0 2000000 0 0 0.0000 sbcl ERROR: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0 0 2000000 0 0 0.0000 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0.0 0 2000000 0 0 0.0000 6 740 15353 26273 17418 325 110 7 3 6 16 0 0 63 3.3 16801168 2000000 15 0 0.0000 Tot 1160 20488 27561 21075 21995 7324 169 9 33 3175 0 0 63 0.3 1070225200 [99.7% of 1073741824 max] GC control variables: *GC-INHIBIT* = true *GC-PENDING* = true *STOP-FOR-GC-PENDING* = false Collection trigger variables: dynamic_space_size = 1073741824 bytes_allocated = 1070225200 auto_gc_trigger = 976094419 bytes_consed_between_gcs = 53687091 fatal error encountered in SBCL pid 35275 pthread 0x70000f554000: Heap exhausted, game over. Error opening /dev/tty: Device not configured I've realised sending to repl is not required. If I don't do anything the memory still overflows.

…

On Wed, Sep 27, 2023 at 3:26 AM nobody-famous ***@***.***> wrote: If you watch the sbcl process in top or something, when does the memory start to spike? — Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC54L5VLUXVI2TKNZ5S3QCTX4OFF5ANCNFSM6AAAAAA5FDRFUA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[nobody-famous]

OK, that actually makes more sense. My current theory is that my parser has a bug that's causing it to get stuck and try to create an infinitely long token. I've found one bug already, but it caused a failure so that wasn't it.

In the output panel, if you switch it to the 'Alive Client' log, how far does it get? That log shows the communication between the server and VSCode. Oh, you may have to put "lisp.trace.server": "verbose" in your settings.

If you open a different lisp file does it still do it?

[ab-10]

Here's the Alive Client log

If you open a different lisp file does it still do it?

Just realized that it does not. Here are the contents of the file causing the problem:

defmacro fib! (x)
        `(if (<= ,x 2)
             1
             (+ (fib! (- ,x 1))
                (fib! (- ,x 2)))))

(fib! 6)
(macroexpand-1  `(fib! 5))

[nobody-famous]

Oh, the joys of lisp...

OK, the problem is the (fib! 6) line. In order to get diagnostics, the extension asks sbcl to compile the file without loading it so that it doesn't execute everything. Apparently, when it compiles a macro it tries to fully expand it. In this case, that causes infinite expansion and it eventually blows up. The same thing happens if I run sbcl from the command line and do (compile-file "fib.lisp"). Here's the fun part, it doesn't do that if it loads the file instead of compiling it. With your code, if I comment out that line then nothing breaks. If I then run (fib! 6) from the repl, it works fine.

I might need to put in an option to turn off automatic diagnostics. I've been hoping to avoid that, but it might not be avoidable. I know emacs doesn't do diagnostics as you type because of this.

[ab-10]

I'm not sure whether this is a byproduct of automatic diagnosis, but if I (manually) compile the same defun in sly, it throws an error when SBCL runs out of heap memory and then gracefully recovers (do let me know if you'd like more info on sly behaviour here). IMO, the ideal behaviour for Alive would be the same. I can imagine it being more tricky in automatic diagnosis use-case. What do you think about such direction?

[ab-10]

#169 should be fixed by the behaviour I described in the comment above as well.

[nobody-famous]

Alive does the same thing if you manually compile the file, it goes into the debugger and has an option to abort the thread. I'll hopefully have time this weekend to play around with it. Hopefully there's a specific out of memory error that it's using. The auto diagnostics can't use the debugger because it popping up over and over gets very annoying, very quickly. Calling a restart programmatically aborts the compiler and no further errors/warnings are shown, which isn't ideal. If it has a more specific error that can be caught, then aborting the compiler would be fine. I can still see wanting an option to turn it off because it's going to spike the CPU every time it runs.

I don't think it would help with the other issue, which is the repl getting stuck in an infinite loop. The only options there are to programmatically kill the thread, which can take forever because the CPU is maxed, or kill and restart the repl. Since the LSP is the repl, that's done by reloading the window.

[ab-10]

Thanks for taking the time to explain, @nobody-famous, this exchange has been very educational for me.