*** stack smashing detected ***: terminated Fatal error 6: Aborted about tree-sitter

[jesse-c]

For anyone wanting to downgrade on Arch, I've been able to get Emacs non-crashing again with sudo pacman -U file:///var/cache/pacman/pkg/tree-sitter-0.22.2-1-x86_64.pkg.tar.zst [1].

[1] https://wiki.archlinux.org/title/downgrading_packages

[1player]

If you don't have the old version of tree-sitter locally, you can download it from the Arch Linux archive at https://archive.archlinux.org/packages/t/tree-sitter/tree-sitter-0.22.2-1-x86_64.pkg.tar.zst

Downgrading to this version fixes it for me, for now.

[maxbrunsfeld]

The Tree-sitter C library's ABI has indeed changed (due to the TSQueryCursor struct field addition). I think that the Arch Emacs package should probably pin a specific version of the Tree-sitter package. I really don't want Emacs users to be broken by changes to Tree-sitter. Does anyone know the maintainers of the Arch packages?

In this repo, we try to not make breaking changes unnecessarily, but the library is pre 1.0, in its Semver versioning, and even once it goes 1.0, it's going to be easier to maintain API stability than full-on ABI stability.

[MatthewGentoo]

Breaking ABI compatibility is not an issue at all, especially pre-1.0.

It would make things easier for consumers if the library's soname was bumped when ABI changed though. Having a different soname allows for the incompatible versions to be installed side by side, and is a signal that consumers need to be rebuilt when it changes.

[maxbrunsfeld]

It sounds like in the future, we should bump the library's minor version when changing any structs in api.h, because with our current strategy, the soname is based on the minor version.

[thesamesam]

I suspect we've hit the same thing in Gentoo as https://bugs.gentoo.org/930039 - it looks like 0.22.2 vs 0.22.4 breaks ABI?

EDIT: quoting some details from the Gentoo bug...

libabigail's abidiff output is:

$ abidiff /var/tmp/portage/dev-libs/tree-sitter-0.22.{2,4}/image/usr/lib64/libtree-sitter.so.0.22 --fail-no-debug-info --debug-info-dir1 /var/tmp/portage/dev-libs/tree-sitter-0.22.2/image/usr/lib/debug --debug-info-dir2 /var/tmp/portage/dev-libs/tree-sitter-0.22.4/image/usr/lib/debug
Functions changes summary: 0 Removed, 4 Changed (45 filtered out), 0 Added functions
Variables changes summary: 0 Removed, 0 Changed, 0 Added variable

4 functions with some indirect sub-type change:

  [C] 'function const size_t* ts_parser_cancellation_flag(const TSParser*)' at parser.c:1921:1 has some indirect sub-type changes:
    parameter 1 of type 'const TSParser*' has sub-type changes:
      in pointed to type 'const TSParser':
        in unqualified underlying type 'typedef TSParser' at api.h:45:1:
          underlying type 'struct TSParser' at parser.c:90:1 changed:
            type size hasn't changed
            1 data member insertion:
              'bool has_scanner_error', at offset 11616 (in bits) at parser.c:113:1

  [C] 'function void ts_query_cursor_delete(TSQueryCursor*)' at query.c:2986:1 has some indirect sub-type changes:
    parameter 1 of type 'TSQueryCursor*' has sub-type changes:
      in pointed to type 'typedef TSQueryCursor' at api.h:48:1:
        underlying type 'struct TSQueryCursor' at query.c:301:1 changed:
          type size changed from 1152 to 1216 (in bits)
          15 data member changes:
            type of 'TSTreeCursor cursor' changed:
              underlying type 'struct TSTreeCursor' at api.h:105:1 changed:
                type size changed from 192 to 256 (in bits)
                1 data member change:
                  type of 'uint32_t context[2]' changed:
                    type name changed from 'uint32_t[2]' to 'uint32_t[3]'
                    array type size changed from 64 to 96
                    array type subrange 1 changed length from 2 to 3
            'struct {QueryState* contents; uint32_t size; uint32_t capacity;} states' offset changed from 256 to 320 (in bits) (by +64 bits)
            'struct {QueryState* contents; uint32_t size; uint32_t capacity;} finished_states' offset changed from 384 to 448 (in bits) (by +64 bits)
            'CaptureListPool capture_list_pool' offset changed from 512 to 576 (in bits) (by +64 bits)
            'uint32_t depth' offset changed from 832 to 896 (in bits) (by +64 bits)
            'uint32_t max_start_depth' offset changed from 864 to 928 (in bits) (by +64 bits)
            'uint32_t start_byte' offset changed from 896 to 960 (in bits) (by +64 bits)
            'uint32_t end_byte' offset changed from 928 to 992 (in bits) (by +64 bits)
            'TSPoint start_point' offset changed from 960 to 1024 (in bits) (by +64 bits)
            'TSPoint end_point' offset changed from 1024 to 1088 (in bits) (by +64 bits)
            'uint32_t next_state_id' offset changed from 1088 to 1152 (in bits) (by +64 bits)
            'bool on_visible_node' offset changed from 1120 to 1184 (in bits) (by +64 bits)
            'bool ascending' offset changed from 1128 to 1192 (in bits) (by +64 bits)
            'bool halted' offset changed from 1136 to 1200 (in bits) (by +64 bits)
            'bool did_exceed_match_limit' offset changed from 1144 to 1208 (in bits) (by +64 bits)

  [C] 'function TSTreeCursor ts_tree_cursor_copy(const TSTreeCursor*)' at tree_cursor.c:695:1 has some indirect sub-type changes:

  [C] 'function TSTreeCursor ts_tree_cursor_new(TSNode)' at tree_cursor.c:153:1 has some indirect sub-type changes:

I've filed a bug upstream with libabigail after discussion on IRC wrt why it didn't flag it as a breaking change (https://sourceware.org/bugzilla/show_bug.cgi?id=31642).

Truncated backtrace (full at https://bugs.gentoo.org/930039#c2) from our pkgcheck tool which uses tree-sitter-bash:

tests/checks/test_codingstyle.py::TestStaticSrcUri::test_no_report[${P}]
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff49a3726 in ts_tree_cursor_init (self=0x555555650008, node=...) at lib/src/tree_cursor.c:167
167       array_push(&self->stack, ((TreeCursorEntry) {
(gdb)
#0  0x00007ffff49a3726 in ts_tree_cursor_init (self=0x555555650008, node=...) at lib/src/tree_cursor.c:167
#1  ts_tree_cursor_reset (_self=0x555555650008, node=...) at lib/src/tree_cursor.c:160
#2  ts_query_cursor_exec (self=0x555555650000, query=0x555555d8f3f0, node=...) at lib/src/query.c:3045
#3  0x00007ffff529dce0 in query_captures (self=0x7ffff2618210, args=(<tree_sitter.Node at remote 0x7ffff2180b70>,), kwargs=0x0) at tree_sitter/binding.c:2112
#4  0x00007ffff7b0c603 in method_vectorcall_VARARGS_KEYWORDS (func=<method_descriptor at remote 0x7ffff260f740>, args=0x7ffff21f8a98, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:344
#5  0x00007ffff7af816e in _PyObject_VectorcallTstate (tstate=0x555555576f90, callable=<method_descriptor at remote 0x7ffff260f740>, args=0x7ffff21f8a98, nargsf=<optimized out>, kwnames=0x0) at ./Include/cpython/abstract.h:114
[...]

Valgrind output:

tests/checks/test_codingstyle.py::TestStaticSrcUri::test_no_report[${P}] ==602430== Invalid write of size 8
==602430==    at 0xDC597D4: ts_query_cursor_set_byte_range (query.c:3064)
==602430==    by 0xD609CA2: query_captures (binding.c:2110)
==602430==    by 0x49A2602: method_vectorcall_VARARGS_KEYWORDS (descrobject.c:344)
==602430==    by 0x498E16D: UnknownInlinedFun (abstract.h:114)
==602430==    by 0x498E16D: UnknownInlinedFun (abstract.h:123)
==602430==    by 0x498E16D: UnknownInlinedFun (ceval.c:5893)
==602430==    by 0x498E16D: _PyEval_EvalFrameDefault (ceval.c:4198)
==602430==    by 0x49D4AE9: UnknownInlinedFun (pycore_ceval.h:46)
==602430==    by 0x49D4AE9: UnknownInlinedFun (genobject.c:213)
==602430==    by 0x49D4AE9: gen_iternext (genobject.c:580)
==602430==    by 0x498E39F: _PyEval_EvalFrameDefault (ceval.c:4001)
==602430==    by 0x49D4E02: UnknownInlinedFun (pycore_ceval.h:46)
==602430==    by 0x49D4E02: gen_send_ex2 (genobject.c:213)
==602430==    by 0x4990580: _PyEval_EvalFrameDefault (ceval.c:2586)
==602430==    by 0x49D4AE9: UnknownInlinedFun (pycore_ceval.h:46)
==602430==    by 0x49D4AE9: UnknownInlinedFun (genobject.c:213)
==602430==    by 0x49D4AE9: gen_iternext (genobject.c:580)
==602430==    by 0x49DCFC4: list_extend (listobject.c:960)
==602430==    by 0x49AC14C: method_vectorcall_O (descrobject.c:460)
==602430==    by 0x498E16D: UnknownInlinedFun (abstract.h:114)
==602430==    by 0x498E16D: UnknownInlinedFun (abstract.h:123)
==602430==    by 0x498E16D: UnknownInlinedFun (ceval.c:5893)
==602430==    by 0x498E16D: _PyEval_EvalFrameDefault (ceval.c:4198)
==602430==  Address 0xa430078 is 24 bytes after a block of size 944 in arena "client"
==602430==

valgrind: m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 1008, hi = 18446744069414584320.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.
```

[ObserverOfTime]

Downstream emacs issue: https://gitlab.archlinux.org/archlinux/packaging/packages/emacs/-/issues/2

[amaanq]

there's not much in this backtrace that we can go off of. are you able to build with debug syms and the frame pointer enabled? (-fno-omit-frame-pointer, if using clang you might also need -mno-omit-leaf-frame-pointer)

[jostillmanns]

I suspect we've hit the same thing in Gentoo as https://bugs.gentoo.org/930039 - it looks like 0.22.2 vs 0.22.4 breaks ABI?

tree-sitter 0.22.4 also breaks things for me (crashes emacs) vs 0.22.2 works fine.

[seanhagen]

Been running into this on two different machines; opening any file that tries to use tree-sitter to handle syntax highlighting causes Emacs to crash with the same stack trace posted above.

[maxbrunsfeld]

could anyone attach a debugger to emacs and help give some insight into the crash?

[maxbrunsfeld]

The public TSTreeCursor struct has changed, so I do think that will change the ABI.

[maxbrunsfeld]

Just so I understand - on Arch Linux, Emacs uses tree-sitter via a dynamic library? Is the Arch package based on Git tags in the Tree-sitter repository? So if we push a tag to this repository, emacs users on Arch just start using the new version, based on the fact that it is a patch release?

[seanhagen]

I think the only way Emacs can use tree-sitter is via dynamic library, I'm not sure it can be statically linked. At least, I can't find any information from some quick searching on how to have Emacs statically link dependencies.

The Arch package I'm using is extra/tree-sitter, and the PKGBUILD file in the package repository seems to indicate it checks out this repo using a specific tag ( currently 0.22.5 for the latest ). I think the package maintainer has to update the PKGBUILD file to point it to a new version. I'm not sure how the package is updated, but it does seem to be kept up-to-date with the latest version in this repository.

[maxbrunsfeld]

That makes sense. Does the Emacs package specify a Semver range of Tree-sitter library that it is compatible with?

[xxyzz]

Here is the backtrace from gdb on Arch Linux with emacs 29.3-2 and tree-sitter 0.22.5-1:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff3aab393 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff3a5a6c8 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff3a424b8 in __GI_abort () at abort.c:79
#4  0x00007ffff3a43395 in __libc_message_impl (fmt=fmt@entry=0x7ffff3bbb161 "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:132
#5  0x00007ffff3b3273b in __GI___fortify_fail (msg=msg@entry=0x7ffff3bbb179 "stack smashing detected") at fortify_fail.c:24
#6  0x00007ffff3b33a56 in __stack_chk_fail () at stack_chk_fail.c:24
#7  0x000055555579e063 in Ftreesit_subtree_stat (node=<optimized out>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/treesit.c:3498
#8  0x00007fffe61d3832 in F747265657369742d666f6e742d6c6f636b2d666f6e746966792d726567696f6e_treesit_font_lock_fontify_region_0 () at /usr/lib/emacs/29.3/native-lisp/29.3-4b59e32f/treesit-37439c61-97df641d.eln
#9  0x000055555570a61e in Ffuncall (nargs=4, args=0x7fffffff8000) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#10 0x00007fffef16cf42 in F666f6e742d6c6f636b2d666f6e746966792d73796e746163746963616c6c792d726567696f6e_font_lock_fontify_syntactically_region_0 ()
    at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/font-lock-895216f6-1f3b244f.eln
#11 0x000055555570a61e in Ffuncall (nargs=4, args=0x7fffffff8170) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#12 0x00007fffef16abef in F666f6e742d6c6f636b2d64656661756c742d666f6e746966792d726567696f6e_font_lock_default_fontify_region_0 ()
    at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/font-lock-895216f6-1f3b244f.eln
#13 0x000055555570a61e in Ffuncall (nargs=4, args=0x7fffffff8220) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#14 0x00007fffef169873 in F666f6e742d6c6f636b2d666f6e746966792d726567696f6e_font_lock_fontify_region_0 () at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/font-lock-895216f6-1f3b244f.eln
#15 0x000055555575b8be in exec_byte_code (fun=0x2cd73, args_template=4294967295, nargs=2, args=0x7fffee400040) at /usr/src/debug/emacs/emacs-29.3-wayland/src/bytecode.c:809
#16 0x000055555570a61e in Ffuncall (nargs=2, args=args@entry=0x7fffffff8480) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#17 0x000055555570ec31 in run_hook_wrapped_funcall (nargs=<optimized out>, args=0x7fffffff8480) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2777
#18 0x000055555570967c in run_hook_with_args (nargs=2, args=0x7fffffff8480, funcall=0x55555570ec10 <run_hook_wrapped_funcall>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2858
#19 0x00007fffeedcec88 in F6a69742d6c6f636b2d2d72756e2d66756e6374696f6e73_jit_lock__run_functions_0 () at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/jit-lock-8a988e43-a9956d8b.eln
#20 0x000055555570a61e in Ffuncall (nargs=3, args=0x7fffffff85a0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#21 0x00007fffeedcf59a in F6a69742d6c6f636b2d666f6e746966792d6e6f77_jit_lock_fontify_now_0 () at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/jit-lock-8a988e43-a9956d8b.eln
#22 0x000055555570a61e in Ffuncall (nargs=3, args=0x7fffffff86b0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#23 0x00007fffeedce98f in F6a69742d6c6f636b2d66756e6374696f6e_jit_lock_function_0 () at /usr/bin/../lib/emacs/29.3/native-lisp/29.3-4b59e32f/preloaded/jit-lock-8a988e43-a9956d8b.eln
#24 0x000055555570a61e in Ffuncall (nargs=nargs@entry=2, args=args@entry=0x7fffffff8770) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:2999
#25 0x00005555557f28d1 in internal_condition_case_n.constprop.0
    (bfun=0x55555570a520 <Ffuncall>, nargs=nargs@entry=2, args=args@entry=0x7fffffff8770, hfun=hfun@entry=0x5555555a1790 <safe_eval_handler>, handlers=0x30)
    at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:1558
#26 0x00005555555a190b in safe__call (inhibit_quit=inhibit_quit@entry=false, nargs=2, func=<optimized out>, ap=ap@entry=0x7fffffff87d0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:3059
#27 0x00005555555a7a10 in safe_call (nargs=<optimized out>, func=<optimized out>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:3074
#28 0x00005555555abd96 in safe_call1 (arg=0x6, fn=<optimized out>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:3085
#29 handle_fontified_prop (it=0x7fffffff89c0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:4601
#30 0x00005555555b1a76 in handle_stop (it=0x7fffffff89c0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:4134
#31 0x00005555555a8976 in start_display (it=0x7fffffff89c0, w=0x55555653c8e8, pos=...) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:3751
#32 0x00005555555caaf9 in try_window (window=window@entry=0x55555653c8ed, pos=..., flags=flags@entry=1) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:20748
#33 0x00005555555d2a02 in redisplay_window (window=<optimized out>, just_this_one_p=<optimized out>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:20140
#34 0x00005555555c34c3 in redisplay_window_0 (window=window@entry=0x55555653c8ed) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:17623
#35 0x0000555555707aec in internal_condition_case_1 (bfun=0x5555555c3490 <redisplay_window_0>, arg=0x55555653c8ed, handlers=<optimized out>, hfun=0x5555555c34d0 <redisplay_window_error>)
    at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:1498
#36 0x00005555555c3418 in redisplay_windows (window=0x55555653c8ed) at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:17592
#37 0x00005555555c715c in redisplay_internal () at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:17042
#38 0x0000555555679d5e in redisplay () at /usr/src/debug/emacs/emacs-29.3-wayland/src/xdisp.c:16225
#39 read_char (commandflag=1, map=0x5555585f1d23, prev_event=0x0, used_mouse_menu=0x7fffffffe0db, end_time=0x0) at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:2636
#40 0x0000555555682ee1 in read_key_sequence (keybuf=<optimized out>, prompt=0x0, dont_downcase_last=<optimized out>, can_return_switch_frame=true, fix_current_buffer=true, prevent_redisplay=false) at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:10084
#41 0x000055555567282f in command_loop_1 () at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:1384
#42 0x0000555555707a57 in internal_condition_case (bfun=0x5555556725d0 <command_loop_1>, handlers=<optimized out>, hfun=0x555555670bd0 <cmd_error>) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:1474
#43 0x000055555566ddae in command_loop_2 (handlers=handlers@entry=0x90) at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:1133
#44 0x000055555570799a in internal_catch (tag=<optimized out>, func=0x55555566dd80 <command_loop_2>, arg=0x90) at /usr/src/debug/emacs/emacs-29.3-wayland/src/eval.c:1197
#45 0x00005555556710d9 in command_loop () at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:1111
#46 0x000055555580740b in recursive_edit_1.isra.0 () at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:720
#47 0x00005555556714b1 in Frecursive_edit () at /usr/src/debug/emacs/emacs-29.3-wayland/src/keyboard.c:803
#48 0x0000555555586d19 in main (argc=<optimized out>, argv=0x7fffffffe748) at /usr/src/debug/emacs/emacs-29.3-wayland/src/emacs.c:2521

The emacs build file doesn't seem to set the dependency package version.

[maxbrunsfeld]

Good point @MatthewGentoo - Do you know if there is a good tool for automatically detecting ABI changes in a header file? It would be cool to have our version bump script automatically bump the soname when warranted, before creating a new tag.

[thesamesam]

@maxbrunsfeld The output in #3296 (comment) is from abidiff (from libabigail) which is pretty good. Just in this instance, it didn't seem to catch the problem as an explicit breaking change (it noted diffs. but not that it was breaking). I've reported that upstream - who are great. There is also https://github.com/lvc/abi-compliance-checker.

[ObserverOfTime]

Breaking ABI compatibility is not an issue at all, especially pre-1.0.

It is an issue when there's no such indication.
It might be a good idea to revert 09d2b23 and reserve it for ABI 15.

It would be cool to have our version bump script automatically bump the soname when warranted

The soname matches the version minus the patch. (0.22)

[ObserverOfTime]

You can also try rebuilding emacs and see if that fixes it.

[jesse-c]

That didn't work for me, but hopefully does for others.

[ObserverOfTime]

There is also https://github.com/lvc/abi-compliance-checker.

That one catches the breaking change and produces a nice HTML report.
Unfortunately, we can't use the HTML report in CI (except as an artifact).
We might be able to use the XML report with some post-processing though.

abi-dumper /tmp/ts/0.22.2/lib/libtree-sitter.so -vnum 0.22.2 -o 0222.dump \
  -dir -public-headers "$PWD/lib/include" -include-paths "$PWD/lib/src;lib/include"
abi-dumper /tmp/ts/0.22.5/lib/libtree-sitter.so -vnum 0.22.5 -o 0225.dump \
   -dir -public-headers "$PWD/lib/include" -include-paths "$PWD/lib/src;$PWD/lib/include"
abi-compliance-checker -abi -l tree-sitter -old 0222.dump -new 0225.dump -check-private-abi # -report-format xml

However, it does not include line/column numbers in the report like abidiff does.

[ObserverOfTime]

To be precise: the soname is currently only the major version but the file extension also includes the minor so that multiple versions can be installed side-by-side. I would not mind changing the soname (or extension) to an arbitrary number but we will need a workflow that checks for breaking ABI changes so we know when to bump it.