let-def / texpresso

TeXpresso: live rendering and error reporting for LaTeX

TeXpresso: live rendering and error reporting for LaTeX

Note: TeXpresso is still in an early development phase. Changes and bug fixes are happening frequently, check the CHANGELOG.md.

Important: this repository uses submodules. Clone using git clone --recurse-submodules.

About

TeXpresso provides a "live rendering" experience when editing LaTeX documents in a supported editor: change something in the .tex file, the render window will update almost immediately with your change. Write something invalid, you get an error message immediately.

This can radically improve the LaTeX editing experience compared to the usual rebuild-and-wait-for-viewer-to-update experience, especially for large documents.

See the screencasts at the end of this file for a visual demo of TeXpresso capabilities.

Install

TeXpresso has been tested on Linux and macOS and should work with both AMD64 and Apple Silicon architectures. See INSTALL.md for dependency and build instructions.

Design

The TeXpresso system is built of the following parts:

1. A TeX engine that renders LaTeX documents into PDF; we use a modified version of the Tectonic engine, modified to interact with the TeXpresso driver.

   This is in the tectonic/ git-submodule, and it produces the texpresso-tonic helper binary

2. A PDF renderer that renders PDF documents into images. We use MuPDF.

3. A viewer that shows the rendered images and allows simple user commands (see Viewer controls below), built with libSDL.

4. A driver program that talks to the editor to be notified of changes to the LaTeX document, maintains an incremental view of the document and the rendering process (supporting incrementality, rollback, error recovery, etc.), talks to the LaTeX engine to re-render the modified portions of the document, and synchronizes with the viewer.

   The driver is where the "live" magic lives. It is the texpresso binary, whose sources are in this repository.

The driver sends information between the editor and the renderer in both directions. In particular, it is possible to ask the editor to jump to a specific place in the LaTeX document by clicking on the viewer window or, conversely, to refresh the viewer window to display the document at the editor position.

Viewer controls

Keyboard controls:

• ←, →: change page
• p (for "page"): switch between "fit-to-page" and "fit-to-width" zoom modes
• c ("crop"): crop borders
• q ("quit"): quit
• i ("invert"): dark mode
• I : toggle theming
• t ("top"): toggle stay-on-top (keeping TeXpresso above the editor window)
• b ("border"): toggle window borders
• F5: start fullscreen presentation (leave with ESC)

Mouse controls:

• click: select text in window (TODO: move Emacs buffer with SyncTeX)
• control+click: pan page
• wheel: scroll page
• control+wheel: zoom

Supported editors

Emacs

TeXpresso comes with an Emacs mode. The source can be found in emacs/texpresso.el. Load this file in Emacs (using M-X load-file; it is also compatible with require).

Start TeXpresso with M-x texpresso. The prompt will let you select the master/root TeX file. It will try to start the texpresso command. If it is not possible, it will open (customize-variable 'texpresso-binary) to let you set the path to texpresso binary (<where you cloned the repository>/build/texpresso).

To work correctly, texpresso needs texpresso-tonic helper; when copying them, make sure they are both in the same directory.

M-x texpresso-display-output will open a small window listing TeX warnings and errors on the current page. Use M-x texpresso-next-page and M-x texpresso-previous-page to move between pages without leaving Emacs.

Neovim

A Neovim mode is provided in a separate repository texpresso.vim. It is not yet compatible with vanilla Vim, patches are welcome 🙇.

Screencasts

Neovim integration. Launching TeXpresso in vim:

txp-start.mp4

Live update during edition:

txp-edit.mp4

Using Quickfix window to fix errors and warnings interactively:

txp-quickfix.mp4

Synchronization from Document to Editor (SyncTeX backward):

synctex-backward.mp4

Synchronization from Editor to Document (SyncTeX forward):

synctex-forward.mp4

Theming, Light/Dark modes: 😎

txp-theme.mp4

Emacs integration. Here is a sample recording of me editing and browsing @fabiensanglard Game Engine Black Book: Doom in TeXpresso (using my emacs theme):

compress.mp4

texpresso's Issues

Clicking on references/links?

Does the TeXpresso viewer support jumping to the destination of a reference?

(When working on scientific LaTeX documents it is very useful to be able to follow references, and also to have a shortcut to go back to where we where before following it.)

`-I` argument: support load-path customisation

Add -I dir to look up source files in other directories.

Aborts on test

It compiles, but when trying to run the test I get an abort and a coredump (NB: the "IOT instruction" means SIGABRT).

I run Arch btw. The same behaviour occurs when running texpresso installed via the PKGBUILD.

~/src/texpresso % build/texpresso test/simple.tex
[info] working directory: /home/philipp/src/texpresso
[info] executable path: /home/philipp/src/texpresso/build/texpresso
[info] document path: /home/philipp/src/texpresso/test
[info] document name: simple.tex
texpresso logo: 512x512
[info] tectonic path: /home/philipp/src/texpresso/build/texpresso-tonic
[1]    673395 IOT instruction (core dumped)  build/texpresso test/simple.tex

~/src/texpresso % coredumpctl info
           PID: 673395 (texpresso)
           UID: 1000 (philipp)
           GID: 1000 (philipp)
        Signal: 6 (ABRT)
     Timestamp: Mon 2023-08-07 13:35:20 CEST (1min 5s ago)
  Command Line: build/texpresso test/simple.tex
    Executable: /home/philipp/src/texpresso/build/texpresso
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (philipp)
       Storage: /var/lib/systemd/coredump/core.texpresso.1000.c93ab394c2f04b39a352be378571b081.673395.1691408120000000.zst (present)
  Size on Disk: 27.0M
       Message: Process 673395 (texpresso) of user 1000 dumped core.

                Stack trace of thread 673395:
                #0  0x00007ff750c8e83c n/a (libc.so.6 + 0x8e83c)
                #1  0x00007ff750c3e668 raise (libc.so.6 + 0x3e668)
                #2  0x00007ff750c264b8 abort (libc.so.6 + 0x264b8)
                #3  0x00005619af1b5815 n/a (/home/philipp/src/texpresso/build/texpresso + 0x48815)
                #4  0x00005619af1ba7f3 n/a (/home/philipp/src/texpresso/build/texpresso + 0x4d7f3)
                ELF object binary architecture: AMD x86-64

Install error

make -C src texpresso
make[1]: 进入目录“/home/paren/texpresso/src”
gcc -O2 -ggdb -I. -fPIC -c -o ../build/objects/driver.o -Idvi/ driver.c
gcc -O2 -ggdb -I. -fPIC -c -o ../build/objects/main.o -Idvi/ main.c
main.c: In function ‘find_diff’:
main.c:514:27: error: dereferencing pointer to incomplete type ‘fz_buffer’ {aka ‘const struct fz_buffer_s’}
514 | int i, len = fz_mini(buf->len, size);
| ^~
main.c: In function ‘realize_change’:
main.c:558:19: error: dereferencing pointer to incomplete type ‘fz_buffer’ {aka ‘struct fz_buffer_s’}
558 | uint8_t *p = b->data;
| ^~
make[1]: *** [Makefile:34：../build/objects/main.o] 错误 1
make[1]: 离开目录“/home/paren/texpresso/src”

libmupdf-dev 已经是最新版 (1.16.1+ds1-1ubuntu1)。
libsdl2-dev 已经是最新版 (2.0.10+dfsg1-3)。

SyncTeX: indicate when a file is closed

There is no way usinc SyncTeX output to tell when we are done with an input file.

This can be a problem when forwarding a position from the editor: if the position is beyond the last line reported by any SyncTex entry, TeXpresso has no way to tell when it can stop searching, until reaching the end of the whole document.
This is wasteful for large documents, and it can happen a lot when the document is split in multiple files.

A solution would be to add a SyncTeX entry indicating that an input file has been closed.
Alternatively, TeXpresso could record when it receives a "close" request and correlate this with SyncTeX output, but that is much more complicated.

Language Server Protocol support

Hey!

I see that texpresso supports multiple different editors via ad-hoc plugins made separately for each one. Would it make sense to unify the interface and provide texpresso as an LSP?

This way, it would trivially work with many different editors with no need to add support one-by-one. And the LSP interface seems sufficient for texpresso's functionality.

Ivan

SyncTeX forward synchronization

SyncTeX backward synchronization is implemented: our quick'n'dirty synctex parser does a good enough job that we can find source files when clicking on the window.

Synchronizing in the other direction is more complex because there needs to be coordination between synctex and the engine (progress as long as the forwarded position has not been found), and it should happen concurrently (the GUI should stay responsive) and be cancellable (maybe the position is nowhere to be found because it refers to a file that is not used in the document; the user will move to some file and we need to resume work as if nothing happened).

VIM mode

It would be nice to have a (neo)vim mode :).

It is probably quite easy on TeXpresso side (one would have to implement a parser for sexp-based TeXpresso protocol, that I should document!), but requires someone knowledgeable with (modern) vim internals.

UI: shortcuts cheatsheet

Increase discoverability by drawing an help screen, for instance when pressing '?'.

Install problem

What is the cause of this error and how can I solve it? Thanks!

########################################################################
*make texpresso
make[1]: Entering directory '/home/yangdewu/texpresso'
make -C src texpresso
make[2]: Entering directory '/home/yangdewu/texpresso/src'
gcc -O2 -ggdb -I. -fPIC -c -o ../build/objects/main.o -Idvi/ main.c
main.c: In function ‘find_diff’:
main.c:516:27: error: dereferencing pointer to incomplete type ‘fz_buffer’ {aka ‘const struct fz_buffer_s’}
516 | int i, len = fz_mini(buf->len, size);
| ^~
main.c: In function ‘realize_change’:
main.c:560:19: error: dereferencing pointer to incomplete type ‘fz_buffer’ {aka ‘struct fz_buffer_s’}
560 | uint8_t p = b->data;
| ^~
main.c: In function ‘interpret_command’:
main.c:866:7: warning: implicit declaration of function ‘SDL_SetWindowAlwaysOnTop’ [-Wimplicit-function-declaration]
866 | SDL_SetWindowAlwaysOnTop(ui->window, SDL_TRUE);
| ^~~~~~~~~~~~~~~~~~~~~~~~
main.c: In function ‘texpresso_main’:
main.c:1190:44: error: ‘SDL_MouseWheelEvent’ {aka ‘struct SDL_MouseWheelEvent’} has no member named ‘preciseX’
1190 | ui_mouse_wheel(ps->ctx, ui, e.wheel.preciseX, e.wheel.preciseY, mx, my,
| ^
main.c:1190:62: error: ‘SDL_MouseWheelEvent’ {aka ‘struct SDL_MouseWheelEvent’} has no member named ‘preciseY’
1190 | ui_mouse_wheel(ps->ctx, ui, e.wheel.preciseX, e.wheel.preciseY, mx, my,
| ^
make[2]: *** [Makefile:34: ../build/objects/main.o] Error 1
make[2]: Leaving directory '/home/yangdewu/texpresso/src'
make[1]: *** [Makefile:8: texpresso] Error 2
make[1]: Leaving directory '/home/yangdewu/texpresso'
make: *** [Makefile:2: all] Error 2
#########################################################

Workaround for SyncTeX and (fragile) Beamer slides

When fragile, Beamer uses an intermediate file to temporarily save the content of the slide.

This cause SyncTeX to incorrectly report the location of this file rather than the original source. This file will have disappeared by the time SyncTeX output is processed, and the same filename is used for all slides, so SyncTeX is useless in this situation.

However TeXpresso has a detailed view of the whole process: when the files are created and removed, and which instance of the file was "alive" when a specific SyncTeX instruction was emitted. By correlating all these informations, it should be possible to recreate an approximate location of the actual source.

Minimal test faild, pdftex.map can't download

[info] working directory: /Users/projects/personal/texpresso
[info] executable path: /Users/projects/personal/texpresso/build/texpresso
[info] document path: /Users/projects/personal/texpresso/test
[info] document name: simple.tex
texpresso logo: 512x512
[info] tectonic path: /Users/yin/projects/personal/texpresso/build/texpresso-tonic
[dvi] loading pdftex.map
note: "version 2" Tectonic command-line interface activated
note: not in a document workspace; using the built-in default bundle
note: downloading SHA256SUM
note: downloading pdftex.map
warning: failure downloading "pdftex.map" from network
caused by: error decoding response body: operation timed out
caused by: operation timed out

Dynamically change loaded document

Right now the only way to display another TeX file is to start a new process.
Having a command to load a different document without restarting anything, recreating a window etc, could come handy.

\cite not work

Thank you for the amazing project!

But I meet problems in citation. I use \cite to cite references in .bib file. References can't be found in TeXpresso. The same project works well in Overleaf.

UI: Display a spinner during initialization

On first start, TeXpresso downloads necessary TeX/LaTeX files from Tectonic repository.
This can take some time, during which TeXpresso appears to be frozen.
We should display some information or a spinner to keep users waiting.

One difficulty is that there are two sources of downloads and none of them have an easy to display information:

• from tectonic process downloading packages and metrics
• from renderer downloading graphical resources (mostly fonts)
  On tectonic side, we have no direct connection to the UI. On renderer side, we are in a synchronous function waiting to load a font, from which it would be difficult to update the window.

As a first approximation, I am thinking to display a simple spinner with a fixed message when there is nothing else to display (document is being processed and no page has been produced yet).

Adding file to the filesystem can cause crashes

If TeXpresso tries to read foo.tex (for instance because it is processing \include{foo.tex}), when foo.tex does not exist at first, and it is then created, TeXpresso will crash.

Single-threaded downloader

Tectonic uses Reqwest to download TeX data, and Reqwest uses a background thread for downloading data.

On Linux, stopping the thread before forking was sufficient to get a correct behavior.
On macOS, forking is not possible from a multi-threaded process.

TeXpresso needs another backend. My thought:

• create a downloading process, by forking early during the initialization
• define some ad-hoc protocol that workers can use to request files from the downloading process

Race condition with sub-processes

Snapshotting is done by forking the TeX process (the worker) at relevant points. All processes share the same line of communication, a bidirectional pipe.
A parent wait for their child to finish before resuming communication, therefore at any point in time, the pipe is morally owned by the most recent child process. In the meantime, the TeXpresso driver can also decide to kill sub-processes if their work is no longer needed.

This happens to cause race conditions that are difficult to precisely pinpoint. Fortunately they happen quite rarely, but the end result is a desynchronisation between the driver and the workers (guesses: partial writes by workers on the communication pipe, likely due to the delay between the decision to kill a worker and the kill signal being delivered).

Short term: Right now, the driver aborts when this happen. It would be better for the end-user if, after logging the error, it stayed alive and restarted document processing from scratch.

Long term: The shared pipe approach is inherently brittle, I am tempted to switch to UNIX domain sockets and pass a unique FD to each worker.
On top of robustness, this would simplify the workers implementation since they are relieved of process management. Each could work in isolation, with the driver being the only one aware of the running jobs.
It would also fix the linear nature of the current snapshots. By using only fork and wait, processes always form a stack: the last forked process will be the first to finish ("LIFO"). This is not efficient when processing long documents: when editing the first pages, it might seem relevant to fork the worker. Editing the following pages, a new fork might seem relevant again. Etc. We keep forking, but if we never go back, we keep pushing on the stack and there is no opportunity to cleanup the processes: either we abuse resources (processes and memory), or we have to decide at some point to backtrack, cleanup the stack, and start processing the document from an old checkpoint, temporarily making the application unresponsive.

PDF graphics interpreter: handle shading, XObjects

Our PDF graphics interpreter only handles basic primitives.

TikZ/PGF (and hence beamer) often uses shading. The difficulty with this is that is it not done using new instructions (that we could implement), but by defining and referencing resources external to the instruction stream.
Our interpreter lacks both features: no notion of resources, no way to define nor reference them.

Question: Does texpresso works well on IEEE/Elesvier etc. publication template?

as title.

UI: continuous pages

Add an option to stack all pages vertically and continuously scroll between them.

build error

22.04.1-Ubuntu SMP PREEMPT_DYNAMIC 2 x86_64 x86_64 x86_64 GNU/Linux

make config
make clean  texpresso

error message

rm -rf build/objects/*
make -C src texpresso
make[1]: Entering directory '/home/quchunhe/Git/texpresso/src'
gcc -O2 -ggdb -I. -fPIC -c -o ../build/objects/driver.o -Idvi/ driver.c
gcc -O2 -ggdb -I. -fPIC -c -o ../build/objects/main.o -Idvi/ main.c
main.c:795:1: error: static declaration of ‘SDL_SetWindowAlwaysOnTop’ follows non-static declaration
795 | SDL_SetWindowAlwaysOnTop(SDL_Window *window, SDL_bool state)
| ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/SDL2/SDL_events.h:33,
from /usr/include/SDL2/SDL.h:41,
from main.c:25:
/usr/include/SDL2/SDL_video.h:1123:30: note: previous declaration of ‘SDL_SetWindowAlwaysOnTop’ with type ‘void(SDL_Window *, SDL_bool)’
1123 | extern DECLSPEC void SDLCALL SDL_SetWindowAlwaysOnTop(SDL_Window * window,
| ^~~~~~~~~~~~~~~~~~~~~~~~
make[1]: *** [Makefile:34: ../build/objects/main.o] Error 1
make[1]: Leaving directory '/home/quchunhe/Git/texpresso/src'
make: *** [Makefile:8: texpresso] Error 2

missing dependencies in Ubuntu install instructions

While trying to install this on Ubuntu 22.04.4 LTS, I was missing dependencies leptonica and harfbuzz and it was not so clear to me from the instructions that I would need to install them, and how.
Do you want to add the packages libleptonica-dev and libharfbuzz-dev to this line ?

Unable to test TeXpresso

I am reporting an issue as I am unable to perform the test build/texpresso test/simple.tex

After running, the program enters an infinite loop with the last few lines as follows:

success:1 size:35469
[dvi] loading cmmi8
success:1 size:1520
[dvi] loading cmmi8
success:0 size:9
bundle_serve_hooks_cat: error loading cmmi8.vf: not found
(flush)

optimized bounds: 0.000000,0.000000 - 700.000000,900.000061

I am running on Linux Mint 21.2 Cinnamon, with x86_64.
The model is Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz

SDL Window disables composition under KWin by default

Hello, first I would like to thank you for this great software! :)
Made editing LaTeX files in Emacs really pleasant experience.

The default behavior for SDL2 window is to disable composition if the Window Manager understands the hint.
KDE's KWin does (probably is in the minority). The workaround is to either set the environment variable:
export SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR=0

or execute SDL_SetHint(SDL_HINT_VIDEO_X11_NET_WM_BYPASS_COMPOSITOR, "0");
before creating the window as someone mentioned here: mosra/magnum#184 (comment)

Not sure what version of SDL you are targeting so I'm not expecting you'll add this line of code - just wanted to put it here so others may find it if they would like to workaround the issue with the env. variable.

Thanks,
Adam.

Root file should exist or TeXpresso will just give up

When starting the process with texpresso foo.tex, foo.tex should already exist.
This is understandable when used as a standalone viewer, but not ideal when remote controlled from an editor: the buffer might have been created in the editor but not yet saved on disk.

TeXpresso protocol allows the editor to communicate the contents of the buffer without writing to disk, however the file name should at least exist on disk (a simple touch foo.tex is sufficient).

When remote controlled, TeXpresso should not exit and rather behave as if the file existed and was empty (waiting for the editor to communicate the actual contents).

Downloading new Tectonic files is broken

When using a new package or a feature that requires downloading new resources to the tectonic cache, TeXpresso is likely to freeze.
This is likely because of a bad interaction between forking and downloading: the socket used for downloading files will be shared by all fork of the worker, quickly leading to corruption.
It could also be the cache that gets corrupted: file A is missing, process P2 is forked from P1, P2 downloads A, later P2 dies and P1 is resumed: it still thinks A is missing from cache, potentially corrupting some internal state. (Less likely, downloading file should be idempotent, so at worst the file is downloaded twice).

I am not yet sure how to address that: tectonic code needs to be instrumented to close the downloading socket after a fork. The drawback is that we might start more short-lived TCP connections to download files. This is probably not a problem (it happens at most once per file, and this is already what happens when using tectonic -X bundle cat).
Alternatively, this could be handled by forking before downloading, so that a single downloading process is shared by all the workers. The drawback is more IPC.

I tend to favor the first option (short-lived TCP connections).

vscode editor support

Hey!

Nice to see this project! I seen texpresso support Emacs and Neovim.
Have any plan to support vscode?

Yin

Use sequence alignment to refine SyncTeX position

The source positions reported by SyncTeX are quite imprecise (one could say they suck :P).

Once we have a located the candidate lines (a mapping from a range of lines to a range of pages), we could use a sequence alignment algorithm to find a fine-grained mapping from source code characters to DVI characters.

UI: Move with HJKL

Scroll the pages using keyboard with HJKL.
Also add the possibility to scroll by a screen/half a screen.

SyncTeX backward synchronization sometimes off-by-one

Our SyncTeX parser often jumps one line before the right position, but not always.
I don't know why, nor if this behavior also happens with the official parser.

Error report: batch changes and compute delta

Logging of TeX messages to the editor is almost append-only: during a run, content can only be appended, but when backtracking the current output is truncated, before appending again.
To prevent flickering or accidentally scrolling the output, it is necessary for the editor to wait a bit before reflecting these changes.

The heuristic implemented by emacs mode is to truncate only at the end of a run, and replace the output line by line during the run. This provides a decent look'n'feel, but can still feel slow if the output is updated too fun, without any benefits (e.g to receive 100 lines, it can be the case that the editor refresh the screen 100 times, once per line).

This heuristic with extra batching (e.g. wait for a buffer to fill up and delay updates by a bunch of milliseconds) is probably better implemented in TeXpresso itself rather than in the editor mode. Furthermore, if other editors were to be supported in the future (e.g. (n)vim), it is better to place as much logic as possible in TeXpresso itself.

Snapshot positioning is too naive

To observe the problem:

1. Edit a line by inserting contents, from left-to-right, for some time
2. Go up a few lines, and start editing from that point.

Step 1. will happen smoothly: when adding consecutive characters, a snapshot very close to the text caret is ready to reprocess the current page. Step 2. is likely to cause a slowdown: not problematic, but enough to be observable (100-200ms on my tests).

This situation should be better handled, for instance by keeping a snapshot close to the beginning of the page. The problem is that right now, the positioning is too naive: even if we keep enough snapshots, they are either too close or too far.

BibTex support?

Hi,
Thanks for the nice work! I am trying to use it for a journal paper, but it doesn't properly render the citations (I use .bibtex). Here is the template package I am using: https://www.ctan.org/tex-archive/macros/latex/contrib/mnras

Thanks.