chimehq / neon

A Swift library for efficient, flexible content-based text styling

Neon

A Swift library for efficient, flexible content-based text styling.

• Lazy content processing
• Minimal invalidation calculation
• Support for up to three-phase highlighting via fallback, primary, and secondary sources
• Support for versionable text data storage
• A hybrid sync/async system for targeting flicker-free styling on keystrokes
• tree-sitter integration
• Text-system agnostic

Neon has a strong focus on efficiency and flexibility. It sits in-between your text system and wherever you get your semantic token information. Neon was developed for syntax highlighting and it can serve that need very well. However, it is more general-purpose than that and could be used for any system that needs to manage the state of range-based content.

Many people are looking for a drop-in editor View subclass that does it all. This is a lower-level library. You could, however, use Neon to drive highlighting for a view like this.

Warning: The code on the main branch is still in beta. It differs significantly from the 0.6.x releases. Both your patience and bug reports are very appreciated.

Installation

dependencies: [
    .package(url: "https://github.com/ChimeHQ/Neon", branch: "main")
],
targets: [
    .target(
        name: "MyTarget",
        dependencies: [
            "Neon",
            .product(name: "TreeSitterClient", package: "Neon"),
            .product(name: "RangeState", package: "Neon"),
        ]
    ),
]

Concepts

Neon is made up of three parts: the core library, RangeState and TreeSitterClient.

RangeState

Neon's lowest-level component is called RangeState. This module contains the core building blocks used for the rest of the system. RangeState is built around the idea of hybrid synchronous/asynchronous execution. Making everything async is a lot easier, but that makes it impossible to provide a low-latency path for small documents. It is content-independent.

• Hybrid(Throwing)ValueProvider: a fundamental type that defines work in terms of both synchronous and asynchronous functions
• RangeProcessor: performs on-demand processing of range-based content (think parsing)
• RangeValidator: building block for managing the validation of range-based content
• RangeInvalidationBuffer: buffer and consolidate invalidations so they can be applied at the optimal time
• SinglePhaseRangeValidator: performs validation with a single data source (single-phase highlighting)
• ThreePhaseRangeValidator: performs validation with primary, fallback, and secondary data sources (three-phase highlighting)

Many of these support versionable content. If you are working with a backing store structure that supports efficient versioning, like a piece table, expressing this to RangeState can improve its efficiency.

It might be surprising to see that many of the types in RangeState are marked @MainActor. Right now, I have found no way to both support the hybrid sync/async functionality while also not being tied to a global actor. I think this is the most resonable trade-off, but I would very much like to lift this restriction. However, I believe it will require language changes.

Neon

The top-level module includes systems for managing text styling. It is also text-system independent. It makes very few assumptions about how text is stored, displayed, or styled. It also includes some components for use with stock AppKit and UIKit systems. These are provided for easy integration, not maximum performance.

• TextViewHighlighter: simple integration between NSTextView/UITextView and TreeSitterClient
• TextViewSystemInterface: implementation of the TextSystemInterface protocol for NSTextView/UITextView
• LayoutManagerSystemInterface, TextLayoutManagerSystemInterface, and TextStorageSystemInterface: Specialized TextKit 1/2 implementations TextSystemInterface
• TextSystemStyler: a style manager that works with a single TokenProvider
• ThreePhaseTextSystemStyler: a true three-phase style manager that combines a primary, fallback and secondary token data sources

There is also an example project that demonstrates how to use TextViewHighlighter for macOS and iOS.

TextKit Integration

In a traditional NSTextStorage-backed system (TextKit 1 and 2), it can be challenging to achieve flicker-free on-keypress highlighting. You need to know when a text change has been processed by enough of the system that styling is possible. This point in the text change lifecycle is not natively supported by NSTextStorage or NSLayoutManager. It requires an NSTextStorage subclass. Such a subclass, TSYTextStorage is available in TextStory.

But, even that isn't quite enough unfortunately. You still need to precisely control the timing of invalidation and styling. This is where RangeInvalidationBuffer comes in.

I have not yet figured out a way to do this with TextKit 2, and it may not be possible without new API.

Performance

Neon's performance is highly dependant on the text system integration. Every aspect is important as there are performance cliffs all around. But, priority range calcations (the visible set for most text views) are of particular importance. This is surprisingly challenging to do correctly with TextKit 1, and extremely hard with TextKit 2.

TreeSitterClient

This library is a hybrid sync/async interface to SwiftTreeSitter. It features:

• UTF-16 code-point (NSString-compatible) API for edits, invalidations, and queries
• Processing edits of String objects, or raw bytes
• Invalidation translation to the current content state regardless of background processing
• On-demand nested language resolution via tree-sitter's injection system
• Background processing when needed to scale to large documents

Tree-sitter uses separate compiled parsers for each language. There are a variety of ways to use tree-sitter parsers with SwiftTreeSitter. Check out that project for details.

Token Data Sources

Neon was designed to accept and overlay token data from multiple sources simultaneously. Here's a real-world example of how this is used:

• First pass: pattern-matching system with ok quality and guaranteed low-latency
• Second pass: tree-sitter, which has good quality and could be low-latency
• Third pass: Language Server Protocol's semantic tokens, which can augment existing highlighting, but is high-latency

Theming

A highlighting theme is really just a mapping from semantic labels to styles. Token data sources apply the semantic labels and the TextSystemInterface uses those labels to look up styling.

This separation makes it very easy for you to do this look-up in a way that makes the most sense for whatever theming formats you'd like to support. This is also a convenient spot to adapt/modify the semantic labels coming from your data sources into a normalized form.

Usage

TreeSitterClient

Here's a minimal sample using TreeSitterClient. It is involved, but should give you an idea of what needs to be done.

import Neon
import SwiftTreeSitter
import TreeSitterClient

import TreeSitterSwift // this parser is available via SPM (see SwiftTreeSitter's README.md)

// assume we have a text view available that has been loaded with some Swift source

let languageConfig = try LanguageConfiguration(
    tree_sitter_swift(),
    name: "Swift"
)

let clientConfig = TreeSitterClient.Configuration(
    languageProvider: { identifier in
        // look up nested languages by identifier here. If done
        // asynchronously, inform the client they are ready with
        // `languageConfigurationChanged(for:)`
        return nil
    },
    contentProvider: { [textView] length in
        // given a maximum needed length, produce a `Content` structure
        // that will be used to access the text data

        // this can work for any system that efficiently produce a `String`
        return .init(string: textView.string)
    },
    lengthProvider: { [textView] in
        textView.string.utf16.count

    },
    invalidationHandler: { set in
        // take action on invalidated regions of the text
    },
    locationTransformer: { location in
        // optionally, use the UTF-16 location to produce a line-relative Point structure.
        return nil
    }
)

let client = try TreeSitterClient(
    rootLanguageConfig: languageConfig,
    configuration: clientConfig
)

let source = textView.string

let provider = source.predicateTextProvider

// this uses the synchronous query API, but with the `.required` mode, which will force the client
// to do all processing necessary to satisfy the request.
let highlights = try client.highlights(in: NSRange(0..<24), provider: provider, mode: .required)!

print("highlights:", highlights)

Contributing and Collaboration

I would love to hear from you! Issues or pull requests work great. A Discord server is also available for live help, but I have a strong bias towards answering in the form of documentation.

I prefer collaboration, and would love to find ways to work together if you have a similar project.

I prefer indentation with tabs for improved accessibility. But, I'd rather you use the system you want and make a PR than hesitate because of whitespace.

By participating in this project you agree to abide by the Contributor Code of Conduct.

neon's Issues

Cannot invalidate highlight at startup and/or with manual text settings

I am experimenting with some code that is derived directly from the iOS sample code.
I am using a custom tree-sitter grammar I am building for the ABC music notation.
For reference, I am using a UITextView in a UIViewController, embedded in a UIViewControllerRepresentable.
Here is the whole code:
https://gist.github.com/nutsmuggler/c1d50e04c894324b1f0cd9b6a1502cfb

Highlighting is working, but only after I edit the text in the text view (a space or a delete are sufficient).

I tried to call the new .invalidate() method of TextViewHighlighter, but it has no effect.
For debugging purposes, I dispatched on the main queue with some delays, to make sure the text was there.
Here it is, it's very dirty but I wanted to see what happened:

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        DispatchQueue.main.asyncAfter(deadline: .now() + 2) {
            let text = self.textView.text
            self.textView.text = "jsk"
            DispatchQueue.main.asyncAfter(deadline: .now() + 2) {
                self.textView.text = text

                self.highlighter.invalidate()
            }
        }
    }

It looks like highlighting invalidations is not triggered when the text is modified programmatically.

Variable 'version' needs fix

Version variable is always assigned to zero and does not have any effect.

#11 (comment)

Screenshot link for "painting it black" in README broken

The reference of @danielpunkass's repo and its readme-example folder for the "Paint it Black" screenshot is broken.

Text highlighting after edit does not work

When adding new text, it is not styled. It will still get styled if the scrollview is resized. It looks like commit 45d3d62 is the last with working styling after typing. This issue also occurs in the demo project

async version of currentTree

Hi I have this snippet in my code

extension TreeSitterClient {
    func currentTree() async throws -> Tree? {
        try await withCheckedThrowingContinuation { continuation in
            currentTree() { result in
                switch result {
                case .failure: continuation.resume(returning: nil)
                case .success(let tree):
                    continuation.resume(returning: tree)
                }
            }
        }
    }
}

Do you think it would make sense to have this function in the repo? I can open a PR

Direct text storage attribute modifications

#28 and #31 brings up the common problem of attribute invalidation. Using temporary attributes has a big advantage, because it gives you another "layer" of attributes.

Many users have run into issues when not using temporary attributes in the past. Let's talk about possible solutions.

'staleContent' error in highlighter

I am getting an error while updating the font size of textview

self.textView.font = .systemFont(ofSize: size, weight: .regular)

The error says failed to get tokens: staleContent

---

Tried debugging and noticed that the issue does not occur in the Example project. While comparing the stack side by side by putting breakpoints on the line which prints error, it seems like the guard comparison fails on the following line

Neon/Sources/TreeSitterClient/TreeSitterClient.swift

Line 350 in 4bc0fe4

guard startedVersion == self.version else {

In debugger startedVersion = 2 and self.version = 3. What could be the probable reason?
The code works fine despite the error

Crash on Updating Text from updateNSView(_ view: NSTextView, context: Context)

I am using NSTextView in SwiftUI and using TextViewHighlighter for Syntax Highlighting. It works well if we dont update the text from func updateNSView(_ view: JSONEditor, context: Context)
but if we do then I am getting a crash everytime
Thread 1: Assertion failed: range must not exceed limit in RangeMutation at line number 24
(assert(range.max <= l, "range must not exceed limit"))

I am pasting the screenshot of call stack

Just let me know if you need anything else from my end to fix this or any idea how to fix this would be a great help.

Repo example code returns nil for highlight method

I'm trying to use the example code provided in the repo. For some reason that is not clear to me, the following line will return nil (and in the example case, crash because it's force unwrapped)

let highlights = try client.highlights(in: NSRange(0..<24), provider: provider, mode: .required)!

I had a look at the code but couldn't find an immediate cause for it. I'm using the main branch to test this, as it's the one used in the example code.

I also tried using the sample project. There, the TextViewHighlighter example seems to work as expected. The other method provided (setUpTreeSitter) has the same issue though.

Am I doing anything wrong here? Or should I stick with 0.6.0 for now?

Proposal : TextViewSystemInterface to class

Neon/Sources/Neon/TextViewSystemInterface.swift

Line 15 in 4bc0fe4

public let textView: TextView

Hi. Will it make sense to convert TextViewSystemInterface to class and mark textView as a weak property?

Validation/Invalidation coordination problem with TreeSitterClient and TextSystemStyler

There is currently no good way to coordinate changes between TreeSitterClient and a TextSystemStyler.

On change both a client and the styler need to be updated.

• If you update the styler first, it will end up trying to validate synchronously (ok), but the client hasn't yet been updated so that fails.
• If you update the client first, it will produce invalidations that will result in the styler performing work without yet processing the change.

This coordination problem existed with Neon 0.6 as well. The solution was either to wait for the next runloop turn to apply styling (easy, could result in styling flicker), or to use precise NSTextStorage/NSLayoutManger event hooks that aren't easy to set up (without TSYTextStorage from TextStory).

SwiftUI example

Hi there,

Is there a SwiftUI example project?

I'm having difficulties to get going with this. I'm using a NSTextViewDelegate in a SwiftUI project.

Any help appreciated.

TextStorageSystemInterface does not style textedits that are not at end of string

When using TextStorageSystemInterface, initial highlighting works fine. Adding text to the end of the file works fine too. Adding text somewhere else doesn't seem to highlight it.

I did some digging and couldn't find an immediate reason why it's happening. In the beginValidation function (RangeValidator.swift), the following line will return .none, which is probably unexpected.

guard let neededRange = nextNeededRange(in: set, prioritizing: range) else { return .none }

Reproducing the issue: open the sample project, in TextViewSystemInterface comment out the first two interfaces in effectiveInterface, so the TextStorageSystemInterface is used. Then, try editing the text somewhere in the middle.

Coalesce attribute changes into one block

I noticed that multiple changes to the attributes during the same edit operation aren't batched up.

See:

1. beginEditing
2. (apply styles here, potentially nesting calls)
3. endEditing

Once the outermost pair of these calls finishes, delegates and the layout manager are notified.

TextSystemInterface.applyStyles(to:) loops over all tokens in a batch and applies them to the text. This would reduce NSTextStorageDelegate callbacks a lot already if we wrap the loop in begin/end editing calls.

But when pasting longer parts of text, applyStyles(to:) is called multiple times. I believe the architecture around that is mostly to support asynchronous highlighting?

So -- I don't yet see a way to make e.g. pasting the whole Highlighter.swift file into the sample app produce only one such delegate x (and layout manager) callback.

@mattmassicotte Would you still be interested in batch-processing changes? If so, I'm not sure how to best model this 🤔 Suggestions:

• Add a function to the TextSystemInterface protocol like groupStyleChanges(_ unitOfWork: () -> Void) and surround the for-in-loop. TextViewSystemInterface would need to provide the implementation, then, because it has access to the text storage.
• Send notifications before/after the loop instead. Would not match the rest of the architecture IMHO.
• Execute (optional?) begin/end functions instead. Could also be closure properties of the TextSystemInterface protocol. Would require 2 functions or 2 properties, though, and IMHO that's a weird amount of API surface . Also conforming types need to declare stored properties, so it's also annoying :)

Rendering HTML

I am interested in using a project like this that uses treesitter for a Vapor web application.
I am still pretty new to the world of Swift but I was trying to reason about if the APIs provided in this library could be used to render HTML (and if this can even run on the server)

I would love some thoughts on what HTML rendering might look like and if it would use Neon or the lower level SwiftTreeSitter

Crash on async access

Crash happens on async access to currentTree

Created a minimum reproducible https://github.com/kaunteya/TreeSitter-crash-minreprex

Demo crashes when editing text

The demo project provided in the repo will crash when trying to edit the text.
Some checks seem to fail, and it looks like it depends where in the text something is edited.

Here are the cases that I found:

• Put cursor at end of last line, press enter:
  assertionFailure (RangeMutation.swift -> transform(location:), line 84)

• Put cursor after let, type a:
  assertionFailure("location is greater than end") (String+Data.swift -> data(at:limit:using:chunkSize:), line 19)