Is `runLexer` incremental? about lexer-applicative

[UnkindPartition]

Thank you for this lovely work!

Glad you like it!

As the title, I wonder if runLexer, or the whole package in general, is capable of converting a stream of chars into tokens incrementally?

Depends on what precisely you mean by "incrementally", but I'd say yes — in the sense that it emits tokens as soon as they are recognized, without consuming the whole input, or any more input than necessary:

import Language.Lexer.Applicative
import Text.Regex.Applicative
import Data.Char

lexer =
  token (longest (psym (not . isSpace))) <>
  whitespace (longest (psym isSpace))

main = do
  -- Normal input
  putStrLn . take 30 . show $ runLexer lexer "-" "a a a"
  -- Infinite input
  putStrLn . take 30 . show $ runLexer lexer "-" (cycle "a ")
  -- Input ending with an error
  putStrLn . take 30 . show $ runLexer lexer "-" ("a a a " ++ undefined)

[banacorn]

That's great!
Because I want to feed those tokens to attoparsec, otherwise it would defeat the purpose of paring with an incremental parser.