s-expressionists / eclector Goto Github PK

By printing signaled conditions whenever an operation is supposed to signal.

If the value of *read-suppress* is true, read, read-preserving-whitespace, read-delimited-list, and read-from-string all return a primary value of nil when they complete successfully

Is it implemented?

beach want's to talk with Bike and symtym about precisely which error and so I will leave this here.

The with-preservered-backquote-context can be moved out of the if at the same time.

(defun sharpsign-dot (stream char parameter)
  (declare (ignore char))
  (unless (null parameter)
    (warn 'numeric-parameter-supplied-but-ignored
          :parameter parameter
          :macro-name 'sharpsign-dot))
  (with-preserved-backquote-context
      (if *read-suppress*
          (progn
            (read stream t nil t)       ; throw away the result
            (values))
          (eval (read stream t nil t)))))

Consider: (newest eclector, current sbcl)

("eclector")
* (let ((*read-base* 4))
           (eclector.reader:read-from-string "2140.969"))
36.969
8
* (let ((*read-base* 4))(read-from-string "2140.969"))
2140.969
8
* (let ((*read-base* 4))
           (eclector.reader:read-from-string "2140"))
36
4

Seems that the digits before the dot are read with base 4, and the digits after the dot with base 10 (since they don't fit into base 4).

probabyly a problem for all read-base different from 10.

23.2.13 read-base says:
The value of read-base, called the current input base, is the radix in which integers and ratios are to be read by the Lisp reader. The parsing of other numeric types (e.g., floats) is not affected by this option.

So I believe it is an error to apply read-base to floats at all

Some clients implement custom behavior for packages and symbols by defining methods on interpret-symbol-token, interpret-symbol, wrap-in-quote and friends. These clients can have their own notion of the current package (e.g. stored in the client object). There is (at least) one blind spot though: the sharpsign-plus-minus macro function, when reading feature expressions, changes the current package by binding cl:*package*. A generic function (call-with-current-package client thunk package-designator) should be called instead.

(eclector.reader:read-from-string (string #\Backspace)) returns a symbol with
a single letter in its name. The way I read the standard is that it should
signal an error.

Came up in #52.

According to HyperSpec section 2.3.3 (The Consing Dot), a token consisting solely of multiple dots (more than one dot, no escapes) is illegal.

A suggestion: document methods which have default implementation with side effects. It would be useful for implementation of side-effect-free parser. I know of #. and interning of symbols, perhaps there are other corners of the Spec to be aware of?

i have trouble loading maxima with the cst version of clasp, while it loads fine with the ast-version.
The cst-version of clasp uses eclector.concrete-syntax-tree:cst-read to read lisp-sources.

The problem seems to be the following (tested with sbcl and eclector from git)

Define a marco character "_" (the silly pprints are just to test)

(ql:quickload :eclector)
(ql:quickload "eclector-concrete-syntax-tree")
(set-macro-character #\_ (lambda (stream char)
                             (pprint `(char = ,char))
                             (let ((was (peek-char nil stream nil nil t)))
                               (pprint `(peek = ,was))
			       (case was
				 (#\" (values))
				 (#\N (read-char stream t nil t) (values))
				 (otherwise '_))))
                       t)

Use that

* (read-from-string "_N\"The message-lookup\"")
(CHAR = #\_)
(PEEK = #\N)
"The message-lookup"
22

but than in eclector:

* (eclector.reader:read-from-string "_N\"The message-lookup\"")
_N
2
* (with-input-from-string (stream "_N\"The message-lookup\"")
  (eclector.concrete-syntax-tree:cst-read stream))
#<CONCRETE-SYNTAX-TREE:ATOM-CST raw: _N {10046D7143}>
NIL

Am I missing something or are user-defined macro-characters not used?

In eclector:
(input-stream &optional (eof-error-p t) eof-value recursive-p)
in clhs
21.2.17 read-char | Function
read-char &optional input-stream eof-error-p eof-value recursive-p char

The &rest arguments stuff is unhelpful for clients and handling it is more trouble than it used to be.

• sub-char-must-not-be-a-decimal-digit
• char-must-be-a-dispatching-character
• unknown-macro-sub-character

When taking the README example with the following input:

(with-input-from-string (stream "1 ") ;; with an extra space
  (eclector.parse-result:read (make-instance 'my-client) stream))

the source-location of the 1 is considered to be the range (0 . 2) where I think it should be (0 . 1).
I think removing this when might fix the issue, but I am not sure it serves another purpose.

SBCL:

(princ (read (make-string-input-stream "#\\Pile_of_Poo")))
| 💩
#\PILE_OF_POO

Eclector:

(princ (eclector.reader:read (make-string-input-stream "#\\Pile_of_Poo")))
|- ECLECTOR.READER:UNKNOWN-CHARACTER-NAME

But seriously, the client should probably have control over character lookup. Maybe a generic function interpret-character-name in the spirit of interpret-symbol.

Reproducible by e.g., (eclector.concrete-syntax-tree:cst-read (make-string-input-stream (format nil "(~{~A~^ ~})" (make-list 20000 :initial-element 1)))), though of course the required size might vary.

I believe this is due to the recursion in make-cons-cst in read-cst.lisp. On Clasp, the stack trace looks like

[...]
frame #19640: 0x000000010c7beaf8 clasp`::LAMBDA^COMMON-LISP^FN^^() at read-cst.lisp:0
frame #19641: 0x000000010c7beaf8 clasp`::LAMBDA^COMMON-LISP^FN^^() at read-cst.lisp:0
frame #19642: 0x000000010c7beaf8 clasp`::LAMBDA^COMMON-LISP^FN^^() at read-cst.lisp:0
frame #19643: 0x000000010c7beaf8 clasp`::LAMBDA^COMMON-LISP^FN^^() at read-cst.lisp:0
frame #19644: 0x000000010c7be2d7 clasp`::MAKE-EXPRESSION-RESULT^ECLECTOR.PARSE-RESULT^((CST-CLIENT T T T))^METHOD^^() at read-cst.lisp:0

i.e. the make-expression-result is the last intelligible call.

To consolidate duplicated code between (at least):

• sharpsign-backslash
• sharpsign-colon
• read-token

The symbol SET-SYNTAX-FROM-CHAR is shadowed and exported, but
the function with that name is not implemented.

Needed for implementation-specific extensions such as

#+(and some-lisp (some-lisp-version-at-least "1.2.3"))

Wrap the signaled condition in an Eclector condition?

I'd like to be able to parse a source file into a tree structure which includes all information required to build the source (including whitespace and comments). This would be easier if every parsed CST element has a valid associated source range. Currently that isn't the case for (at least) QUOTE elements. See:

SEL/LISP-DIFF> (labels ((maptree (fn tree)
                          (if (concrete-syntax-tree:consp tree)
                              (cons
                               (maptree fn (concrete-syntax-tree:first tree))
                               (maptree fn (concrete-syntax-tree:rest tree)))
                              (funcall fn tree))))
                 (maptree #'concrete-syntax-tree:source
                          (with-input-from-string (in "'(1 2 3 4)")
                            (cst-read in))))
(NIL ((2 . 3) (4 . 5) (6 . 7) (8 . 9)))

I've taken a shot at building this using cst-read and record-skipped-input with some custom code to fold everything together and add whitespace. See https://github.com/GrammaTech/sel/blob/master/ast-diff/lisp-diff.lisp#L71, is there a better way?

Thanks!

The sharpsign-a reader macro calls (length 1) at some point, signaling a type error instead of the appropriate eclector.reader:read-object-type-error.

Including signaled conditions and their reports.

• semicolon
• single-quote
• double-quote
• backquote
• comma
• left-parenthesis
• right-parenthesis
• sharpsign-single-quote
• sharpsign-left-parenthesis
• sharpsign-dot
• sharpsign-backslash
• read-rational
• sharpsign-b
• sharpsign-x
• sharpsign-o
• sharpsign-r
• sharpsign-asterisk
• sharpsign-vertical-bar
• sharpsign-a
• sharpsign-colon
• sharpsign-c
• sharpsign-p
• sharpsign-plus-minus
• sharpsign-invalid
• sharpsign-equals
• sharpsign-sharpsign

Not 100% sure about this, but it is a deviation from other readers
latest sbcl, latest eclector from git

* (eclector.reader:read-from-string  "-0.0")
0.0
4
but sbcl gives
* (Read-from-string "-0.0")
-0.0
4

• invalid-context-for-backquote
• comma-not-inside-backquote
• unquote-splicing-in-dotted-list
• unquote-splicing-at-top
• invalid-context-for-consing-dot
• consing-dot-most-be-followed-by-object
• multiple-objects-following-consing-dot
• invalid-context-for-right-parenthesis
• sub-char-must-not-be-a-decimal-digit
• char-must-be-a-dispatching-character
• symbol-does-not-exist
• symbol-is-not-external
• symbol-name-must-not-end-with-package-marker
• two-package-markers-must-be-adjacent
• two-package-markers-must-not-be-first
• symbol-can-have-at-most-two-package-markers
• uninterned-symbol-must-not-contain-package-marker
• numeric-parameter-supplied-but-ignored
• numeric-parameter-not-supplied-but-required
• unknown-character-name
• digit-expected
• invalid-radix
• invalid-default-float-format
• too-many-elements
• no-elements-found
• incorrect-initialization-length
• single-feature-expected
• sharpsign-invalid
• sharpsign-equals-label-defined-more-than-once
• sharpsign-sharpsign-undefined-label

In latest sbcl, eclector git master

Interaction of read-suppress and #R:
I believe no error should be signaled as READ-FROM-STRING from sbcl does
23.2.16 read-suppress
...
Except as noted below, any standardized reader macro2 that is defined to read2 a following object or token will do so, but not signal an error if the object read is not of an appropriate type or syntax.

* (WITH-STANDARD-IO-SYNTAX
  (LET ((*READ-SUPPRESS* T))
    (eclector.reader:READ-FROM-STRING "#0r0")))

debugger invoked on a ECLECTOR.READER:INVALID-RADIX in thread #<THREAD "main thread" RUNNING {10005185B3}>: 0 is too small to be a radix.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

(ECLECTOR.BASE:%READER-ERROR #<SB-IMPL::STRING-INPUT-STREAM {1004BA3233}> ECLECTOR.READER:INVALID-RADIX :RADIX 0)
   source: (APPLY (FUNCTION ERROR) DATUM :STREAM STREAM :STREAM-POSITION STREAM-POSITION (ALEXANDRIA.0.DEV:REMOVE-FROM-PLIST ARGUMENTS :STREAM-POSITION))
0] 0
* (WITH-STANDARD-IO-SYNTAX
  (LET ((*READ-SUPPRESS* T))
    (READ-FROM-STRING "#0r0")))
NIL
4

• eclector.parse-result
  • eclector.parse-result:read
  • eclector.parse-result:read-preserving-whitespace
  • eclector.parse-result:read-from-string
  • Update tests
  • Update documentation
• eclector.concrete-syntax-tree
  • eclector.concrete-syntax-tree:read
  • eclector.concrete-syntax-tree:read-preserving-whitespace
  • eclector.concrete-syntax-tree:read-from-string
  • Update tests
  • Update documentation
  • Deprecate eclector.concrete-syntax-tree:cst-read

First time: read-common
Second time: call-reader-macro

• Add tests
• Document
• Add example to documentation

I believe the following should not fail:

(in-package :eclector.reader)
(multiple-value-list
              (WITH-STANDARD-IO-SYNTAX
                (LET ((*READ-SUPPRESS* T))
                  (READ-FROM-STRING "#garbage"))))
->
debugger invoked on a ECLECTOR.READTABLE:UNKNOWN-MACRO-SUB-CHARACTER in thread #<THREAD "main thread" RUNNING {10005205B3}>: g is not a sub-character of the dispatch macro character #.

• read-token
• interpret-symbol
• interpret-token

This is a deviation from sbcl and ansi-tests, but don't really know what is correct:

(in-package :eclector.reader)
* (WITH-STANDARD-IO-SYNTAX
         (READ-FROM-STRING ".||"))

debugger invoked on a ECLECTOR.READER:INVALID-CONTEXT-FOR-CONSING-DOT in thread #<THREAD "main thread" RUNNING {10005205B3}>: A consing dot appeared in an illegal position.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

(ECLECTOR.BASE:%READER-ERROR #<SB-IMPL::STRING-INPUT-STREAM {10044D9503}> ECLECTOR.READER:INVALID-CONTEXT-FOR-CONSING-DOT)
   source: (APPLY (FUNCTION ERROR) DATUM :STREAM STREAM :STREAM-POSITION STREAM-POSITION (ALEXANDRIA.0.DEV:REMOVE-FROM-PLIST ARGUMENTS :STREAM-POSITION))
0] 0
* (cl:read-from-string ".||") 
|.|
3

This is very broad and may have to be split into multiple issues.

Some initial thoughts:

• What should the client-facing interface be? A recover restart?
• Problems with right parentheses seem to be the most common obstacle for recovery.
• Do we need dedicated condition sub-classes? Dedicated result objects?

(with-input-from-string (stream "#1=(#1#)")
    (eclector.concrete-syntax-tree:cst-read stream))

evaluates to a cst whose raw data is #1=(#1#), but whose FIRST cst has the raw data (t)

(eclector.reader:read (make-string-input-stream "#+(or) #|skipme|# 1 2")) => 1

but should be 2.

CLHS section 2.3.5 (Valid Patterns for Tokens) says ppppp:[:]nnnnn are undefined token patterns.