s-expressionists / concrete-syntax-tree Goto Github PK

This is a question, so feel free to close is it if goes a bit too much off topic/open ended or you don't have time.

I would like to solve the following problem: Given some CL program, let's say for example:

(defparameter str "(defun id (x) x)") ;; line 1                                       
(with-input-from-string (is str) ;; line 2                
    (eclector.concrete-syntax-tree:cst-read is)) ;; line 3

I would like to know that str on line 2 is referring to str on line 1 and is on line 3 is referring to is on line 2 , is on line 2 is essentially a new binding. And as final goal essentially cross referencing a whole project. The whole macro business is making thinks NOT easier.
The path I wanted to take is something along the lines of
construct ast with file-location annotation -> macroexpand -> figure out what is bound and what is free.
So I was wondering if this currently possible with this library (or at least parts of it)?
I am also wondering if it is always possible to determine these cross references.
Feel free to also suggest a completely different approach.

canonicalize-declaration-specifier ignores declarations it doesn't specifically handle. At the moment this seems to include all declarations other than the standard ones. This includes types, like the (declare (cons x)) synonym for (declare (type cons x)). The overall consequence is that this sort of declaration is silently ignored by Cleavir, which is bad.

This behavior is caused by the default method here: https://github.com/robert-strandh/Concrete-Syntax-Tree/blob/master/declarations.lisp#L18-L20

Brief review of the declaration semantics: (typespec var*) is an abbreviation for (type typespec var*). according to the Notes on the page for the type declaration. I don't see this specified anywhere else, so I suppose technically this isn't part of the standard (because Notes sections aren't), but given how universal the syntax is I think supporting it is basically required.

Now typespec there is a type specifier, but there's no good way to tell what is and isn't a valid type specifier really, or at least certainly not one that won't involve interrogating the environment, and interrogating the environment is outside the scope of CST.

An additional wrinkle is the declaration proclamation (i.e. not valid in local declare), which can be used by a programmer to indicate that a declaration identifier is valid but should be ignored. I've never seen this in code younger than I am but it's part of the standard.

It would be possible to move this functionality out of CST to somewhere that can deal with environments, but I don't think that's necessary. An easier solution would be to treat any unrecognized declaration identifier, at least if it's a symbol or cons, as a type declaration. This is simple and I believe almost entirely correct in reference to real code. If this isn't the case it's likely whatever's using CST (e.g. Cleavir) would check the validity of type specifiers while processing canonicalized type declarations anyway.

To support declaration, an additional parameter of proclaimed declaration identifiers could be added. Then the canonicalizer could simply ignore any identifier in this list, and treat the rest as type specifiers.

unfortunately I can only reproduce this in clasp

(let ((x nil))
    (macrolet ((%m ((&whole w arg))
                   `(progn (setq x (quote ,w))
                           ,arg)))
      (values (%m (1)) x)))
->
COMMON-LISP-USER> (let ((x nil))
    (macrolet ((%m ((&whole w arg))
                   `(progn (setq x (quote ,w))
                           ,arg)))
      (values (%m (1)) x)))
; caught ERROR:
;   No applicable method for PARAMETER-GROUP-BINDINGS with arguments
;    (#<CLASP-CLEAVIR::CLASP-64BIT> #<WHOLE-PARAMETER-GROUP parameter: #<CONCRETE-SYNTAX-TREE:SIMPLE-VARIABLE> @0x12fd5e899> #:G787516)

Condition of type: NO-APPLICABLE-METHOD-ERROR
No applicable method for PARAMETER-GROUP-BINDINGS with arguments
 (#<CLASP-CLEAVIR::CLASP-64BIT> #<WHOLE-PARAMETER-GROUP parameter: #<CONCRETE-SYNTAX-TREE:SIMPLE-VARIABLE> @0x12fd5e899> #:G787516)

Available restarts:
(use :r1 to invoke restart 1)

but the following methd really seem to be missing:

(defmethod parameter-group-bindings
    (client (parameter-group whole-parameter-group)
     argument-variable)
...)

This enhancement consists of writing a generic function named
CANONICALIZE-DECLARATION-SPECIFIER. It should take four parameters:

• SYSTEM. An object representing the Common Lisp system.

• DECLARATION-IDENTIFIER. A symbol that identifies what kind of
  declaration specifier to be canonicalized. Methods will have EQL
  specializers for this parameter. When the declaration specifier is a
  type specifier, the identifier will always be the symbol TYPE. No
  implicit type specifiers will ever be passed to this function.

• DECLARATION-IDENTIFIER-CST. A CST that has DECLARATION-IDENTIFIER
  as its associated RAW data.

• DECLARATION-DATA. A CST that has the CDR of the declaration
  specifier to be canonicalized as its associated RAW data.

It should return a list of CSTs representing canonicalized declaration
specifiers. A canonicalized declaration specifier is one of the
following:

• (declaration name)
• (dynamic-extent var)
• (dynamic-extent (function fn))
• (ftype type function-name)
• (ignore var)
• (ignore (function fn))
• (ignorable var)
• (ignorable (function fn))
• (inline function-name)
• (notinline function-name)
• (optimize (quality value))
• (special var)
• (type typespec var)

plus any canonicalized declaration specifier that is specific to the
client system. The CST representing the CAR of each element of the
list should be that of the second argument to the function. The CST
representing the canonicalized declaration data should be taken from
the corresponding information in the DECLARATION-DATA CST.

In addition to the definition of the generic function itself, the
enhancement consists of writing default methods (with the SYSTEM
parameter being unspecialized) for all standard Common Lisp type
identifiers.

A set of tests should be written to verify that the function works
for all Common Lisp declaration identifiers.
[I initially accidentally wrote "type identifiers" instead of
"declaration identifiers". Sorry about that.]

In file conditions.lisp, define a condition named CONS-CST-REQUIRED,
modeled after the condition EXPRESSION-CST-REQUIRED already present in
that file.

In the file cst.lisp, define default methods on FIRST and SECOND
modeled after the default method on RAW. The new methods should
signal the new condition CONS-CST-REQUIRED.

Quick reproduction:

(with-input-from-string (*standard-input* "#1=(#1#)")
  (eclector.concrete-syntax-tree:cst-read))
(cst:reconstruct '#1=(#1#) * nil)
(cst:reconstruct nil * nil)

The middle form is a workaround for s-expressionists/Eclector#8 i stumbled into - gets you an actually circular CST.

Anyway, this is due to add-atoms probably, here https://github.com/s-expressionists/Concrete-Syntax-Tree/blob/master/reconstruct.lisp#L83 There's a seen table but it's not used.

Would this library be suitable for inferring variable types, for example, can I use it to determine the type of a in:

(let ((a 5)) a) ; I'd expect (EQL 5)
;;; OR
(let ((a 5)) ; I'd expect (OR (EQL 5) (EQL 6))
  (setq a 6)
  a)

If so, is there any example code about how I can achieve this?

In fact they ignore the name entirely.

Hi,

the parse-macro-lambda-list fails on a more involved lambda list.

(ql:quickload :eclector-concrete-syntax-tree)
(ql:quickload :concrete-syntax-tree)

(defun read-one-cst (string)
  (with-input-from-string (is string)
    (eclector.concrete-syntax-tree:cst-read is)))

(defparameter *client* (make-instance 'cst::client))

(let* ((program
         "(defmacro loser (x &optional ((&optional a b &rest c)) &rest z)
            `(list ,x ,a ,b (list ,@c) (list ,@z)))")
       (cst (read-one-cst program))
       (lambda-list (cst:third cst)))
  (cst:parse-macro-lambda-list *client* lambda-list))

the example is taken from the CLHS page for defmacro.

&optional parameters have their -p variables unconditionally bound to nil, apparently due to a typo here. Original clasp issue.

Macro bodies do not allow declarations; see robert-strandh/SICL#141

Attempting to parse a macro lambda list with a complex &key parameter, e.g., (&key ((:a (b c)) '(nil nil))), results in a parse error being signaled. Original clasp issue.

Provide unit tests that verify that the condition
EXPRESSION-CST-REQUIRED is signaled when RAW is called on some object
other than an EXPRESSION-CST.

Provide unit tests that verify that the condition CONS-CST-REQUIRED is
signaled when FIRST or REST is called on some object other than a
CONS-CST.