okamsn / loopy Goto Github PK

I've had a craving for using the loop command within the loop command. It seems intuitive to me. Also, I don't like having to declare the whole loopy macro all over again when I'm already in the macro. And it shortens the syntax for nested loops significantly.

(loopy (loop (list store petstores)
	       (append (loop (list dog dogs)
			     ...
			     (collect dog)))
	       (loop (list cat cats)
		     ...
		     (do (some-side-effect)))))
		     (do (message "meow")))))

Even Iterate doesn't allow drivers lower than the top level.

;; Error: can't use `for' except at the top level.
(iter (for i from 1 to 10)
  (when (evenp i)
    (for j from 11 to 20)
    (collect j)))

After testing, it makes sense to do likewise. Consider:

(loopy (list i '(1 2 3 4 5))
       (when (cl-evenp i)
         (if (some-random-condition) (leave))
         (list j '(6 7 8 9))
         (collect j)))

A naive approach would be to update the variable holding '(6 7 8 9) after the main loop body when (cl-evenp i), but that isn't always correct for cases where the remaining instructions in the cycle are skipped.

It would be nice to have a way to leave the loop without affecting the returned value of the block. For example, a leave command could stop the loop without changing the implicit return.

One way to do this is with a cl-tagbody that jumps to the end of the cl-block, but is that efficient?

Might not want all of these, but some could be useful. iterate commands have
many options which is harder to do with how loopy currently parses commands,
but could probably be added using keywords like :by or :test, but that's not
high priority. It is more important to get the basic features first, and then
come back for more niches features.

Iteration

• index-of-{vector,sequence,string}

  • Sets VAR to indices of sequence instead to elements of sequence.
  • Added in #73.

• for (key val) in-hashtable table

  • Can just use the map.el library for this Implemented with map.el, but not as efficient as using maphash.

• for VAR in-file NAME

  • Opens the file name (which may be a string or pathname) for input, and
    iterates over its contents. reader defaults to read, so by default var will
    be bound to the successive forms in the file. The iterate body is wrapped in
    an unwind-protect to ensure that the file is closed no matter how the
    iterate is exited.

• for VAR in-stream STREAM

  • Like for... in-file, except that stream should be an existing stream object
    that supports input operations.

• for pvar previous var &optional initially init back n

  • Sets pvar to the previous value of var, which should be a driver variable, a
    variable from another for... previous clause, or a variable established by a
    for... =, for... initially... then or for... first... then clause (see
    Variable Binding and Setting). Initially, pvar is given the value init
    (which defaults to nil). The init expression will be moved outside the loop
    body, so it should not depend on anything computed within the loop. pvar
    retains the value of init until var is set to its second value, at which
    point pvar is set to var's first value; and so on.

    The argument n to back must be a constant, positive integer, and defaults
    to 1. It determines how many iterations back pvar should track var. For example,
    when n is 2, then pvar will be assigned var's first value when var is set to its
    third value.

    A for... previous clause may occur after or before its associated driver
    clause. for... previous works with generators as well as ordinary drivers.

  • Added as prev-expr command.

• for var initially init-expr then then-expr

  • Would prefer to have an init special macro argument that sets the initial
    value of any var, similar to how with declares variables. We can already use the with special macro argument to initialize variables. This is easier and cleaner than having a bunch of :init keywords for each command.
  • We cannot use with for variables created by the split flag. Ergo, it is better to have an :init keyword argument for some commands like expr and sum.
  • An init keyword was added to expr, which is equivalent to this feature.

Accumulation:

• an at place option?

  • Added to new and existing accumulation commands.

• multiply, like sum Added by Luis-Henriquez-Perez.

• reducing expr by func &optional initial-value init-val into var

  • Would be (reduce VAR FUNC &optional INIT), though INIT could also be
    set by an init special macro argument.
  • (reduce VAR EXPR FUNC &key init)

• adjoining exptr &optional into var test test at place result-type type

  • Like collect, but only adds the value of exptr if it is not already
    present. test, which defaults to #'eql, is the test to be used with
    member.

• union and nunion from cl-union and cl-nunion (destructive).

Boolean Tests

• first-iteration-p: variables loopy-first-iteration or
  loopy-first-cycle

  • added variable loopy-first-iteration and alias loopy-first-iteration-p.
  • Removed this variable. The command set works for this. See #108 and 5608825.

• first-time-p: Returns t the first time it is evaluated, then nil,
  like

  (prog1
     loopy-first-time
   (if loopy-first-time
       (setq loopy-first-time nil)))

  • This function won't be added. A variable in the command set works just as well.

• always, never, thereis: We could probably add these now. Added, but using the simpler cl-loop behavior of returning t for always and never.

  • always: If expr ever evaluates to nil, then nil is immediately returned; the
    epilogue code is not executed. If expr never evaluates to nil, the epilogue
    code is executed and the last value of expr (or t if expr was never
    evaluated) is returned (whereas loop would constantly return t).

  • never: Like (always (not expr)), except it does not influence the last value
    returned by a possible other always clause. That is,

    (iter (repeat 2)
             (always 2)
             (never nil)) => 2 ; not t

Control Flow

• if-first-time then else: Do then the first time and else other times.
  • Not going to add this. The commands set and if already work for this.

    ;; => ((1) (2 3))
    (loopy (list i '(1 2 3))
       (set first-cycle t nil)
       (if first-cycle
           (collect first i)
         (collect rest i))
       (finally-return first rest))

Sometimes I end up using multiple collects. Would be nice to use one.

(defun use-package-handler/:silence (name _ fns rest state)
  "Generate forms that silence output of FNS.
FNS is a list of function symbols."
  `(,@(loopy (list fn fns)
	     (collect (use-package%autoload fn name))
	     (collect `(xl-silence-output #',fn)))
    ,@(use-package-process-keywords name rest state)))

One collect.

(defun use-package-handler/:silence (name _ fns rest state)
  "Generate forms that silence output of FNS.
FNS is a list of function symbols."
  `(,@(loopy (list fn fns)
	     (collect (use-package%autoload fn name) `(xl-silence-output #',fn)))
    ,@(use-package-process-keywords name rest state)))

While writing loopy loops sometimes I want to append to the beginning of the list instead of at the end. This might be generally useful enough to include in loopy itself.

As I was using loopy it occurred to me that there are cases where I may want to use some of the features provided by dash's destructuring. Perhaps others using loopy might want to as well. Right now loopy and dash (not to mention cl-destructuring-bind and pcase) implement their own destructuring). What would be cool is if loopy could easily use destructuring provided by another package.

As a side-issue--not sure if this should be the responsibility of loopy, perhaps this should be a package. There should be a destructuring package that implements a cutomizable destructuring and is designed for other packages to use if they need destructuring for some purpose (like loopy does). This would make it so that packages don't each implement their own destructuring; also, it would provide a uniform destructuring experience among several packages. By customizable I mean that it should be easy to add your own "unorthodox" destructuring idioms (like dash does for example).

Perhaps it would be a nice feature if we could bind functions temporarily in a with clause.

While I was working on #25. I wanted to check if loopy without the dash destructuring enabled destructured variables in its with clause. I could have sworn it did. But anyway, I realized I had already set loopy-default-destructuring-function and loopy-default-accumulation-parsing-function to the corresponding dash functions and I found it inconvenient to switch back.

Perhaps instead of asking the user to set these functions we can just have a loopy-default-flags custom variable. The variable would contain a list of tags to apply to all loopy loops (unless you override it in particular loop).

Thus enabling certain loopy features would be as simple as:

(setq loopy-default-flags '(dash split))

I'd like to devise a better syntax for nested loops as it stands, I would not be very fond writing double nested loops (or even loops of a greater nest) for loopy. The syntax is a lot and it does not scale well for N-nested loops. I'd like a way to specify the lists in a "flat" way as opposed to nesting forms.

(loopy outer-loop
       ((list inner-list '((1 2 3) (1 bad-val? 1) (4 5 6)))
        (do (loopy ((list i inner-list)
                    (when (eq i 'bad-val?)
                      (return-from outer-loop 'bad-val?)))))))

Here's a clearer picture of what I mean. Of course their is -table-flat from dash, but the downside is it is not as performant as looping. I bet we can use macros to get a flat, scalable syntax that is also performant (ie. expands to while loop/dolist).

;; Does not scale well for N nests because you have to keep nesting forms.
(dolist (i '(1 2 3 ))
  (dolist (j '(4 5 6 7))
    (ignore 'this)))

;; Stays the same for N nests. But the problem is it is not as efficient as the former.
(loopy ((list (i j) (-table-flat #'list '(1 2 3) '(4 5 6 7)))
	(do (ignore 'this))))

I notice that in the iteration and looping commands, there is no command for looping while an expression is true. The closes example I could find in the README is:

(loopy (with  (j 0))
       ((do (cl-incf j))
        (when (> j 5)
          (return j)))) ; => 6

So it achieves an infinite loop but it does not check the "truthness" if any condition.
I think such an inclusion would be useful.

I was reading the source code of iterate the other day and I was pleasantly surprised at how short it is. I thought maybe we overestimated the difficulty in implementing this. I wanted to create this issue just as a place to discuss what it would take to implement this for loopy. The goal would be to get more concrete idea of what this would entail without making any commitments.

One easy step I could do is just start porting some of iterate's helper function to elisp. For many of them this is trivial. And then, try to understand how it works.

In the readme, you've already provided some real-life examples with loopy as well as contrived, didactic examples. I had been thinking that it might be useful to create a wiki with user examples of loops they've used. I could put some from my own config that I've used so solve problems. Or even interesting clause ideas could go there.

Even though I've suggested this idea, I'm not sure if it's a good idea or not, but I wanted to toss it in the air nevertheless.

This is helpful when we only want some parts of a destructured value.

We can't just not create instructions, though, as we still need to update the
value holders so that we pop the correct part from the list when assigning the
values of concern.

I was writing the following function. The relevant part to this issue is the with clause. When I used the dash flag I expected the with clause to use destructuring.So i expected the ((bind-tokens apply-tokens) ...) form to be properly destruction. However, according to the macro expansion it expands into a normal let (which in this case results in an error because let* only accepts symbols).

side note: I was not sure whether to make this a separate issue or include it in #20. If you mind, let me know.

(defun bind:apply-tokens (clause)
  "Apply appliers to CLAUSE."
  (loopy (flag dash)
	 (with (tokens (-splice #'bind:subclause-p #'bind:apply-tokens clause))
	       ((bind-tokens apply-tokens) (-separate #'bind:bind-token-p clause)))
	 ((list (token &as key . _) apply-tokens)
	  (expr name (xl-keyword-name key))
	  (expr apply-fn (xl-symbol-intern 'bind:apply- name))
	  (expr bind-tokens (funcall apply-fn token bind-tokens)))
	 (return bind-tokens)))

(defun bind:apply-tokens (clause)
  "Apply appliers to CLAUSE."
  (let*
      ((tokens
	(-splice
	 (function bind:subclause-p)
	 (function bind:apply-tokens)
	 clause))
       ((bind-tokens apply-tokens)
	(-separate
	 (function bind:bind-token-p)
	 clause)))
    (let
	((bind-tokens nil)
	 (apply-fn nil)
	 (name nil)
	 (_ nil)
	 (key nil)
	 (--dash-source-229-- nil)
	 (token nil))
      (let*
	  ((list-2952 apply-tokens))
	(cl-block nil
	  (while
	      (consp list-2952)
	    (setq token
		  (car list-2952)
		  --dash-source-229-- token key
		  (pop --dash-source-229--)
		  _ --dash-source-229--)
	    (setq name
		  (xl-keyword-name key))
	    (setq apply-fn
		  (xl-symbol-intern 'bind:apply- name))
	    (setq bind-tokens
		  (funcall apply-fn token bind-tokens))
	    (setq list-2952
		  (cdr list-2952)))
	  nil)
	bind-tokens))))

Variables have been re-named and added (e.g., implicit returns). Make sure that
documentation about customization in the Info document and the readme is still
up to date.

Right now loopy uses gensym to create generalized variables and this is fine. However, macroexpansion could be made more readable if instead loopy used make-symbol to create named references. For example, the generalized variable used to hold items in the remaining list in the list clause could be generated with (make-symbol "rest").

It's a small change but I think worth it.

When creating a while loop with loopy where the EXPR contains the same variable as that of the WHILE--in this case rest, the loop never runs because rest is set to nil at the beginning of the loop. I wrote I want a loop that continues based on some dynamic circumstance.

This surprised me but I get why it happens. Is there some way for loopy to account for this case and not bind REST to nil initially?

If there is not a good way of intelligently accounting for this, I want to add this to the wiki. Maybe in a "gotcha"/"potentially suprising behaviors" section.

(defun xl-define-key::clause (rest)
  "Convert clause into proper general-def arguments."
  (loopy ((while rest)
	  (list identify-fn xl-define-key::identifiers-alist)
	  (expr (clause rest) (funcall identify-fn rest)))))

(let
      ((clause nil)
       (rest nil) ; => set to nil
       (identify-fn nil))
    (let*
	((list-395 xl-define-key::identifiers-alist))
      (cl-block nil
	(cl-tagbody
	 (while
	     (consp list-395)
	   (if rest nil
	     (go loopy--non-returning-exit-tag))
	   (setq identify-fn
		 (car list-395))
	   (setq rest
		 (funcall identify-fn rest))
	   (setq clause
		 (pop rest))
	   (setq rest
		 (car rest))
	   (setq list-395
		 (cdr list-395)))
	 loopy--non-returning-exit-tag)
	nil)))

I realize this could be fixed by doing the following but it seems too wordy.

(defun xl-define-key::clause (rest)
  "Convert clause into proper general-def arguments."
  (loopy ((while rest)
	  (list identify-fn xl-define-key::identifiers-alist)
	  (expr (clause new-rest) (funcall identify-fn rest))
	  (do (setq rest new-rest)))))

One of the consequences of having the keyword do is that it is confused with cl-do. This doesn't matter in terms of evaluating the loopy loop because loopy interpret it correctly. However, the indentation of cl-do is different than that of progn and since emacs thinks that a loopy do is cl-do it indents do improperly.. Also if someone ends up using the actual cl-do in a loopy loop it might look wierd. Perhaps, we should have an alternative alias for this?

The idea is this. Sometimes I want to use an unnamed collect throughout loopy. But in the after-do or else-do clause I want to reference the list collect just one time. This would be even more annoying the more collects I used (this is assuming, however, that collects will accumulate into the same variable #21) I often lament having to named the list just to reference it one time at the end. Perhaps an anaphoric variable could be of use for this case. Maybe $result or <result>? Don't know.

Perhaps it would be nice provide an alias the list clause as for. for would look more like what's used in the other looping constructs which are similar to loopy including in cl-loop. Also it arguably is more intuitive than list.

Requested in #35.

From that discussion:

I agree that it is noticeably less convenient. The special macro arguments are
hard-coded right now, so one can't currently add an alias for them. That's
easy to change for arguments that don't affect how loop commands are expanded,
like finally-return, finally-do, before-do, and else-do. Aliases might
not work well for with and flag.

These new commands should error on loopy--in-sub-level, and would allow for
aliases as discussed in #41.

I wrote this function to begin automating finding the load-path of mu4e. Weird thing is that it succeeds on the first call (it returns a path), but on every call afterwards it throws an error void-variable base-dir.

Notably when I just run the body of this function--aka the loopy loop--it always succeeds. I'm not sure exactly what's going on but i wanted to point this out.

(defun mu4e:load-path-guix ()
  "Return load-path for mu4e.
This assumes that you're on guix."
  (loopy (with (regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")
	       (base-dir "/gnu/store/"))
	 ((list file (directory-files base-dir))
	  (expr full-path (f-expand file base-dir))
	  (when (and (string-match-p regexp file) (f-dir-p full-path))
	    (expr mu4e-path (f-expand "share/emacs/site-lisp/" full-path))
	    (when (f-dir-p mu4e-path)
	      (return mu4e-path))))))

This is the corresponding backtrace.

Debugger entered--Lisp error: (void-variable base-dir)
  (directory-files base-dir)
  (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil))
  (let ((mu4e-path nil) (full-path nil) (file nil)) (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil)))
  (let* ((regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")) (let ((mu4e-path nil) (full-path nil) (file nil)) (let* ((list-2794 (directory-files base-dir))) (cl-block nil (while (consp list-2794) (setq file (car list-2794)) (setq full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (setq mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (cl-return-from nil mu4e-path))) (setq list-2794 (cdr list-2794))) nil))))
  (loopy (with (regexp "\\`[[:alnum:]]+-mu-\\(?:[[:digit:]]\\|\\.\\)+")) ((list file (directory-files base-dir)) (expr full-path (f-expand file base-dir)) (when (and (string-match-p regexp file) (f-dir-p full-path)) (expr mu4e-path (f-expand "share/emacs/site-lisp/" full-path)) (when (f-dir-p mu4e-path) (return mu4e-path)))))
  mu4e:load-path-guix()
  eval((mu4e:load-path-guix) nil)
  elisp--eval-last-sexp(t)
  eval-last-sexp(t)
  eval-print-last-sexp(nil)
  funcall-interactively(eval-print-last-sexp nil)
  call-interactively(eval-print-last-sexp nil nil)
  command-execute(eval-print-last-sexp)

Not all commands can be used together. The CL Iterate package warns the user
when they attempt to use the commands in a way that won't work. We should as
well.

Allow users to create custom aliases in a loopy-custom-aliases alist.
Requested in #35.

From that discussion:

Maybe the function could be something like (loopy-defalias NEW-NAME OLD-NAME) which just adds a cons cell of (NEW-NAME . OLD-NAME) to a user
option loopy-custom-aliases, which would be an alist. I think that that
would work well.

Then, the order for finding the meaning of symbols would be

1. custom aliases
2. custom commands
3. built-in commands

There are general musings about inconveniences I've faced while looping.

Often as I'm writing elisp, I want to iterate by more than one element at once. Something like this:

(dolist (trio (-partition 3 '(1 2 3 4 5))
  ...)

The value of trio in the first iteration would be (1 2 3).

It's not a huge deal to do this; it gets the job done. But it is sub-optimal. Partitioning the list, then looping through it when I could in theory collect sets of three elements during the first pass.
Maybe it would be nice to have a partition clause like (partition element list n), where you traverse LIST N elements at a time, putting the (-take N list) in ELEMENT. Then again, maybe I'm the only one who feels this is a problem.

Another candidate for a good possible clause is one that abstracts key value pairs such as alists, plists and hash tables. It would provide an element of the collection as a cons cell of the form (KEY . VALUE).

(loopy ((key-val-clause (key . value) hash-or-alist-or-plist)
	(do-something-with key value)))

Sometimes I want access to the rest of the list. My use case is inserting an item at a particular place in a plist. In the example below I try to use loopy to insert a keyword and value after a particular place in a plist. I use an undeclared variable REST (so this code doesn't work) that I would like to contain the rest of the list.

(defun lleaf-insert-after (list place keyword value)
  "Insert KEYWORD and VALUE after PLACE."
  (loopy ((list element list)
	  ;; accumulate already-traversed elements into HEAD.
	  (collect head element)
	  (when (eq element place)
	    ;; when we find the PLACE, the keyword we want to insert after
	    ;; get it's value, (pop list), and put it in HEAD
	    (collect head (pop rest))
	    ;; Then put KEYWORD and VALUE into HEAD 
	    (append head (list keyword value))
	    ;; Return the plist with KEYWORD and VALUE inserted
	    (return (append head rest))))
	 (return list)))

This has to do with (-drop N list) and (-take N list).

I haven't thought of a good way to deal with this yet.

Some clauses in iter don't accept arbitrary clauses such as before-do.

For example, I would expect to be able to use collect within initially-do. But iter does not check in before-do (and I'm assuming it also does not do so for similar clauses like after-do either). I think it should.

(cl-defun defun%parse-hook (init _ hooks &key depth append local)
  (let-alist init
    (iter (with (orig-name (format "xl--%s-hook" .name)))
          ;; LOOK HERE.
	  (initially (collect `(cl-defun ,orig-name ,.args ,.docstring ,@.body)))
	  (each hook hooks)
	  (expr hook-fn (xl-symbol-intern hook '& .name))
	  (collect `(defalias ',hook-fn ',orig-name))
	  (collect `(add-hook ',hook #',hook-fn ,(or append depth) ,local)))))

I am not sure if it is worth having a distinction between built-in commands and custom commands. Main benefit I see of it is the user won't accidentally get rid of built-in commands.

So, any is a good (slightly better in my opinion) name than thereis. I made thereis modeling the specification of the iterate thereis clause. However, it contrasts from never and always in that it actually returns a value. Maybe, to be consistent the behavior with always and never, an any clause should strictly return a boolean.

I don't know whether this is overkill or not, but I could whip up a PR with alwaysis neveris clauses which contrast with their counterparts in that they return values as opposed to booleans.

Currently, the macro creates constructs even when they aren't needed, such as symbol macros. This can create warnings of unused variables when compiling.

It would be better to conditionally construct the returned code, though this will probably decrease readability.

I think it's safe to say one of loopy's goals is to to write everything that concerns a loop within the loopy macro. That is why clauses like with and before-do exist even though they could be done outside loopy. With this in mind, i was thinking maybe the when clause could work outside of a loopy loop.

(loopy (with (my-list '((1 2 3))))
       (when (some-condition))
       ((list i '(1 2 3 4))
	(collect (+ i i))))

;; Would be equivalent to below.

(when (some-condition)
  (loopy (with (my-list '((1 2 3))))
	 ((list i '(1 2 3 4))
	  (collect (+ i i)))))

In fact we can go even further and support arbitrary wrappers. For instance, typically in emacs you want to wrap a loop with save-excursion or save-match-data or save-restrictions.

(loopy (wrap ((when (= 1 2)) (save-excursion) (save-match-data)))
       ((while (re-search-forward "someregexp" nil t nil)
	  (collect (match-data)))))
;; =>

(when (= 1 2)
  (save-excursion
    (save-match-data
      (loopy
       ((while (re-search-forward "someregexp" nil t nil)
	  (collect (match-data))))))))

These links might be automatically formatted as "see link", which can
interrupt the sentence. For example,

some text (see [[info:my-thing#section]])

can become

some text (see see "(my-thing)section")

which is wrong.

This is not a concrete idea. But it has been on my mind and I think would be useful so I wanted to explain it as best as I can.

The idea is to have a recur clause which can trigger a the current loopy loop, overwriting any variables declared.

A recur clause would take in a set of bindings. Basically recur would execute the same loopy loop, but with those bindings instead.

(recur &rest bindings)

;; In practice it would look like this:

;; => 120
(loopy (with (n 5))
       (loop (return (if (zerop n) 1 (* n (recur (n (1- n))))))))

I think that first of all this looks really cool. But in all seriousness, this could addresses the problem that sometimes you want to use recursion, but you want to do something to the result of the recursion. Normally, you'd have to wrap the recursive function in another function in that case because otherwise any post manipulation would recurse as well. This is why I think cl-labels partly was created. Also, perhaps it is possible for loopy to recure more efficiently than with function calls. I got some inspiration for this by looking at the package recur. Which, now that I mention it, could be a useful for #16 because it showcases "code-walking" which is what would be needed for arbitrarily nested clauses.

The idea is to allow destructuring in the seq/list clause. I haven't explored it thoroughly, but seems cool at a glance.

(loopy ((seq (a . b) '((1 . 2) (3 . 4)))
        (collect keys a)
	(collect values b))
       (return (list keys values)))
;; => '((1 3) (2 4))

The macro expansion has changed drastically.

This

(pcase-let* ((`(,a (,_ ,b)) '(1 (2 3))))
         `(,a (,_ ,b)))

becomes

(progn
  (ignore (consp '(1 (2 3))))
  (let* ((x326 (car-safe '(1 (2 3))))
         (x327 (cdr-safe '(1 (2 3)))))
    (progn (ignore (consp x327))
           (let* ((x328 (car-safe x327)))
             (progn
               (ignore (consp x328))
               (let* ((x330 (cdr-safe x328)))
                 (progn
                   (ignore (consp x330))
                   (let* ((x331 (car-safe x330))
                          (x332 (cdr-safe x330)))
                     (progn
                       (ignore (null x332))
                       (let* ((x333 (cdr-safe x327)))
                         (progn
                           (ignore (null x333))
                           (let ((a x326) (b x331))
                             nil nil
                             `(,a (,_ ,b))))))))))))))

instead of something like the below, as it did previously.

(let* ((x326 (car-safe '(1 (2 3))))
       (x327 (cdr-safe '(1 (2 3))))
       (x328 (car-safe x327))
       (x330 (cdr-safe x328))
       (x331 (car-safe x330))
       (x332 (cdr-safe x330))
       (x333 (cdr-safe x327)))
  (let ((a x326)
        (b x331))
    `(,a (,_ ,b))))

In my running emacs instance I pulled the latest loopy version, rebuilt it and to make sure it was in fact the one being applied, evaled the loopy.el buffer. I didn't want to restart emacs since I'm working on my binding macro.

When I try to run this loop (it's the body of a function but I only include the loop), I get an error that the dash source variables are not defined.

(loopy (flag dash)
	 (without clause)
	 ((while clause)
	  (expr elt (pop clause))
	  (cond ((bind:is-valid-token-p elt)
		 (collect elt))
		((listp elt)
		 (collect (bind:parse-clause elt)))
		(t
		 (expr (clause . tokens) (bind:read-token (cons elt clause)))
		 (append tokens)))))

It seems that the error lies in the form which seems to reference --dash-source-4123-- which was not defined previously. Also, the variable pairings look off because it seems to also be setting nil to loopy--explicit-vars.

(setq loopy--explicit-vars --dash-source-4123-- nil loopy--explicit-vars clause nil loopy--explicit-vars tokens nil loopy--main-body setq --dash-source-4123--
		   (bind:read-token
		    (cons elt clause))
		   clause
		   (pop --dash-source-4123--)
		   tokens --dash-source-4123--)

This is the full macro expansion.

(let
      ((loopy--implicit-accumulation-updated nil))
    (let*
	((elt nil)
	 (loopy-result nil)
	 (loopy--main-body nil)
	 (loopy--explicit-vars nil))
      (cl-block nil
	(cl-tagbody
	 (while t
	   (if clause nil
	     (go loopy--non-returning-exit-tag))
	   (setq elt
		 (pop clause))
	   (cond
	    ((bind:is-valid-token-p elt)
	     (setq loopy-result
		   (cons elt loopy-result)))
	    ((listp elt)
	     (setq loopy-result
		   (cons
		    (bind:parse-clause elt)
		    loopy-result)))
	    (t
	     (setq loopy--explicit-vars --dash-source-4122-- nil loopy--explicit-vars clause nil loopy--explicit-vars tokens nil loopy--main-body setq --dash-source-4122--
		   (bind:read-token
		    (cons elt clause))
		   clause
		   (pop --dash-source-4122--)
		   tokens --dash-source-4122--)
	     (setq loopy-result
		   (nconc
		    (reverse tokens)
		    loopy-result)))))
	 (setq loopy-result
	       (nreverse loopy-result))
	 (setq loopy--implicit-accumulation-updated t)
	 loopy--non-returning-exit-tag
	 (if loopy--implicit-accumulation-updated nil
	   (setq loopy-result
		 (nreverse loopy-result))))
	loopy-result)))

I get the following backtrace:

Debugger entered--Lisp error: (void-variable --dash-source-4121--)
  (setq loopy--explicit-vars --dash-source-4121-- nil loopy--explicit-vars clause nil loopy--explicit-vars tokens nil loopy--main-body setq --dash-source-4121-- (bind:read-token (cons elt clause)) clause (car-safe (prog1 --dash-source-4121-- (setq --dash-source-4121-- (cdr --dash-source-4121--)))) tokens --dash-source-4121--)
...

In the tests I see you do (eval (quote ...)). Why is this instead of simply the loopy form itself?

(ert-deftest wrap-pcase-let ()
  (should
   (equal '(1 2 3 4 5 6)
          (eval (quote (loopy-iter (for list i '((1 2) (3 4) (5 6)))
                                   (pcase-let ((`(,a ,b) i))
                                     (accum collect a)
                                     (accum collect b))))))))

I was working on the adjoining command mentioned in #48. And I wanted to relay my thoughts on the experience.

1. distinguishing optional values

I'm having trouble distinguishing between the following cases.

(adjoining var val)
(adjoining var val testfn)
(adjoining val)
(adjoining val testfn)

Maybe the solution to this is to require the keyword :test when using testfn so there is no ambiguity.

On a side note: better logic pattern matching is needed in emacs lisp. I probably need to talk to emacs devel about this. Specifically pcase needs to be more powerful. I'd like to have something like (adjoining ,(and (pred symbolp) ,var) ,var ,(optional testfn))). Or even something like (integer-sum ,(one-or-more (pred (integerp)))). There should be similar keywords to what rx has.

2. Adding a new accumulation keyword in a thorough way can be very involved. I'll likely add more examples to adding custom commands in a PR.

I haven't added destructuring for this command yet. Not sure if I need to?

(cl-defun loopy%adjoining-command-parser (arg)
  "Add ELT to VAR is ELT is not already in VAR."
  (let (val var)
    (cond ((= 2 (length (cdr arg)))
	   (setq var (cl-second arg))
	   (setq val (cl-third arg))
	   `((loopy--accumulation-vars . (,var ,nil))
	     (loopy--implicit-return . ,var)
	     (loopy--main-body . (setq ,var (cl-adjoin ,val ,var :test #'eql)))
	     (loopy--implicit-return . (setq ,var (nreverse ,var)))))
	  ((and (= 1 (length (cdr arg))) loopy--split-implied-accumulation-results)
	   (setq var (gensym "adjoining-list-"))
	   (setq val (cl-second arg))
	   `((loopy--main-body . (setq ,var (cl-adjoin ,val ,var :test #'eql)))
	     (loopy--implicit-return . (nreverse ,var))))
	  ((= 1 (length (cdr arg)))
	   (setq var (gensym "adjoining-list-"))
	   (setq val (cl-second arg))
	   `((loopy--main-body . (setq ,var (cl-adjoin ,val ,var)))
	     (loopy--implicit-accumulation-final-update . (setq ,var (nreverse ,var)))
	     (loopy--implicit-return . ,var)))
	  (t (error "invalid args: %S" arg)))))

(add-to-list 'loopy-custom-command-parsers '(adjoining . loopy%adjoining-command-parser))

Sometimes, destructuring create a variable to hold a value, even though it only
actually copies the value of an existing variable (instead of the return value
of a function).

• Is it worthwhile to try to avoid this variable? How things are destructured
  might need to be changed.
• Can it be detected just by checking whether a value is a symbol? How do
  symbol macros affect this?

Currently, updates are run after the main body of the loop. This is what cl-loop does.

It might be better to update a value-holding variable immediately after setting a main variable.

This might allow iterations commands to work as expected in nested commands and could be better for arbitrary nesting like in Iterate.

For example, (list i '(1 2 3)) would have one instruction to inintialize the value holder and one main-body instruction like

(setq  i (car value-holder)
    value-holder (cdr value-holder))

However, as noted elsewhere, this is complicated by the skip command, which skips to the end of the main body but which must not skip the latter body, since otherwise variables in iteration commands wouldn't correctly update.

Maybe it would be better to update before the main body? That might mean adding junk data.

More examples are needed.

In loopy's info manual you showed how you can conveniently modify a sequence in place via setf.

          (loopy (with (my-seq '(1 2 3 4)))
                 (loop (seq-ref i my-seq)
                       (do (setf i 7)))
                 (return my-seq)) ; => '(7 7 7 7)

I'd like to know how to remove a particular value in place--if it is possible.

;; I made `removef` up.
          (loopy (with (my-seq '(1 2 3 4)))
                 (loop (seq-ref i my-seq)
                       (when (= i 2) (removef i)))
                 (return my-seq)) ; => '(1 3 4)

Implicit returns can probably be implemented by creating a new special variable
(loopy--implicit-returns) and using it in place of the nil located at the
bottom of the cl-block.

This will generally work, and leaves only a few questions:

1. How should implicitly returned values be returned? Should any implicit value
   be returned in a list?

2. Should accumulation commands have different calling forms for implicit return
   values?

   • cl-loop does this, but for does not. for seems to require a variable
     name and returns all implicit variables. We should probably do the same,
     but are not required to because we put the command name before the variable
     name.

       (for:for ((i repeat 10)
                 (randoms collecting (random 10))
                 (randoms2 collect 1)))

3. How should implicit returns interact with early-exit commands like leave
   and leave-from? We can think of these as meaning "stop the loop", but what
   they really do is force the cl-block to return nil early. I think it
   would be best if these two commands did not affect implicit returns, and this
   would give them a more meaningful use when compared to the return and
   return-from commands.

I am writing a general macro to bind keys for myself (maybe I will create a package for it). In it I wrote this function.

(defun bind:parse-clause (parsers clause)
  "Return list of tokens from parsing CLAUSE with PARSERS."
  (loopy (without clause)
	 ((while clause)
	  (expr elt (pop clause))
	  (cond ((bind:is-valid-token-p elt)
		 (collect elt))
		((listp elt)
		 (collect (bind:parse-clause parsers elt)))
		(t
		 (expr (elt . clause) (bind:read-token parsers (cons elt clause)))
		 (collect elt))))))

I thought this would work like this and as far as I can tell it should. Right?

(defun bind:parse-clause (parsers clause)
  "Return list of tokens from parsing CLAUSE with PARSERS."
  (let (elt parsed)
    (while clause
      (setq elt (pop clause))
      (cond ((bind:is-valid-token-p elt)
	     (push elt parsed))
	    ((listp elt)
	     (push (bind:parse-clause parsers elt) parsed))
	    (t
	     (-setq (elt . clause) (bind:read-token parsers (cons elt clause)))
	     (push elt parsed))))
    (reverse parsed)))

Weird thing is I get completely different results for the two.

It looks like it's making extra lists in the macroexpansion

(let
    ((elt nil))
  (let*
      ((collect-implicit-149406 nil)
       (collect-implicit-149404 nil)
       (collect-implicit-149403 nil))
    (cl-block nil
      (cl-tagbody
       (while t
	 (if clause nil
	   (go loopy--non-returning-exit-tag))
	 (setq elt
	       (pop clause))
	 (cond
	  ((bind:is-valid-token-p elt)
	   (setq collect-implicit-149403
		 (cons elt collect-implicit-149403)))
	  ((listp elt)
	   (setq collect-implicit-149404
		 (cons
		  (bind:parse-clause parsers elt)
		  collect-implicit-149404)))
	  (t
	   (setq clause
		 (bind:read-token parsers
				  (cons elt clause)))
	   (setq elt
		 (pop clause))
	   (setq clause clause)
	   (setq collect-implicit-149406
		 (cons elt collect-implicit-149406)))))
       loopy--non-returning-exit-tag)
      (list
       (nreverse collect-implicit-149406)
       (nreverse collect-implicit-149404)
       (nreverse collect-implicit-149403)))))

My version of loopy is at commit:

1cf221e * master origin/master Add test case to make sure repeated evaluation doesn’t raise errors.

I noticed from the macro expansion of the collect command, that loopy uses append to accumulate values into a list. I thought the most efficient way to do this is using push and nreverse (https://github.com/alphapapa/emacs-package-dev-handbook#collecting-items-into-a-list). Should this be changed?

As it stands these things seem equivalent:

(loopy (loop (list a '(1 2 3 4))
	     (collect a))
       (loop (list b '(5 6 7))
	     (collect b)))

;=> (1 5 2 6 3 7)

(loopy (loop (list a '(1 2 3 4))
	     (list b '(5 6 7))
	     (collect a)
	     (collect b)))

;=> (1 5 2 6 3 7)

But maybe they shouldn't be. I understand that the latter form returns what it does. But I expected two different loops in the former. So two consecutive while loops. Perhaps this could be optional behaviour designated by a flag.

In this case, the loopy loop would return this.

(loopy (loop (list a '(1 2 3 4))
	     (collect a))
       (loop (list b '(5 6 7))
	     (collect b)))

;=> (1 2 3 4 5 6 7)

cl-loop provides clauses that wrap functions like mapatoms and map-keymap. These can be added by wrapping the loop body in a lambda after it is constructed. This probably best done after implementing #2.

Texinfo has more mark-up features then what Org can do. It might be an
improvement to use these features via Org macros.

For example, use the @dfn command to create definitions. This might allow
conveniences in the future.

This is low priority.

Tests with nconc sometimes pass and sometimes fail.

Sometimes it reports a circular list is reported, but this error goes away if the tests.el buffer is evaluated, reloading the test definitions.

These errors do not happen when evaluating the tests manually, such as with eval-last-sexp.

I have a loop where I want to return one thing under a certain condition, but something else if the loop finishes successfully.

(defun xl--keyword-macro-args (body)
  "Return list of (docstring KEYWORD-ARGS BODY)."
  (loopy (with (docstring (when (stringp (car body)) (pop body))))
	 ((cons (key value . rest) body #'cddr)
	  (unless (keywordp key)
	    (return docstring plist `(,key ,value ,@rest)))
	  (append plist (list key value)))
	 (return list docstring plist rest)))

In other words, I want the return statement to be at the end of the expanded cl-block. As I show below.

(let ()
  (cl-block nil
    (while ... (return docstring plist rest))
    plist
    ;; Where I want return value to go instead.
    )
  ;; Current return value
  (list docstring plist ret))

;; What I want.
(let ()
  (cl-block nil
    (while ... (return docstring plist rest))
    plist
    (list docstring plist ret)))

So far return and finally-return both put it at the end outside of the cl-block. And after-do puts it before the default cl-block return value, plist. I couldn't find a way to put it where I wanted.

I propose that return should put it's return value at the end of the cl-block, overriding the cl-block's default return value (plist). So, return could be circumvented by return statements in the loop. Whereas, finally-return would still always be returned as it does now.

;; What I propose return should do
(let ()
  (cl-block nil
    (while ... (return docstring plist rest))
    plist
    ;; This was a returned value.
    (list some-return-value))
  ;; finally-return would still go here.
  )

There's currently no equivalent of a progn command for loop commands, but it
might be useful to have one. For example, as the first sub-command in the if
command.

Here's a quick example:

(loopy ((list i '(2 4 5))
        (if (cl-oddp i)
            (group
               (expr failure t)
               (do (message "Warning: odd found")))
          (expr failure nil))
        (count failures failure))
       (return failures)) ; => 1

progn is currently an alias for the do command. Should that be changed to
this?