org-ql-agenda API proposal about org-ql

[mpereira]

Hey @alphapapa,

Thanks for the heads up. I'll take a look at it as soon as I get a free weekend! Thanks for working on these libraries, and feel free to close this issue.

[alphapapa]

Hi,

Thanks for the kind words, and for your ideas.

I think that some of what you're suggesting is supported already, or almost is, by using :super-groups to group them. For example, something like:

(org-ql-agenda (org-agenda-files)
  (and (done)
       (closed >= (- today 28)))
  :super-groups '((:name "Done today" :closed today)
                  (:name "Done earlier this month" :anything t)))

The highlights.org part is unclear to me.

functions that take org entries returned by org-ql as input, do arbitrary filtering and return org entries.

The issue with that idea is that the functions would be duplicating some work already done by org-ql selectors, but doing it in a different way since they'd be post-processing the results. That would essentially require maintaining two parallel implementations of the same logic. I've thought about using a macro to define them this way, in multiple implementations in a concise way, but I'm not sure that it would be worth it.

Also, org-super-agenda is essentially already that: functions that filter the output of Org Agenda and org-ql. I think, to implement the example you gave, would need a few new group selectors in org-super-agenda, and that that would probably be a better place to implement them than here.

In general, I've thought about this issue before, wanting to select items and group them at the same time. Maybe there could be a way to do grouping in org-ql directly, something like:

(org-ql-agenda (org-agenda-files)
  (grouping (group (and (done)
                        (closed = today)))
            (group (and (done)
                        (closed >= (- today 28))))))

The issue here is performance. Using multiple, independent predicates like that would mean running the predicates multiple times per item, and that means running a lot of regexp searches, result processing and looking-up things, etc, because Org buffers are essentially text, not structured data. Doing that would probably cause the query to take n-queries times as long, which would probably be prohibitive for users who have enough data to benefit from using org-ql.

One strategy for dealing with that is to first do a "lowest-common denominator" search in the Org buffer, then do further matching based on the processed, structured results. But that, again, goes back to requiring multiple implementations for a single selector.

In fact, I have a branch that begins to implement that, but sort of in reverse: the second implementation is a regexp search that is run on the whole buffer, to go directly to potential matches, rather than mapping the predicates across each entry.

So doing that as well could potentially mean 3 separate implementations for a single selector: one for the whole-buffer regexp search, one for the per-entry matching, and one for post-processing matches.

Now there's no reason that couldn't work, but it would be a lot of work to maintain 3 different "modes" for each selector, and I don't know if it would be worth it. And I don't know if that would be useful enough to me to do all that work myself.

One idea this gives me is to define selectors with a macro that automatically applies text properties to each item that can be easily matched on by a post-processing version of each selector. That would at least provide some standardization and make the second implementations easier to maintain, and it probably wouldn't reduce performance much.

That way it would be possible to build arbitrarily complex agendas, with groups possibly using distinct sources and queries.

To do this in a very basic way could be as simple as:

(let ((inhibit-read-only t))
  (with-current-buffer (get-buffer-create "*org-ql-complex*")
    (erase-buffer)
    (--each '((and (done)
                   (closed = today))
              (and (done)
                   (closed >= (- today 28))))
      (--> (eval `(org-ql "~/org/main.org"
                    ,it
                    :markers t))
           (--map (concat (org-ql-agenda--format-element it) "\n") it)
           (insert (apply #'concat it) "\n\n")))
    (pop-to-buffer (current-buffer))))

[alphapapa]

@mpereira If you haven't, please take a look at the recent additions to org-ql and org-super-agenda. The new timestamp predicates in org-ql are powerful and allow matching like in your examples.

That way it would be possible to build arbitrarily complex agendas, with groups possibly using distinct sources and queries.

Also, the org-ql-agenda--agenda function takes an :entries argument now, which can be the result of several org-ql-select queries in a single list. So you can have a combined view of several queries with a little bit of coding. I'll probably make better ways to do that in the future.