querying clocked in time about org-ql

[alphapapa]

I've merged this to master along with a new ts predicate. Please let me know if you have any more feedback.

[seanfarley]

Woah, neat!! I'll try this out tonight!

[alphapapa]

Those are interactive commands for manipulating text timestamps in a buffer.

[alphapapa]

Sorry if this is a noise, but I just noticed that encode-time allows out-of-range parameters, that is to say, one can just increase the day parameter by 1, and then encode and decode the time, to increase the date by 1 day. And I think this is the approach used by org-timestamp-change as I can tell.

Yes, but org-timestamp--to-internal-time calls encode-time, which converts to the internal time value, in the (HIGH LOW . IGNORED) format, which can't be so easily incremented. So we could write our own version of org-timestamp--to-internal-time that optionally adds 1 to the minute value before calling encode-time, and then we could use time-less-p. But unless there were a substantial performance difference, I don't think it would be worth it. Unix timestamps are much simpler to work with.

In any case, thanks for the great package. :-D

You're welcome! I would love to hear about how you're finding it useful. Please let me know if you have any suggestions.

[alphapapa]

@seanfarley What do you think about changing the arguments to keywords, like this?

;; Select all clocked entries
(org-ql (current-buffer)
    (clocked))

;; Select entries clocked on or after given timestamp
(org-ql (current-buffer)
    (clocked :from "2018-09-15"))

;; Select entries clocked on or before given timestamp
(org-ql (current-buffer)
    (clocked :to "2018-09-20"))  

;; Select entries clocked within the range of given timestamps
(org-ql (current-buffer)
    (clocked :from "2018-09-15" :to "2018-09-20"))

;; Select entries clocked on a certain day
(org-ql (current-buffer)
    (clocked :on "2018-09-15"))

[alphapapa]

Hi,

This is an interesting idea. Probably the simplest way to do it is to search for org-clock-line-re in the buffer and compare each timestamp that's found.

org-ql currently works by looking at every heading in a buffer, which is a lot slower than searching the buffer with re-search-forward, so implementing this in org-ql now would be slow. However, I have a branch that does a very basic optimization when possible by replacing selectors with a "preamble" that uses re-search-forward to skip irrelevant entries. So implementing this feature in that branch could be relatively fast, I think.

After finding CLOCK: lines, the slow part would be comparing timestamps, but I guess that would probably still be fast enough to be useful, depending on how big the Org file is and how many timestamps it has. This feature would basically be like the Org Agenda's log mode, which seems to work pretty well for me.

[seanfarley]

Yeah, this is basically org agenda which works fine for most simple cases. The problem I have with org-agenda is when I want to gather entries that span more than, say, a week. Or when I want to fine tune an invoice for a client. The latter is what I'm truly after: a (simple) org parsing tool that I can use to generate reports.

[alphapapa]

Ok, I have a prototype in the https://github.com/alphapapa/org-ql/tree/feature/clocked-p branch. Please take a look. You can use it like this:

;; Select all clocked entries
(org-ql (current-buffer)
    (clocked))

;; Select entries clocked after given timestamp
(org-ql (current-buffer)
    (clocked "2018-09-15"))

;; Select entries clocked before given timestamp
(org-ql (current-buffer)
    (clocked nil "2018-09-20"))  

;; Select entries clocked within the range of given timestamps
(org-ql (current-buffer)
    (clocked "2018-09-15" "2018-09-20"))

[seanfarley]

This is amazing! I'll try to combine this with atheriel/org-clock-csv so I can generate pretty invoices now.

P.S. I noticed that org-is-habit-p wasn't defined and required me to require 'org-habit when running org-ql-agenda.

[alphapapa]

This is amazing! I'll try to combine this with atheriel/org-clock-csv so I can generate pretty invoices now.

Okay, but be advised that "I have not tested the code, only proved it correct." haha ;) That is, I don't promise that it will provide completely correct results. Comparing timestamps is not as straightforward as it seems. e.g. at first I tried to use Emacs-native time objects, comparing with time-less-p, but that presented a problem: if a clocked timestamp were the same as the comparison timestamp, IIUC, that would mean that it would not be "less than" according to that function, so it wouldn't be returned. And there is no other Emacs function for comparing time values. So instead I decided to use Unix timestamps and compare with <=. I think that will work correctly, but I've only done basic testing. And then, comparing time ranges, which Org clocked lines are, presents more complexities, but I think those are handled correctly.

In general, processing time and date values in Emacs and Org is never straightforward (at least, not to me). So, use it however you like, but with regard to generating important, money-related things like invoices...use it at your own risk. :)

P.S. I noticed that org-is-habit-p wasn't defined and required me to require 'org-habit when running org-ql-agenda.

I get reports about that issue fairly often, and I don't know what to do about it. The error shouldn't happen unless the user tries to use the habit selector, and if the user's using org-habit, he should already have it loaded. I don't want to load org-habit myself, because that should only be done by the user. So I think it's essentially a configuration error on the part of the user, but I'm not completely sure.

[JSDurand]

What about org-element-timestamp-parser? It returns a lisp object (a list whose car is timestamp and whose cadr is a plist) representing the timestamp at point. I think it can be used to compare timestamps? Or is it too slow to be used here?

[alphapapa]

@JSDurand The code already uses timestamp objects by calling (org-element-property :value (org-element-context)). That doesn't help with comparing timestamps, because AFAIK there is no function that compares timestamp plist objects. (And if there were, it would still be very difficult to increment them, because how would you easily increment a timestamp like (:year 2018 :month 12 :day 31 :hour 23 :minute 59 :second 59)? And imagine needing to account for leap days and leap seconds...) So we convert them to Unix timestamps: https://github.com/alphapapa/org-ql/blob/feature/clocked-p/org-ql.el#L273

@seanfarley Reading a comment on /r/Emacs today, I ended up reading https://orgmode.org/manual/The-clock-table.html, and you may find it helpful. It already has the ability to build clocktables with starting and ending dates, etc.

[seanfarley]

Yeah, I think that's what I'll have to end up doing for actual invoices (move / copy billable items to its own client file). It's nice, though, to have a query language. I'll have to keep playing around with it.

[alphapapa]

Well, you could use the :action argument to org-ql to refile or copy and paste subtrees to other places.

[seanfarley]

[alphapapa]

Uh...I'll take that as a good sign...LOL...

[JSDurand]

@alphapapa Thanks for the explanation. I agree that it is more convenient to convert to UNIX timestamps now. But FWIW, there are functions org-timestamp-up-day and org-timestamp-up which can be exploited to some extent I suppose.

[JSDurand]

Sorry if this is a noise, but I just noticed that encode-time allows out-of-range parameters, that is to say, one can just increase the day parameter by 1, and then encode and decode the time, to increase the date by 1 day. And I think this is the approach used by org-timestamp-change as I can tell.

In any case, thanks for the great package. :-D

[seanfarley]

I mean, sure, that's fine with me; it your project after all!

[seanfarley]

Just wanted to circle back and see if there was any blockers to getting this in master?

[alphapapa]

Not really. I guess I’ll make the change to keyword args and merge it. It will probably be a few days before I have time.

[alphapapa]

Well, actually, I’m not sure it’s ready yet. I think it needs more testing, and maybe more design too.

For example, imagine a task that was clocked on 1 Dec and 10 Dec (times are unimportant for this example). Then a query is run, like (clocked :from "2018-12-05" :to "2018-12-06"). I think the task should not match that query, but the way it's currently written, it would.

What do you think?

[seanfarley]

Ah, I see. That's a good question and I don't know what to suggest.

[alphapapa]

I rewrote the selector, so now it works like this:

(org-ql "test-clocked.org"
  (clocked))  ;=> ((headline ...))

(org-ql "test-clocked.org"
  (clocked :from "2018-12-11"))  ;=> nil
(org-ql "test-clocked.org"
  (clocked :from "2018-11-10"))  ;=> ((headline ...))

(org-ql "test-clocked.org"
  (clocked :to "2018-11-30"))  ;=> nil
(org-ql "test-clocked.org"
  (clocked :to "2018-12-30"))  ;=> ((headline ...))

(org-ql "test-clocked.org"
  (clocked :on "2018-11-11"))  ;=> nil
(org-ql "test-clocked.org"
  (clocked :on "2018-12-02"))  ;=> ((headline ...))

(org-ql "test-clocked.org"
  (clocked :from "2018-12-03" :to "2018-12-11"))  ;=> ((headline ...))

Using this test-clocked.org data file:

* test heading 
:LOGBOOK:
CLOCK: [2018-12-01 Sat 00:00]--[2018-12-10 Mon 23:59] => 239:59
:END:

Please let me know what you think.

[seanfarley]

Awesome! I'll check it out this weekend.

[alphapapa]

@seanfarley Did you ever take a look at this? Do you think it's ready for merging as-is?

[seanfarley]

@seanfarley Did you ever take a look at this? Do you think it's ready for merging as-is?

Yeah! I'm at a conference now and was traveling; apologies for the late reply. I've been using this as-is since you rewrote the selector and it does what I want and I've been loving it :-)

Thanks for the merge!