Make style selector annotation smart about org-ref-cite

[jkitchin]

I guess we are doing different things. I was interested in better previews on tooltips that better reflect the best guess of the intended export.

I did get that to work. And, I was able to adapt it to make the style annotation a little smarter. Now if I try to change the style on

with #+cite_export: csl

I get this annotation (with an author/date csl style):

In contrast, with my default exporter (latex) I see

It seems fast enough to me.

[jkitchin]

I think that push fully supports org-cite like styles. There are a lot of style options, since it supports every combination of full and abbreviated style. Everything I tried worked for me, but I won't be surprised if I missed some things that rely on the old style code.

I am going to take a break for a few hours probably. if you spot an issue, I can probably fix it later this afternoon, or when I get back in a week.

[jkitchin]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

[jkitchin]

I think the function in https://github.com/jkitchin/org-ref-cite/blob/main/org-ref-cite-core.el#L304 should cover this.

[jkitchin]

Even if you don't want to do this yourself, @jkitchin, I thought I'd pull this into a separate issue, in part because export processors are another area I have not yet worked with.

If this can work, the advantage is the UI can adapt to the user setup, and so it can be a general solution, without need for user config.

WDYT?

Also, little thing: how do I parse a string into a citation object?

Here is one way to what I think you asking for (it is literally what you asking for, but I am not sure why you would want to do this, what do you need it for?)

#+BEGIN_SRC emacs-lisp :results code
(let ((c "[cite/t:global prefix; @lin-2021-exper-theor suffix]"))
  (with-temp-buffer (insert c) (goto-char 1) (org-element-citation-parser)))
#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
(citation
 (:style "t" :begin 1 :post-blank 0 :end 53 :prefix
	 (#("global prefix" 0 13
	    (:parent #0)))
	 :contents-begin 23 :contents-end 52))
#+end_src

I feel like this is not possible. What would you do in an org-file that is going to be used for multiple exports, e.g. There will be a pdf version (via latex), an html version, an ODT version, and a plain text version. What does an accurate preview even mean? It will be different depending on the output.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

For my export to latex, it is possible for it to map to literally how it will export (at least on the style part). On the keys I am less concerned about it being just right, there it is sufficient for me to know what the key points too.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style).

The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

The only way I can see this working in practice is that the tooltip uses a default exporter (defined by the user), or the cite_export keyword if it is defined. Otherwise, you don't know what the export is until it is done. Sometimes, a single document is even exported to more than one format, so there isn't a single format that is even right.

I think it would be easy to implement this, but I am having trouble getting a csl string from a citation at the moment. I feel like it might be an error in my local setup, but can't figure it out right now.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'latex t)
#+END_SRC

#+RESULTS:
: \cite{lin-2021-exper-theor}


This results in an error.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
#+END_SRC

Might even involve curated input example, to be sure to show what some of the variants (notable "full" and "caps) will produce.

Originally posted by @bdarcus in #6 (comment)

[bdarcus]

Here is one way to what I think you asking for (it is literally what you asking for, but I am not sure why you would want to do this, what do you need it for?)

Thank you.

To experiment with a preview using pre-cooked data. See below.

The idea was just to run it to produce the plain text output, since it's not intended to be WYWIWG; more just help the user know what style to choose.

For my export to latex, it is possible for it to map to literally how it will export (at least on the style part). On the keys I am less concerned about it being just right, there it is sufficient for me to know what the key points too.

Yes, I agree the keys, or even the actual citation item in CSL, is not important for this. Users just need to see what to expect.

Why I asked the question above.

So for the TeX processors, would produce the same output as we have in our current UIs, but CSL would produce the kind of output you see in the csl-activate processor (which is also not a precise representation, and uses a default style).
The user would likely be able to choose which to display, and/or it would be auto-configured based on the export processors the user has set up.

The only way I can see this working in practice is that the tooltip uses a default exporter (defined by the user), or the cite_export keyword if it is defined. Otherwise, you don't know what the export is until it is done. Sometimes, a single document is even exported to more than one format, so there isn't a single format that is even right.

Does that matter?

Just to make sure I'm being clear, what I have in mind is specific to the annotation function for the style UI, and which only needs ascii

1. iterate through org-cite-supported-styles
2. for each oc export processor (how this is determined or configured TBD), run the ascii export process, and insert results in the annotation EDIT: see #9 (comment) for a simpler first step based on your test example

So if you have org-ref-cite and csl configured for 2, the annotation might look like:

\citep{doa2019}           (Doa, 2019)

... but the user wouldn't have to configure any of that, and it would work across export processors.

Also, fallback behavior for unsupported style and/or variants would automatically be accounted for.

I think it would be easy to implement this, but I am having trouble getting a csl string from a citation at the moment. I feel like it might be an error in my local setup, but can't figure it out right now.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'latex t)
#+END_SRC

#+RESULTS:
: \cite{lin-2021-exper-theor}


This results in an error.

#+BEGIN_SRC emacs-lisp
(org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
#+END_SRC

I don't get that error, FWIW.

 (org-export-string-as "[cite/t:@lin-2021-exper-theor]" 'html t)
"<p>
(NO_ITEM_DATA:lin-2021-exper-theor)</p>
"

But per above, all we care about for the annotation is the ascii; right?

On the "pre-cooked" data issue from above, I was thinking of this for two reasons:

1. because I use org-cite-make-insert processor, I can't get access to the citation, and I don't want to write all that.
2. pre-cooked data can better and more consistently show some of the variants; most notably "full" and "caps"

[bdarcus]

PS - I guess I get this because CSL is my default export processor?

 (org-export-string-as "[cite/t:@low2001]" 'ascii t)
"(Low, 2001)
"

If yes, then maybe the simplest and most general approach is to only render the default processor in the annotation, using org-export-string-as?

Aside: note how it renders with the fallback behavior here. Because the first iteration of oc-csl doesn't have support for citet citations, it falls back to the detault.

Once Andras merges his updates, that will correctly render.

Hmm .. seems slow; previews might need to be cached.

Also, it doesn't only generate output for the default backend.

(foo/bar--make-previews)
("[citeauthor:@low2001]
" "[citeauthor/f:@low2001]
" "[citeauthor/a:@low2001]
" "[citenoauthor:@low2001]
" "[citenoauthor/c:@low2001]
" "[citenoauthor/bc:@low2001]
" "[citenoauthor/b:@low2001]
" "[citetext:@low2001]
" "[citetext/bcf:@low2001]
" "[citetext/cf:@low2001]
" "[citetext/bf:@low2001]
" "[citetext/bc:@low2001]
" "[citetext/f:@low2001]
" "[citetext/c:@low2001]
" "[citetext/b:@low2001]
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
" "(Low, 2001)
")

[jkitchin]

I had something weird going on, the commits above seem to do what I want. at least I get a latex export by default for the tooltip, and I get a csl preview with a cite_export keyword set to csl.

[bdarcus]

So this would be trivial to extend for this?

[bdarcus]

WDYT?

I like it; I'd just make the CSL one ascii of course ;-)

So is this pretty straightforward then?

I guess I'm wondering if I should try to adapt your code to my group-based UI, or if you want such a group-function here? If the latter, I can send you a PR that is at least the start.

[jkitchin]

It could be interesting to see what it would look like. I don't know what I would do with a group function here at the moment.

[bdarcus]

Hook it up?

Like annotation, It's ignored if not supported, but if running Emacs 28, you get the groups.

[bdarcus]

BTW, in case it's helpful to see where it's going, this the patch for oc-csl that Andras is working on.

https://gist.github.com/andras-simonyi/c79865da0b3252199b55d093f5a4ad64

[bdarcus]

I tried to test org-ref-cite-select-style from IELM, and also org-ref-cite-annotate-style.

The latter gave me:

(org-ref-cite-annotate-style "text")
*** Eval error ***  Wrong type argument: integer-or-marker-p, nil

... and the former a stringp type error.

[bdarcus]

I also just tried outside IELM using org-cite-insert, as I would put in a README, and of course got a similar error, this time stringp.

(org-cite-register-processor 'bibtex-actions-alt-insert
  :insert (org-cite-make-insert-processor 
                #'oc-bibtex-actions-insert
                #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'bibtex-actions-alt-insert)

[jkitchin]

(org-ref-cite-annotate-style "text")

I would not expect that to work. My annotator requires the point to be on a citation, so it can get the references, and make an appropriate preview for it.

This is what I referred to in the mail list, it requires the key selector to insert the citation first, and then to update the style on it.

[bdarcus]

Right, but per subsequent comment, I also get the errors when in org, at-point.

[jkitchin]

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

[bdarcus]

That seems to work, but I only get the natbib previews. Is that intended?

I thought, from previous discussion, and looking at your code, it should use csl if either that is default, or it's defined in the document?

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

If you just use org-cite-support-styles does that not (at least partially) avoid you having to worry about this?

[jkitchin]

That seems to work, but I only get the natbib previews. Is that intended?

I thought, from previous discussion, and looking at your code, it should use csl if either that is default, or it's defined in the document?

It looks for a cite_export keyword and uses that, or defaults to latex. I have not figured out how to find what the default exporter is otherwise, because it depends on what you choose to export too, which is done "at run time". I could add a defcustom where you choose the backend you use most often, and fall through to that.

I don't have "text" as a style. I pushed a few more probable fixes for this to make it work when there is not a citation at point, and to be more robust when a style is not defined.

If you just use org-cite-support-styles does that not (at least partially) avoid you having to worry about this?

Not really. That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite. It doesn't make sense to me to have all the aliases, and to have to generate the combinations (e.g. when you asked how to get a flat list of styles before, it is not obvious).

#+BEGIN_SRC emacs-lisp :results code
(org-cite-processor-cite-styles (org-cite--get-processor 'org-ref-cite))
#+END_SRC

#+RESULTS:
#+begin_src emacs-lisp
("t" "p" "num" "a" "a/f" "a/c" "a/cf" "na/b" "na" "nocite" "t/b" "t/f" "t/bf" "t/c" "t/cf" "t/bc" "t/bcf" "/b" "/bf" "/bc" "/bcf" "/f" "/c" "/cf" nil)
#+end_src

[bdarcus]

BTW, this is another, related, topic, but the tooltip previews do work well, except since I use biblatex for the bib files, they omit the crucial date field.

How do you actually generate the previews there? With oc-csl? So then the formatting is done in citeproc-el?

If yes, that will be getting full biblatex support "any day", so that issue will be solved upstream.

If you are formatting it here, be sure to consider missing data, and maybe support "date" (though I know it's a slippery slope).

That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite.

I wonder if we should suggest changing this?

Andras had the same problem figuring out how to define it properly, and it's definitely a PITA to work with on our end.

To be clear, the style+variant distinction does matter to ensure proper fallback behavior, but that's an internal thing, and maybe simple strings at this level will work fine, and be much easier?

Because the idea of an export processor exposing the styles it supports is valuable.

WDYT?

If you support the idea, I can post it to the org list.

[bdarcus]

I have not figured out how to find what the default exporter is ...

What about this?

(cadr (assoc t org-cite-export-processors))

[jkitchin]

I have not figured out how to find what the default exporter is ...

What about this?

(cadr (assoc t org-cite-export-processors))

That is the fallthrough case, not the default. I think the best way is to look for cite_export, which is explicit when available, or default to a user chosen value.

[bdarcus]

That is the fallthrough case, not the default.

Forgive the ignorance: what's the difference?

[bdarcus]

I wonder if we should suggest changing this?

Hmm ... I guess he did it that way to support full names and short-cuts.

That might get too verbose as just strings.

But there must be some way to simplify.

[jkitchin]

BTW, this is another, related, topic, but the tooltip previews do work well, except since I use biblatex for the bib files, they omit the crucial date field.

How do you actually generate the previews there? With oc-csl? So then the formatting is done in citeproc-el?

If yes, that will be getting full biblatex support "any day", so that issue will be solved upstream.

If you are formatting it here, be sure to consider missing data, and maybe support "date" (though I know it's a slippery slope).

I don't do any formatting. It is generated from `org-export-string-as' with the export backend.

That format is very inconvenient to work with (and overly complicated imo), and I guess I have not defined the styles correctly (I use a simple, flat list of strings, with no aliases) so it returns nil for org-ref-cite.

I wonder if we should suggest changing this?

Andras had the same problem figuring out how to define it properly, and it's definitely a PITA to work with on our end.

I should think about it. The current data structure is maximally flexible in matching full and abbreviated styles. But, doing that requires what I consider an upper echelon of elisp coding via pcase and pattern matching. It also breaks the DRY idea, as the styles are defined in multiple places, e.g. in org-cite-natbib--style-to-command and where the processor is defined, which I avoided, at the expense that I didn't (but could, and it would be long as you mention above) support the full names.

To be clear, the style+variant distinction does matter to ensure proper fallback behavior, but that's an internal thing, and maybe simple strings at this level will work fine, and be much easier?

Because the idea of an export processor exposing the styles it supports is valuable.

WDYT?

If you support the idea, I can post it to the org list.

[bdarcus]

Here's an idea to consider, that would solve the DRY issue AND make this easier;

Nicolas should add a function that does the conversion to string, expose it, and use it in the included tex export processors.

Might that be the best balance?

[jkitchin]

That is the fallthrough case, not the default.

Forgive the ignorance: what's the difference?

My default for export will 99% of the time be to latex for a publication. The fall-through case is for an export backend you haven't defined in org-cite-export-processors, e.g. say I need a word doc, and export to odt. That isn't defined in there, so the the fall-through processor will be used. The recommended processor in the documetation is basic. You want this fall through to be maximally compatible with an arbitrary exporter, e.g. ascii, md, odt, jupyter notebook, calender, ...

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

[jkitchin]

Here's an idea to consider, that would solve the DRY issue AND make this easier;

Nicolas should add a function that does the conversion to string, expose it, and use it in the included tex export processors.

Might that be the best balance?

It might be better to see if we can adapt the approach I took, which is to combine the styles in Nicolas fashion with the commands they map on to. Then, you have a way to lookup the output command, and you can generate the styles in the processor definition.

I will see if I can do that this morning. I feel like it is all that is left in org-ref-cite. I am on vacation starting tomorrow, and it would be nice to wrap it up!

[bdarcus]

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

So maybe another suggestion to include in org-cite?

[jkitchin]

So, in general there is not a way to say what the desired default is without a defcustom specifying it (which I added), or by using cite_export I think.

So maybe another suggestion to include in org-cite?

I feel like I got the cite_export idea from org-cite already.

The citation triplet can also be set with the CITE_EXPORT keyword.
E.g.,

#+CITE_EXPORT: basic note numeric

or

#+CITE_EXPORT: basic

In that case, `basic' processor is used on every export, independently on the
back-end."

[bdarcus]

I will see if I can do that this morning. I feel like it is all that is left in org-ref-cite. I am on vacation starting tomorrow, and it would be nice to wrap it up!

Yes, I remember.

Maybe if you come up with a good general solution, you can submit as a patch to org when you get back?

[bdarcus]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so.

I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

[jkitchin]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so.

I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

what are you doing that causes that?

[bdarcus]

I'm starting to lose track of the different pieces of code I'm tracking, so possible this is my fault. But I don't think so.
I'm getting a stringp type error when using the update, so am thinking there's a little bug somewhere.

what are you doing that causes that?

Just M-x org-cite-insert on the "cite". Same on a key correctly opens bibtex-actions-read.

Using this:

(org-cite-register-processor 'bibtex-actions-alt
  :insert (org-cite-make-insert-processor #'oc-bibtex-actions-insert
					  #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'bibtex-actions-alt)

[jkitchin]

Does this work:

(org-cite-register-processor 'my-inserter
 :insert (org-cite-make-insert-processor #'org-ref-cite--complete-key-28
					 #'org-ref-cite-select-style))

(setq org-cite-insert-processor 'my-inserter)

in [cite:@rose-2019-pybliom]

if you are on "cite" it should change the style. Does (org-ref-cite-select-style) work?

this is the combinatorial explosion of trying to support modularity...

[bdarcus]

Does this work ...
in [cite:@rose-2019-pybliom]

if you are on "cite" it should change the style. Does (org-ref-cite-select-style) work?

No. I get a "wrong type" error on eval, similar to what I saw in my own code when I load order issues.

this is the combinatorial explosion of trying to support modularity...

I still don't understand why this is an issue here, but admittedly haven't looked at this code closely.

Maybe just remove that "28" function, and see if we can get the basics working?

Earlier today, what you had was working correctly, so I suspect some subtle thing changed since.

[bdarcus]

I did just try to first load only the org-cite-make-insert-processor bit, and then the setq, but that also didn't work; same error.

But it's a stringp error, which from my experience is not the error that arises from org-cite-make-insert-processor (when it complains currently, it gives a cryptic functionp error).

[jkitchin]

We also getting bitten by the inability to debug what comes out of org-cite-make-insert-processor. For me, the following things work.

(org-ref-cite--complete-key-28 nil) -> get one key as a string

(org-ref-cite--complete-key-28 t) -> get multiple keys as list of strings

(org-ref-cite-select-style) -> gets a style string

Are you saying that (org-ref-cite-select-style) is the problem? What commit is your org-ref-cite on?

[bdarcus]

Hmm ... outdated. e014ad0

Let me try again.

[jkitchin]

no worries. the style code is too clever IMO. It is very hard to debug.

[bdarcus]

Commit: 1092aa6

Same stringp error, even from IELM using org-ref-cite-select-style. So the issue is not related to org-cite-make-insert-processor.

[jkitchin]

I don't know what to tell you. that works for me. You will probably have to edebug it to see where it is failing. You can do that by going to the definition, and typing C-u C-xC-e

Then when you run the command it will let you step through the function by pressing n, and you can see the output and exactly where it is failing. it works for me with vanilla, vertico, ivy, and helm-mode.

I assume you have (require 'org-ref-cite-core) somewhere right?

[jkitchin]

Does this code work?

(defun test-select-style ()
  "Select a style with completion."
  (interactive)
  (let ((candidates (cl-loop for (style . command) in (org-ref-cite-get-combinatorial-style-commands)
			     collect
			     (concat (car style)
				     (when (cdr style) (format "/%s" (cdr style)))))))
    (completing-read "Style: " candidates)))

[bdarcus]

Does this code work?

(defun test-select-style ()
  "Select a style with completion."
  (interactive)
  (let ((candidates (cl-loop for (style . command) in (org-ref-cite-get-combinatorial-style-commands)
			     collect
			     (concat (car style)
				     (when (cdr style) (format "/%s" (cdr style)))))))
    (completing-read "Style: " candidates)))

ELISP> (test-select-style)
*** Eval error ***  Wrong type argument: listp, "t"
ELISP> (org-ref-cite-get-combinatorial-style-commands)
*** Eval error ***  Wrong type argument: listp, "t"

[jkitchin]

I don't understand why it works for me. What is org-ref-cite-styles set to?

[bdarcus]

ELISP> org-ref-cite-styles
(("t" . "\\cite")
 ("p" . "\\citep")
 ("num" . "\\citenum")
 ("a" . "\\citeauthor")
 ("a/f" . "\\citeauthor*")

... etc.

I'll see if I can figure it out later, when you're on vacation ;-)

I am running a recent version of Emacs 28, though I seldom run into bugs related to that. But it's possible.

[jkitchin]

oh, that is wrong. something is not updating right. see https://github.com/jkitchin/org-ref-cite/blob/main/org-ref-cite-core.el#L32.

or, something is getting loaded from somewhere else.

[bdarcus]

Ok, that would explain it. Let me see if I can figure that out.

[jkitchin]

If you have been running one session all this time, you probably have an old version in memory. You might be able to fix it by going to that file and typing C-u C-xC-e on the defcustom. it doesn't update automatically when you just require it. Maybe M-x eval-buffer would do it too.

as annoying as it is, I usually close and re-open emacs to make sure I don't have old definitions hanging around.

[bdarcus]

It's not that; seems something is wrong with straight and the git repo.

I'll just wipe that directory and use the "local" repo.

[bdarcus]

OK, progress! Is this basically what I should see?

[jkitchin]

Do the inserter's work now?

[bdarcus]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

Key thing is basic approach is working, which is great!

So something easy enough to iterate.

I'll play with it while you're gone. If I come up with some good ideas, I may send a PR.

I definitely prefer long descriptions for the group titles. Do you think that needs to be configurable?

Do the inserter's work now?

Yes; with bibtex-actions-read! I haven't looked at what you added to your README, but I'm guessing code examples should work, which is always nice.

What doesn't work is the preview mode; I still just get natbib. But should be easy enough to fix later I'd guess.

Thanks for your attention to details; enjoy the break!

[jkitchin]

yes, that is it! I don't love the grouping, but that is a bike shed to build another day!

Key thing is basic approach is working, which is great!

So something easy enough to iterate.

I'll play with it while you're gone. If I come up with some good ideas, I may send a PR.

I definitely prefer long descriptions for the group titles. Do you think that needs to be configurable?

Anything reasonable works for me. I pushed some code that does a better job grouping. It might be fragile to changes and I predict doesn't do the right thing if the styles are changed much. The order in the cond matters. A solution here might be to move this out similar to the org-ref-cite-annotate-style function. Same with the sort function I guess.

Do the inserter's work now?

Yes; with bibtex-actions-read! I haven't looked at what you added to your README, but I'm guessing code examples should work, which is always nice.

What doesn't work is the preview mode; I still just get natbib. But should be easy enough to fix later I'd guess.

Thanks for your attention to details; enjoy the break!

I think to get another preview you can try:

1. (setq org-ref-cite-default-preview-backend 'html) which will use the export processor you have defined for the 'html backend. Presumably this would be set to 'csl for you?
2. Use #+cite_export: csl as a keyword in the document.

Hopefully it is all working now, and there are just few refinements left!

[bdarcus]

1. which will use the export processor you have defined for the 'html backend. Presumably this would be set to 'csl for you?

I was actually debating with someone else today how best to configure these. Right now I just have definitions for latex and t (as you say the fallback, which is the CSL one). Do you think I should explicitly define CSL for html and odt too?

[jkitchin]

There are two things happening here I think. One is when you export a document, you need to pick an org-cite-processor for the backend to use. org-cite-export-processors is used to get it. In your case, for latex you would get a specific processor, say org-ref-cite, or natbib or biblatex (or whatever you set it to be), but for all others you would get csl. It is unambiguous to me what should happen here because you have chosen a backend to export to. If latex is the only exception to csl, what you have done will do what you need.

The other thing I am using org-cite-export-processors for is previewing. Here it is ambiguous what backend to use for making the preview, because you have not chosen a backend by exporting yet, and we have to infer what the most reasonable choice is. For me, the fallback case is not what I ever want to see, because most of the time I will export to latex, so that should be the default for preview for me (and it is set in org-ref-cite-default-preview-backend which is a defcustom). In the event I have a document that is going to be exported to html though, the latex preview is not helpful, and then I would use #+cite_export: csl for example in the document, and then the previews would use the local definition and use csl to generate the previews.

I also don't want to make the fallback case to something 'org-ref-cite, because when I export to ascii, e.g. in emails, or to odt, etc, I don't want to see latex exports there. I want either basic or csl.

I simpler solution to this I just thought of would be to add a (preview processor) line to org-cite-export-processors. Assuming that doesn't cause an issue, then you would be able to declare what processor to use for previews by default. It might be a little more awkward to change via #+cite_export: csl, and isn't a proper backend for export though.

[bdarcus]

I'm asking, BTW, not only for me and for this code, but because I set this in the Doom PR. E.g. what I do will be what a lot of other people end up doing, so I figure should do it right.

If latex is the only exception to csl, what you have done will do what you need.

So far, the only export processors are tex-based, and CSL, and the former can't do HTML or ODT, so that was my thinking.

[bdarcus]

I'm going to keep this as a running list of comments on the latest UI, that I'll update over the next week, as I think about things:

1. Given I don't use natbib at all, or have it configured anywhere for oc, I should not see these styles; but I do. Beyond tweaking the logic that determines what gets displayed, maybe we need a defcustom for the default if it can't be 100% reliable?
2. should avoid duplicate full/shortcut variants, as it dramatically expands the number of style candidates, for no obvious benefit. If necessary, maybe this could also be configurable, but maybe not initially?
3. Group titles should be as informative and output-agnostic as possible; why I replaced "text" with "Textual/Narrative". I'll think more about specifics on this.
4. Not liking the "misc" group; I had this "default"? Another thing I'll think about.
5. Probably should sort the groups based on how common somehow? LIke, "default" should certainly be first, and maybe "text" second? This is less an issue if 2 is addressed though, and I do realize if construct the UI from org-cite-supported-styles; but "default" applies to all export processors.

[bdarcus]

On 2 above, here's what I currently see; that first group should only have four candidates.

Maybe could have a defcustom that toggled short vs long for style and variant, with possible values of:

• long-long
• long-short (probably default)
• short-short

Am I right this should be straightforward to handle?

I guess the only issue is existing documents that don't use the same pattern; but that seems a minor issue on balance best to ignore for now?

cc @andras-simonyi

[jkitchin]

I'm going to keep this as a running list of comments on the latest UI, that I'll update over the next week, as I think about things:

1. Given I don't use natbib at all, or have it configured anywhere for oc, I should not see these styles; but I do. Beyond tweaking the logic that determines what gets displayed, maybe we need a defcustom for the default if it can't be 100% reliable?

I see the issue, but not a solution at the moment. The issue is that my code uses the variable org-ref-cite-styles to populate the selection, and not (org-cite-supported-styles). These two things are not currently compatible because in org-ref-cite I combine the style and export command to avoid duplication elsewhere, and (org-cite-supported-styles) does not provide an export command. In the annotate function I have tried setting it up to use export to generate these, which might help a little on annotation, but it is very slow for csl/html for some reason at the moment.

This way, in principle the solution might be to set org-ref-cite-styles to the styles you use.

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

2. should avoid duplicate full/shortcut variants, as it dramatically expands the number of style candidates, for no obvious benefit. If necessary, maybe this could also be configurable, but maybe not initially?

The simplest way to avoid this is to setq org-ref-cite-styles to have what you want in it, e.g. don't list the ones you don't want to see. So, if you want long-short, it would look like: ((("text") ("b")) . "\\citealt") for example.

The problem I see is there is only a convention that you put ("long" "short") in that order, it is not required. So, there is no way to be sure a setting like long-short will actually be that. Something like first-second would be more consistent, but less obvious too.

I set this up with a default to short-short (my preference) to try it out anyway.

3. Group titles should be as informative and output-agnostic as possible; why I replaced "text" with "Textual/Narrative". I'll think more about specifics on this.

I don't have strong feelings here, at least until ivy supports groups.

4. Not liking the "misc" group; I had this "default"? Another thing I'll think about.

There is only one that ends up there at the moment (the one that maps to \cite). I don't feel real strongly about it, but the grouping function is pretty fragile imo, and relies on heuristics defined in org-ref-cite. It won't work for arbitrary styles defined by others. For that, we would have to redefine the style data structure to include group info. in any case there needs to be a fall-through group to catch everything that doesn't end up in a group, and that group needs a name.

The really bare \cite could be default, it is actually what we use the most because of a latex package called natmove which only works on \cite commands I think (and not on \citet). I haven't gotten to use this to write an actual paper yet though, so I haven't had a chance to see how the current choices actually work in practice.

5. Probably should sort the groups based on how common somehow? LIke, "default" should certainly be first, and maybe "text" second? This is less an issue if 2 is addressed though, and I do realize if construct the UI from org-cite-supported-styles; but "default" applies to all export processors.

I guess this could also be a defcustom. text is first for me always. Miscellaneous is actually second, because we use citenum somewhat frequently, e.g. see Ref. \citenum{key} so the citation number is regular text, and not stylized. Clearly this is user dependent.

I don't see how one sorts groups in completing read though. Is there a way to do that?

[bdarcus]

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

So there's two questions I see here, assuming there's value in a general style selector untied to any particular export-processor:

1. a basic UX one: what styles and previews should the style selector present to the user?
2. if it's at least optionally the styles supported by the active export processor, should we propose org-cite-supported-styles add an optional parameter to limit the returned result to some export processor?

Because the design of org-cite and its included processors assumes consistent style and variant names across export processors, and has a fallback logic for unsupported styles and/or variants, my assumption on 1 is just present all of what org-cite-supported-styles returns, complete with preview, which will also indicate whether fallback is active.

Like, I have oc-biblatex and oc-csl configured. The former has a wider range of style/variants, but I have no problem seeing and using those, even if my primary export processor is oc-csl.

Does that make sense?

If there's also value in restricting the displayed style options only to whatever export processor is active, should we propose 2 (or if you have FSF copyright assignment, just submit a patch)?

cc @andras-simonyi

[bdarcus]

I don't see how one sorts groups in completing read though. Is there a way to do that?

I'm actually a little unclear on some details here, but this is what I have:

                 `(metadata
                   (annotation-function . oc-bibtex-actions--style-preview-annote)
                   (cycle-sort-function . identity)
                   (display-sort-function . identity)
                   (group-function . oc-bibtex-actions--styles-group-fn))

It puts "default" and then "textual" first, but I'm not clear why now that I look.

[jkitchin]

It would be a lot of redirection to try to get the styles from (org-cite-supported-styles) (again having to guess which export processor to grab them from), and then construct the desired list (combinations of long/short), and then export each candidate accordingly.

So there's two questions I see here, assuming there's value in a general style selector untied to any particular export-processor:

1. a basic UX one: what styles and previews should the style selector present to the user?
2. if it's at least optionally the styles supported by the active export processor, should we propose org-cite-supported-styles add an optional parameter to limit the returned result to some export processor?

org-cite-supported-styles already supports this, you just pass a list of export processors that you want styles for.

We are still stuck guessing what the export backend/processor is here though because it isn't actually defined until you run an export. I added some code to try doing this. It sort of works, but feels slow at the moment for annotation.

Because the design of org-cite and its included processors assumes consistent style and variant names across export processors, and has a fallback logic for unsupported styles and/or variants, my assumption on 1 is just present all of what org-cite-supported-styles returns, complete with preview, which will also indicate whether fallback is active.

Like, I have oc-biblatex and oc-csl configured. The former has a wider range of style/variants, but I have no problem seeing and using those, even if my primary export processor is oc-csl.

Does that make sense?

Why do you want to have long-long, long-short, etc... filtering of the styles then?

If there's also value in restricting the displayed style options only to whatever export processor is active, should we propose 2 (or if you have FSF copyright assignment, just submit a patch)?

I don't think I did the right thing in the commit a4809a2. I need to make it only filter what is shown, not remove them. Otherwise, two users with different preferences won't be able to use the same document because the styles won't match. I don't like the complexity this adds.

cc @andras-simonyi

[bdarcus]

org-cite-supported-styles already supports this, you just pass a list of export processors that you want styles for.

Ah, right.

We are still stuck guessing what the export backend/processor is here though because it isn't actually defined until you run an export. I added some code to try doing this. It sort of works, but feels slow at the moment for annotation.

So then isn't the easy solution just have it configurable, and let the user decide what they want to see?

(setq org-ref-cite-styles-include '(org-ref-cite csl biblatex))

Why do you want to have long-long, long-short, etc... filtering of the styles then?

Just that I think it's confusing, particularly for new users that won't understand.

And for me personally, it's also kind of overwhelming.

It doesn't bother you?

Edit: with forthcoming enhancements to oc-csl, the list of csl and biblatex styles is awfully similar:

ELISP> (org-cite-supported-styles '(biblatex))
((("author" "a")
  ("caps-full" "cf")
  ("full" "f")
  ("caps" "c"))
 (("locators" "l")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b"))
 (("noauthor" "na"))
 (("text" "t")
  ("caps" "c"))
 (("nil")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b")))

ELISP> (org-cite-supported-styles '(csl))
((("author" "a")
  ("caps-full" "cf")
  ("caps" "c")
  ("full" "f"))
 (("noauthor" "na")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b"))
 (("year" "y")
  ("bare" "b"))
 (("text" "t")
  ("caps-full" "cf")
  ("full" "f")
  ("caps" "c"))
 (("nil")
  ("bare-caps" "bc")
  ("caps" "c")
  ("bare" "b")))

[jkitchin]

I don't see how one sorts groups in completing read though. Is there a way to do that?

I'm actually a little unclear on some details here, but this is what I have:

                 `(metadata
                   (annotation-function . oc-bibtex-actions--style-preview-annote)
                   (cycle-sort-function . identity)
                   (display-sort-function . identity)
                   (group-function . oc-bibtex-actions--styles-group-fn))

It puts "default" and then "textual" first, but I'm not clear why now that I look.

I can't tell either, I think that the group order comes from the lexical ordering of the candidate strings, in other words that the candidates are sorted, then grouped, and the groups show in the order they appear in the candidates.

If you change display-sort-function you can change the order of the groups, e.g. by reverse alphabetically sorting them. To prescribe a group order, you have to sort the candidates by group, say by assigning an integer to each group in the order you want them in. Here is an example of getting the group order to be medium, low, high. I probably wouldn't support doing this though.

(completing-read "choose: "
		 (lambda (str pred action)
		   (if (eq action 'metadata)
		       `(metadata
			 (display-sort-function . (lambda (candidates)
						    (sort candidates (lambda (a b)
								       (let ((g1 (cond
										  ((string< a "3")
										   2)
										  ((string< a "6")
										   1)
										  (t
										   3)))
									     (g2 (cond
										  ((string< b "3")
										   2)
										  ((string< b "6")
										   1)
										  (t
										   3))))
									 (< g1 g2))))))

			 (group-function . (lambda (candidate transform)
					     (if transform
						 candidate
					       (cond
						((string< candidate "3")
						 "low")
						((string< candidate "6")
						 "medium")
						(t
						 "high"))))))
		     (complete-with-action action '("5" "9" "1" "2" "4" "08") str pred))))

[bdarcus]

I think that the group order comes from the lexical ordering of the candidate strings, in other words that the candidates are sorted, then grouped, and the groups show in the order they appear in the candidates.

That makes sense, and explains why the "/" ones go first (though why does "text" go second for me?).

I wonder if it's possible to sort groups by recency of member selection (which is default behavior in a lot of completion UIs without the grouping)?

In any case, group order is much less of an issue if there are only 10 or 15 candidates rather than 40 or 50. This is what I currently have, which is based on static previews, showing the long-short options only.

[bdarcus]

This should work for the configurable candidates list; just need error handling when a processor isn't defined but specified for PROC ...

(defcustom org-ref-cite-styles-format 'long
  "Style format; whether to use full style names or shortcuts.
Variants will always use shortcuts."
  :type '(choice
          (const long)
          (const short)))

(defcustom org-ref-cite-style-targets nil
  "Export processor targets to include in styles list.

If nil, use 'org-cite-supported-styles' without arg."
  :group 'org-ref-cite
  :type '(repeat :tag "org-cite export processor" symbol))

(defun org-ref-cite--flat-supported-styles (&optional proc)
  "Return a flat list of supported styles.

Converts 'org-cite-supported-styles' to a flat list for use
as completion candidates.

With PROC list, limits to specific processors."
  (let ((styles (list)))
    (cl-loop for s in
             (org-cite-supported-styles
              (or proc org-ref-cite-style-targets)) do
             (let* ((style-name
                     (if (eq 'long org-ref-cite-styles-format)
                         (caar s)(cadar s)))
                    (style
                     (if (string= "nil" style-name) "/" style-name)))
               (push (if (string= "" style) "/" style) styles)
               (cl-loop for v in (cdr s) do
                        (push (concat style "/" (cadr v)) styles))))
    styles))

Result:

("year/b" "year" "/b" "/c" "/bc" "/" "text/c" "text/f" "text/cf" "text" "noauthor/b" "noauthor/c" "noauthor/bc" "noauthor" "locators/b" "locators/c" "locators/bc" "locators" "author/f" "author/c" "author/cf" "author")

[bdarcus]

I got some input from the author of the group function (and of vertico), on how group-sorting works.

The group order follows the candidate order. The group of the first candidate comes first. If you want the groups to be sorted, you should sort the candidates first and specifiy display-sort-function=identity. The reason for this is, that completion styles like flex actively reorder candidates in order to get the best match first. Furthermore the completion UI may sort by history position. Grouping should not interfere with these abilities. If you set display/cycle-sort-function to identity you opt out of this, which may be a good idea in some special cases, e.g. consult-line. But I doubt you have such a case here and I generally discourage interfering with sorting.

Basically, your hunch was right.

But if we don't specify a display-sort function at all, the completion UI will usually order the groups based on the history.

E.g. if you use the "text" styles most frequently, that group will come first.

I just tested this with vertico, and it works well.

Upshot: remove the display-sort/cycle-sort lines and we don't have to worry about it.

Can also do this, for the initial display:

          ;; Sort the list upfront, but let completion UI handle beyond that.
         (let ((styles (sort (foo--flat-supported-styles) 'string-lessp))))

[bdarcus]

BTW, this is an experiment, but here's using faces to distinguish styles sans variants from styles with variants; the same faces I use to distinguish main bib entry metadata from secondary.

Just to give a hint to users. I'm still not sure though.