pcubillos / bibmanager Goto Github PK

Version 1.2 will add some further level of auto-completion for bibm search.
How about adding dynamic autocomplete? e.g.,
If the user types:

bibm search
author:"Lastname" year: 

On tab-completion, I'd like to show the years available only for the entries matching author.

Hallo,

thanks for this awesome project. Especially in combination with the ads it seems to be a very helpful tool!
Do you think it would be easily possible to include subfiles in the latex compilation?
I am just writing my masterthesis and would love to use it. However as my thesis will be longer then a few pages its hopeless to put all text into a single tex file.

The code should ignore mismatched braces when dealing with comments.

Here's a MWE:

text = """@ARTICLE{2010A&A...518L...1P,
   author = {{Pilbratt}, G.~L. and others},
    title = "{Herschel Space Observatory. An ESA facility for far-infrared and submillimetre astronomy}",
  journal = {\aap},
archivePrefix = "arXiv",
   eprint = {1005.5331},
 primaryClass = "astro-ph.IM",
 keywords = {space vehicles, space vehicles: instruments, infrared: general, submillimetre: general},
     year = 2010,
    month = jul,
   volume = 518,
      eid = {L1},
    pages = {L1},
      doi = {10.1051/0004-6361/201014759},
   adsurl = {http://adsabs.harvard.edu/abs/2010A%26A...518L...1P},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Any text like this one outside an entry (e.g., [at]article{...}) count as comments
{Riedinger}, J.~R. and {Passvogel}, T. and
       {Crone}, G. and {Doyle}, D. and {Gageur}, U. and {Heras}, A.~M. and
       {Jewell}, C. and {Metcalfe}, L. and {Ott}, S. and {Schmidt}, M.
       }"""

import bibmanager.bib_manager as bm
bibs = bm.loadfile(text=text)

ValueError: Mismatched braces in line 23:
'       }'

Need a config file to handle user-set variables

The Bibtex entry type "misc" has no required fields (https://en.wikipedia.org/wiki/BibTeX#Entry_types) but bibm refuses to take this bibtex file:

@MISC{Nasa_undated-bs,
  title  = "{NASA} {SMD} Annual Report 2018",
  author = "{NASA}"
}

$ bibm merge test.bib                                                                        (py37)
Traceback (most recent call last):
  File "/Users/klay6683/miniconda3/envs/py37/bin/bibm", line 10, in <module>
    sys.exit(main())
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/__main__.py", line 748, in main
    args.func(args)
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/__main__.py", line 83, in cli_merge
    bm.merge(bibfile=args.bibfile, take=args.take)
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 623, in merge
    new = loadfile(bibfile)
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 499, in loadfile
    bibs = [Bib(entry) for entry in entries]
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 499, in <listcomp>
    bibs = [Bib(entry) for entry in entries]
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 133, in __init__
    raise ValueError(f"Bibtex entry '{self.key}' is missing author, "
ValueError: Bibtex entry 'Nasa_undated-bs' is missing author, title, or year.

Note, i'm not saying this is a good bibtex entry, as obviously that NASA report has authors, years etc. But formally speaking bibm shouldn't reject this entry, as it's defined by BibTex to not require any field. Of course bibm could decide that @misc entries are out of scope for itself.

I tried cloning and installing and immediately ran into:

 python setup.py install
Traceback (most recent call last):
  File "setup.py", line 58, in <module>
    import bibmanager as bibm
  File "/Users/adam/repos/bibmanager/bibmanager/__init__.py", line 7, in <module>
    from . import bib_manager
  File "/Users/adam/repos/bibmanager/bibmanager/bib_manager/__init__.py", line 4, in <module>
    from .bib_manager import *
  File "/Users/adam/repos/bibmanager/bibmanager/bib_manager/bib_manager.py", line 31, in <module>
    from prompt_toolkit.formatted_text import PygmentsTokens
ModuleNotFoundError: No module named 'prompt_toolkit.formatted_text'

Do I need a particular version of prompt_toolkit?

Trigger a cleaner, more informative error output than:

$ bibm latex hst_nuv_hd209458b.tex
Traceback (most recent call last):
  File "/home/pcubillos/py36/bin/bibm", line 11, in <module>
    load_entry_point('bibmanager==1.0.0', 'console_scripts', 'bibm')()
  File "/home/pcubillos/py36/lib/python3.6/site-packages/bibmanager/__main__.py", line 668, in main
    args.func(args)
  File "/home/pcubillos/py36/lib/python3.6/site-packages/bibmanager/__main__.py", line 171, in cli_latex
    lm.compile_latex(args.texfile, args.paper)
  File "/home/pcubillos/py36/lib/python3.6/site-packages/bibmanager/latex_manager/latex_manager.py", line 272, in compile_latex
    missing = build_bib(f'{texfile}.tex')
  File "/home/pcubillos/py36/lib/python3.6/site-packages/bibmanager/latex_manager/latex_manager.py", line 177, in build_bib
    with open(f"{path}/{texfile}.tex", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/pcubillos/IWF/proposals/2019_hst/hst_hd207217b.tex'

Enable command-line interface for available features (most of if except ADS management).

After one installs bibmanager using pip install bibmanager, the bibmanager.bib_manager.init() function is not executed, and therefore the database is never initialized.
This will result into an error stating that No section: 'BIBMANAGER' whenever anything related to the command line commands is used, as it cannot find the config file.
Executing python -c import bibmanager; bibmanager.bib_manager.init() will fix this problem, although it will raise an error stating that the ~/.bibmanager.bm_bibliography.bib file cannot be found (which normally was ignored in the setup.py file and thus does not matter).

This problem is also the one that causes the problem in #45.

An edited book has many contributions. Each contribution has its own DOI number, but also shares its parent book's ISBN. Contributions and parent reference are now regarded as duplicates in bibmanager and merge always triggers asking. I have tried to disable exporting ISBN of contributions to bibfile. And this solved the annoyance of asking. But if only focusing on bibfile, this workaround seems lost ISBN information then. Is there a better way to settle this problem down?

Generate bibfile from \cite calls in a tex file.

Bibmanager sets the protocol of pickle as pickle.HIGHEST_PROTOCOL and it is 4 for Python version <=3.7, while it is 5 for Python 3.8 (https://docs.python.org/3/library/pickle.html#data-stream-format). Because of this, the bib database created with Python 3.8 does not work if bibmanager is used in an environment using Python 3.6 or 3.7. In fact, I got the following error.

Traceback (most recent call last):
  File "/Users/monodera/anaconda3/envs/astroconda/bin/bibm", line 8, in <module>
    sys.exit(main())
  File "/Users/monodera/anaconda3/envs/astroconda/lib/python3.6/site-packages/bibmanager/__main__.py", line 1074, in main
    pickle_ver = bm.get_version()
  File "/Users/monodera/anaconda3/envs/astroconda/lib/python3.6/site-packages/bibmanager/bib_manager/bib_manager.py", line 693, in get_version
    dummy = pickle.load(handle)
ValueError: unsupported pickle protocol: 5

Is there a way to recover the database for the latter environment (Python 3.6 or 3.7)?

Additionally, as the protocol 4 is common among the supported Python versions, is there any major problems if the protocol is set as 4 explicitly?

Hola Patricio!

Es Andrés Guzmán, estaba probando este tremendo software q hiciste. La mansa pega.

Tengo mi archivo bibliografia.bib y le puse
> bibm merge ~/Library/texmf/bibtex/bib/bibliografia.bib
Merged 60 new entries.
Merged BibTeX file '/Users/andres/Library/texmf/bibtex/bib/bibliografia.bib' into bibmanager database.

O sea me agarró 60, pero tengo claramente más
> grep -c ARTICLE ~/Library/texmf/bibtex/bib/bibliografia.bib
928

Creo que el problema es que sólo pescó las que están escritas así

@Article{Adams1985ApJ,
author = {{Adams}, F.~C. and {Shu}, F.~H.},
title = "{Infrared emission from protostars.}",
journal = {\apj},
keywords = {Infrared Stars, Pre-Main Sequence Stars, Protostars, Stellar Radiation, Near Infrared Radiation, Optical Thickness, Radiative Transfer, Ray Tracing, Spectral Energy Distribution, Stellar Envelopes, Stellar Temperature, Temperature Profiles, Astrophysics},
year = 1985,
month = sep,
volume = {296},
pages = {655-669},
doi = {10.1086/163483},
adsurl = {https://ui.adsabs.harvard.edu/abs/1985ApJ...296..655A},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Pero tengo muchas más escritas en una misma línea, así

@Article{Adams1991ApJ, author = {{Adams}, F.~C.}, title = "{Asymptotic theory for the spatial distribution of protostellar emission}", journal = {\apj}, keywords = {ASYMPTOTIC METHODS, EMISSION SPECTRA, PROTOSTARS, SPATIAL DISTRIBUTION, STAR FORMATION, COMPUTATIONAL ASTROPHYSICS, DENSITY DISTRIBUTION, PRE-MAIN SEQUENCE STARS, STELLAR ENVELOPES, STELLAR LUMINOSITY, TEMPERATURE DISTRIBUTION}, year = 1991, month = dec, volume = 382, pages = {544-554}, doi = {10.1086/170741}, adsurl = {http://adsabs.harvard.edu/abs/1991ApJ...382..544A}, adsnote = {Provided by the SAO/NASA Astrophysics Data System} }

Efectivamente, cuando cuanto las que están escritas con "newlines" separando las entradas de bibtex
> grep -cP '^\s*@[^,]+,\s*$' ~/Library/texmf/bibtex/bib/bibliografia.bib
60

son justo las 60 que considera.

¿Es esto un comportamiento esperado del programa, o pensarías en arreglarlo? Bibtex reconoce las referencias puestas en una sola línea como válidas.

Saludos!

Hi,

since some people prefer conda over pip, what do you think about packaging it for anaconda?
Packaging a pypi project for anaconda is actually quite simple and well explained: https://github.com/conda-forge/staged-recipes.
The conda forge bot will even automatically update the recipe once there is a new version on pypi.

Thanks again for this incredible tool! Its in my daily use for organising my references.

My usual way to add a paper is the following:
bibm ads-search
bibm ads-add (I always use AuthorYear as key)
bibm open (to fetch the pdf)

I have a quick thought:
Why not link these commands? Maybe by using a format specifier (e.g. AuthorYear) for the citekey?
If overwriting ads-search isn't feasable, then maybe add a new command, like ads-search-add

What do you think? 🤔

Again: Very nice tool!

I recently discovered that ads-update deletes all tags (only applies to bib entries which have an ADS url)

To reproduce:

1. bibm edit or bibm tag -> add tag to entry (entry needs to have an ADS url)
2. bibm ads-update

Hi folks, my .bib file has some 2500 papers. The pdfs exist for all but a dozen or so. The bibtex file has file= entries but the paths are relative paths, not absolute paths. Can I still import these in a way that I will be able to open them with bibmanager?

Best,
R Haynes

Take a look at this:
https://argcomplete.readthedocs.io/en/latest/

Hi,

For systematic literature reviews, it is useful to fetch all papers matching some keywords published within some period of time, and read them all.

I imagined bibmanager could help me here, even downloading the associated PDFs.

However, I cannot figure out how to automatically add ADS search results. I would have expected something like this:

bibm ads-search "citation_count:[300 TO 1000000]" |bibm ads-add

or

bibm ads-search --add "citation_count:[300 TO 1000000]"

However, ads-search does not seem to

• support adding all results
• support passing search keywords by argument (non-interactive use)

ads-add does not seem to

• support batch mode, adding multiple bibcodes

Hello,

When I run ads-update all tag information is lost in the database. I'm hoping this is an easy fix but if not no worries.

Thanks for this great package,
Girish.

This input breaks the ads-search code:

bibm ads-search 'author:"^arnaud" year:1985'

Find out why, and fix it. Seems to be related to the screen-print-out, not to the ADS querry.

I just installed via pip: pip install bibmanager into my Python 3.7 environment.

I received bibmanager-1.1.5.

Then I created an ads_token and used the command as given in the docs:

$ bibm config ads_token my_ads_token_replaced_for_privacy                             (py37)
Traceback (most recent call last):
  File "/Users/klay6683/miniconda3/envs/py37/bin/bibm", line 10, in <module>
    sys.exit(main())
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/__main__.py", line 585, in main
    default=cm.get('paper'))
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/site-packages/bibmanager/config_manager/config_manager.py", line 132, in get
    raise ValueError(f"'{key}' is not a valid bibmanager config parameter.\n"
  File "/Users/klay6683/miniconda3/envs/py37/lib/python3.7/configparser.py", line 674, in options
    raise NoSectionError(section) from None
configparser.NoSectionError: No section: 'BIBMANAGER'

Hi folks, bibm merge balks at merging a .bib entry for a book if it doesn't have an author. In my case it has an editor and the bibtex documentation says that the required field is an author OR and editor.

Best,
R Haynes

Required features for basic database management:

• Create bib database from input bibfile.
• Merge new bibfile into database.
• Export database into bibfile.
• Add entries manually.
• Edit database manually.
• Enable basic querries from database (by author, year, and title).
• document functions.

Need to setup documentation:

• Basics/Intro
• Tutorials
• Examples (same as tutorials?)
• Logo

I do really like using bibmanager for managing my BibTeX entries, but it kinda annoys me that it automatically uses ~/.bibmanager as the directory to store all information in.
As I use several different machines, I would like to use only one database across every single one of them.
Currently, I am forced to make a softlink between ~/.bibmanager and the actual address where the directory is stored, which is then synchronized on all of my computers.

It would be quite a bit easier if I could setup bibmanager manually to directly point to the actual address by using one of its command line commands, which would simply default to ~/.bibmanager if the user does not set it.

As pointed out by @Serenade600 if you could put in your coding source some

encoding="utf-8"

right ahead the right brackets in

open(bibfile,****)

to encode with UTF-8, which finally become

open(bibfile,****,encoding="utf-8")

, I would much more appreciate it. In last two versions I had to add them myself every time after updating. I am making this suggestion as it does no harm to other functions but only add popularity with little work.

Enable latex-bibtex-latex-latex calls by bm.
This requires to enable the command-line interface first.

Need to handle errors.

Latest version 1.3.4. I'm not sure I am using it properly. But when using merge function, the following appears. If deleting ISBN duplicates in bibfile , this would not happen.

Traceback (most recent call last):
  File "e:\python\anzhuang\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "e:\python\anzhuang\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "E:\python\anzhuang\Scripts\bibm.exe\__main__.py", line 7, in <module>
  File "e:\python\anzhuang\lib\site-packages\bibmanager\__main__.py", line 1163, in main
    args.func(args)
  File "e:\python\anzhuang\lib\site-packages\bibmanager\__main__.py", line 94, in cli_merge
    bm.merge(bibfile=args.bibfile, take=args.take)
  File "e:\python\anzhuang\lib\site-packages\bibmanager\bib_manager\bib_manager.py", line 844, in merge
    new = read_file(bibfile)
  File "e:\python\anzhuang\lib\site-packages\bibmanager\bib_manager\bib_manager.py", line 620, in read_file
    remove_duplicates(bibs, "isbn")
  File "e:\python\anzhuang\lib\site-packages\bibmanager\bib_manager\bib_manager.py", line 471, in remove_duplicates
    doi_multis = np.where((doi_counts > 1) & (ubib != ""))[0]
ValueError: operands could not be broadcast together with shapes (2,) (5,)

Hi folks, I have a rather large .bib file with all my references (approx 2200). Not surprisingly a certain percentage have "problems" - some missing entries (missing publisher etc), it is great if they are flagged for future updates but it would be great if I was still able to continue with the merge. Currently bibmanager quits when it encounters such an entry.

Not all latex files require compiling a bibtex bibliography.

When compiling a latex file that does not have a bibliography call, don't raise an error, but rather throw a warning and complete the latex calls.

Was just wondering if, in the long term, this project could add the pdf management of the document for the bibtex reference as well?
Would be fantastic, as it could potentially replace the half-working paperpile.com, Papers.app and ReadCube apps.
In connection with an editor plugin for searching the database, that's all I ever wanted from a reference manager. ;)

For example
@Article{2008A&A...485..137C,
author = {{Caratti o Garatti}, A. and {Froebrich}, D. and {Eisl{"o}ffel}, J. and
{Giannini}, T. and {Nisini}, B.},
title = "{Molecular jets driven by high-mass protostars: a detailed study of the IRAS 20126+4104 jet}",
journal = {\aap},
keywords = {stars: pre-main-sequence, infrared: ISM, ISM: jets and outflows, ISM: kinematics and dynamics, individual objects: IRAS 20126+4104, Astrophysics},
year = 2008,
month = jul,
volume = {485},
number = {1},
pages = {137-152},
doi = {10.1051/0004-6361:200809515},
archivePrefix = {arXiv},
eprint = {0803.2012},
primaryClass = {astro-ph},
adsurl = {https://ui.adsabs.harvard.edu/abs/2008A&A...485..137C},
adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

I get
bibm search
(Press 'tab' for autocomplete)
author:"^Caratti o Garatti"
``

Hi - I hope things are well. I wanted to try the newest version, so I cloned the repo from github, unfortunately I get an error:

python setup.py develop
File "setup.py", line 57
"""
^
SyntaxError: invalid syntax

Is it possible that I am missing something?

basically, a new option for bibm search.
Maybe:
bibm search [-b | --bibcode] bibcode
bibm search [-k | --key] key

I currently annotate my hard copies of papers with personally defined tags, similar to the Unified Astronomy Thesaurus but more flexible because I can write down any tag I want, and keep track of which papers are associated with which tags by writing them down in a notebook. This is very useful when I want to review the literature on a particular topic, but of course not ideal because I do not want to print every paper I read.

Is it possible to add user-defined tags along with the citation key when adding entries, and to later incorporate these tags like ADS keywords during a bibm search?

Hi - many thanks for the tool. It would be fantastic if there was a way to incorporate the tool with mathscinet for example? Would this be difficult?

The stem of identical ADS bibcodes can differ, though they are the same entry. Thus, the bibcode is a more faithful field for comparison.

Hi,

I have a small bug to report. Sometimes, when I hit bibm ads-update and the shell returns

There were 1 entries updated from ArXiv to their peer-reviewed version.

I get the following error upon compiling a bibliography (bibm pdflatex):

! Undefined control sequence.
l.466   [\eprint
                [arXiv]{2009.09444}]

here is the corrisponding bibliography entry:

pdf: Notsu2020_arXiv_2009_9444.pdf
@ARTICLE{Notsu2020,
       author = {{Notsu}, Shota and {Eistrup}, Christian and {Walsh}, Catherine and
         {Nomura}, Hideko},
        title = "{The composition of hot Jupiter atmospheres assembled within chemically evolved protoplanetary discs}",
      journal = {\mnras},
     keywords = {astrochemistry, protoplanetary discs, ISM: molecules, planets and satellites: gaseous planets, planets and satellites: atmospheres, Astrophysics - Earth and Planetary Astrophysics, Astrophysics - Solar and Stellar Astrophysics},
         year = 2020,
        month = sep,
          doi = {10.1093/mnras/staa2944},
archivePrefix = {arXiv},
       eprint = {2009.09444},
 primaryClass = {astro-ph.EP},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2020MNRAS.tmp.2754N},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

I can silene the error by removing the eprint line.

Tested:
removed entry completely using bibm edit and readded with

bibm ads-add 2020MNRAS.tmp.2754N Notsu2020

The error is still the same. Looks like this is an ADS error?

When storing the bibmanager database, save BibTeX entries as a list of dictionaries, rather than a list of Bib() objects, to avoid versioning issues with pickle.

I have a long list of ADS bib file but there are some incorrectly formatted entries that I like to fix. Is there a way bibm can point which line in the input file is causing trouble?

$ bibm cleanup ref.bib 

Traceback (most recent call last):
  File "/home/jp/miniconda3/envs/py3/bin/bibm", line 8, in <module>
    sys.exit(main())
  File "/home/jp/miniconda3/envs/py3/lib/python3.7/site-packages/bibmanager/__main__.py", line 1135, in main
    args.func(args)
  File "/home/jp/miniconda3/envs/py3/lib/python3.7/site-packages/bibmanager/__main__.py", line 201, in cli_cleanup
    bibs = bm.loadfile(args.bibfile)
  File "/home/jp/miniconda3/envs/py3/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 556, in loadfile
    bibs = [Bib(entry, **meta) for entry,meta in zip(entries,meta_info)]
  File "/home/jp/miniconda3/envs/py3/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 556, in <listcomp>
    bibs = [Bib(entry, **meta) for entry,meta in zip(entries,meta_info)]
  File "/home/jp/miniconda3/envs/py3/lib/python3.7/site-packages/bibmanager/bib_manager/bib_manager.py", line 135, in __init__
    raise ValueError(f'Invalid month value ({value})')
ValueError: Invalid month value (x)

Required features for ADS IO:

• Enable queries from ADS.
• Add entries into database from ADS keys.
• Update existing entries with ADS query (either eprint -> accepted or everything).

Is it possible to catch this issue?
(maybe fix it)

$ bibm ads-add 2019A&A...000A..00N NameEtal2019aaWASP000b
A...623A..57S: command not found
Error: Invalid input, 'bibm ads-add' expects either zero or two arguments.

So, this looks like an awesome project, but when I write latex in my favorite editor (vscode), how would I search through the bibmanager database to get the citekey? Anybody created an extension yet for any of the better editors? (Textmate2, SublimeText, Atom, VSCode?)

Add example files and docs

Need to automate testing framework.
pytest + travis CI + hypothesis (?)

Required features for packaging:

• Make Python package structure.
• pip searchable package (not really a git task).

Hi,

Can it fetch PDFs files from the journal and/or arxiv and link them as a file= entry in the bibtex?

The old Jabref had a great extension called LocalCopy https://github.com/lehner/LocalCopy that did this marvelously. Alas, old Jabref went away.

Cheers,
Johannes

First I would like to thank you for making such an awesome tool.

When I use bibmanager, I want to have an overview interface for all items in the database. Optionally, it may further allow users to select one (or multiple) items an export them with key-binding or cursor. I think this would make bibmanager easier to use.