citation-js / bibtex-parser-experiments

Experiments to determine a new BibTeX parser formula for Citation.js -- to be applied to other formats as well

Citation.js converts formats like BibTeX, Wikidata JSON and BibJSON to CSL-JSON to convert to other formats like APA, Vancouver, RIS and back to BibTeX.

Read the paper Citation.js: a format-independent, modular bibliography tool for the browser and command line.

---

Site • Repo • Getting Started • Documentation • Demo

---

Packages

citation-js/citation-js	replaces	larsgw/citation.js
This repository contains the npm package @citation-js/core and several other components.		This repository contains the npm package citation-js that wraps the aforementioned components for backwards compatibility.

Core

Core functionality:

• Cite: reference manager
• plugins: plugins manager
• util: several utility functions and classes
• version

Plugins

Plugin	Description
plugin-bibjson	Plugin for BibJSON formats for Citation.js
plugin-bibtex	Plugin for BibTeX formats for Citation.js
plugin-csl	Plugin for CSL output for Citation.js
plugin-doi	Plugin for DOI input for Citation.js
plugin-ris	Plugin for RIS formats for Citation.js
plugin-wikidata	Plugin for Wikidata for Citation.js

Plugins in other repositories

Plugin	Description
plugin-software-formats	Plugin for CFF, Zenodo JSON, input from GitHub and npm URLs
plugin-isbn	Plugin for ISBNs (from Google Books, OpenLibrary)
plugin-orcid	Plugin for ORCID profiles (metadata from DOIs and ISBNs)
plugin-pubmed	Plugin for PubMed and PubMed Central identifiers
plugin-quickstatements	Plugin for output to Wikidata QuickStatements
plugin-zotero-translation-server	Plugin for Zotero JSON and interfacing with a Zotero translation server
plugin-refer	Plugin for the refer file format
plugin-refworks	Plugin for the RefWorks tagged format

CLI

CLI:

Usage: citation-js [options]

Options:
  -V, --version                   output the version number
  -i, --input <path>              Input file. If all input options are omitted, it uses stdin
  -t, --text <string>             Input text. If all input options are omitted, it uses stdin
  -u, --url <string>              Deprecated in favor of -t, --text. If all input options are omitted, it uses stdin
  -o, --output <path>             Output file (omit file extension). If this option is omitted, the output is written to stdout
  -R, --output-non-real           Output as a text file
  -f, --output-type <option>      Output structure type: string, html, json (default: "json")
  -s, --output-style <option>     Output scheme. A combination of --output-format json and --output-style citation-* is considered invalid. Options: csl (Citation Style Lanugage JSON), bibtex, citation-* (where * is any formatting style) (default: "csl")
  -l, --output-language <option>  Output language. [RFC 5646](https://tools.ietf.org/html/rfc5646) codes (default: "en-US")
  -h, --help                      output usage information

Acknowledgements

• Thanks to the JS.ORG DNS service for the site url!

• Thanks to BrowserStack for the free Open Source plan, allowing me to automate testing browser support, and avoid issues like this one!

bibtex-parser-experiments's Issues

npm install fails

When running npm install on this repo (on Mac), I get:

> [email protected] install /Users/emile/github/bibtex-parser-experiments/node_modules/chokidarAt2/node_modules/fsevents
> node install.js

internal/modules/cjs/loader.js:1033
  throw err;
  ^

Error: Cannot find module 'nan'
Require stack:
- /Users/emile/github/bibtex-parser-experiments/node_modules/chokidarAt2/node_modules/fsevents/[eval]
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:1030:15)
    at Function.Module._load (internal/modules/cjs/loader.js:899:27)
    at Module.require (internal/modules/cjs/loader.js:1090:19)
    at require (internal/modules/cjs/helpers.js:75:18)
    at [eval]:1:1
    at Script.runInThisContext (vm.js:131:18)
    at Object.runInThisContext (vm.js:295:38)
    at Object.<anonymous> ([eval]-wrapper:10:26)
    at Module._compile (internal/modules/cjs/loader.js:1201:30)
    at evalScript (internal/process/execution.js:98:25) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/emile/github/bibtex-parser-experiments/node_modules/chokidarAt2/node_modules/fsevents/[eval]'
  ]
}
gyp: Call to 'node -e "require('nan')"' returned exit status 1 while in binding.gyp. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)
gyp ERR! System Darwin 19.6.0
gyp ERR! command "/usr/local/Cellar/node/14.5.0/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/emile/github/bibtex-parser-experiments/node_modules/chokidarAt2/node_modules/fsevents
gyp ERR! node -v v14.5.0
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 

BBT parser

WRT the issues reported here on the BBT parser:

• does not seem to support all diacritics (errors on {\r a}) is fixed now
• does not seem to support chained concatenations (a # b # c) I can't replicate this.

The sample below imports in BBT since 5.1.154

@string{j = {a space between this }}
@string{a = { string a}}
@string{b = { string b}}
@string{c = { string c}}
@article{key,
    author  = "Author",
    title   = "{\r a}Title" # a # b # c,
    year    = 1990,
    journal = j # "and this"
}

but the concat part of the title imported before that too.

Sharing Bib(La)TeX test cases?

Do you want to share the Bib(La)TeX test cases as well? Or are they already somewhere online available, but I couldn't find them?

I wanted to analyze Zotero's behavior on them and use them for possible improvements of its BibTeX import translator (e.g. supporting electronic as it seems to be an alias for online).

Argument commands

The citationjs parser needs to allow for more different kinds of commands, mostly argument commands. Arguments seem to be treated the same always: it either takes in a braced block or the first character of text. Exceptions are math blocks: \url takes in the dollar sign verbatim while \emph does not.

What should I run so I can run the tests?

I've ran

npm run babel
npm run nearley
npm run astrocite
npm test

but I still get

Error: Cannot find module './grammar'

closedown biblatex-cslc-onverter

Hey, I just discovered this chart. I have been participating in the maintenance of biblatex-csl-converter over the past few years. Based on your chart it looks like Idea (reworked) gives the same output quality as biblatex-csl-converter. Does that mean that it can be used as a drop in replacement and that it covers all the same features? If that is the case, is there any reason why I would continue to maintain biblatex-csl-converter?

idea-reworked: diacritics misplaced

When parsing

publisher =  { D{\u{o}}\}\"ead Po$_{eee}$t Society},

the mark that should be above the o ends up above the }

BBT test suite

I am going through the BBT test suite again (at least, the files in retorquere/bibtex-parser/__tests__/better-bibtex/import). @retorquere, some questions:

• One file contains {\d} which would normally be {\dh}, \d being a diacritic expecting an argument. Do you handle that normally or is that covered by error recovery?
• Does EndNote really export without labels...
• The rest works now after some updates on my side

Update result for Zotero

After the merge of PR zotero/translators#2020 most of the issues for Zotero translator should be addressed and therefore one could update the results here with the new version.

@comment behavior in BibTeX vs BibLaTeX

biblatex and natbib treat @comment entries differently. However, neither treat them as I implemented based on a summary I found, namely from @comment to the end of the line (source). biblatex seems to treat it as a regular entry (e.g. expecting opening braces and ignoring everything including other entries until the end brace), while natbib seems to just ignore the @comment text, so if an entry starts with @comment instead of @book it does nothing. However, it does not require @comment to be an entry, as biblatex does. Also, natbib behavior differs from the aforementioned summary in that it still counts entries that start on the same line as @comment, while the summary states "that everything from the @Comment and to the end of line is ignored".

Examples:

@comment {
  @misc{label,
    author = "name",
    title = "displays with natbib, not with biblatex",
    year = 2019
  }
}

@comment{ @misc{label2,
    author = "name",
    title = "displays with natbib, not with biblatex",
    year = 2019
  }
}

@comment @misc{label3,
    author = "name",
    title = "errors (fatally) with biblatex, displays with natbib",
    year = 2019
  }

@comment {} @misc{label4,
    author = "name",
    title = "displays with biblatex, natbib",
    year = 2019
  }

Progress on the active parser ("citationjs")

2020-09-15: update below

One big problem is the question of what should be parsed when parsing syntax, and what should parsed when mapping to CSL. Consider also that Bib.TXT should be able to use the same mapping.

• diacritics: when parsing syntax, as Bib.TXT and some BibTeX supports utf8
• other known symbol commands and ligatures: when parsing syntax
• except, fields tagged as verbatim or url in the specification should not have commands parsed, and then the syntax parser has to know about all the different fields.
• although field data is available, URL escaping should be handled when mapping since Bib.TXT should probably have that behavior too
• name field parsing: should probably be when mapping
• list field parsing (splitting on " and "): should probably be when mapping
• markup: should be done when mapping, as markup differs between formats
• crossref: should be done when mapping

Less crucial things, maybe:

• let people extend the constants (i.e., add commands, diacritics, ligatures)
• (only) warn for mis-matched entry brackets

Add pandoc to the test suite

Zotero parser comparison

The Zotero parser isn't really compared in the same way as the others -- it starts a full-blown translation server which does a lot more. The Zotero translators aren't really meant to be ran standalone but I've created a haphazard test runner for the bibtex translator here, and that runs on long.bib in 7s.