ezpaarse-project / ezpaarse-middlewares Goto Github PK

J'utilise le middlware deduplicator (pour OpenEditon) mais je n'arrive pas à le faire fonctionner (ou il y a un bug).

Ci dessous un exemple :

• le fichier de log : test-duplicates.log

  • 5 lignes de logs
  • la ligne 3 est identique à la ligne 2, sauf datetime : ligne 3 + 1 sec. par rapport à ligne 2
  • les lignes 4 et 5 sont identiques

• le résultat EZPaarse : 6256b350_2022-11-03_17h27.job-ecs.csv

  • je m'attends à trouver 3 EC dans le résultat (duplicate ligne 2/3 et 4/5) mais il y en a 5.

• Les conf utilisées

  • 01-fr-open-edition-02.json.txt
  • config.local.dev.json.txt

Il y a peut-être quelque chose d'évident que je ne vois pas.‌..
Merci
Jean-François

Most ECs from Istex have a quoted SID, this causes troubles to bibliomap and kibana. The middleware should unquote it.

I'm not sure where to plug in this data so I'll start with an issue here. It is hypothetically possible to also extract database IDs from the "bdata=" Ebscohost url parameter. Have you looked at extracting or converting the longer hashed string IDs into database id's that can be used as part of the publisher knowledge base lookup? Reverse engineering this is really tough, since if end user has all of the Ebschost databases selected the bdata=.... will be a longer list of hashed string database IDs. However the hashed database IDs are not a one to one ratio. So if end user has 30 EDS databases selected there may be 20 unique IDs in the bdata= string. I could hypothetically create a pull request for the list.json attaching to the end of the list some of my bdata= IDs that I've been able to match to the actual database name. But from a programmatic standpoint, I don't think I should do that since list.json is probably mapped to the url parameter "db=", but could you code it to also look for "bdata=..." and the corresponding hashed database ID string? bdata=JmRiPWFzbiZ0eXBlPTAmc2VhcmNoTW9kZT1TdGFuZGFyZCZzaXRlPWVob3N0LWxpdmU for instance describes the a basic search of Academic Search Ultimate. bdata=JmRiPWFzbiZ0eXBlPTEmc2VhcmNoTW9kZT1TdGFuZGFyZCZzaXRlPWVob3N0LWxpdmU would be an advanced search (JmRiPWFzbiZ0eXBlPTE) of Academic Search Ultimate.

Istex use case : possibility to fill the SID with data extracted from the user agent

Although Ebsco has always provided a titles list webpage to scrape and feed into the Ebsco list.json, they have now reformatted their title list webpage and they are also linking to xls formatted versions of their database specific title lists. Does this open up any new programmatic avenues for refreshing the Ebscohost list.json and also updating the dozens of Ebsco pkb files that we can harvest as xls rather than html? Take a look to see what I mean https://www.ebsco.com/title-lists It seems like this could be an easier source of Ebsco database shortcode conversion to long name conversion, but also a method of updating the Ebscohost pkb files. Thanks for taking a look at this. :)

Currently the readthedocs middleware info for on-campus-counter links out to wikipedia when it should be linking internally to other documentation pages. https://ezpaarse.readthedocs.io/en/master/development/middlewares.html