XME format should export full deleted record about inspire

[tsgit]

but XME format is also publicly accessible and advertised, so deleted records should clearly be separate, no !? 980__a:deleted isn't obvious to external users

[michamos]

Is XME advertised? on the records detailed view, clicking on MARCXML opens the XM format, not XME.

[kaplun]

XME is advertized in the FAQ. However there are 2 different XME dumps. The one publicly accessible e.g. from the dump advertized in the FAQ or via /export/xme, and this already today is protecting hidden fields and can keep on avoiding exporting deleted records.

Then we are creating the full-dump via the bibtasklet. This can be extended to also included content from deleted records.

[jacquerie]

This is currently causing the biggest error on Sentry: https://sentry.cern.ch/inspire-sentry/inspire-nightly/group/619214/.

While clearly this data is not useful for end users it's useful data for curation algorithms: for example, deleted records contain records that were determined to be duplicates. So it would be cool to have this as soon as possible.

[kaplun]

Yeah. The only issue is that this data will surely contain more outliers and crap because it won't have been curated (whoah what a tense is that) for a long time.

[jacquerie]

I've seen things you people wouldn't believe. Records with three first authors (all referring to the same guy). I watched exceptions bubble up the stack because records had two longitudes and no latitude.

Seriously, I don't expect this data to be any worse than what we've already found : )

[kaplun]

You won't believe your eyes...

[jacquerie]

\o/

ETA for a new prodsync dump with the deleted records?

Answered in #277 (comment).