Comments (3)
I ran into similar issues (too big XML files) in the past and I remember that there are tools addressing this problem specifically; one I remember is xml_split.
On debian/ubuntu it seems to be available with the xml-twig-tools
package.
$ metha-sync https://yareta.unige.ch/oai && metha-cat https://yareta.unige.ch/oai | xml_split -l 1 -s 50kb
$ ls -l
total 456K
drwxrwxr-x 2 tir tir 4.0K Jun 1 13:29 ./
drwxr-xr-x 39 tir tir 56K Jun 1 13:28 ../
-rw-rw-r-- 1 tir tir 346 Jun 1 13:29 out-00.xml
-rw-rw-r-- 1 tir tir 50K Jun 1 13:29 out-01.xml
-rw-rw-r-- 1 tir tir 54K Jun 1 13:29 out-02.xml
-rw-rw-r-- 1 tir tir 51K Jun 1 13:29 out-03.xml
-rw-rw-r-- 1 tir tir 50K Jun 1 13:29 out-04.xml
-rw-rw-r-- 1 tir tir 51K Jun 1 13:29 out-05.xml
-rw-rw-r-- 1 tir tir 51K Jun 1 13:29 out-06.xml
-rw-rw-r-- 1 tir tir 51K Jun 1 13:29 out-07.xml
-rw-rw-r-- 1 tir tir 23K Jun 1 13:29 out-08.xml
The resulting XML is valid, but slightly modified:
$ xmllint --format out-01.xml 2> /dev/null | head -4
<?xml version="1.0"?>
<xml_split:root xmlns:xml_split="http://xmltwig.com/xml_split">
<record xmlns="http://www.openarchives.org/OAI/2.0/">
<header status="">
Does this help?
PS: Thanks for using metha! I'm just curious (and collecting uses of metha) - if possible, can you share the project name in which metha is used for data acquisition?
from metha.
@miku Thanks a lot for your answer. We'll try this out!
We are using metha
for our linked open research data project Connectome.
from metha.
Related Issues (20)
- Why is data only harvested up to the last day? HOT 3
- base-dir argument for metha-cat HOT 1
- Bad page state in metha-sync (arm)
- authorization // character limit HOT 5
- Support for basic auth HOT 2
- two different resumptionTokens? HOT 1
- metha-cat - can not open the "dir" extablished in .cache/metha HOT 3
- Urlencode resuptionToken HOT 2
- Migration from Goodtables to Frictionless Repository
- Question: Can metha auto harvest all formats/metadataPrefixes? HOT 2
- Client Timeout HOT 4
- Dependency Issue with Version 0.2.37 HOT 2
- metha-sync should catch SIGINT HOT 3
- Extend metha-cat to extract metadata records HOT 3
- Selective Harvesting and metha-cat HOT 2
- decode failed due to XML header HOT 1
- conflicting namespace prefixes during ListRecords HOT 2
- `-format` not respected? HOT 2
- "FATA[0000] illegal base64 data at input byte 0" for metha-ls and metha-cat HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metha.