datacoon / undatum Goto Github PK

Add command with XML structure analysis. It should allow to detect XML structure and find data container and children tags to make XML file conversion easier.

Right now to convert XML file you need to provide tagname - name of XML tag that will be assumed as tag that contain data which we interested in. But name of this tag is a user knowledge only. Command that analyze XML structure and prints it shoud help user with choice.

More commands needed:

• sort - sort by number of columns
• merge - merge two or more data files
• join - join two or more data files
• print - multiple ways to print values in data files
• query - simple yet effective query command
• search - search for value using substring and regex

On MacOs, brew install undatum does not work:

$ brew install undatum
Error: No available formula with the name "undatum"
==> Searching for a previously deleted formula (in the last month)...
Error: No previously deleted formula found.
==> Searching for similarly named formulae...
Error: No similarly named formulae found.
==> Searching taps...
==> Searching taps on GitHub...
Error: No formulae found in taps.

For example using same code https://www.kdnuggets.com/2022/07/parallel-processing-large-file-python.html

Support Excel files XLS and XLSX with analyze command and report on it's metadata
Add support for commands uniq, frequency, select and other text processing commands with Excel files

There seem to be a missing dependancy when installed using pip install --upgrade undatum

➜  undatum --help
Traceback (most recent call last):
  File "/usr/local/bin/undatum", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/undatum/__main__.py", line 10, in main
    from .core import cli
  File "/usr/local/lib/python3.9/site-packages/undatum/core.py", line 9, in <module>
    from .cmds.analyzer import Analyzer
  File "/usr/local/lib/python3.9/site-packages/undatum/cmds/analyzer.py", line 13, in <module>
    import xmltodict
ModuleNotFoundError: No module named 'xmltodict'

After installing the dep manually (pip install xmltodict) it worked

Need to focus on:

• identify is JSONL object flat
• use "--fields" options
• use "skip" or "flatten" or "tables" strategy
• if "skip" just ignore all complex fields
• if "flatten" than merge complex fields with different separator
• if "tables" then generate several CSV files from objects, need to think how to do that right

Key questions:

• Do we need simple CSV or Data package with a schema like Frictionless Data package?

Add support of ORC files https://cwiki.apache.org/confluence/display/hive/languagemanual+orc

Transfer code from https://github.com/datacoon/datadifflib and implement diff and apply commands to generate and apply patches to JSONl, BSON and CSV files

Rewrite code and make data conversion processes universal. Right now it's partially implemented with IterableData and DataWriter classes, but they aren't used in data conversion functions.

An ideas how to implement it:

• analyze existing data formats to understand cross convertability.
• write short research notes and convertability comparison table
• create generic classes for data source and data writer
• rewrite current convert code to the more universal approach for all used data formats
• write documentation how to add more data sources/destinations for other data formats

Add support of Avro files https://avro.apache.org/docs/1.2.0/spec.html

More info https://github.com/iqlusioninc/veriform

Following list:

• BSON
• JSON
• JSON lines
• XDR
• ASN
• ASN.1 DER
• Cap'N Proto
• Veriform
• NoProto
• CBOR
• csexp
• MessagePack
• Protobuf
• Avro
• Orc
• Parquet
• Flatbuffers
• Bincode
• CSV
• XML
• UBJSON
• Thrift
• Sereal
• Bencode
• Gobs
• ROOT (https://root.cern)
• HDF5
• Velocitypack

Consider to add table detection and extraction from non-table files like .docx file format. For example using docx2csv command https://github.com/ivbeg/docx2csv

Review other possible data sources and data formats

Add support for YAML files for stats, conversion and other operations

Add support of Cap'N Proto protocol https://capnproto.org

It would be nice to have options for compression. Looks like there is no compression by default?

parq RS_2008-04.parquet 

 # Metadata 
 <pyarrow._parquet.FileMetaData object at 0x7f5d6f635490>
  created_by: parquet-cpp-arrow version 7.0.0
  num_columns: 124
  num_rows: 167472
  num_row_groups: 1
  format_version: 1.0
  serialized_size: 53334

Dump MongoDB, ArangoDB, SQL DBMS and other databases directly to BSON, JSON lines, CSV and e.t.c formats.

NoProto is a new data serialization definition https://github.com/only-cliches/NoProto

Add more file conversion types, like:

• jsonl to bson
• bson to jsonl
• xls to bson
• xlsx to bson
  and e.t.c.