hay / dataknead Goto Github PK

First of all awesome package, as this is definitely a common issue and I'm surprised there is nothing comparable out there. Anyhow, I am getting the error message Exception: Data of type dict can not be processed, needs to be list when trying to run a map over the JSON.

I believe this line https://github.com/hay/dataknead/blob/master/dataknead/knead.py#L71 needs to be changed to check if the instance is a dict or not set the check_instance?

EDIT: My overall goal is to check for keys in the nested dict/json then replace the value from other files that have that key with "real" values.

i.e. find key/value named "reference_id": "fake-value" and then open another JSON (or few JSONs) then find the id key by searching the value of reference_id ("id": "fake-value" {"value_1": "fake", "value2": "fake2"}) and replace the reference_id's value with the value(s) from the id dict.

I gave up trying to find what I wanted and wrote the desired functionality in python. First, I wrote a generator that will untwist any JSON datasource of any size and complexity. Then I added two-pass functionality, the first pass finds the deepest node, the second pass goes through every line in the generator and pads lines shorter than the deepest node with zeroes. I cannot post the source code here because it's proprietary to my employer.

When entering a PosixPath (e.g. from using pathlib) as input to a new Knead instance because it is not a string dataknead will think this is Python data (a native type), and will give back a stringified path as data instead of the file contents. This should be changed so that instances of PosixPath are recognised and treated as file paths.

Thank you for this tool! Hoping this is an easy fix ...

Here's how I used the tool:

>>> from dataknead import Knead
>>> Knead("../test.xml").write("../test.json")

starting contents of "test.xml" (input to knead):

<?xml version="1.0" encoding="UTF-8"?>
<job version="1.0">
   <name>test</name>
   <time/>
   <src>job composer</src>
   <priority>normal</priority>
   <trigger enabled="false">
      <dir>c:\trigger_in</dir>
      <mask>*.*</mask>
      <job_name>Template-%1%</job_name>
   </trigger>
   <segment idx="1">
      <crop enabled="false"
            top="0"
            left="0"
            width="0"
            height="0"
            if_width="0"
            if_height="0"/>

starting contents of "test.json" (output from knead):
{"job": {"@version": "1.0", "name": "test", "time": null, "src": "job composer", "priority": "normal", "trigger": {"@enabled": "false", "dir": "c:\\trigger_in", "mask": "*.*", "job_name": "Template-%1%"}, "segment": {"@idx": "1", "crop": {"@enabled": "false", "@top": "0", "@left": "0", "@width": "0", "@height": "0", "@if_width": "0", "@if_height": "0"},

expected contents of "test.json" (output from knead):
{"job": {"version": "1.0", "name": "test", "time": null, "src": "job composer", "priority": "normal", "trigger": {"enabled": "false", "dir": "c:\\trigger_in", "mask": "*.*", "job_name": "Template-%1%"}, "segment": {"idx": "1", "crop": {"enabled": "false", "top": "0", "left": "0", "width": "0", "height": "0", "if_width": "0", "if_height": "0"},

Thank you for looking into this!

Performing a simple xml to csv conversion results in error that the xml loader cannot be found.

In [1]: from dataknead import Knead

In [2]: Knead('cd_catalog.xml').write('cd_catalog.csv')
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-2-90e8ad4393dd> in <module>
----> 1 Knead('cd_catalog.xml').write('cd_catalog.csv')

c:\users\japbo\appdata\local\programs\python\python37-32\lib\site-packages\dataknead\knead.py in __init__(self, inp, parse_as, read_as, is_data, **kwargs)
     25                 read_as = Path(inp).suffix[1:]
     26
---> 27             loader = self._get_loader(read_as)
     28
     29             with open(inp) as f:

c:\users\japbo\appdata\local\programs\python\python37-32\lib\site-packages\dataknead\knead.py in _get_loader(self, extension)
     41                 return loader
     42
---> 43         raise Exception("Could not find loader for type '%s'" % extension)
     44
     45     def apply(self, fn):

Exception: Could not find loader for type 'xml'

Tired of Data Conversion Hassles? Discover the Universal Data Converter - The All-in-One Tool for Data Analysts! JSON, CSV, XML – We've Got You Covered!

Universal Data Converter | Contribute on GitHub

Say Goodbye to Data Format Headaches and Hello to Seamless Conversions!

As of v0.13.0, xmltodict uses native python dictionaries instead of OrderedDict martinblech/xmltodict/pull/295.

Like this:

Knead("data.csv").map("title,url").write("title-url.csv")

So it allows you to do something like

knead input.json -m text -o input.csv

OS - MacOS X 10.15.7
Python version - 3.8.5
Install mechanism:
poetry - 1.0.5
pip - 20.2.3

Error Message:

clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/var/pyenv/versions/3.8.5/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/var/pyenv/versions/3.8.5/lib build/temp.macosx-10.15-x86_64-3.8/pandas/_libs/join.o -o build/lib.macosx-10.15-x86_64-3.8/pandas/_libs/join.cpython-38-darwin.so
building 'pandas._libs.lib' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include -Ipandas/_libs/src/klib -Ipandas/_libs/src -Ipandas/_libs/tslibs/src -Ipandas/_libs/tslibs -I/Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/lib/python3.8/site-packages/numpy/core/include -I/Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/include -I/usr/local/var/pyenv/versions/3.8.5/include/python3.8 -c pandas/_libs/lib.c -o build/temp.macosx-10.15-x86_64-3.8/pandas/_libs/lib.o -Wno-unused-function
In file included from pandas/_libs/lib.c:634:
In file included from /Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:
In file included from /Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1832:
/Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it with "
^
In file included from pandas/_libs/lib.c:636:
pandas/_libs/src/parse_helper.h:141:26: error: implicit declaration of function 'tolower_ascii' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
for (; *p; ++p) *p = tolower_ascii(*p);
^
pandas/_libs/lib.c:39057:39: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_Seen.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39077:44: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_Validator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39091:48: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_BoolValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39105:51: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_IntegerValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39119:56: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_IntegerFloatValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39133:49: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_FloatValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39147:50: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_StringValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39161:51: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_UnicodeValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39175:49: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_BytesValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39191:52: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_TemporalValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39205:52: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_DatetimeValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39218:54: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_Datetime64Validator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39232:53: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_TimedeltaValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39245:56: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_AnyTimedeltaValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39258:48: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_DateValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39271:48: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_TimeValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39285:50: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_PeriodValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
pandas/_libs/lib.c:39298:52: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
__pyx_type_6pandas_5_libs_3lib_IntervalValidator.tp_print = 0;
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
^
/usr/local/var/pyenv/versions/3.8.5/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
#define Py_DEPRECATED(VERSION_UNUSED) attribute((deprecated))
^
19 warnings and 1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/40/c8b3smrj5fz8cv1zqqlc863wh1j1cg/T/pip-install-lebu_z7p/pandas/setup.py'"'"'; file='"'"'/private/var/folders/40/c8b3smrj5fz8cv1zqqlc863wh1j1cg/T/pip-install-lebu_z7p/pandas/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/40/c8b3smrj5fz8cv1zqqlc863wh1j1cg/T/pip-record-wgykc9_b/install-record.txt --single-version-externally-managed --compile --install-headers /Users/rohitus/Documents/Projects/KendravsESComparison/google_blogs_input_based/convert_xml_to_json/.venv/include/site/python3.8/pandas Check the logs for full command output.

Steps to repeat:

I have tried all the following mechanisms and they all give the same error:

• poetry add dataknead
• poetry add [email protected]
• poetry add pandas==0.24.2
• 2 step process
  -- poetry add numpy==1.18.5
  -- poetry add pandas==0.24.2
• 2 step process
  -- poetry run pip install numpy==1.18.5
  -- poetry run pip install pandas==0.24.2

Let me know what other information is needed.

Using knead results in a numpy error, e.g.

  File "/usr/local/lib/python3.11/site-packages/openpyxl/compat/numbers.py", line 16, in <module>
    NUMERIC_TYPES = NUMERIC_TYPES + (numpy.short,
                                     ^^^^^^^^^^^
AttributeError: module 'numpy' has no attribute 'short'

I believe the numpy in question is: numpy-1.24.3

See https://levelup.gitconnected.com/fix-attributeerror-module-numpy-has-no-attribute-float-d7d68c5a4971

E.g.

Knead(['one', 'two', three']).write("numbers.txt")

Results in:

one
two
three

csv.Sniffer has an option has_header that sometimes doesn't detect a header in a CSV file. The constructor should be able to pass options like has_header = True to the loader so you can overwrite standard behaviour.

Unfortunately, it only works on eeeasy and siiiimple, it won't work on the results of a graphql query like this:

query FilterOutNulls
{
customSQLTables
(permissionMode: FILTER_RESULTS)
{
id
name
columns
(
permissionMode: FILTER_RESULTS
)
{
id
name
referencedByFields
{
datasource
{
... on PublishedDatasource
{
name
downstreamOwners
{
name
email
}
downstreamWorkbooks
(permissionMode: FILTER_RESULTS)
{
name
}
}
... on EmbeddedDatasource
{
name
downstreamOwners
{
name
email
}
downstreamWorkbooks
(permissionMode: FILTER_RESULTS)
{
name
}
}
}
}
}
}
}

Some loaders pass on additional arguments to the libraries they use to load the files (e.g. ExcelLoader passes on all arguments to openpyxl). Unfortunately not all loaders do this, so you can't pass on options like giving a delimiter to the CsvLoader.

Loaders affected:

• CsvLoader
• JsonLoader (only for read)
• TomlLoader
• XmlLoader (only for read)

Right now it's not possible to replace a default loader (e.g. for CsvLoader) except by using hidden/private methods. For example:

from dataknead import Knead
from my_csvloader import CsvLoader

Knead._loaders["csv"] = CsvLoader

# Convert a csv with semicolons to json
print(Knead("./cities-semicolon.csv", delimiter = ";"))

Hay, When running:
Knead(file_in).write(file_out)
with file_in being an XML file, the output shows the OrderedDict as text (?).
I used this XML file: http://www.nvon-biologie-itembank.nl/xml/nvon.xml

thanks, Lieke