chhantyal / parquet-cli Goto Github PK

Is it possible to relax the requirements (lower and upper bound) or are the exact pinned versions needed?

Hi! Is there a workaround for the list datatypes?

Here's the error I'm seeing:
$ parq calls.parquet --schema
Traceback (most recent call last):
File "/home/clande/.local/bin/parq", line 10, in
sys.exit(main())
File "/home/clande/.local/lib/python2.7/site-packages/parq/main.py", line 38, in main
pq_table = pq.read_table(cmd_args.file)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 1156, in read_table
use_pandas_metadata=use_pandas_metadata)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 214, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 737, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: No support for reading columns of type list<array: string not null>

Content of my command line while trying to get parquet-cli to run on Ubuntu 18.04.1 LTS:

machine:~$ pip install parquet-cli
Collecting parquet-cli

... [installing stuff]

Installing collected packages: pytz, six, python-dateutil, numpy, pandas, futures, pyarrow, parquet-cli
Successfully installed futures-3.2.0 numpy-1.15.4 pandas-0.22.0 parquet-cli-1.2 pyarrow-0.11.1 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0

machine:~$ parq --help

Command 'parq' not found, did you mean:

  command 'par2' from deb par2
  command 'par' from deb par
  command 'parl' from deb libpar-packer-perl

Try: sudo apt install <deb name>

It's unintuitive to specify True or False after command line flags, which are traditionally used without further argument. I think you want the action='store_true' behavior of ArgumentParser.

i.e. I should be able to do parq -s <parquet_file> rather than having to do parq -s True <parquet_file>.

I am trying to install the package but getting below error:

H:>pip install parquet-cli
Collecting parquet-cli
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Could not find a version that satisfies the requirement parquet-cli (from versions: )
No matching distribution found for parquet-cli

My use case is mostly to cat from s3 and pipe that into your parq utility. This is what I did to accept stdin if someone else want to work this into the code or just know how to patch for stdin.

Example leaving file arg out completely:

$ mc cat s3/us-east-1.dev-pull/DataLake/parquet/run-S3bucket_node3-74-part-block-0-r-00000-snappy.parquet | parq --head 1

$ diff /usr/local/lib/python3.8/dist-packages/parq/main.py ./parq-main.py.backup
8d7
< import pyarrow as pa
42,48c41
<     if not (cmd_args.file):
<         print ("file arg not passed. using stdin as passed by cat")
<         reader = pa.BufferReader(sys.stdin.buffer.read())
<         pq_table = pq.read_table(reader)
<     else:
<         pq_table = pq.read_table(cmd_args.file)
<
---
>     pq_table = pq.read_table(cmd_args.file)
101d93
<                         nargs="?",