GithubHelp home page GithubHelp logo

chhantyal / parquet-cli Goto Github PK

View Code? Open in Web Editor NEW
162.0 162.0 9.0 66 KB

Command line (CLI) tool to inspect Apache Parquet files on the go

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

parquet-cli's People

Contributors

chhantyal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

parquet-cli's Issues

No support for reading columns of type list

Hi! Is there a workaround for the list datatypes?

Here's the error I'm seeing:
$ parq calls.parquet --schema
Traceback (most recent call last):
File "/home/clande/.local/bin/parq", line 10, in
sys.exit(main())
File "/home/clande/.local/lib/python2.7/site-packages/parq/main.py", line 38, in main
pq_table = pq.read_table(cmd_args.file)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 1156, in read_table
use_pandas_metadata=use_pandas_metadata)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 214, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 737, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: No support for reading columns of type list<array: string not null>

"Command 'parq' not found" after successful "pip install parquet-cli"

Content of my command line while trying to get parquet-cli to run on Ubuntu 18.04.1 LTS:

machine:~$ pip install parquet-cli
Collecting parquet-cli

... [installing stuff]

Installing collected packages: pytz, six, python-dateutil, numpy, pandas, futures, pyarrow, parquet-cli
Successfully installed futures-3.2.0 numpy-1.15.4 pandas-0.22.0 parquet-cli-1.2 pyarrow-0.11.1 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0

machine:~$ parq --help

Command 'parq' not found, did you mean:

  command 'par2' from deb par2
  command 'par' from deb par
  command 'parl' from deb libpar-packer-perl

Try: sudo apt install <deb name>


Error during installation

I am trying to install the package but getting below error:

H:>pip install parquet-cli
Collecting parquet-cli
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Could not find a version that satisfies the requirement parquet-cli (from versions: )
No matching distribution found for parquet-cli

accept stdin

My use case is mostly to cat from s3 and pipe that into your parq utility. This is what I did to accept stdin if someone else want to work this into the code or just know how to patch for stdin.

Example leaving file arg out completely:

$ mc cat s3/us-east-1.dev-pull/DataLake/parquet/run-S3bucket_node3-74-part-block-0-r-00000-snappy.parquet | parq --head 1

$ diff /usr/local/lib/python3.8/dist-packages/parq/main.py ./parq-main.py.backup
8d7
< import pyarrow as pa
42,48c41
<     if not (cmd_args.file):
<         print ("file arg not passed. using stdin as passed by cat")
<         reader = pa.BufferReader(sys.stdin.buffer.read())
<         pq_table = pq.read_table(reader)
<     else:
<         pq_table = pq.read_table(cmd_args.file)
<
---
>     pq_table = pq.read_table(cmd_args.file)
101d93
<                         nargs="?",

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.