chhantyal / parquet-cli

Command line (CLI) tool to inspect Apache Parquet files on the go

parquet-cli

Command line (CLI) tool to inspect Apache Parquet files on the go

Apache Parquet is a columnar storage format commonly used in the Hadoop ecosystem.

parq is small, easy to install, Python utility to view and get basic information from Parquet files.

Current features set are what I need, please use Github issues for any requests/suggestions.

Install

pip install parquet-cli

An executable script called parq will be installed.

Use

Once installed, you can use parq command.

View Parquet file metadata:

$ parq input.parquet

# Metadata
<pyarrow._parquet.FileMetaData object at 0x1014879a8>
created_by: parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
num_columns: 13
num_rows: 1000
num_row_groups: 1
format_version: 1.0
serialized_size: 1125

Get schema information:

$ parq input.parquet --schema

# Schema
<pyarrow._parquet.ParquetSchema object at 0x1048b9a88>
registration_dttm: INT96
id: INT32
name: BYTE_ARRAY UTF8
email: BYTE_ARRAY UTF8
...
ip_address: BYTE_ARRAY UTF8
country: BYTE_ARRAY UTF8

Get total rows count:

$ parq input.parquet --count

1025

Get top N records (head)

$ parq input.parquet --head 10

Get bottom N records (tail)

$ parq input.parquet --tail 10

Help

$ parq --help

usage: usage: parq file [-s [SCHEMA] | --head [HEAD] | --tail [TAIL] | -c [COUNT]]

positional arguments:
  file                  Parquet file

optional arguments:
  -h, --help            show this help message and exit
  -s [SCHEMA], --schema [SCHEMA]
                        get schema information
  --head [HEAD]         get first N rows from file
  --tail [TAIL]         get last N rows from file
  -c [COUNT], --count [COUNT]
                        get total rows count

parquet-cli's Issues

"Command 'parq' not found" after successful "pip install parquet-cli"

Content of my command line while trying to get parquet-cli to run on Ubuntu 18.04.1 LTS:

machine:~$ pip install parquet-cli
Collecting parquet-cli

... [installing stuff]

Installing collected packages: pytz, six, python-dateutil, numpy, pandas, futures, pyarrow, parquet-cli
Successfully installed futures-3.2.0 numpy-1.15.4 pandas-0.22.0 parquet-cli-1.2 pyarrow-0.11.1 python-dateutil-2.7.5 pytz-2018.7 six-1.11.0

machine:~$ parq --help

Command 'parq' not found, did you mean:

  command 'par2' from deb par2
  command 'par' from deb par
  command 'parl' from deb libpar-packer-perl

Try: sudo apt install <deb name>

No support for reading columns of type list

Hi! Is there a workaround for the list datatypes?

Here's the error I'm seeing:
$ parq calls.parquet --schema
Traceback (most recent call last):
File "/home/clande/.local/bin/parq", line 10, in
sys.exit(main())
File "/home/clande/.local/lib/python2.7/site-packages/parq/main.py", line 38, in main
pq_table = pq.read_table(cmd_args.file)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 1156, in read_table
use_pandas_metadata=use_pandas_metadata)
File "/home/clande/.local/lib/python2.7/site-packages/pyarrow/parquet.py", line 214, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 737, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: No support for reading columns of type list<array: string not null>

Relax constraints on requirements

Is it possible to relax the requirements (lower and upper bound) or are the exact pinned versions needed?

Schema and Count Arguments Unintuitive

parquet-cli/parq/main.py

Lines 62 to 68 in 588b738

	group.add_argument("-s",
	"--schema",
	nargs="?",
	type=bool,
	const=True,
	help="get schema information",
	)

It's unintuitive to specify True or False after command line flags, which are traditionally used without further argument. I think you want the action='store_true' behavior of ArgumentParser.

i.e. I should be able to do parq -s <parquet_file> rather than having to do parq -s True <parquet_file>.

Error during installation

I am trying to install the package but getting below error:

H:>pip install parquet-cli
Collecting parquet-cli
Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=3, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=1, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Retrying (Retry(total=0, connect=None, read=None, redirect=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/parquet-cli/
Could not find a version that satisfies the requirement parquet-cli (from versions: )
No matching distribution found for parquet-cli

accept stdin

My use case is mostly to cat from s3 and pipe that into your parq utility. This is what I did to accept stdin if someone else want to work this into the code or just know how to patch for stdin.

Example leaving file arg out completely:

$ mc cat s3/us-east-1.dev-pull/DataLake/parquet/run-S3bucket_node3-74-part-block-0-r-00000-snappy.parquet | parq --head 1

$ diff /usr/local/lib/python3.8/dist-packages/parq/main.py ./parq-main.py.backup
8d7
< import pyarrow as pa
42,48c41
<     if not (cmd_args.file):
<         print ("file arg not passed. using stdin as passed by cat")
<         reader = pa.BufferReader(sys.stdin.buffer.read())
<         pq_table = pq.read_table(reader)
<     else:
<         pq_table = pq.read_table(cmd_args.file)
<
---
>     pq_table = pq.read_table(cmd_args.file)
101d93
<                         nargs="?",

Also support directories