Reading CSV/Parquet files written by Spark Job from Pandas about koalas

[ch-nickgustafson]

This could be because when writing a parquet with Spark, several non parquet files are produced. e.g.

/path/to/data/my_data.parquet/_SUCCESS
/path/to/data/my_data.parquet/_committed_12345678
/path/to/data/my_data.parquet/_started_12345678
/path/to/data/my_data.parquet/_part-00000-98178901734kjalksjd.snappy.parquet

The workaround/kludge I've used if I want to then read these parquet output using pandas is to copy all of the *.parquet files to a new directory. You should then be able to read the parquet files with pandas.

[rxin]

Can you paste an example file and exception? I suspect this is more of an issue with those readers in Pandas rather than Spark or Koalas issue.

[thunterdb]

From seeing the tickets, it looks like the workaround downstream will be an upgrade of the parquet reador or pandas.