hrivnac / spark-fits

FITS data source for Spark SQL and DataFrames

FITS Data Source for Apache Spark

Latest news

• [01/2018] Launch: spark-fits is created!
• [03/2018] Release: version 0.3.0
• [04/2018] Paper:
• [04/2018] Talk: Statistical challenges for large-scale structure in the era of LSST, Oxford. slides
• [05/2018] Release: version 0.4.0
• [06/2018] New location: spark-fits is an official project of AstroLab!
• [07/2018] Release: version 0.5.0, 0.6.0

spark-fits

This library provides two different tools to manipulate FITS data with Apache Spark:

• A Spark connector for FITS file.
• A Scala library to manipulate FITS file.

The user interface has been done to be the same as other built-in Spark data sources (CSV, JSON, Avro, Parquet, etc). See our website for more information.

Currently available:

• Read fits file and organize the HDU data into DataFrames.
• Automatically distribute bintable rows over machines.
• Automatically distribute image rows over machines. new in 0.4.0
• Automatically infer DataFrame schema from the HDU header.

Header Challenge!

The header tested so far are very simple, and not so exotic. Over the time, we plan to add many new features based on complex examples (see here). If you use spark-fits, and encounter errors while reading a header, tell us (issues or PR) so that we fix the problem asap!

TODO list

• Define custom Hadoop InputFile.
• Adapt to S3 usage?

Support