steven-varga / h5cpp11

an easy to use c++11 templates between popular matrix algebra systems and HDF5 datasets

Hierarchical Data Format prevalent in high performance scientific computing, sits directly on top of sequential or parallel file systems, providing block and sequential operations on standardized or custom binary/text objects. Scientific computing platforms such as Julia, Matlab, R, Python, C/C++, Fortran come with the necessary libraries to read write HDF5 dataset. However the C/C++ API provided by HDF Group requires detailed understanding the file format and doesn't support popular c++ scientific libraries.

HDF5 CPP is to simplify object persistence by implementing CREATE,READ,WRITE,APPEND operations on fixed or variable length N dimensional arrays. This header only implementation supports raw pointers | armadillo | eigen3 | blaze | blitz++ | it++ | dlib | uBlas | std::vector by directly operating on the underlying data-store, avoiding intermediate/temporary memory allocations. The api is doxygen documented, furnished with examples, as well as profiled.

supported classes:

T := ([unsigned] ( int8_t | int16_t | int32_t | int64_t )) | ( float | double  )
S := T | c/c++ struct | std::string
ref := std::vector<S> 
	| arma::Row<T> | arma::Col<T> | arma::Mat<T> | arma::Cube<T> 
	| Eigen::Matrix<T,Dynamic,Dynamic> | Eigen::Matrix<T,Dynamic,1> | Eigen::Matrix<T,1,Dynamic>
	| Eigen::Array<T,Dynamic,Dynamic>  | Eigen::Array<T,Dynamic,1>  | Eigen::Array<T,1,Dynamic>
	| blaze::DynamicVector<T,rowVector> |  blaze::DynamicVector<T,colVector>
	| blaze::DynamicVector<T,blaze::rowVector> |  blaze::DynamicVector<T,blaze::colVector>
	| blaze::DynamicMatrix<T,blaze::rowMajor>  |  blaze::DynamicMatrix<T,blaze::colMajor>
	| itpp::Mat<T> | itpp::Vec<T>
	| blitz::Array<T,1> | blitz::Array<T,2> | blitz::Array<T,3>
	| dlib::Matrix<T>   | dlib::Vector<T,1> 
	| ublas::matrix<T>  | ublas::vector<T>
ptr 	:= T* 
accept 	:= ref | ptr 

In addition to the standard data types offered by BLAS/LAPACK systems and POD struct -s, std::vector also supports std::string data-types mapping N dimensional variable-length C like string HDF5 data-sets to std::vector<std::string> objects.

how to use:

sudo make install will copy the header files and h5cpp.pc package config file into /usr/local/ or copy them and ship it with your project. There is no other dependency than hdf5 libraries and include files. However to activate the template specialization for any given library you must include that library first then h5cpp. In case the auto detection fails turn template specialization on by defining:

#define [ H5CPP_USE_BLAZE | H5CPP_USE_ARMADILLO | H5CPP_USE_EIGEN3 | H5CPP_USE_UBLAS_MATRIX 
	| H5CPP_USE_UBLAS_VECTOR | H5CPP_USE_ITPP_MATRIX | H5CPP_USE_ITPP_VECTOR | H5CPP_USE_BLITZ | H5CPP_USE_DLIB | H5CPP_USE_ETL ]

short examples:

to read/map a 10x5 matrix from a 3D array from location {3,4,1}

#include <armadillo>
#include <h5cpp/all>
...
hid_t fd h5::open("some_file.h5",H5F_ACC_RDWR);
	/* the RVO arma::Mat<double> object will have the size 10x5 filled*/
	try {
		auto M = h5::read<arma::mat>(fd,"path/to/matrix",{3,4,1},{10,1,5});
	} catch (const std::runtime_error& ex ){
		...
	}
h5::close(fd);

to write the entire matrix back to a different file

#include <Eigen/Dense>
#include <h5cpp/all>
...
hid_t fd = h5::create("output.h5")
	h5::write(fd,"/result",M);
h5::close(fd);

to create an dataset recording a stream of struct into an extendable chunked dataset with GZIP level 9 compression:

#include <h5cpp/core>
	#include "your_data_definition.h"
#include <h5cpp/io>
...
hid_t ds = h5::create<some_type>(fd,"bids",{H5S_UNLIMITED},{1000}, 9);

to append records to an HDF5 datastream

#include <h5cpp/core>
	#include "your_data_definition.h"
#include <h5cpp/io>

auto ctx = h5::context<some_struct>( dataset );
for( record:entire_dataset)
			h5::append(ctx, record );

Templates:

create dataset within an opened hdf5 file

using par_t = std::initializer_list<hsize_t>

template <typename T> hid_t create(  hid_t fd, const std::string& path, const T ref ) noexcept;
template <typename T> hid_t create(hid_t fd, const std::string& path, par_t max_dims, par_t chunk_dims={},
															const int32_t deflate = H5CPP_NO_COMPRESSION ) noexcept;

read a dataset and return a reference of the created object

using par_t = std::initializer_list<hsize_t>

template <typename T> T read(const std::string& file, const std::string& path ); 
template <typename T> T read(hid_t fd, const std::string& path ); 
template <typename T> T read(hid_t ds ) noexcept; 
template <typename T> T read(hid_t ds, par_t offset, par_t count  ) noexcept; 
template <typename T> T read(hid_t fd, const std::string& path, par_t offset, par_t count  );

write dataset into a specified location

using par_t = std::initializer_list<hsize_t>

template <typename T> hid_t write(hid_t ds, const T* ptr, const hsize_t* offset, const hsize_t* count ) noexcept;
template <typename T> hid_t write(hid_t ds, const T* ptr, par_t offset, par_t count) noexcept;
template <typename T> hid_t write(hid_t ds, const T& ref, par_t offset, par_t count) noexcept;
template <typename T> hid_t write(hid_t fd, const std::string& path, const T& ref) noexcept;
template <typename T> hid_t write(hid_t fd, const std::string& path, const T& ref, par_t offset, par_t count) noexcept;
template <typename T> hid_t write(hid_t fd, const std::string& path, const T* ptr, par_t offset, par_t count) noexcept;

append to extentable C++/C struct dataset]

#include <h5cpp/core>
	#include "your_data_definition.h"
#include <h5cpp/io>

template <typename T> context<T>( hid_t ds);
template <typename T> context<T>( hid_t fd, const std::string& path);
template <typename T> void append( h5::context<T>& ctx, const T& ref) noexcept;

HOW TO ADD XYZ linear algebra library?

If you're aware of a qualifying linear algebra library that should be included and it isn't please send me an email with the followings:

• where to find it
• list of data structures: mylib::vector, mylib::matrix<T,?,..?..>, mylib::cube<T,...>
• accessors to number of rows, columns, slice, ..., size, read/write pointer to underlying data

**Copyright (c) 2018 vargaconsulting, Toronto,ON Canada**