GithubHelp home page GithubHelp logo

libnpy's Introduction

libnpy

libnpy is a simple C++ library for reading and writing of numpy's .npy files.

Refer to format.py for a detailed description of the .npy format.

This libraries primary purpose is writing numerical data easily and efficiently into the .npy format. It also allows reading .npy files, although only a very limited subset of data types are supported.

Features

  • Writing C++ vectors (std::vector) to .npy files
  • Reading (some) simple .npy files into C++ vectors

Supported data types and mapping

Only scalar numeric data types are supported. There is no natural way to represent more complex objects and parsing the header becomes tremendously more complex. Supported types:

  • unsigned integer
  • signed integer
  • floating point
  • complex floating point (std::complex, ...)

Usage

libnpy is a header only library. You only need to download npy.hpp into your include path. Neither special compiler flags nor a specific build system are required.

Optional: If you use meson, you can use the provided meson.build file to declare the dependency on libnpy.

The API has changed in the last release. The old C-style API is still available, but might get removed in the future.

Reading data:

#include "npy.hpp"
#include <vector>
#include <string>

int main() {
  
  const std::string path {"data.npy"};
  npy::npy_data d = npy::read_npy<double>(path);

  std::vector<double> data = d.data;
  std::vector<unsigned long> shape = d.shape;
  bool fortran_order = d.fortran_order;
}

Writing data:

#include "npy.hpp"
#include <vector>
#include <string>

int main() {
  const std::vector<double> data{1, 2, 3, 4, 5, 6};

  npy::npy_data d;
  d.data = data;
  d.shape = {2, 3};
  d.fortran_order = false; // optional

  const std::string path{"out.npy"};
  npy::write_npy(path, d);
}

This will involve an additional copy of the data, which might be undesireable for larger data. The copy can be avoided by using npy::npy_data_ptr as follows.

#include "npy.hpp"
#include <vector>
#include <string>

int main() {
  const std::vector<double> data{1, 2, 3, 4, 5, 6};

  npy::npy_data_ptr<double> d;
  d.data_ptr = data.data();
  d.shape = {2, 3};
  d.fortran_order = false; // optional

  const std::string path{"out.npy"};
  npy::write_npy(path, d);
}

See test/ for further examples. C++14 is required.

Tests

The tests can be build with meson>=0.55 and depend on catch2.

cd tests
meson setup builddir
meson test -Cbuilddir

Known limitations

  1. Only a few data types are supported.

  2. The numpy header is a literal Python dictionary and the Python syntax is very permissive. libnpy's parser was only tested with numpy's implemenation of the .npy format.

Contributing

Feel free to send me a pull request, open an issue, or contact me directly.

The code is formatted with clang-format. Please test your changes by running the tests and static analysis. Meson automatically builds a target for clang-tidy:

cd tests
meson setup builddir
ninja -C builddir clang-tidy

License

The project is licensed under the MIT license

libnpy's People

Contributors

anthonybarbier avatar bananafructa avatar caic99 avatar drelatgithub avatar llohse avatar mattisinit avatar pauljurczak avatar ptrunschke avatar uberlambda avatar ziembla avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

libnpy's Issues

unable to npy.hpp file

Debuger Error;

abort() has been called

490 line;
read_header0x00b739c0{Project1.exe!npy::read_header(std::basic_istream<char,std::char_traits> &)}

unable to access read_header function, parameter

License?

Hey,

thanks for this library!

Just wanted to inquire what the license is, as I couldn't find it in the repo.

Cheers,

Wolf

Malformed .npy header due to comma separated digit grouping.

I'm using v1.0.1 and I'm getting malformed headers for dimensions larger than 999, e.g.

npy::npy_data_ptr<float> npyData{.data_ptr=depthMap.data(), .shape={2621, 401}};

will produce this header

�NUMPY��F�{'descr': '<f4', 'fortran_order': False, 'shape': (2,621, 401), }

which has the wrong dimensions of [2, 621, 401]. This problem appears only in certain contexts. A simple test in a minimal context doesn't reproduce this problem.

I tried to force the C standard formatting with std::cout.imbue(std::locale("C")); to no effect.

duplicate symbol under GCC-10

In my project, we include npy.hpp in different source files, generate .o files, and link them together.
However, gcc-10 defaults to -fno-common
. This part is raising duplicate symbol error while linking

// typestring magic

To reproduce:
case.zip
Based on libnpy/tests, split test.cpp into 3 cpp files, each containing one function, i.e. load, save, and main.
Compile with g++-10 -c, and link. The linker generates:

duplicate symbol 'npy::has_typestring::dtype' in:
test_load.o
test_save.o
duplicate symbol 'npy::has_typestring::dtype' in:
test_load.o
test_save.o
...

for each has_typestring<T>.

[bug] Reading unsigned char as (signed) char

In read_header(), header length is read as an unsigned, little-endian short int, which is currently done by reading two char and concatenating them into an unsigned int.

However the sign-ness of char in C++ is not specified (implementation defined). When char is treated as signed char, the code immediately overflows for byte in range [128, 255], and causing all latter buffer allocating go wrong.

  uint32_t header_length;
  if (version == version_t{1, 0}) {
    char header_len_le16[2];
    istream.read(header_len_le16, 2);
    header_length = (header_len_le16[0] << 0) | (header_len_le16[1] << 8); // header_length incorrect for signed char!
  .......

[clang] constexpr variable cannot have non-literal type

Any plans to make it compatible with clang?

Problematic line: https://github.com/llohse/libnpy/blob/master/npy.hpp#L67

-- Check for working C compiler: /root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang
-- Check for working C compiler: /root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -- works

-- Check for working CXX compiler: /root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++
-- Check for working CXX compiler: /root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -- works

/root/workspace/neo-ai-dlr/3rdparty/libnpy/npy.hpp:66:30: error: constexpr variable cannot have non-literal type 'const std::array<char, 3>'
constexpr std::array<char,3> endian_chars = { little_endian_char, big_endian_char, no_endian_char };
                             ^
/root/workspace/neo-ai-dlr/3rdparty/libnpy/npy.hpp:66:30: error: implicit instantiation of undefined template 'std::__ndk1::array<char, 3>'
/root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/__tuple:223:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;
                                                               ^
In file included from /root/workspace/neo-ai-dlr/demo/cpp/run_resnet.cc:16:
/root/workspace/neo-ai-dlr/3rdparty/libnpy/npy.hpp:67:30: error: constexpr variable cannot have non-literal type 'const std::array<char, 4>'
constexpr std::array<char,4> numtype_chars = { 'f', 'i', 'u', 'c' };
                             ^
/root/workspace/neo-ai-dlr/3rdparty/libnpy/npy.hpp:67:30: error: implicit instantiation of undefined template 'std::__ndk1::array<char, 4>'
/root/workspace/android-ndk-r19c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/__tuple:223:64: note: template is declared here
template <class _Tp, size_t _Size> struct _LIBCPP_TEMPLATE_VIS array;

More on the issue: https://stackoverflow.com/questions/46576847/clang-vs-gcc-crtp-constexpr-variable-cannot-have-non-literal-type

C++ named requirements: LiteralType: https://en.cppreference.com/w/cpp/named_req/LiteralType

"'std::runtime_error' what(): formatting error: typestrings not matching",

I find i can't load the numpy file due to "terminate called after throwing an instance of "'std::runtime_error' what(): formatting error: typestrings not matching", and i tried "std::vector<std::complex> data", std::vector<std::complex> data, std::vector data, std::vector data and so on, it didn't work too, the size of my numpy file is (1, 25, 224,224).
here is my code:
npy::LoadArrayFromNumpy("00000_1_video.npy", video.shape, video.fortran_order, video.data);

[Feature request] Save std::vector<torch::Tensor> to .npy file

Foremost, I should have thank you for an impressive project. However, for a wider approach to this project, it seems like there has no support for saving libtorch datatypes into .npy without catching errors yet. Should there be a feature like this in the future, it would be very helpful for those using torch via C++. Regards.

Does libnpy/1.0.1 support C++11?

Hi.
I am writing a recipe for libnpy for a package manager called conan.

Let me ask a question that has arisen while writing the recipe.

Version 1.0.1 appears to be buildable in C++11.
Does libnpy officially support C++14 and not C++11?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.