xtensor-stack / xtensor-python

Python bindings for xtensor

Python bindings for the xtensor C++ multi-dimensional array library.

• xtensor is a C++ library for multi-dimensional arrays enabling numpy-style broadcasting and lazy computing.

• xtensor-python enables inplace use of numpy arrays in C++ with all the benefits from xtensor

  • C++ universal function and broadcasting
  • STL - compliant APIs.
  • A broad coverage of numpy APIs (see the numpy to xtensor cheat sheet).

The Python bindings for xtensor are based on the pybind11 C++ library, which enables seamless interoperability between C++ and Python.

Installation

xtensor-python is a header-only library. We provide a package for the mamba (or conda) package manager.

mamba install -c conda-forge xtensor-python

Documentation

To get started with using xtensor-python, check out the full documentation

http://xtensor-python.readthedocs.io/

Usage

xtensor-python offers two container types wrapping numpy arrays inplace to provide an xtensor semantics

• pytensor
• pyarray.

Both containers enable the numpy-style APIs of xtensor (see the numpy to xtensor cheat sheet).

• On the one hand, pyarray has a dynamic number of dimensions. Just like numpy arrays, it can be reshaped with a shape of a different length (and the new shape is reflected on the python side).

• On the other hand pytensor has a compile time number of dimensions, specified with a template parameter. Shapes of pytensor instances are stack allocated, making pytensor a significantly faster expression than pyarray.

Example 1: Use an algorithm of the C++ standard library on a numpy array inplace.

C++ code

#include <numeric>                        // Standard library import for std::accumulate
#include <pybind11/pybind11.h>            // Pybind11 import to define Python bindings
#include <xtensor/xmath.hpp>              // xtensor import for the C++ universal functions
#define FORCE_IMPORT_ARRAY
#include <xtensor-python/pyarray.hpp>     // Numpy bindings

double sum_of_sines(xt::pyarray<double>& m)
{
    auto sines = xt::sin(m);  // sines does not actually hold values.
    return std::accumulate(sines.begin(), sines.end(), 0.0);
}

PYBIND11_MODULE(xtensor_python_test, m)
{
    xt::import_numpy();
    m.doc() = "Test module for xtensor python bindings";

    m.def("sum_of_sines", sum_of_sines, "Sum the sines of the input values");
}

Python Code

import numpy as np
import xtensor_python_test as xt

v = np.arange(15).reshape(3, 5)
s = xt.sum_of_sines(v)
print(s)

Outputs

1.2853996391883833

Working example

Get the working example here:

• CMakeLists.txt
• main.cpp
• example.py

Example 2: Create a universal function from a C++ scalar function

C++ code

#include <pybind11/pybind11.h>
#define FORCE_IMPORT_ARRAY
#include <xtensor-python/pyvectorize.hpp>
#include <numeric>
#include <cmath>

namespace py = pybind11;

double scalar_func(double i, double j)
{
    return std::sin(i) - std::cos(j);
}

PYBIND11_MODULE(xtensor_python_test, m)
{
    xt::import_numpy();
    m.doc() = "Test module for xtensor python bindings";

    m.def("vectorized_func", xt::pyvectorize(scalar_func), "");
}

Python Code

import numpy as np
import xtensor_python_test as xt

x = np.arange(15).reshape(3, 5)
y = [1, 2, 3, 4, 5]
z = xt.vectorized_func(x, y)
print(z)

Outputs

[[-0.540302,  1.257618,  1.89929 ,  0.794764, -1.040465],
 [-1.499227,  0.136731,  1.646979,  1.643002,  0.128456],
 [-1.084323, -0.583843,  0.45342 ,  1.073811,  0.706945]]

Installation

We provide a package for the conda package manager.

conda install -c conda-forge xtensor-python

This will pull the dependencies to xtensor-python, that is pybind11 and xtensor.

Project cookiecutter

A template for a project making use of xtensor-python is available in the form of a cookiecutter here.

This project is meant to help library authors get started with the xtensor python bindings.

It produces a project following the best practices for the packaging and distribution of Python extensions based on xtensor-python, including a setup.py file and a conda recipe.

Building and Running the Tests

Testing xtensor-python requires pytest

py.test .

To pick up changes in xtensor-python while rebuilding, delete the build/ directory.

Building the HTML Documentation

xtensor-python's documentation is built with three tools

• doxygen
• sphinx
• breathe

While doxygen must be installed separately, you can install breathe by typing

pip install breathe

Breathe can also be installed with conda

conda install -c conda-forge breathe

Finally, build the documentation with

make html

from the docs subdirectory.

Dependencies on xtensor and pybind11

xtensor-python depends on the xtensor and pybind11 libraries

xtensor-python	xtensor	pybind11
master	^0.25.0	>=2.6.1,<3
0.27.0	^0.25.0	>=2.6.1,<3
0.26.1	^0.24.0	~2.4.3
0.26.0	^0.24.0	~2.4.3
0.25.3	^0.23.0	~2.4.3
0.25.2	^0.23.0	~2.4.3
0.25.1	^0.23.0	~2.4.3
0.25.0	^0.23.0	~2.4.3
0.24.1	^0.21.2	~2.4.3
0.24.0	^0.21.1	~2.4.3

These dependencies are automatically resolved when using the conda package manager.

License

We use a shared copyright model that enables all contributors to maintain the copyright on their contributions.

This software is licensed under the BSD-3-Clause license. See the LICENSE file for details.

xtensor-python's Issues

doxygen warning for pyvectorize docs

See https://xtensor-python.readthedocs.io/en/latest/pyvectorize.html

doxygenfunction: Unable to resolve multiple matches for function “xt::pyvectorize” with arguments () in doxygen xml output for project “xtensor-python” from directory: ../xml. Potential matches:
<elided>

support for noconvert

It seems noconvert is not supported? I tried adding noconvert to my (now classic) summer.cc:

#include <numpy/arrayobject.h>
#include "pybind11/numpy.h"     // complex
#include "pybind11/pybind11.h"
#include "pybind11/operators.h"
#include "numpy/arrayobject.h"
//#include "ndarray/pybind11.h"
#include "xtensor/xtensor.hpp"
#include "xtensor/xcontainer.hpp"
#include "xtensor-python/pyarray.hpp"

#include <complex>

namespace py = pybind11;

PYBIND11_PLUGIN (summer) {
  if (_import_array() < 0) {
    PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import");
    return nullptr;
  }

  py::module m("summer", "pybind11 example plugin");

  // m.def("sum", [](xt::pyarray<double> const& x) {
  //     xt::xtensor<double, 1> tmp(x);
  //     double sum = 0;
  //     for (auto e : tmp)
  //       sum += e;
  //     return sum;
  //   });

  m.def("sum", [](xt::pyarray<std::complex<double>> const& x) {
      std::complex<double> sum = 0;
      for (auto e : x)
        sum += e;
      return sum;
    },
    py::arg("inp").noconvert()
    );

  m.def("sum", [](xt::pyarray<double> const& x) {
      double sum = 0;
      for (auto e : x)
        sum += e;
      return sum;
    },
    py::arg("inp").noconvert()
    );


  return m.ptr();
}

It seems to have no effect. If I reverse the order of the defs (double is 1st), then when passing a complex array I get the numpy warning of discarding imag part. If I use the order above, the time for processing float array is much more than for complex array, suggesting there is conversion.

poor benchmark (lambda issue)

The attached code benchmarks about 10x slower than my earlier code (using ndarray), I don't know why

#include <numpy/arrayobject.h>
#include "pybind11/pybind11.h"
#include "pybind11/stl.h"
#include "xtensor/xarray.hpp"
#include "xtensor/xtensor.hpp"
#include "xtensor/xcontainer.hpp"
#include "xtensor/xbroadcast.hpp"
#include "xtensor/xbuilder.hpp"
#include "xtensor-python/pyarray.hpp"
#include "xtensor-python/pytensor.hpp"
#include <algorithm>            // ?

namespace py = pybind11;

template<typename flt_t>
inline int nint (flt_t x) {
  return x >= 0.0 ? int (x + 0.5) : int (x - 0.5);
}

//xt::pytensor<int,1> x (xt::zeros<int> ({1}));
void F(){
  xt::pytensor<int,2> x (xt::zeros<int> ({1,1}));
}

template<typename flt_t>
struct histogram_2d {
  histogram_2d (flt_t _minx, flt_t _maxx, flt_t _miny, flt_t _maxy,
                          flt_t _deltax, flt_t _deltay, bool _clip=true) :
    minx (_minx), maxx (_maxx), miny (_miny), maxy (_maxy),
    deltax (_deltax), deltay (_deltay), clip (_clip),
    buckets (xt::zeros<int> ({int ((_maxx - _minx)/_deltax + 1), int ((_maxy - _miny)/_deltay + 1)}))
  {}

  histogram_2d (xt::pytensor<int,2> const& _buckets, flt_t _minx, flt_t _maxx,
    flt_t _miny, flt_t _maxy, flt_t _deltax, flt_t _deltay, bool _clip=true) :
    minx (_minx), maxx (_maxx), miny (_miny), maxy (_maxy),
    deltax (_deltax), deltay (_deltay), clip (_clip), buckets (_buckets)
  {}

  flt_t apply (flt_t x, flt_t min, flt_t max) {
    if (x > max)
      if (clip)
        return max;
      else
        throw std::runtime_error ("histogram: out of range");
    else if (x < min)
      if (clip)
        return min;
      else
        throw std::runtime_error ("histogram: out of range");
    else
      return x;
  }

  auto operator() (flt_t x, flt_t y, int w=1) {
    int x_index = nint ((apply (x, minx, maxx) - minx) / deltax);
    int y_index = nint ((apply (y, miny, maxy) - miny) / deltay);
    buckets(x_index, y_index) += w;
    return *this;
  }

  //  auto operator() (xt::xtensor<flt_t,1> x, xt::xtensor<flt_t,1> y, 

  flt_t minx, maxx, miny, maxy, deltax, deltay;
  bool clip;
  xt::pytensor<int,2> buckets;
  
};

PYBIND11_PLUGIN (histogram2d_c) {
  if (_import_array() < 0) {
    PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import");
    return nullptr;
  }
  py::module m("histogram2d_c", "pybind11 example plugin");


  typedef histogram_2d<double> hist_t;
  py::class_<hist_t> (m, "histogram2d")
    .def(py::init<double,double,double,double,double,double,bool>(),
         py::arg("minx"), py::arg("maxx"), py::arg("miny"), py::arg("maxy"),
         py::arg("dx"), py::arg("dy"), py::arg("clip")=true)

    .def("__call__", [](hist_t& h, double x, double y, int w) {
        return h (x, y, w); },
         py::arg("x"), py::arg("y"), py::arg("w")=1)

    .def("__call__", [](hist_t& h, xt::pytensor<double,1> const& x,
                                   xt::pytensor<double,1> const& y,
                                   int w) {
        if (x.shape() != y.shape())
          throw std::runtime_error ("histogram2d::shape mismatch");
        //auto ws = xt::broadcast (w, {x.shape()[0]});
        auto xpt = x.xbegin();
        auto ypt = y.xbegin();
        auto xend = x.xend();   // even moving this out of loop doesn't help
        for (;
             xpt != xend; ++xpt, ++ypt) {
          h (*xpt, *ypt, w);
        }
      },
      py::arg("x"), py::arg("y"), py::arg("w")=1)

    .def("__call__", [](hist_t& h, xt::pytensor<double,1> const& x,
                                   xt::pytensor<double,1> const& y,
                                   xt::pytensor<int,1> const& w) {
        if (x.shape() != y.shape() or x.shape() != w.shape())
          throw std::runtime_error ("histogram2d::shape mismatch");
        auto xpt = x.xbegin();
        auto ypt = y.xbegin();
        auto wpt = w.xbegin();
        for (;
             xpt != x.xend(); ++xpt, ++ypt, ++wpt) {
          h (*xpt, *ypt, *wpt);
        }
      })

    .def_readonly ("buckets", &hist_t::buckets)
    .def ("__getstate__", [](hist_t const& h) {
        return py::make_tuple (h.buckets, h.minx, h.maxx, h.miny, h.maxy, h.deltax, h.deltay, h.clip);
      })
    .def(py::init<xt::pytensor<int,2>const&,double,double,double,double,double,double,bool>())
    .def ("__setstate__", [](hist_t & h, py::tuple t) {
        if (t.size() != 8)
          throw std::runtime_error("Invalid state!");
        new (&h) hist_t (t[0].cast<xt::pytensor<int,2>>(),
                         t[1].cast<double>(),
                         t[2].cast<double>(),
                         t[3].cast<double>(),
                         t[4].cast<double>(),
                         t[5].cast<double>(),
                         t[6].cast<double>(),
                         t[7].cast<bool>()
                         );
      })
    .def_readonly ("minx", &hist_t::minx)
    .def_readonly ("maxx", &hist_t::maxx)
    .def_readonly ("miny", &hist_t::miny)
    .def_readonly ("maxy", &hist_t::maxy)
    .def_readonly ("deltax", &hist_t::deltax)
    .def_readonly ("deltay", &hist_t::deltay)
    .def_readonly ("clip", &hist_t::clip)

    .def_property_readonly ("axes", [](hist_t& h) {
        xt::pytensor<double,1> x (xt::arange (h.minx, h.maxx+h.deltax, h.deltax));
        xt::pytensor<double,1> y (xt::arange (h.miny, h.maxy+h.deltay, h.deltay));
        return py::make_tuple (x, y);
      })
    ;

  return m.ptr();
}

profile shows:

pprof --text /usr/bin/python3 ./test_histogram2d.prof 
Using local file /usr/bin/python3.
Using local file ./test_histogram2d.prof.
Total: 953 samples
     193  20.3%  20.3%      894  93.8% pybind11::cpp_function::initialize::{lambda#3}::_FUN [clone .lto_priv.119]
     119  12.5%  32.7%      344  36.1% PyArray_NewFromDescr_int
     101  10.6%  43.3%      101  10.6% __memmove_avx_unaligned_erms
      47   4.9%  48.3%       47   4.9% PyArray_MultiplyList
      45   4.7%  53.0%       45   4.7% PyMem_RawRealloc

Bindings for views (or xstrided_views)

I was trying to get an array view from cpp and call a python function back with it.
After some help, I managed to do it by converting the view to a xt::pytensor before proceeding with the Python call (See code below).

void cpp_f(xt::pytensor<double, 2> my_arr, py::object py_f) {
	auto v = xt::pytensor<double, 1>(xt::view(my_arr, 0, xt::all()));
	py_f(v);
}

def test_view():
    def py_f(v):
        print(v)
    cpp_f(np.array([ [1,2,3], [4,5,6], ]), py_f)

It would be nice to have some way to pass the view directly to the python function, instead of having to convert them. It seems that it is not possible to make bindings for xt::view directly, though. I'll leave the original discussion here for reference:

Tarcísio Fischer @tarcisiofischer 14:49
Do you know if there'll be bindings for the xt::view anytime soon?

Johan Mabille @JohanMabille 14:50
the thing is numpy view and xtensor views are not exactly the same so we cannot bind view directly

Sylvain Corlay @SylvainCorlay 14:50
Not as a numpy array, but there is some hope for strided views on pyarrays / pytensor.

Johan Mabille @JohanMabille 14:50
we have xstrided_views that works like numpy views, we can make bindings for them

trying to broadcast

Trying out broadcast. The test is trying to broadcast the 3rd arg (int) to an array matching the shape of the 1st arg, x. x and y are 1d arrays.
The error seems to be coming from the py::print

    .def("__call__", [](hist_t& h, xt::pyarray<double> const& x, xt::pyarray<double> const& y, int w) {
        if (x.shape() != y.shape())
          throw std::runtime_error ("histogram2d::shape mismatch");
        auto ws = xt::broadcast (w, {x.shape()[0]});
        py::print (ws.shape());
        for (size_t s = 0; s < x.size(); ++s)
          h (x(s), y(s), ws(s));
      },
      py::arg("x"), py::arg("y"), py::arg("w")=1)

python3 ./test_histogram2d.py 
Traceback (most recent call last):
 File "./test_histogram2d.py", line 5, in <module>
   h (np.zeros (10), np.zeros(10))
RuntimeError: make_tuple(): unable to convert arguments of types 'std::tuple<std::array<unsigned long, 1ul> const&>' to Python object

Pure C++ tests

Adding test suites for pyarray and pytensor used just like xarray and xtensor would probably be the best way to check basic funcitonnality.

[xoptional] Support for numpy masked arrays as optional expressions

This should be done via the xoptional assembly proposed in xtensor-stack/xtensor#238.

Inplace reshape & resize with pytensor should check dimension and assign to m_shape

change default install path?

Hi guys, I find that xtensor-python will be installed under the envs/include directory rather than envs/include/python/ when using conda, and user will need to specify the include path if they do not manually install it.

pybind11 is installed in envs/include/python/ by default.

Iterator pairs de-referencing to proxy references on lower-dimensional expressions.

We need to be able to iterate over an axis of an expression, with iterators that resolve to proxy references on xviews along that axis.

The most general case might require a dynamic xview.

Transpose method for pyarray

@wolfv added a transpose method to the two main containers of xtensor. We should probably add the same to pyarray.

pystrides adaptor is missing reverse iterator pairs (rbegin / rend) and const counterparts

example 2 has ImportError: undefined symbol xtensor_python_ARRAY_API

Hi guys, I tried the example 2. But I got this ImportError. It seems that xtensor_python_ARRAY_API is not defined. I tried grep this in both xtensor and xtensor-python but I cannot find its definition.

It is used here in pycontainer.hpp

#define PY_ARRAY_UNIQUE_SYMBOL xtensor_python_ARRAY_API

Can't use immediate evaluation strategy w/ reductions with xt::pyarray

Attempting to compile the following code results in an error (using xtensor 0.15.9 but the bug was still present when trying against 0.15.2):

mean_ = xt::sum(xt::cast<double>(x), xt::evaluation_strategy::immediate()) / x.size();

.../xtensor/include/xtensor/xreducer.hpp:84:28: error: 'class xt::xfunction<xt::detail::cast<double>::functor<float>, double, const xt::pyarray<float>&>' has no member named 'data'; did you mean 'at'?
             auto begin = e.data().begin();

xtensor-python and array_t

• rewrite pybind::array's accessors to not require creating a temporary array, by using the same method as in xtensor.

Binding array of arrays

I'm attempting to bind a xt::pyarray<xt::pyarray<double>> to a Numpy array of arrays, but I am getting compilation errors instantiating the pyarray. The error is a bit long to include here, the relevant bit is

pybind11/include/pybind11/numpy.h:260:81: error: no member named 'dtype' in 'pybind11::detail::npy_format_descriptor<xt::pyarray<double, 16>, void>'
return detail::npy_format_descriptor<typename std::remove_cv::type>::dtype();

coming from the call to pybind11::dtype::of<T> in the return ofpyarray::raw_array_t. Unfortunately, I don't know enough about the inner workings of pybind11 to determine if the problem lies there, or whether this is a xtensor-python issue.

The problem I am trying to solve is to provide Python bindings to a code that uses xtensor xarray. Using xt::xarray<xt::xarray<double>> in a C++-only situation works as expected. Actually, my case is limited in that I always have three inner arrays, so in C++ I would prefer to use std::array<xt::xarray<double>, 3> (or perhaps xt::xtensor<xt::xarray<double>, 3>) but I do not know how to transparently (without copies) expose those as an ndarray of three ndarrays to Python.

Example code that produces the compilation error:

#include <numeric>
#include "pybind11/pybind11.h"
#include "xtensor/xarray.hpp"
#include "xtensor/xmath.hpp"
#include "xtensor-python/pyarray.hpp"

double sum_of_dim(xt::pyarray<xt::pyarray<double>> & m, const int dim)
{
    auto d = m(dim); 
    return std::accumulate(d.begin(), d.end(), 0.0);
}

PYBIND11_PLUGIN(example1_m)
{
    pybind11::module m("example1_m", "Array of array test");
    m.def("sum_of_dim", sum_of_dim, "Array of array function test");
    return m.ptr();
}

[testing] xtensor-python-cookiecutter CI should generate test builds

This would be really helpful with the testing of xtensor-python.

Test failures with xtensor version 0.10.9+

See the Debian CI test logs for version 0.10.9 and version 0.10.10.

[==========] Running 32 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 15 tests from pyarray
[ RUN      ] pyarray.initializer_constructor
[       OK ] pyarray.initializer_constructor (0 ms)
[ RUN      ] pyarray.shaped_constructor
[       OK ] pyarray.shaped_constructor (0 ms)
[ RUN      ] pyarray.strided_constructor
[       OK ] pyarray.strided_constructor (0 ms)
[ RUN      ] pyarray.valued_constructor
[       OK ] pyarray.valued_constructor (0 ms)
[ RUN      ] pyarray.strided_valued_constructor
[       OK ] pyarray.strided_valued_constructor (0 ms)
[ RUN      ] pyarray.copy_semantic
[       OK ] pyarray.copy_semantic (0 ms)
[ RUN      ] pyarray.move_semantic
[       OK ] pyarray.move_semantic (0 ms)
[ RUN      ] pyarray.extended_constructor
[       OK ] pyarray.extended_constructor (0 ms)
[ RUN      ] pyarray.reshape
[       OK ] pyarray.reshape (0 ms)
[ RUN      ] pyarray.access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:321: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 9
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:316: row_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:332: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:327: column_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:343: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:338: central_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:354: Failure
      Expected: vec(2, 0, 0)
      Which is: 16
To be equal to: vec(2, 0)
      Which is: -1
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:349: unit_shape access
[  FAILED  ] pyarray.access (0 ms)
[ RUN      ] pyarray.indexed_access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:390: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 9
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:385: row_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:400: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:395: column_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:410: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:405: central_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:422: Failure
      Expected: vec(2, 0, 0)
      Which is: 16
To be equal to: vec[id1]
      Which is: -1
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:415: unit_shape access
[  FAILED  ] pyarray.indexed_access (0 ms)
[ RUN      ] pyarray.broadcast_shape
[       OK ] pyarray.broadcast_shape (0 ms)
[ RUN      ] pyarray.iterator
[       OK ] pyarray.iterator (0 ms)
[ RUN      ] pyarray.initializer_list
[       OK ] pyarray.initializer_list (0 ms)
[ RUN      ] pyarray.zerod
[       OK ] pyarray.zerod (0 ms)
[----------] 15 tests from pyarray (0 ms total)

[----------] 14 tests from pytensor
[ RUN      ] pytensor.initializer_constructor
[       OK ] pytensor.initializer_constructor (0 ms)
[ RUN      ] pytensor.shaped_constructor
[       OK ] pytensor.shaped_constructor (0 ms)
[ RUN      ] pytensor.strided_constructor
[       OK ] pytensor.strided_constructor (0 ms)
[ RUN      ] pytensor.valued_constructor
[       OK ] pytensor.valued_constructor (0 ms)
[ RUN      ] pytensor.strided_valued_constructor
[       OK ] pytensor.strided_valued_constructor (0 ms)
[ RUN      ] pytensor.copy_semantic
[       OK ] pytensor.copy_semantic (0 ms)
[ RUN      ] pytensor.move_semantic
[       OK ] pytensor.move_semantic (0 ms)
[ RUN      ] pytensor.extended_constructor
[       OK ] pytensor.extended_constructor (0 ms)
[ RUN      ] pytensor.reshape
[       OK ] pytensor.reshape (0 ms)
[ RUN      ] pytensor.access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:321: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 9
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:316: row_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:332: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:327: column_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:343: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec(2, 1)
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:338: central_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:354: Failure
      Expected: vec(2, 0, 0)
      Which is: 16
To be equal to: vec(2, 0)
      Which is: -1
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:349: unit_shape access
[  FAILED  ] pytensor.access (0 ms)
[ RUN      ] pytensor.indexed_access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:390: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 9
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:385: row_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:400: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:395: column_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:410: Failure
      Expected: vec(2, 1, 0)
      Which is: 20
To be equal to: vec[index1]
      Which is: 2
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:405: central_major access
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:422: Failure
      Expected: vec(2, 0, 0)
      Which is: 16
To be equal to: vec[id1]
      Which is: -1
Google Test trace:
/tmp/autopkgtest-virt-lxc.shared.v25qf1sr/downtmp/autopkgtest_tmp/test_common.hpp:415: unit_shape access
[  FAILED  ] pytensor.indexed_access (1 ms)
[ RUN      ] pytensor.broadcast_shape
[       OK ] pytensor.broadcast_shape (0 ms)
[ RUN      ] pytensor.iterator
[       OK ] pytensor.iterator (0 ms)
[ RUN      ] pytensor.zerod
[       OK ] pytensor.zerod (0 ms)
[----------] 14 tests from pytensor (1 ms total)

[----------] 3 tests from pyvectorize
[ RUN      ] pyvectorize.function
[       OK ] pyvectorize.function (0 ms)
[ RUN      ] pyvectorize.lambda
[       OK ] pyvectorize.lambda (0 ms)
[ RUN      ] pyvectorize.complex
[       OK ] pyvectorize.complex (0 ms)
[----------] 3 tests from pyvectorize (0 ms total)

[----------] Global test environment tear-down
[==========] 32 tests from 3 test cases ran. (1 ms total)
[  PASSED  ] 28 tests.
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] pyarray.access
[  FAILED  ] pyarray.indexed_access
[  FAILED  ] pytensor.access
[  FAILED  ] pytensor.indexed_access

 4 FAILED TESTS

fixed-dimensionality pytensor

Just as xtensor provides both xarray<T> and xtensor<T, n>, it would be nice if xtensor-python provided a fixed-dimensionality container (that would raise at call time if an array(-like) with invalid dimensionality was passed in from python).

interoperate with numpy arrays?

How can I use xtensor-python with numpy arrays? I'd want to call functions implemented with xtensor c++ from python passing numpy arrays, and also return numpy arrays to python.

Is there a discussion or mail list for these questions?

Thanks

Returning python tuples of arrays causes segfault

As described in #30, binding arrays of complex works fine with pybind11 2.0.1 but crashes with the master branch. The bug is triggered after many calls, like in the benchmarks.

The benchmarks should be added to the CI scripts in order to trigger the bug.

Compatibility between xt::pytensor and xt::xtensor

Thank you for developing xtensor and xtensor-python, which are very practical libraries.
In some cases, I want to write the following functions and export to Python with pybind11:

void foo(const xt::pytensor<double, 1> &a)
{
  xt::xtensor<double, 1> b = 0.5 * a;
}

This function can be compiled with g++ (5.4.0@ubuntu 16.04).
However, the following Python code stops with Segmentation Fault (core dumped):
foo(np.array([1]))
although this code works correctly:
foo(np.array([1,2,3]))

If you have some ideas about this bug and don't have enough time for debugging,
I would like to try fixing according to your comments.

Provide custom base for pyarray and pytensor

This would avoid the need to call pybind11 API in the type_caster.

pytensor construct?

I want to construct a pytensor of all zeros. Should the following compile (it doesn't):

void F(){
  xt::pytensor<int,2> x (xt::zeros<int> ({1,1}));
}

slow iteration

Test files are here:
https://gist.github.com/nbecker/13c6ca2869d8b5266b6a1e5ae64e9806

This is not much different than previous benchmark.
The actual benchmark is at the end of test_logsumexp.py

Can't reshape pyarray with shape of different input type

Looks like #120 still isn't fixed as of latest xtensor-python master (w/ xtensor 0.15.9).

Taking the result of xt::pyarray<float>.shape() and calling reshape(<that value>) on another array produces the following error

In file included from ?
warning: narrowing conversion of '((const xt::xcontainer<xt::pyarray<float> >*)(& x))->xt::xcontainer<xt::pyarray<float> >::size()' from 'xt::xcontainer<xt::pyarray<float> >::size_type {aka long unsigned int}' to 'double' inside { } [-Wnarrowing]
           auto out = xt::pyarray<double>({x.size()});
                      ^~~~~
warning: narrowing conversion of '((const xt::xcontainer<xt::pyarray<float> >*)(& x))->xt::xcontainer<xt::pyarray<float> >::size()' from 'xt::xcontainer<xt::pyarray<float> >::size_type {aka long unsigned int}' to 'double' inside { } [-Wnarrowing]
 #define BH_TT xt::pyarray
           auto out = xt::pyarray<double>({x.size()});
                      ^~~~~
...
.../xtensor-python/include/xtensor-python/pycontainer.hpp:345:30: error: reinterpret_cast from type 'xt::detail::xbuffer_storage<long unsigned int*, std::allocator<long unsigned int> >::const_pointer {aka const long unsigned int*}' to type 'npy_intp* {aka long int*}' casts away qualifiers
         PyArray_Dims dims = {reinterpret_cast<npy_intp*>(shape.data()), static_cast<int>(shape.size())};

I'm using GCC7.

Similarly, instantiating a new pyarray using the shape of another array fails with the following error:

error: no matching function for call to 'xt::pyarray<double>::pyarray(const inner_shape_type&)'

Doc: add a section for code exposed to both C++ and python

This section should describe the best practices when functions need to be exposed to both C++ (operating on xtensor / xarray) and Python (operating on pytensor / pyarray)

Problem when running pip install ./my_package following the xtensor cookiecutter procedure: gcc failed because of pystrides_adaptor.hpp (operator[])

(very first issue on github, be indulgent please :) )
When running
(on Sierra 10.12.6, in a conda env in which I previously installed xtensor 0.10.9 and xtensor-python 0.12.1):
pip install ./my_package_created_thanks_to_the_cool_cookiecutter_github_repository (it is named "dyna"),
the following error is returned:
"
Processing ./dyna
Requirement already satisfied: pybind11>=2.0.1 in ./env/lib/python3.6/site-packages (from dyna==0.0.1)
Requirement already satisfied: numpy in ./env/lib/python3.6/site-packages (from dyna==0.0.1)
Installing collected packages: dyna
Running setup.py install for dyna ... error
Complete output from command /Users/jcalais/Prog/workspace/dyna/env/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/pip-geq3z0bp-build/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/pip-fon09ep0-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
creating var
creating var/folders
creating var/folders/x7
creating var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn
creating var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include/python3.6m -c /var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/tmpxc3s13qj.cpp -o var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/tmpxc3s13qj.o -std=c++14
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include/python3.6m -c /var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/tmp3qr7nhob.cpp -o var/folders/x7/h2yj87t94x9gb6nkm0qg1ck40000gn/T/tmp3qr7nhob.o -fvisibility=hidden
building 'dyna' extension
creating build
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/src
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include -arch x86_64 -I/Users/jcalais/Prog/workspace/dyna/env/include/python3.6m -I/Users/jcalais/.local/include/python3.6m -I/Users/jcalais/Prog/workspace/dyna/env/lib/python3.6/site-packages/numpy/core/include -I/Users/jcalais/Prog/workspace/dyna/env/include -I/Users/jcalais/Prog/workspace/dyna/env/Library/include -I/Users/jcalais/Prog/workspace/dyna/env/include/python3.6m -c src/main.cpp -o build/temp.macosx-10.7-x86_64-3.6/src/main.o -stdlib=libc++ -mmacosx-version-min=10.7 -DVERSION_INFO="0.0.1" -std=c++14 -fvisibility=hidden
In file included from src/main.cpp:3:
In file included from /Users/jcalais/Prog/workspace/dyna/env/include/xtensor/xmath.hpp:20:
In file included from /Users/jcalais/Prog/workspace/dyna/env/include/xtensor/xoperation.hpp:18:
/Users/jcalais/Prog/workspace/dyna/env/include/xtensor/xstrides.hpp:88:26: error: no viable overloaded operator[] for type 'const const xt::pystrides_iterator<8>'
return arg * strides[dim] + raw_data_offset<size_type, S, dim + 1>(strides, args...);
^~~~~~~ ~~~
/Users/jcalais/Prog/workspace/dyna/env/include/xtensor/xstrides.hpp:116:28: note: in instantiation of function template specialization 'xt::detail::raw_data_offset<unsigned long, const xt::pystrides_iterator<8>, 0, unsigned long>' requested here
return detail::raw_data_offset<size_type, const typename S::const_iterator, dim, Arg, Args...>(view, arg, args...);
^
/Users/jcalais/Prog/workspace/dyna/env/include/xtensor/xcontainer.hpp:377:27: note: in instantiation of function template specialization 'xt::data_offset<unsigned long, xt::pystrides_adaptor<8>, 0, unsigned long>' requested here
size_type index = data_offset<size_type>(strides(), static_cast<size_type>(args)...);
^
src/main.cpp:20:13: note: in instantiation of function template specialization 'xt::xcontainer<xt::pyarray >::operator()' requested here
return m(0);
^_

_ **/Users/jcalais/Prog/workspace/dyna/env/include/xtensor-python/pystrides_adaptor.hpp:83:26: note: candidate function not viable: 'this' argument has type 'const const xt::pystrides_iterator<8>', but method is not marked const

        inline reference operator[] (difference_type n) { return *(p_current + n) / N; }
                         ^
1 error generated.
error: command 'gcc' failed with exit status 1**

"

Proposal for efficient shape_type

The shape_type for pyarray should be a friend class to pyarray that takes a reference to that pyarray, being a proxy on the numpy array shape buffer and dimensionality (by reference).

pyarray and pytensor names in auto-generated docstring

Pyarray and pytensor rely on pybind11 make_caster<T>::name() to determine the string representation of its value_type (which is then used for the auto-generated docstring). However, make_caster assigns the name "int" to every integral type and "float" to every floating type (except for "char"->"str" and "bool"->"bool").

While this makes perfect sens for vanilla Python, this becomes confusing with Numpy where the precise numeric type is important. For example, defining the two overloaded test functions in the module m:

m.def("test", [](xt::pyarray<int>&a){});
m.def("test", [](xt::pyarray<long>&a){});

leads to the following docstring:

    test(*args, **kwargs)
    Overloaded function.
    
    1. test(arg0: numpy.ndarray[int]) -> None
    2. test(arg0: numpy.ndarray[int]) -> None

where the two overloads seem to have an identical signature.

Would it be possible to bypass pybind11 make_caster for types that corresponds to numpy dtypes such that we would get a correct docstring ? (something like: )

    1. test(arg0: numpy.ndarray[int32]) -> None
    2. test(arg0: numpy.ndarray[int64]) -> None

dtype gets implicitly converted, and buffer is copied

When we have a function taking a pyarray but pass in a numpy array of integers, the numpy array is implicitly converted to a pyarray of doubles (but this creates a copy of the buffer).

We should think about emitting a warning in this case, or make the warning user modifiable.
Or at least clearly document it.

no type named 'type' in 'struct std::enable_if<false, xsimd::batch<double, 4> >'

xtensor-0.16.3
xsimd-4.1.6
xtensor-blas-0.11.0

This code:

https://gist.github.com/nbecker/251112eb10e8effbdeb0545eabef6b31

compiled with
g++ -o mag_sqr.cpython-36m-x86_64-linux-gnu.os -c -g -std=c++1z -O3 -march=native -fvisibility=hidden -flto -DNDEBUG -ffast-math -fPIC -DHAVE_CBLAS -DXTENSOR_USE_XSIMD -I/home/nbecker/.local/include/xtensor-blas/flens -I/home/nbecker/.local/include -I/usr/include/eigen3 -I/home/nbecker/.local/lib/python3.6/site-packages/numpy/core/include -I/usr/include/python3.6m mag_sqr.cc

results in errors (rather long, not copied here)

size() is inconsistent with shape() for 1D slice view

If I'm understanding the docs for xarray::size() correctly, then in the 1D case, size() and shape()[0] should always be identical. Is that correct?

In the following pyarray test case, size() is not consistent with its shape(). Notice that xtensor thinks that col0 has a size is 10 even though its shape is only (5,)

I discovered this using the latest releases of xtensor and xtensor-python, but I can reproduce the behavior using the latest master branches.

Test case:

import numpy as np
from example import print_xt_info

rows = np.zeros((5,2), dtype=np.uint32)
print(f"rows:\t size: {rows.size} shape: {rows.shape} strides: {rows.strides}")
print_xt_info(rows)

col0 = rows[:,0]
print(f"col0:\t size: {col0.size} shape: {col0.shape} strides: {col0.strides}")
print_xt_info(col0)

Output:

rows:	 size: 10 shape: (5, 2) strides: (8, 4)
xt_a:	 size: 10 shape: [5,2,] strides: [2,1,]

col0:	 size: 5 shape: (5,) strides: (8,)
xt_a:	 size: 10 shape: [5,] strides: [2,]

Click for C++ module

#include <iostream>
#include <cstdint>
#include "pybind11/pybind11.h"

#define FORCE_IMPORT_ARRAY
#include "xtensor-python/pyarray.hpp"

namespace py = pybind11;
using std::uint32_t;

template <class T>
std::string seq_to_string(T v)
{
    std::ostringstream ss;
    ss << "[";
    for ( auto x : v )
        ss << x << ",";
    ss << "]";
    return ss.str();
}

inline void print_xt_info(xt::pyarray<uint32_t> const & xa)
{
    std::cout << "xt_a:\t"
              << " size: " << xa.size()
              << " shape: " << seq_to_string(xa.shape())
              << " strides: " << seq_to_string(xa.strides())
              << std::endl << std::endl;
}

PYBIND11_PLUGIN(example)
{
    xt::import_numpy();
    py::module m("example", "");
    m.def("print_xt_info", print_xt_info);
    return m.ptr();
}

Tests FTBFS in standalone mode

Looks like when I try to build the test project standalone with the xtensor-python-dev package (which pulls xtensor-dev and pybind11-dev), the necessary include path for the Python headers is missing:

[ 55%] Building CXX object CMakeFiles/test_xtensor_python.dir/main.cpp.o
/usr/bin/c++    -I/home/gvaillan/debian/packages/xtensor-python/build/googletest-src/googletest/include  -march=native -Wunused-parameter -Wextra -Wreorder -Wconversion -std=c++14 -O3 -DNDEBUG   -o CMakeFiles/test_xtensor_python.dir/main.cpp.o -c /home/gvaillan/debian/packages/xtensor-python/test/main.cpp
/home/gvaillan/debian/packages/xtensor-python/test/main.cpp:9:20: fatal error: Python.h: No such file or directory
 #include <Python.h>
                    ^
compilation terminated.

I wonder which part in the root CMakeLists.txt is adding it. Perhaps the Numpy CMake detection which is missing from test/CMakeLists.txt?

Getting Error - No module named 'xtensor_python_test' in Ubuntu

Please see the below commands to learn about this issue.

root@docker-host:/home/vagrant# uname -a
Linux docker-host 3.13.0-151-generic #201-Ubuntu SMP Wed May 30 14:22:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
=======================================
root@docker-host:/opt/anaconda3/pkgs# conda install -c conda-forge xtensor-python
Solving environment: done

# All requested packages already installed.

root@docker-host:/opt/anaconda3/pkgs#

================================
root@docker-host:/home/vagrant# cat > test.py
import numpy as np
import xtensor_python_test  as xt

v = np.arange(15).reshape(3, 5)
s = xt.sum_of_sines(v)
print(s)
=================================
root@docker-host:/home/vagrant# python test.py
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import xtensor_python_test  as xt
ModuleNotFoundError: No module named 'xtensor_python_test'

CMake: find_packages

Hi,

we should improve the way we are currently looking for xtensor-python dependencies:

find_package(xtl REQUIRED)
message(STATUS "Found xtl: ${xtl_INCLUDE_DIRS}/xtl")
find_package(xtensor REQUIRED)
message(STATUS "Found xtensor: ${xtensor_INCLUDE_DIRS}/xtensor")
find_package(pybind11 REQUIRED)
message(STATUS "Found pybind11: ${pybind11_INCLUDE_DIRS}/pybind11")
find_package(NumPy REQUIRED)
message(STATUS "Found numpy: ${NUMPY_INCLUDE_DIRS}")

I think we could add the following:

• add min-versions to each dependency in the find_package call
• expose in the readme that numpy is a direct dependency, it's not documented; which version range is known to work?
• xtl is not only an indirect dependency of xtensor (which would mean we can remove the find_package and take automatically added PUBLIC dependency of the xtensor dependency) but also a direct dependency used in xtensor-python code: expose it in the readme with its version range as well
• docs: does it run with pybind11 2.2.2 as well? :)

Add named constructor from_shape(const S& ...)

As in xarray/xtensor

Create allowed-failure test fetching pybind11 from source

This should make it easier to track undetected pybind11 bugs in the master branch.

Memory leak with std::tuple<xt::pytensor<...>>

Putting an xt::pytensor inside an std::tuple seems to cause a memory leak, e.g.:

void leak() {
    xt::pytensor<float, 1> ones = xt::ones<float>({10});
    std::tuple<xt::pytensor<float, 1>> tup(ones);
}

Using pybind11's py::tuple doesn't cause the same problem.

Am I doing something wrong here or is this a bug?

See https://github.com/daniel-ziegler/tuple_leak/blob/master/tuple_fun.cpp for a full demonstration of the problem; you can install the package and run https://github.com/daniel-ziegler/tuple_leak/blob/master/demo.py. Intriguingly, it looks like the size of the array doesn't affect how much memory gets leaked.

I'm on pybind11 version 2.2.3 and xtensor-python version 0.19.0.

Is there any custom type cast defined for `xtensor` and `xarray`?

I tried this based on example 1, but it seems that there no type casters like pybind11's stl.h which will auto-cast std::vector and etc. to python types.

double _sum_of_sines(xt::xarray<double> &m)
{
  auto sines = xt::sin(m);
  return std::accumulate(sines.cbegin(), sines.cend(), 0.0);
}

double sum_of_sines(xt::pyarray<double> &m)
{
  return _sum_of_sines(m.cast<xt::xarray<double>>());
}

Error: no type named 'storage type' (when using xt::pyarray in class)

When attempting to use xt::pyarray in a class (where substituting xt::xarray works perfectly) I receive the following error during compilation:

.../third_party/xtensor/include/xtensor/xcontainer.hpp:34:78: error: no type named 'storage_type' in 'struct xt::xcontainer_inner_types<xt::pyarray<double> >'
         using storage_type = typename xcontainer_inner_types<D>::storage_type;

I have a class that's more or less as follows:

class SomeClass {
  xt::pyarray<double> mean_;
  xt::pyarray<double> variance_;
};

I'm using the latest xtensor, xtensor-simd, and xtensor-python as of may 4.

Using GCC6 (via homebrew) on OSX.

Configuration feedback

• when installing the latest xtensor using conda, it does not appear to install its header files anywhere in the conda env include
• the cookiecutter template does not refer to any xtensor include path either
• when I clone the latest xtensor master or the 0.8.4 release commit, and add its include path to my cookiecutter instance, I get the following when running 'python setup.py build_ext -fi':

running build_ext building 'pyaabbtree' extension C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\Users\Eelco\Miniconda3\envs\pyaabbtree\Include -IC:\Users\Eelco\AppData\ Roaming\Python\Python36\Include -IC:\Users\Eelco\Miniconda3\envs\pyaabbtree\lib\site-packages\numpy\core\include -IC:\Users\Eelco\Dropbox\Git\xtensor\include -IC:\Users\Eelco\Miniconda3\env s\pyaabbtree\include -IC:\Users\Eelco\Miniconda3\envs\pyaabbtree\include -IC:\Users\Eelco\Miniconda3\envs\pyaabbtree\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUD E" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETF XSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\shared" "-IC:\Program Files (x86)\Windows Kits\8.1\include\\um" "-IC:\Program Files (x86)\Windows Kits\8.1\include \\winrt" /EHsc /Tpsrc/main.cpp /Fobuild\temp.win-amd64-3.6\Release\src/main.obj /EHsc /DVERSION_INFO=\"0.0.1\" main.cpp c:\users\eelco\dropbox\git\xtensor\include\xtensor\xfunction.hpp(132): error C2039: 'layout_type': is not a member of 'xt::pyarray<double>' src/main.cpp(18): note: see declaration of 'xt::pyarray<double>' src/main.cpp(25): note: see reference to class template instantiation 'xt::xfunction<std::plus<double>,double,const xt::pyarray<double> &,xt::xscalar<const int>>' being compiled c:\users\eelco\dropbox\git\xtensor\include\xtensor\xfunction.hpp(132): error C2065: 'layout_type': undeclared identifier c:\users\eelco\dropbox\git\xtensor\include\xtensor\xfunction.hpp(133): error C2039: 'contiguous_layout': is not a member of 'xt::pyarray<double>' src/main.cpp(18): note: see declaration of 'xt::pyarray<double>' c:\users\eelco\dropbox\git\xtensor\include\xtensor\xfunction.hpp(133): error C2065: 'contiguous_layout': undeclared identifier c:\users\eelco\dropbox\git\xtensor\include\xtensor\xfunction.hpp(133): error C2975: 'B': invalid template argument for 'xt::and_c', expected compile-time constant expression c:\users\eelco\dropbox\git\xtensor\include\xtensor\xutils.hpp(210): note: see declaration of 'B' error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\BIN\\x86_amd64\\cl.exe' failed with exit status 2

Am I using incompatible versions of the various packages? Sadly I cant see right away if this is some form of trivial error without familiarizing myself with the codebase first.

Can't reshape with shape of another pyarray

inline auto add(xt::pyarray<double> &a, xt::pyarray<double> &b)
{
    auto res = xt::pyarray<double>();
    res.reshape(a.shape());
    xt::noalias(res) = (a + b);
    return res;
}

fails to compile with:

src/main.cpp: In function ‘auto add(xt::pyarray<double>&, xt::pyarray<double>&)’:
src/main.cpp:26:26: error: no matching function for call to ‘xt::pyarray<double>::reshape(const inner_shape_type&)’
     res.reshape(a.shape());
                          ^
In file included from /home/wolfv/Programs/xtensor-python/include/xtensor-python/pyarray.hpp:20:0,
                 from src/main.cpp:8:
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:222:17: note: candidate: void xt::pycontainer<D>::reshape(const shape_type&) [with D = xt::pyarray<double>; xt::pycontainer<D>::shape_type = std::vector<long unsigned int>]
     inline void pycontainer<D>::reshape(const shape_type& shape)
                 ^~~~~~~~~~~~~~
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:222:17: note:   no known conversion for argument 1 from ‘const inner_shape_type {aka const xt::xbuffer_adaptor<long unsigned int>}’ to ‘const shape_type& {aka const std::vector<long unsigned int>&}’
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:236:17: note: candidate: void xt::pycontainer<D>::reshape(const shape_type&, xt::layout_type) [with D = xt::pyarray<double>; xt::pycontainer<D>::shape_type = std::vector<long unsigned int>]
     inline void pycontainer<D>::reshape(const shape_type& shape, layout_type l)
                 ^~~~~~~~~~~~~~
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:236:17: note:   candidate expects 2 arguments, 1 provided
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:249:17: note: candidate: void xt::pycontainer<D>::reshape(const shape_type&, const strides_type&) [with D = xt::pyarray<double>; xt::pycontainer<D>::shape_type = std::vector<long unsigned int>; xt::pycontainer<D>::strides_type = std::vector<long unsigned int>]
     inline void pycontainer<D>::reshape(const shape_type& shape, const strides_type& strides)
                 ^~~~~~~~~~~~~~
/home/wolfv/Programs/xtensor-python/include/xtensor-python/pycontainer.hpp:249:17: note:   candidate expects 2 arguments, 1 provided

Add support for xtensor_fixed

Hi,

currently xtensor python brings seamless integration of xarray and xtensor types with numpy. However xtensor_fixed type is not supported. Hence the following definitions work (with m an instance of pybind11::module):

m.def("test_xarray", [](){return xt::xarray<int>{1,2};});
m.def("test_xtensor", [](){return xt::xtensor<int, 1>{1,2};});

while this one fails

m.def("test_xtensor_fixed", [](){return xt::xtensor_fixed <int, xt::xshape<2>>{1,2};});

at runtime. When trying to call the method from python, the following error is reported:

TypeError: Unable to convert function return value to a Python type! The signature was
() -> xt::xfixed_container<int, xt::fixed_shape<2ul>, (xt::layout_type)1, xt::xtensor_expression_tag>

Would it be possible to add seamless integration with xtensor_fixed ?

Numpy Array of Strings

How would one go about taking in (and returning) a numpy array of strings using xtensor-python (assuming ASCII)?

The use case is I have a numpy array containing a bunch of Base64 encoded JPEG images. I want to decode this batch using an OpenMP loop in C++. Ideally I should also be able to return a numpy array of strings.

I know I can work around this by creating a 2D numpy array of bytes (each row contains the ASCII string's bytes) but the problem is that it requires two passes since we have to find the max string length. Not to mention string conversions in python.

PyMem_Malloc Allocator

I have run into a few issues with a similar allocator to the one added in #23 and wanted to warn you about them so you can decide to either remove it or fix it.

1. PyMem_(Malloc|Free) requires the GIL.
2. As of 3.6, interpreter shutdown causes memory allocated with PyMem_Malloc to be freed.

The issue with 1 is that it makes it hard to release the gil because any internal allocations will require the gil and it is not obvious to a consumer what can allocate.

The issue with 2 is that if you store an object at global scope whose destructor invokes PyMem_Free you will get a double free or other failure because the memory was already released.

For my application I have just switched to PyMem_Raw(Malloc|Calloc|Free) but those are wrappers around malloc|calloc|free with slightly different NULL and 0 handling. I feel bad for adding this without sufficient testing and I hope you haven't had too many issues because of this.

is there a procedure to install xtensor-python via standard pip on windows ?

maybe it's impossible...

Changing dtype in example causes "Internal error"

First of all, it was really easy to get started with xtensor-python using your cookiecutter template. Thanks!

I tried modifying one of the examples in a seemingly trivial way: I replaced double with uint8. And now it crashes! Any idea what's going on?

Here's the condensed example, adapted from example2():

#include "pybind11/pybind11.h"
#include "xtensor/xarray.hpp"

#define FORCE_IMPORT_ARRAY
#include "xtensor-python/pyarray.hpp"
#include "xtensor-python/pyvectorize.hpp"

namespace py = pybind11;
using std::uint8_t;

// I'm using uint8_t here instead of double
inline xt::pyarray<uint8_t> example2(xt::pyarray<uint8_t> &m)
{
    return m + 2;
}

PYBIND11_PLUGIN(example_module)
{
    using namespace pybind11::literals;
    xt::import_numpy();
    py::module m("example_module", "");
    m.def("example2", example2);
    return m.ptr();
}

I tried to run this test program:

import numpy as np
from example_module import example2

a = np.zeros((10,), np.uint8)
b = example2(a)

And it crashes with the following error:

Traceback (most recent call last):
  File "test2.py", line 5, in <module>
    b = example2(a)
RuntimeError: Precondition violation!
Internal error: trivial_assigner called with unequal types.
  /miniforge/envs/flyem-forge/include/xtensor/xassign.hpp(377)

I tried other dtypes, too:

Crashes: uint8_t, uint16_t, int8_t, int16_t
No crash: uint32_t, uint64_t, int32_t, int64_t, float, double

Setup details: I'm using OSX and clang with libc++, with the following conda packages:

numpy                     1.13.1                   py36_0
python                    3.6.2                         0
xtensor                   0.12.1                        0    conda-forge
xtensor-python            0.14.0                        0    conda-forge
pybind11                  2.1.1                    py36_0    conda-forge

Build issues with version 0.20.0

I am having build issues wit the latest release of xtensor-python (0.20.0)
and xtensor (0.17.4) / xtl (0.4.16).

The lines

#define FORCE_IMPORT_ARRAY
#include "xtensor-python/pyarray.hpp"

are causing:

                 from /home/cpape/Work/software/src/z5/src/python/lib/z5py.cxx:7:
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp: In function ‘constexpr auto xt::detail::trivial_begin(C&)’:
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp:406:25: error: ‘xtl::mpl’ has not been declared
             return xtl::mpl::static_if<has_storage_iterator<C>::value>([&](auto self)
                         ^~~
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp: In function ‘constexpr auto xt::detail::trivial_end(C&)’:
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp:418:25: error: ‘xtl::mpl’ has not been declared
             return xtl::mpl::static_if<has_storage_iterator<C>::value>([&](auto self)
                         ^~~
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp: In function ‘constexpr auto xt::detail::trivial_begin(const C&)’:
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp:430:25: error: ‘xtl::mpl’ has not been declared
             return xtl::mpl::static_if<has_storage_iterator<C>::value>([&](auto self)
                         ^~~
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp: In function ‘constexpr auto xt::detail::trivial_end(const C&)’:
/home/cpape/Work/software/conda/miniconda3/envs/z5-bld/include/xtensor/xiterator.hpp:442:25: error: ‘xtl::mpl’ has not been declared
             return xtl::mpl::static_if<has_storage_iterator<C>::value>([&](auto self)
                         ^~~

I am using Ubuntu 18.04 and gcc 7.3.

Note that this error does not occur with xtensor-python (0.19.1), xtensor (0.16.4) and xtl (0.4.16)

from pytensor to xtensor without copying the underlying data and vice-versa

Is there a way to go from a xt::pytensor to xt::xtensor and back again without copying the underlying data ?
If so, it would be nice with some documentation on how to do it. If not, then I have a feature request :) .