numpy / numpy-user-dtypes

Repository for example user DTypes using the new API

NumPy is the fundamental package for scientific computing with Python.

• Website: https://www.numpy.org
• Documentation: https://numpy.org/doc
• Mailing list: https://mail.python.org/mailman/listinfo/numpy-discussion
• Source code: https://github.com/numpy/numpy
• Contributing: https://www.numpy.org/devdocs/dev/index.html
• Bug reports: https://github.com/numpy/numpy/issues
• Report a security vulnerability: https://tidelift.com/docs/security

It provides:

• a powerful N-dimensional array object
• sophisticated (broadcasting) functions
• tools for integrating C/C++ and Fortran code
• useful linear algebra, Fourier transform, and random number capabilities

Testing:

NumPy requires pytest and hypothesis. Tests can then be run after installation with:

python -c "import numpy, sys; sys.exit(numpy.test() is False)"

Code of Conduct

NumPy is a community-driven open source project developed by a diverse group of contributors. The NumPy leadership has made a strong commitment to creating an open, inclusive, and positive community. Please read the NumPy Code of Conduct for guidance on how to interact with others in a way that makes our community thrive.

Call for Contributions

The NumPy project welcomes your expertise and enthusiasm!

Small improvements or fixes are always appreciated. If you are considering larger contributions to the source code, please contact us through the mailing list first.

Writing code isn’t the only way to contribute to NumPy. You can also:

• review pull requests
• help us stay on top of new and old issues
• develop tutorials, presentations, and other educational materials
• maintain and improve our website
• develop graphic design for our brand assets and promotional materials
• translate website content
• help with outreach and onboard new contributors
• write grant proposals and help with other fundraising efforts

For more information about the ways you can contribute to NumPy, visit our website. If you’re unsure where to start or how your skills fit in, reach out! You can ask on the mailing list or here, on GitHub, by opening a new issue or leaving a comment on a relevant issue that is already open.

Our preferred channels of communication are all public, but if you’d like to speak to us in private first, contact our community coordinators at [email protected] or on Slack (write [email protected] for an invitation).

We also have a biweekly community call, details of which are announced on the mailing list. You are very welcome to join.

If you are new to contributing to open source, this guide helps explain why, what, and how to successfully get involved.

numpy-user-dtypes's Issues

Small string optimization for stringdtype

For strings smaller than sizeof(ss) - 1 we could use some bitshifting tricks to store the string data directly in the array buffer. See this nice visual explanation using a 32 bit big-endian representation for how this could work, although note that's for a dynamic string type that also stores a capacity.

stringdtype: problem displaying a structured array containing a StringDType

I ran into this exception when I tried to use a StringDType in a structured array:

In [47]: dt = np.dtype([('id', int), ('value', float), ('key', StringDType())])

In [48]: dt
Out[48]: ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/py3.11.5/lib/python3.11/site-packages/IPython/core/formatters.py:708, in PlainTextFormatter.__call__(self, obj)
    701 stream = StringIO()
    702 printer = pretty.RepresentationPrinter(stream, self.verbose,
    703     self.max_width, self.newline,
    704     max_seq_length=self.max_seq_length,
    705     singleton_pprinters=self.singleton_printers,
    706     type_pprinters=self.type_printers,
    707     deferred_pprinters=self.deferred_printers)
--> 708 printer.pretty(obj)
    709 printer.flush()
    710 return stream.getvalue()

File ~/py3.11.5/lib/python3.11/site-packages/IPython/lib/pretty.py:410, in RepresentationPrinter.pretty(self, obj)
    407                         return meth(obj, self, cycle)
    408                 if cls is not object \
    409                         and callable(cls.__dict__.get('__repr__')):
--> 410                     return _repr_pprint(obj, self, cycle)
    412     return _default_pprint(obj, self, cycle)
    413 finally:

File ~/py3.11.5/lib/python3.11/site-packages/IPython/lib/pretty.py:778, in _repr_pprint(obj, p, cycle)
    776 """A pprint that just redirects to the normal repr function."""
    777 # Find newlines and replace them with p.break_()
--> 778 output = repr(obj)
    779 lines = output.splitlines()
    780 with p.group():

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:46, in __repr__(dtype)
     45 def __repr__(dtype):
---> 46     arg_str = _construction_repr(dtype, include_align=False)
     47     if dtype.isalignedstruct:
     48         arg_str = arg_str + ", align=True"

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:96, in _construction_repr(dtype, include_align, short)
     69 """
     70 Creates a string repr of the dtype, excluding the 'dtype()' part
     71 surrounding the object. This object may be a string, a list, or
   (...)
     93     provided as the second parameter.
     94 """
     95 if dtype.fields is not None:
---> 96     return _struct_str(dtype, include_align=include_align)
     97 elif dtype.subdtype:
     98     return _subarray_str(dtype)

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:310, in _struct_str(dtype, include_align)
    305 def _struct_str(dtype, include_align):
    306     # The list str representation can't include the 'align=' flag,
    307     # so if it is requested and the struct has the aligned flag set,
    308     # we must use the dict str instead.
    309     if not (include_align and dtype.isalignedstruct) and _is_packed(dtype):
--> 310         sub = _struct_list_str(dtype)
    312     else:
    313         sub = _struct_dict_str(dtype, include_align)

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:297, in _struct_list_str(dtype)
    292     item += "{}, {}".format(
    293         _construction_repr(base, short=True),
    294         shape
    295     )
    296 else:
--> 297     item += _construction_repr(fld_dtype, short=True)
    299 item += ")"
    300 items.append(item)

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:100, in _construction_repr(dtype, include_align, short)
     98     return _subarray_str(dtype)
     99 else:
--> 100     return _scalar_str(dtype, short=short)

File ~/py3.11.5/lib/python3.11/site-packages/numpy/core/_dtype.py:156, in _scalar_str(dtype, short)
    153     return dtype.type.__name__
    155 else:
--> 156     raise RuntimeError(
    157         "Internal error: NumPy dtype unrecognized type number")

RuntimeError: Internal error: NumPy dtype unrecognized type number

Segfault from getitem with array

import os
os.environ["NUMPY_EXPERIMENTAL_DTYPE_API"] = "1"

import numpy as np
from stringdtype import StringDType

arr = np.array(["a", "b", "c", "d", "e"], dtype=StringDType())

arr[np.array([1, 2, 3])]

This script segfaults with the following stack trace (further frames elided for clarity):

#0  0x0000000000000000 in ?? ()
#1  0x00007ffff64fc7c4 in mapiter_trivial_get (self=0x7ffff57ef340, 
    ind=0x7ffff57ef1c0, result=0x7ffff57ef3c0)
    at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:1531
#2  0x00007ffff6503212 in array_subscript (self=0x7ffff57ef340, 
    op=<numpy.ndarray at remote 0x7ffff57ef1c0>)
    at numpy/core/src/multiarray/mapping.c:1568
#3  0x00007ffff7affd01 in PyObject_GetItem (
    o=<numpy.ndarray at remote 0x7ffff57ef340>, 
    key=<numpy.ndarray at remote 0x7ffff57ef1c0>) at Objects/abstract.c:158

mapiter_trivial_get is calling copyswap:

#1  0x00007ffff64fc7c4 in mapiter_trivial_get (self=0x7ffff57ef340, 
    ind=0x7ffff57ef1c0, result=0x7ffff57ef3c0)
    at numpy/core/src/multiarray/lowlevel_strided_loops.c.src:1531
1531	            copyswap(result_ptr, self_ptr, 0, self);

Threadsafety in string dtype

I don't want anyone to waste cycles on this at this point. However, I do not think that the current string dtype implementation is fully thread-safe if one thread mutates the array while another thread also works with it (reading or writing).

Some form of locking seems necessary. This happens when mutating an element, the original one gets free'd so if someone still using it, that would be incorrect.
That could lead to a double-free, but overall the whole "store" operation probably needs to block (both write-lock blocks everything, read-lock blocks writes).

Not sure how this is usually implement. The probability of contesting access should be practically 0, so I would suspect a fine-grained lock that assumes contesting never really happens is best?

(That said, I am not sure mpfdtype is fully either, maybe it is possible for a number to be partially updated, which might mean it sees neither the old nor the new value :).)

Seg fault arising from PyArray_DiscoverDTypeAndShape

The following script currently seg faults during interpreter finalization when numpy is trying to de-allocate the dtype variable:

import os
os.environ["NUMPY_EXPERIMENTAL_DTYPE_API"] = "1"

import numpy as np
from asciidtype import ASCIIDType
dtype = ASCIIDType(6)
a = np.array(["hello", "array"], dtype=dtype)
arr = np.array([a[0], a[1]])

This script doesn't seg fault:

import os
os.environ["NUMPY_EXPERIMENTAL_DTYPE_API"] = "1"

import numpy as np
from metadatadtype import MetadataDType
dtype = MetadataDType("hello")
a = np.array([1, 2], dtype=dtype)
arr = np.array([a[0], a[1]])

I put watchpoints in gdb on the refcount of the dtypes in both scripts and found that the first time the behavior of the two scripts is different is that in the ASCIIDType script, this incref isn't getting executed: https://github.com/numpy/numpy/blob/main/numpy/core/src/multiarray/array_coercion.c#L691

I unfortunately ran out of time today before I could figure out why *out_descr is NULL in one case but not in the other but wanted to get this written up so I can come back to it on Monday. I suspect there's a missing call to Py_INCREF somewhere in Numpy.

stringdtype fails to build on Linux

Hey! While trying to test stringdtype with numpy/numpy#24835, I encountered some build failures, which also occur when trying to build with the latest numpy nightly. Am I doing something wrong?

Here's the failure I get

numpy-user-dtypes/stringdtype on main is 📦 v0.0.1 via 🐍 v3.11.5 (venv) 
❯ python -m build --wheel -Cbuilddir=build
* Creating venv isolated environment...
* Installing packages in isolated environment... (meson-python, meson>=0.63.0, numpy, patchelf, wheel)
* Getting build dependencies for wheel...
* Installing packages in isolated environment... (ninja >= 1.8.2)
* Building wheel...
+ meson setup --reconfigure /home/lysnikolaou/repos/numpy-user-dtypes/stringdtype /home/lysnikolaou/repos/numpy-user-dtypes/stringdtype/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/home/lysnikolaou/repos/numpy-user-dtypes/stringdtype/build/meson-python-native-file.ini
The Meson build system
Version: 1.2.2
Source dir: /home/lysnikolaou/repos/numpy-user-dtypes/stringdtype
Build dir: /home/lysnikolaou/repos/numpy-user-dtypes/stringdtype/build
Build type: native build
Project name: stringdtype
Project version: undefined
C compiler for the host machine: ccache cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
/tmp/build-env-mokwxajm/lib/python3.11/site-packages/mesonbuild/dependencies/python.py:114: DeprecationWarning: path is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
  with importlib.resources.path('mesonbuild.scripts', 'python_info.py') as f:
Program python found: YES (/tmp/build-env-mokwxajm/bin/python)
Library npymath found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.2)
Run-time dependency python found: YES 3.11
Build targets in project: 1

stringdtype undefined

  User defined options
    Native files: /home/lysnikolaou/repos/numpy-user-dtypes/stringdtype/build/meson-python-native-file.ini
    buildtype   : release
    b_ndebug    : if-release
    b_vscrt     : md

/tmp/build-env-mokwxajm/lib/python3.11/site-packages/mesonbuild/modules/python.py:332: DeprecationWarning: read_binary is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
  f.write(importlib.resources.read_binary('mesonbuild.scripts', 'pycompile.py'))
Found ninja-1.11.1.git.kitware.jobserver-1 at /tmp/build-env-mokwxajm/bin/ninja
Cleaning... 0 files.
+ /tmp/build-env-mokwxajm/bin/ninja
[2/6] Compiling C object _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_umath.c.o
FAILED: _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_umath.c.o 
ccache cc -I_main.cpython-311d-x86_64-linux-gnu.so.p -I. -I.. -I/tmp/build-env-mokwxajm/lib/python3.11/site-packages/numpy/core/include -I../stringdtype/src -I/home/lysnikolaou/repos/cpython-3.11.5/build/include/python3.11d -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_umath.c.o -MF _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_umath.c.o.d -o _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_umath.c.o -c ../stringdtype/src/umath.c
../stringdtype/src/umath.c: In function ‘init_ufuncs’:
../stringdtype/src/umath.c:969:49: error: ‘PyArray_Int8DType’ undeclared (first use in this function); did you mean ‘PyArray_Int8Type’?
  969 |             (PyArray_DTypeMeta *)&StringDType, &PyArray_##typename##DType, \
      |                                                 ^~~~
../stringdtype/src/umath.c:1112:5: note: in expansion of macro ‘INIT_MULTIPLY’
 1112 |     INIT_MULTIPLY(Int8, int8);
      |     ^~~~~
../stringdtype/src/umath.c:969:49: note: each undeclared identifier is reported only once for each function it appears in
  969 |             (PyArray_DTypeMeta *)&StringDType, &PyArray_##typename##DType, \
      |                                                 ^~~~
../stringdtype/src/umath.c:1112:5: note: in expansion of macro ‘INIT_MULTIPLY’
 1112 |     INIT_MULTIPLY(Int8, int8);
      |     ^~~~~
[4/6] Compiling C object _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_casts.c.o
FAILED: _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_casts.c.o 
ccache cc -I_main.cpython-311d-x86_64-linux-gnu.so.p -I. -I.. -I/tmp/build-env-mokwxajm/lib/python3.11/site-packages/numpy/core/include -I../stringdtype/src -I/home/lysnikolaou/repos/cpython-3.11.5/build/include/python3.11d -fvisibility=hidden -fdiagnostics-color=always -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O3 -fPIC -MD -MQ _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_casts.c.o -MF _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_casts.c.o.d -o _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_casts.c.o -c ../stringdtype/src/casts.c
../stringdtype/src/casts.c: In function ‘string_to_float32’:
../stringdtype/src/casts.c:862:21: warning: implicit declaration of function ‘PyUFunc_GiveFloatingpointErrors’ [-Wimplicit-function-declaration]
  862 |                 if (PyUFunc_GiveFloatingpointErrors("cast",                  \
      |                     ^~~~~~~~~~~
../stringdtype/src/casts.c:984:1: note: in expansion of macro ‘STRING_TO_FLOAT_CAST’
  984 | STRING_TO_FLOAT_CAST(float32, f32, npy_isinf, npy_float32)
      | ^~~~~~~~
../stringdtype/src/casts.c: In function ‘string_to_datetime’:
../stringdtype/src/casts.c:1052:13: warning: implicit declaration of function ‘NpyDatetime_ParseISO8601Datetime’ [-Wimplicit-function-declaration]
 1052 |         if (NpyDatetime_ParseISO8601Datetime(
      |             ^~~~~~~~~~~~
../stringdtype/src/casts.c:1057:13: warning: implicit declaration of function ‘NpyDatetime_ConvertDatetimeStructToDatetime64’ [-Wimplicit-function-declaration]
 1057 |         if (NpyDatetime_ConvertDatetimeStructToDatetime64(dt_meta, &dts, out) <
      |             ^~~~~~~~~~~~~~~~~
../stringdtype/src/casts.c: In function ‘datetime_to_string’:
../stringdtype/src/casts.c:1106:17: warning: implicit declaration of function ‘NpyDatetime_ConvertDatetime64ToDatetimeStruct’ [-Wimplicit-function-declaration]
 1106 |             if (NpyDatetime_ConvertDatetime64ToDatetimeStruct(dt_meta, *in,
      |                 ^~~~~~~~~~~~~~~~~
../stringdtype/src/casts.c:1114:17: warning: implicit declaration of function ‘NpyDatetime_MakeISO8601Datetime’ [-Wimplicit-function-declaration]
 1114 |             if (NpyDatetime_MakeISO8601Datetime(
      |                 ^~~~~~~~~~~
../stringdtype/src/casts.c: In function ‘get_casts’:
../stringdtype/src/casts.c:756:49: error: ‘PyArray_Int8DType’ undeclared (first use in this function); did you mean ‘PyArray_Int8Type’?
  756 |             (PyArray_DTypeMeta *)&StringDType, &PyArray_##typename##DType);  \
      |                                                 ^~~~
../stringdtype/src/casts.c:1237:5: note: in expansion of macro ‘DTYPES_AND_CAST_SPEC’
 1237 |     DTYPES_AND_CAST_SPEC(i8, Int8)
      |     ^~~~~~~~
../stringdtype/src/casts.c:756:49: note: each undeclared identifier is reported only once for each function it appears in
  756 |             (PyArray_DTypeMeta *)&StringDType, &PyArray_##typename##DType);  \
      |                                                 ^~~~
../stringdtype/src/casts.c:1237:5: note: in expansion of macro ‘DTYPES_AND_CAST_SPEC’
 1237 |     DTYPES_AND_CAST_SPEC(i8, Int8)
      |     ^~~~~~~~
../stringdtype/src/casts.c:756:49: error: ‘PyArray_HalfDType’ undeclared (first use in this function); did you mean ‘PyArray_HalfType’?
  756 |             (PyArray_DTypeMeta *)&StringDType, &PyArray_##typename##DType);  \
      |                                                 ^~~~
../stringdtype/src/casts.c:1264:5: note: in expansion of macro ‘DTYPES_AND_CAST_SPEC’
 1264 |     DTYPES_AND_CAST_SPEC(f16, Half)
      |     ^~~~~~~~
[5/6] Compiling C object _main.cpython-311d-x86_64-linux-gnu.so.p/stringdtype_src_dtype.c.o
ninja: build stopped: subcommand failed.

ERROR Backend subprocess exited when trying to invoke build_wheel

Sub-byte custom element types

I'm wondering if anyone has considered sub-byte user types as part of the API design. The assumption that types have element sizes that are integer numbers of bytes seems to be baked into current user dtypes design.

For example, consider a packed int4 type, whose elements are nibbles (half-bytes). It doesn't seem possible to express such a type in the user dtype API, e.g., the type descriptor requires a byte size for each element, and ufuncs require byte strides. I was toying with the idea of trying to write such a type, but it doesn't seem to be possible without API changes.

I see this was mentioned briefly in NEP 41 (int2) but there is no further discussion I can see. Has there been any more thinking along those lines?

Switch to an arena allocator

Arrow's string arrays use arrow's variable-sized binary layout, which stores the string data in a single contiguous buffer per array along with a strided array of integer offsets. This is a really nice design because you only need one heap allocation per array and the string data are contiguous in memory so it's much easier to write fast ufuncs and exploit SIMD. The down side is that string arrays need to either be immutable or there can be pathological cases where enlarging a single array element leads to reallocating the storage for an entire array. I don't think it would be a big downside for many real-world use cases for string arrays to force people to copy into a new array if they need to mutate elements, but it would be a wart in NumPy's API where one data type has restrictions that others don't.

Numpy doesn't have a concept of per-array heap storage outside the array buffer, however even if we added facilities for that, the dtype API doesn't allow access to the array object, so we would also need to add a pointer to the per-array storage to the ufunc loops in the dtype API as well. I think a more straightforward approach would be to let the per-array storage buffer live on the descriptor instance and ensure that each array has a unique descriptor object.

A first step towards figuring out how hard it would be to have unique descriptors per array would be to add a flag to the dtype that indicates the dtype is associated with an array. When we set this flag at array creation time, if the flag is already set, we raise an error. I think running the strigdtype tests with this flag turned on would probably catch most of the places in numpy or the dtype implementation that would need to be updated. Once we're confident a per-array descriptor is possible, we get per-array buffer storage for free, and I think it would be relatively straightfoward to switch to an arrow-like in-memory representation.

`ssnewemptylen` failure when calling `arr.astype(StringDType())` for certain array sizes

Haven't had enough time to investigate yet, but I'm putting this here so I don't forget:

import numpy as np
from stringdtype import StringDType
import string

n = 1000
indices = np.random.randint(0, len(string.ascii_lowercase), size=(n, 10))
arr = np.array(
    ["".join([string.ascii_lowercase[i] for i in row]) for row in indices]
)

arr.astype(StringDType())

For n > 128 this fails with a MemoryError.

`PyArray_Item_XDECREF` fails with user dtypes that have `NPY_ITEM_REFCOUNT` flag

When calling arr.fill() on an array of dtype StringDType, a failed assertion causes a coredump. Here's a reproducer:

import numpy as np
from stringdtype import StringDType

x = np.array(['a', 'b', 'c'], dtype=StringDType())

x.fill('f')
print(x)

This happens because when x.fill('f') is called, array_fill is executed, which calls PyArray_FillWithScalar, which tries to XDECREF all objects in the object that fills the array using PyArray_Item_XDECREF. This fails because dtypes of different varieties are handled differently, and type_num is used to check what kind of dtype the input is. There's no case for when the dtype is a user dtype, and a safety assertion is triggered, stopping execution.

In short we need to handle the case when user dtypes with the NPY_ITEM_REFCOUNT are passed into PyArray_Item_XDECREF. I guess in principle this issue belongs on the numpy repo, so I can move it there if there's not a workaround.

Two minor exception handling bugs

Posting these here before I forget about them:

1. Here PyErr_Occurred() is called without first calling PyGilState_Ensure. According to the docs, the caller must hold the GIL.
2. In add_promoter, the call to PyCapsule_New can fail. I think we need to check whether the result is NULL and whether the error indicator is set, then DECREF stuff appropriately.