[C++] Importing an extension type without `ARROW:extension:metadata` crashes about arrow

[jorisvandenbossche]

A reproducer just with pyarrow as well (using your registered ext type example):

class DummyArray:
    def __init__(self):
        self._field = pa.field("", pa.null(), metadata={"ARROW:extension:name": "arrow.test"})
        self._array = pa.array([None, None], pa.null())

    def __arrow_c_array__(self, requested_schema=None):
        return self._field.__arrow_c_schema__(), self._array.__arrow_c_array__()[1]

pa.array(DummyArray())

which crashes, and when adding "ARROW:extension:metadata": "" to the dict, it works.

Now, apart from your fix (thanks for that!), I realize from the example that it is right now actually impossible to register an extension type without metadata (at least from Python).

I don't know it we should fix that (e.g. allow __arrow_ext_serialize__ to return None in addition to bytes?), or whether this is OK as long as we ensure this still works when actually receiving data with extension types without metadata (i.e. your fix for this issue)

[jorisvandenbossche]

We should probably also check that this works in other places that imports extension types, like reading IPC.

[jorisvandenbossche]

For IPC it seems to work:

table = pa.Table.from_arrays(
   [pa.array([None, None], pa.null())],
   schema=pa.schema([pa.field("a", pa.null(), metadata={"ARROW:extension:name": "arrow.test"})])
)
with pa.ipc.new_file("/tmp/test_ext_no_meta.arrow", table.schema) as f:
    f.write(table)

with pa.ipc.open_file("/tmp/test_ext_no_meta.arrow") as f:
    result = f.read_all()
print(result.schema)

pa.register_extension_type(DummyExtType())

with pa.ipc.open_file("/tmp/test_ext_no_meta.arrow") as f:
    result = f.read_all()
print(result.schema)

And the same for write/read Parquet (where we get the extension type from the ARROW:schema)

[paleolimbot]

Good call! It looks like the version of this that IPC and Parquet (which I believe uses the IPC Schema encoding) use is here and has more or less the same logic.

arrow/cpp/src/arrow/ipc/metadata_internal.cc

Lines 871 to 892 in 065a6da

	if (metadata != nullptr) {
	// Look for extension metadata in custom_metadata field
	int name_index = metadata->FindKey(kExtensionTypeKeyName);
	if (name_index != -1) {
	std::shared_ptr<ExtensionType> ext_type =
	GetExtensionType(metadata->value(name_index));
	if (ext_type != nullptr) {
	int data_index = metadata->FindKey(kExtensionMetadataKeyName);
	std::string type_data = data_index == -1 ? "" : metadata->value(data_index);
	ARROW_ASSIGN_OR_RAISE(type, ext_type->Deserialize(type, type_data));
	// Remove the metadata, for faithful roundtripping
	if (data_index != -1) {
	RETURN_NOT_OK(metadata->DeleteMany({name_index, data_index}));
	} else {
	RETURN_NOT_OK(metadata->Delete(name_index));
	}
	}
	// NOTE: if extension type is unknown, we do not raise here and
	// simply return the storage type.
	}
	}

(Also good call on the registration not mattering!)

[paleolimbot]

Issue resolved by pull request 41763
#41763