gonum / hdf5

hdf5 is a wrapper for the HDF5 library

Gonum

Installation

The core packages of the Gonum suite are written in pure Go with some assembly. Installation is done using go get.

go get -u gonum.org/v1/gonum/...

Supported Go versions

Gonum supports and tests using the gc compiler on the two most recent Go releases on Linux (386, amd64 and arm64), macOS and Windows (both on amd64).

Note that floating point behavior may differ between compiler versions and between architectures due to differences in floating point operation implementations.

Release schedule

The Gonum modules are released on a six-month release schedule, aligned with the Go releases. i.e.: when Go-1.x is released, Gonum-v0.n.0 is released around the same time. Six months after, Go-1.x+1 is released, and Gonum-v0.n+1.0 as well.

The release schedule, based on the current Go release schedule is thus:

• Gonum-v0.n.0: February
• Gonum-v0.n+1.0: August

Build tags

The Gonum packages use a variety of build tags to set non-standard build conditions. Building Gonum applications will work without knowing how to use these tags, but they can be used during testing and to control the use of assembly and CGO code.

The current list of non-internal tags is as follows:

• safe — do not use assembly or unsafe
• bounds — use bounds checks even in internal calls
• noasm — do not use assembly implementations
• tomita — use Tomita, Tanaka, Takahashi pivot choice for maximimal clique calculation, otherwise use random pivot (only in topo package)

Issues

If you find any bugs, feel free to file an issue on the github issue tracker. Discussions on API changes, added features, code review, or similar requests are preferred on the gonum-dev Google Group.

https://groups.google.com/forum/#!forum/gonum-dev

License

Original code is licensed under the Gonum License found in the LICENSE file. Portions of the code are subject to the additional licenses found in THIRD_PARTY_LICENSES. All third party code is licensed either under a BSD or MIT license.

Code in graph/formats/dot is dual licensed Public Domain Dedication and Gonum License, and users are free to choose the license which suits their needs for this code.

The W3C test suites in graph/formats/rdf are distributed under both the W3C Test Suite License and the W3C 3-clause BSD License.

hdf5's Issues

Getting datasets in a group

What are you trying to do?

Getting a list of datasets from a group for looping.

What did you do?

I believe it's possible to use a combination of the functions here:
https://github.com/gonum/hdf5/blob/master/h5g.go#L108

num := f.NumObjects()

Then looping for the range of num and calling ObjectNameByIndex on each.

What did you expect to happen?

A property in the Group struct here: https://github.com/gonum/hdf5/blob/master/h5g.go#L25

An array with each of the dataset names so that they can be looped and opened easily with a f.OpenDataset("myname")

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

1.10.2

Does this issue reproduce with the current master?

Yes.

hdf5: finalizers are sometimes called before they should

Hello @sbinet. It's me again 😸

Using this library in production, I noticed some major issues that seem to be caused by the finalizers which are used.

Three out of four times, our application is crashing. I looked at the traces and found an interesting thing:

fatal error: unexpected signal during runtime execution
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0xb762bb]

runtime stack:
runtime.throw(0xcd5fec, 0x2a)
        /usr/local/go/src/runtime/panic.go:616 +0x81
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:372 +0x28e

goroutine 4 [syscall]:
runtime.cgocall(0x88e8a0, 0xc42006b698, 0x0)
        /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc42006b658 sp=0xc42006b620 pc=0x401f94
decode/vendor/gonum.org/v1/hdf5._Cfunc_H5Tclose(0x30000000000013e, 0x0)
        _cgo_gotypes.go:913 +0x4d fp=0xc42006b698 sp=0xc42006b658 pc=0x71391d
decode/vendor/gonum.org/v1/hdf5.(*Datatype).Close(0xc42022cbd0, 0xc42006b738, 0x0)
        /go/src/decode/vendor/gonum.org/v1/hdf5/h5t.go:153 +0x37 fp=0xc42006b6d0 sp=0xc42006b698 pc=0x717ac7
decode/vendor/gonum.org/v1/hdf5.(*Datatype).finalizer(0xc42022cbd0)
        /go/src/decode/vendor/gonum.org/v1/hdf5/h5t.go:138 +0x2f fp=0xc42006b728 sp=0xc42006b6d0 pc=0x7179ef
runtime.call32(0x0, 0xce3b48, 0xc420432000, 0x1000000010)
        /usr/local/go/src/runtime/asm_amd64.s:573 +0x3b fp=0xc42006b758 sp=0xc42006b728 pc=0x4535fb
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:222 +0x1eb fp=0xc42006b7e0 sp=0xc42006b758 pc=0x415edb
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42006b7e8 sp=0xc42006b7e0 pc=0x455d81
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:156 +0x62

Looks like the (Datatype).finalizer tries to close an already closed Datatype... hmm... This happened right after the beginning of execution where the finalizer should actually not be called.

Looking at other traces, I noticed that the Close() on Datatype is not always the cause of crash, but a segmentation violation in regard to that Datatype is.

Having extensive logging in place that also spits out a log whenever we call Close() on the Datatype ourselves, I noticed that this actually never happened in those cases. So something else must've closed it before.

So I went to our vendor folder, h5t.go, removed the finalizer related code and BOOOM. The error does not occur any longer. Our application calls Close() when it is actually done writing and everything is fine.

As we both know (latest after the article that you linked recently), finalizers are not optimal. Actually, they seem to be causing issues in regard to this library. Therefore, I would suggest to remove them all over the place and would also take care of it in case you agree.

IMHO everybody that uses this library should actually take care of the Close() calls themselves.

For the record, I am running Go 1.10 and libhdf5 1.10.1.

hdf5: fix []C.hid_t <-> []uint type prunning

there are many places where we do:

func f(vs []uint) {
    cvs := (*C.hid_t)(unsafe.Pointer(&vs[0]))
    C.cfunc(C.int(len(vs)), cvs)
}

(or equivalent.)

either replace all these type prunning snippets or insert a func init() that checks this type prunning is actually working for the platform/OS/compiler setup at hand.

cmd/xyz: rework as proper ExampleXYZ

We should rewrite all the cmd/xyz commands as proper ExampleFoo() examples.

ROS3 Driver Addition

I’m not sure this is really an issue, but given the previous discussions on hdf5 version functionality > 1.8 I figured I would just create one. I’ve added support for the ROS3 driver and wasn’t sure if I should submit a pull request. Fork of the project is here and the changes are in the ros3 branch. I ended up creating a new method for opening files with props (OpenFileWithProps), but since there were comments of “//FIXME: file props” in the current OpenFile method, I wasn't sure if there were changes already being considered. Regardless, let me know if you want a pull request but this functionality does require hdf5 version 1.10.6 or higher. I’m currently running it with 1.10.7.

all: Crash when reading from dataset

What are you trying to do?

I'm trying to write a program to convert h5 files to other formats. The one written in python is very inefficient and starves the memory in my computer

What did you do?

I have this basic program to iterate the contents of the file:

package main

import (
	"log"
	"os"

	"gonum.org/v1/hdf5"
)

func handleErr(err error) {
	if err != nil {
		panic(err)
	}
}

func processDataSet(g *hdf5.Group, name string) {
	ds, err := g.OpenDataset(name)
	handleErr(err)

	defer ds.Close()

	d := []interface{}{}

	err = ds.Read(&d)
	if err != nil {
		log.Printf("error reading dataset: %v", err)
		return
	}

	log.Printf("read: %v", len(d))
}

func processGroup(f *hdf5.File, name string) {
	g, err := f.OpenGroup(name)
	handleErr(err)

	defer g.Close()

	objects, err := g.NumObjects()
	handleErr(err)

	for i := uint(0); i < objects; i++ {
		dsName, err := g.ObjectNameByIndex(i)
		handleErr(err)

		log.Printf("dataset: %v", dsName)

		processDataSet(g, dsName)
	}
}

func main() {
	f, err := hdf5.OpenFile(os.Args[1], 0)
	handleErr(err)

	defer f.Close()

	objects, err := f.NumObjects()
	handleErr(err)

	for i := uint(0); i < objects; i++ {
		groupName, err := f.ObjectNameByIndex(i)
		handleErr(err)

		log.Printf("group: %v", groupName)

		processGroup(f, groupName)
	}
}

It takes the file as first argument, the file that I'm using to test is this one:

https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_th_dim_ordering_th_kernels_notop.h5 (56MB)

What did you expect to happen?

I expect it to not crash

What actually happened?

it crashed with different errors like:

[signal SIGSEGV: segmentation violation code=0x2 addr=0x3c0109033c18 pc=0x104840130]

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

gonum.org/v1/hdf5: master
go version: 1.16
hdf5: Tried both 1.8 and 1.12 (same result on both)
arch: tried arm64 and amd64, same result on both

Does this issue reproduce with the current master?

yes

all: migrate import paths to gonum.org/v1/hdf5

Can't open attributes on the root group

What are you trying to do?

View an attribute on the root path of the HDF5 File "/"

What did you do?

		f, err := hdf5.OpenFile(event.Name(), hdf5.F_ACC_RDONLY)
		if err != nil {
			log.Fatalln("Panic, could not open file")
		}

		d, err := f.OpenGroup("/")
		if err != nil {
			log.Fatalln("Panic, could not open dataset")
			return
		}

		attr, err := d.OpenAttribute("file_version")

OpenAttributes on a group doesn't seem to work. You can however call CreateAttribute, just not read from it.

What did you expect to happen?

The ability to open an attribute like on a dataset.

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

go version go1.10.2 linux/amd64

Does this issue reproduce with the current master?

Unknown

Does this package work on Windows?

DataType: decide what type should be the arg of SetSize (for 32b platforms)

details:

• #10 (comment)

In C, H5Tset_size takes a size_t as argument to set the total size of the DataType.
This was translated into DataType.SetSize(sz uint).

This wasn't taking into account the special value -1 that is used to make a DataType of variable size.

we could settle for Go's int but that would be wrong for 32b platforms.
perhaps use an explicit int64 ?

hdf5: filenames could be more meaningful

I think we can do better than ~single significant letter filenames.

hdf5: figure out how to resolve the Go pointer to Go pointer issue

The GODEBUG=cgocheck=0 incantation is untenable in the long run. So we need to sort out how to deal with types holding pointers.

It is not clear to me how to deal with this using the current approach to calling HDF5 functions where entire values are simply handed off to the library. But anything else involves significant reengineering/reimplementation of the HDF5 code in Go.

Note that the example in the tests that fails this runtime check is already incorrect, since the string field that causes the failure is not correctly round-tripped anyway.

Is there a way to know the type of the object without trying to open first?

What are you trying to do?

I'm reading this file: https://github.com/rcmalli/keras-vggface/releases/download/v2.0/rcmalli_vggface_tf_notop_resnet50.h5

What did you do?

What did you expect to happen?

What actually happened?

I'm using a code based on the code listed in #76 to read the file listed above, the problem is that this file has group inside group but I always try to read datasets inside groups so it fails.
Is there a way to know that the object is a group or dataset?

What I did to solve this was attempting to open a group first if it fails it attempts to open a dataset but I wonder if there's a cleaner way to do this

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

gonum.org/v1/hdf5: master
go version: 1.16

Does this issue reproduce with the current master?

yes

cmd/test*: fail with error while loading shared libraries: libhdf5_serial.so.10: cannot open shared object file: No such file or directory

What are you trying to do?

Run tests.

What did you do?

git checkout 847297cb569e6bb5d2ef9248b222eb4d8d40bd2d
GODEBUG=cgocheck=0 go test ./...

What did you expect to happen?

Successful test completion.

What actually happened?

$ GODEBUG=cgocheck=0 go test ./...
ok  	gonum.org/v1/hdf5	(cached)
--- FAIL: TestCpxCmpd (0.00s)
    main_test.go:24: error: exit status 127
        test-go-cpxcmpd: error while loading shared libraries: libhdf5_serial.so.10: cannot open shared object file: No such file or directory
        
FAIL
FAIL	gonum.org/v1/hdf5/cmd/test-go-cpxcmpd	0.004s
?   	gonum.org/v1/hdf5/cmd/test-go-extend-ds	[no test files]
?   	gonum.org/v1/hdf5/cmd/test-go-hdf5	[no test files]
--- FAIL: TestTableRWCmd (0.00s)
    main_test.go:25: error: exit status 127
        test-go-table-01: error while loading shared libraries: libhdf5_serial.so.10: cannot open shared object file: No such file or directory
        
FAIL
FAIL	gonum.org/v1/hdf5/cmd/test-go-table-01	0.004s
?   	gonum.org/v1/hdf5/cmd/test-go-table-01-readback	[no test files]
ok  	gonum.org/v1/hdf5/cmem	(cached)
FAIL

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

Gonum: 1d8f8b2ee4cec2e174d8abadec9339ec4961b303
HDF5: 847297c
Go version: go version go1.13 linux/amd64

Does this issue reproduce with the current master?

Yes

Additional information

The system-wide install provides a libhdf5_serial.so.100.0.1.

$ dpkg -L libhdf5-100 
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100.0.1
/usr/lib/x86_64-linux-gnu/libhdf5_serial_fortran.so.100.0.1
/usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100.0.0
/usr/lib/x86_64-linux-gnu/libhdf5_serialhl_fortran.so.100.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libhdf5-100
/usr/share/doc/libhdf5-100/changelog.Debian.gz
/usr/share/doc/libhdf5-100/copyright
/usr/share/lintian
/usr/share/lintian/overrides
/usr/share/lintian/overrides/libhdf5-100
/usr/lib/x86_64-linux-gnu/libhdf5_serial.so.100
/usr/lib/x86_64-linux-gnu/libhdf5_serial_fortran.so.100
/usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.100
/usr/lib/x86_64-linux-gnu/libhdf5_serialhl_fortran.so.100
/usr/share/doc/libhdf5-100/RELEASE.txt.gz

Docs not displayed on pkg.go.dev

What are you trying to do?

I attempted to use the new documentation site pkg.go.dev to view this code-base's documentation, but I was unable to do so due to the site being unable to detect the project's license in the root of the repository.

What did you do?

Go to pkg.go.dev/gonum.org/v1/hdf5 and you will see the issue.

Currently the old godoc.org site is still live, but the Go team are "planning to redirect traffic from godoc.org to the corresponding page on pkg.go.dev."

I just wanted to ensure that the developers were aware of this.

What did you expect to happen?

I expected the documentation to load.

What actually happened?

The docs did not load, and a message about license restrictions was displayed

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

Not applicable

Does this issue reproduce with the current master?

Not applicable

travis: build leaves artifacts lying around

The current travis script and testing builds executable in a user-global space rather than locally to the package being tested and the tests then depend on this. This is sort of surprising so we should try not to do that.

Support install on raspbian

On raspbian, hdf5.h and libhdf5_hl.so is under different path from Ubuntu.

// Copyright ©2017 The Gonum Authors. All rights reserved.
// Use of this source code is governed by a BSD-style 
// license that can be found in the LICENSE file.

package hdf5                                                                    
                                                                                
// #cgo LDFLAGS: -lhdf5 -lhdf5_hl
// #cgo darwin CFLAGS: -I/usr/local/include
// #cgo darwin LDFLAGS: -L/usr/local/lib
// #cgo linux CFLAGS: -I/usr/local/include, -I/usr/lib/x86_64-linux-gnu/hdf5/serial/include
// #cgo linux LDFLAGS: -L/usr/local/lib, -L/usr/lib/x86_64-linux-gnu/hdf5/serial/
// #cgo linux arm CFLAGS: -I/usr/local/include, -I/usr/lib/arm-linux-gnueabihf/hdf5/serial/include/
// #cgo linux arm LDFLAGS: -L/usr/local/lib, -L/usr/lib/arm-linux-gnueabihf/hdf5/serial/
// #include "hdf5.h"
import "C"           

String in scalar dataset does not appear to work

Hi,

When I try to run this code:

package main

import (
	"fmt"
	"gonum.org/v1/hdf5"
)

func main() {
	hf, err := hdf5.CreateFile("my_string.h5", hdf5.F_ACC_TRUNC)
	if err != nil {
		fmt.Println("Failed to open file", err)
		return
	}

	dSpaceScalar, err := hdf5.CreateDataspace(hdf5.S_SCALAR)
	if err != nil {
		fmt.Println("Failed to create dataset", err)
		return
	}

	dSet, err := hf.CreateDataset("string_dataset", hdf5.T_GO_STRING, dSpaceScalar)
	if err != nil {
		fmt.Println("Failed to create dataset", err)
		return
	}
	someText := "once upon a time"
	err = dSet.Write(someText)
	if err != nil {
		fmt.Println("Failed to write dataset:")
		return
	}
	dSet.Close()
	dSpaceScalar.Close()
	hf.Close()
	fmt.Println("All good. Closed everything")

	return
}

I get this error:

./teststring
panic: reflect.Value.UnsafeAddr of unaddressable value

goroutine 1 [running]:
reflect.Value.UnsafeAddr(...)
reflect/value.go:1972
gonum.org/v1/hdf5.(*Dataset).WriteSubset(0xc000098360, 0x4be3c0, 0xc000098370, 0x0, 0x0, 0x0, 0x0)
gonum.org/v1/[email protected]/h5d_dataset.go:130 +0x38b
gonum.org/v1/hdf5.(*Dataset).Write(...)
gonum.org/v1/[email protected]/h5d_dataset.go:154
main.main()

This may be related to this comment in your readme:

Known problems
     support for structs with slices and strings as fields is broken

but this obviously isn't a field of a structure.

Thanks,

Matt

Don't have support for resize(H5Dset_extent)

Hi,
I tried to find the wrapper for H5Dset_extent which resize the chunked datasets. I forked the repo to https://github.com/zyc-sudo/hdf5 and implemented the wrapper in h5d_dataset.go as
func (s *Dataset) Resize(dims []uint)

I'm not sure if you guys already implemented it or not. If you already did, could you please tell me where it is? Or if you didn't let me know too so we can do pull request when I finish it to contribute a little to this project

all: consider renaming Datatype into DataType

this was mentionned in #1:

we should probably use a better, Go-conforming, capitalization scheme for Datatype and rename it into DataType.

all: move to Camel Case naming

As discussed e.g. in #35, we should move to the Go-way of using Camel Case, as described in Effective Go.

Right now, we are dealing with a mixture of both Camel Case and more C_style variable names.

[Help wanted] How do I follow a link?

Hey!

I'm using the lib to look at a HDF5 file that is a master file, which references other files. I come as far as stating that a link for a certain sub-file exists, but the path to that file is in something called Link Target Info. How do I extract that from the link?

package main

import (
	"fmt"

	"gonum.org/v1/hdf5"
)

func main() {
	hdf5.DisplayErrors(true)

	// open the file and the dataset
	f, err := hdf5.OpenFile("data/master.h5", hdf5.F_ACC_RDONLY)
	if err != nil {
		panic("err 1 - " + err.Error())
	}
	defer f.Close()

	g1, err := f.OpenGroup("entry")
	if err != nil {
		panic("err 2 - " + err.Error())
	}
	defer g1.Close()

	g2, err := g1.OpenGroup("data")
	if err != nil {
		panic("err 3 - " + err.Error())
	}
	defer g2.Close()

	num, err := g2.NumObjects()
	if err != nil {
		panic("err 4 - " + err.Error())
	}

	for i := uint(0); i < num; i++ {
		name, err := g2.ObjectNameByIndex(i)
		if err != nil {
			panic("err 5 - " + err.Error())
		}

		b := g2.LinkExists(name)
		fmt.Printf("Link for %s exists? %v\n", name, b)
	}
}

Use of HDF5 1.12

What are you trying to do?

I am trying to read HDF5 database (version 1.12.1).

The datatabase was populated using Python's h5py library. The data is pandas dataframe but guess that should be no problem as h5ls and HDFView app read the data without any issues.

What did you do?

I used the example from this repo for reading table. Also tried to use DataSet instead. This is the code excerpt:

package main

import (
  "fmt"
  "gonum.org/v1/hdf5"
)

type ohlcv struct {
  Index      int64   `hdf5:"index"`
  Exchange   string  `hdf5:"exchange"`
  Pair       string  `hdf5:"pair"`
  Timestamp  int64   `hdf5:"timestamp"`
  PriceOpen  float64 `hdf5:"price_open"`
  PriceHigh  float64 `hdf5:"price_high"`
  PriceLow   float64 `hdf5:"price_low"`
  PriceClose float64 `hdf5:"price_close"`
  Volume     float64 `hdf5:"volume"`
}

func main() {
  version, _ := hdf5.LibVersion()
  fmt.Printf("HDF5 version: %s\n", version)

  file, _ := hdf5.OpenFile("tickers.h5", hdf5.F_ACC_RDONLY)
  month, _ := file.OpenGroup("M11")
  day, _ := month.OpenGroup("D07")
  table, _ := day.OpenTable("table")

  recs, _ := table.NumPackets()

  for i := 0; i != recs; i++ {
    p := make([]ohlcv, 1)
    if err := table.Next(&p); err != nil {
      panic(fmt.Errorf("next failed: %s", err))
    }
    fmt.Printf("data[%d]: O:%.2f H:%.2f L:%.2f C:%.2f V:%.2f \n", i, p[0].PriceOpen, p[0].PriceHigh, p[0].PriceLow, p[0].PriceClose, p[0].Volume)

  file.Close()
}

What did you expect to happen?

I expected something like this:

HDF5 version: 1.12.1
data[0]: O:62829.33 H:62858.35 L:62829.32 C:62853.66 V:10.72221
data[1]: O:62853.66 H:62920.04 L:62851.32 C:62896.75 V:10.19546
...
data[1439]: O:63276.08 H:63286.35 L:63250.01 C:63273.59 V:43.11052

What actually happened?

What I get is:

HDF5 version: 1.12.1
data[0]: O:24533265083020748587221761909950877822199906846513430683666835688641707196344354649178734577047675756970784403964996179506865859538714624.00 H:153999479823021862704498665709509248968354775291789269717488570675195022731875416084608859555430794393831940365058635304153349319996889497485119259215244127082639950809210292371944342687481593856.00 L:0.00 C:0.00 V:0.00 
data[1]: O:-0.00 H:11485591669347015527702671166617436553216.00 L:0.00 C:0.00 V:0.00 
data[2]: O:16786184717166469080015018654342952761822206471285346540228583460481189475663073161248768.00 H:116860917747596761471525066204868691258239771993742452785440191
...
data[1433]: O:-9500707167603260.00 H:59636916704940832875429063464307788500085761805873313238334329889516158976.00 L:0.00 C:0.00 V:0.00 
data[1434]: O:5019141222517546172965875509332335194542251462150973230580526953620246797924906675853593091481301449281850864438229980645461301058991845935258992265931878573396108562393519846894098059723698237228341624032758439241216420110000323300523402260850195612038020143058501041685903495909081088.00 H:-14749955137625195020933306096366472509413755436838140703864665704181336175168284990050108620591141868165148740370361311351055003224895967047494566278255701287614003245498688726479296978595350966497450734598780051966490510138076888153184354240036864.00 L:0.00 C:0.00 V:0.00 
...
data[1438]: O:-0.00 H:40804893379413961024208896.00 L:0.00 C:0.00 V:0.00 
data[1439]: O:11485478191699172345758915201790495424512.00 H:-0.00 L:0.00 C:0.00 V:0.00 

Also when trying to access Exchange or Pair attributes, I get the following error:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x405fbc9]

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

go version go1.17 darwin/amd64
gonum.org/v1/hdf5
h5cc -showconfig (excerpt)
General Information:
-------------------
HDF5 Version: 1.12.1
Configured on: Mon Jul 12 08:05:03 BST 2021
Configured by: brew@BigSur
Host system: x86_64-apple-darwin20.4.0
Uname information: Darwin BigSur 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64
Byte sex: little-endian
Installation point: /usr/local/Cellar/hdf5/1.12.1

Does this issue reproduce with the current master?

Yes, it does!

Getting dtype in terms of int32 or int64

What are you trying to do?

I am trying to get the exact size of a datatype

What did you do?

Tried using Datatype.GoType(), but this aliases everything to golang int

What did you expect to happen?

Expected to get an exact datatype

What actually happened?

Didn't get an exact datatype

HDF5 version 1.10 support?

How likely is 1.10 support? What needs to be done to support 1.10?

Example of writing a dataset

What are you trying to do?

I'm trying to write an array in a dataset, what I do is reading it, applying some operations to the data and then I want to write it back. At the moment of writing it back I've tried:

		dataToWrite := ts.Data().([]float32)
		err = ds.WriteSubset(dataToWrite, nil, nil)

		dataToWrite := ts.Data().([]float32)
		err = ds.WriteSubset(&dataToWrite, nil, nil)

		dataToWrite := ts.Data().([]float32)
		err = ds.WriteSubset(dataToWrite, dspace, nil)

etc...

but all of them fail with either:

panic: reflect.Value.UnsafeAddr of unaddressable value

or

panic: code -1

so I'm not sure what I'm missing here

thanks in advance

all: consider renaming GetType into Type

This was mentioned in #1:

we should probably rename

GetType() Indentifier

into:

Type() Indentifier

test-go-cpxcmpd: output does not match with h5ls/h5dump/h5py

What are you trying to do?

I'm examining the test-go-cpxcmpd code and the generated SDScompound.h5 file so to understand how things work.

What did you do?

I run the example:

$ go run cmd/test-go-cpxcmpd/main.go
:: data: [{0 0 1 [0 0 0]} {1 1 0.5 [1 2 3]} {2 4 0.3333333333333333 [2 4 6]} {3 9 0.25 [3 6 9]} {4 16 0.2 [4 8 12]} {5 25 0.16666666666666666 [5 10 15]} {6 36 0.14285714285714285 [6 12 18]} {7 49 0.125 [7 14 21]} {8 64 0.1111111111111111 [8 16 24]} {9 81 0.1 [9 18 27]}]
:: file [SDScompound.h5] created (id=72057594037927936)
:: dset (id=360287970189639680)
:: dset.Write...
:: dset.Write... [ok]
:: data: [{0 0 1 [0 0 0]} {1 1 0.5 [1 2 3]} {2 4 0.3333333333333333 [2 4 6]} {3 9 0.25 [3 6 9]} {4 16 0.2 [4 8 12]} {5 25 0.16666666666666666 [5 10 15]} {6 36 0.14285714285714285 [6 12 18]} {7 49 0.125 [7 14 21]} {8 64 0.1111111111111111 [8 16 24]} {9 81 0.1 [9 18 27]}]

What did you expect to happen?

$ h5ls -d SDScompound.h5
ArrayOfStructures        Dataset {10}
    Data:
         {0, 0, 1, [0,0,0]}, {1, 1, 0.5, [1,2,3]}, {2, 4, 0.333333333333333, [2,4,6]}, {3, 9, 0.25, [3,6,9]}, {4, 16, 0.2, [4,8,12]},
         {5, 25, 0.166666666666667, [5,10,15]}, {6, 36, 0.142857142857143, [6,12,18]}, {7, 49, 0.125, [7,14,21]}, {8, 64, 0.111111111111111, [8,16,24]},
         {9, 81, 0.1, [9,18,27]}

What actually happened?

$ h5ls -d SDScompound.h5
ArrayOfStructures        Dataset {10}
    Data:
         {0, 0, 1, [0,0,0]}, {1, 1, 0.5, [1,0,2]}, {2, 4, 0.333333333333333, [2,0,4]}, {3, 9, 0.25, [3,0,6]}, {4, 16, 0.2, [4,0,8]},
         {5, 25, 0.166666666666667, [5,0,10]}, {6, 36, 0.142857142857143, [6,0,12]}, {7, 49, 0.125, [7,0,14]}, {8, 64, 0.111111111111111, [8,0,16]},
         {9, 81, 0.1, [9,0,18]}

Note how the arrays of integers are completely wrong, ex. [1,0,2] instead of [1,2,3]. h5dump is consistent with h5ls but also h5py (the Python bindings) is. Hence my suspects on gonum/hdf5.

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

Go: 1.16
gonum/hdf5.git: 496fefe

Does this issue reproduce with the current master?

Yes.

I locally removed the string from the dataset so to work around #12.

These are the local changes:

diff --git cmd/test-go-cpxcmpd/main.go cmd/test-go-cpxcmpd/main.go
index 7c80377..3c65cdf 100644
--- cmd/test-go-cpxcmpd/main.go
+++ cmd/test-go-cpxcmpd/main.go
@@ -25,7 +25,6 @@ type s1Type struct {
 	b float32
 	c float64
 	d [3]int
-	e string
 }
 
 type s2Type struct {
@@ -48,7 +47,6 @@ func main() {
 			b: float32(i * i),
 			c: 1. / (float64(i) + 1),
 			d: [...]int{i, i * 2, i * 3},
-			e: fmt.Sprintf("--%d--", i),
 		}
 		//s1[i].d = []float64{float64(i), float64(2*i), 3.*i}}
 	}

hdf5/h5a: consider dropping the Datatype arg in Attribute.Read/Write

right now we have this API for Attribute:

type Attribute struct {
        Identifier
}

// Read reads raw data from a attribute into a buffer.
func (s *Attribute) Read(data interface{}, dtype *Datatype) error { ... }

// Write writes raw data from a buffer to an attribute.
func (s *Attribute) Write(data interface{}, dtype *Datatype) error { ... }

we already use reflect for the interface{} part (to discover its underlying type) so the *Datatype is not strictly needed in Go. (even in the not (yet?) handled case of conversions).

hdf5: start handleing references

In earlier discussions, such as #15, we talked about about working with an "HDF5 DataType that describes a pointer to x". As it is right now, whenever we are receiving a pointer, we are dereferencing it and are working with the underlying value.

The code that was at the center of the discussion was:

when creating a DataType

case reflect.Ptr:
     return NewDataTypeFromType(t.Elem())

and when performing encoding in our cmem package:

case reflect.Ptr:
	return enc.Encode(rv.Elem())

Looking at the HDF5 Documentation describing that topic, one notices that this is a bit more complex than we thought in our discussions. The guys there have a separate interface for handling the issue: Reference Interface.

Therefore, if we want to support those referencing types, we would need to create a wrapper for the named interface.

What still would not be resolved by adding the interface, is the issue of what happens if someone would try to e.g. append a pointer to a Go value without going through such an interface. Would we error? Or would we keep the status quo?

hdf5/cmem: TestEncode restores the wrong endianness value

package cmem test fails with master:

cmem/encoder_test.go:43:4: oldEndian declared but not used

that's because:

oldEndian := nativeEndian
nativeEndian = binary.LittleEndian
defer func() { oldEndian = nativeEndian }()

it restores the wrong endianness :)

Not able to read string attribute from dataset

What are you trying to do?

I try to read a string attribute from a dataset.

What did you do?

Here is a code snippet:

    var version int64
    versionAttr, _ := dset.OpenAttribute("version")
    versionAttr.Read(&version, hdf5.T_STD_U64LE)
    fmt.Println(version)

    var start string
    startAttr, _ := dset.OpenAttribute("start")
    startAttr.Read(&start, hdf5.T_GO_STRING)
    fmt.Println(start)

Reading the version attribute is fine. Reading the 'start' attribute results in empty string.
Here the snippet from h5dump:

      ATTRIBUTE "start" {
         DATATYPE  H5T_STRING {
            STRSIZE H5T_VARIABLE;
            STRPAD H5T_STR_NULLTERM;
            CSET H5T_CSET_UTF8;
            CTYPE H5T_C_S1;
         }
         DATASPACE  SCALAR
         DATA {
         (0): "2018-09-04T02:27:43"
         }
      }
      ATTRIBUTE "version" {
         DATATYPE  H5T_STD_I64LE
         DATASPACE  SCALAR
         DATA {
         (0): 1
         }
      }

What did you expect to happen?

I am excepting a variable start with this content "2018-09-04T02:27:43".

What actually happened?

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

go:
hdf5: master (c257073)

Problem occurs either with libhdf5: 1.8.20 and libhdf5: 1.10.2

Does this issue reproduce with the current master?

yes.

go get gonum.org/v1/hdf5 failed

What are you trying to do?

go get gonum.org/v1/hdf5

What did you do?

go get gonum.org/v1/hdf5

What did you expect to happen?

successed

What actually happened?

gonum.org/v1/hdf5

go\pkg\mod\gonum.org\v1\[email protected]\cgoflags.go:14:11: fatal error: hdf5.h: No such file or directory
14 | // #include "hdf5.h"
| ^~~~~~~~
compilation terminated.

What version of Go, Gonum, Gonum/netlib and libhdf5 are you using?

go version go1.16.3 windows/amd64
[email protected]

Does this issue reproduce with the current master?

yes