tbeu / matio Goto Github PK
View Code? Open in Web Editor NEWMATLAB MAT File I/O Library
Home Page: https://matio.sourceforge.io
License: BSD 2-Clause "Simplified" License
MATLAB MAT File I/O Library
Home Page: https://matio.sourceforge.io
License: BSD 2-Clause "Simplified" License
@sebastien-villemot Do you have access to a MATLAB installation on a big endian machine (like old Mac on PPC or even Solaris)?
Currently, matio does not support reading UTF16 strings. Unfortunately, decently recent Windows computer use UTF16 and when Matlab writes a .mat file, it will default to UTF16 for strings containing non-ascii characters.
This means that with the current version of matio, trying to read a .mat file created by Matlab on Windows that contains non-ascii characters (like the '°' for temperatures or like some German accentuated letters) will write lots of error messages ("Character data not supported type") as well as fail to read the offending strings (which becomes even more problematic when these should contain the units). As I am stuck with some files with such properties (they come from an operational weather forecast toolchain and should be forwarded into another operational toolchain, so I can absolutely not ask for anything to be changed), my only hope is to have some support implemented in matio.
As I see it, they are a few options:
Of course, you might have other ideas or prefer some other options!
I get this error when running the make command, I have hdf5 installed via brew:
main-10-230-5-186:matio-1.5.2 aaron$ make
Making all in src
/Applications/Xcode.app/Contents/Developer/usr/bin/make all-am
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -g -O2 -c -o mat73.lo mat73.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I. -g -O2 -c mat73.c -fno-common -DPIC -o .libs/mat73.o
mat73.c:599:69: error: too few arguments to function call, expected 4, have 3
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
~~~~~~~~~~~~~~ ^
/usr/local/include/H5Rpublic.h:80:8: note: 'H5Rdereference2' declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^
mat73.c:904:58: error: too few arguments to function call, expected 4, have 3
ref_ids+l);
^
/usr/local/include/H5Rpublic.h:80:8: note: 'H5Rdereference2' declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^
mat73.c:1033:73: error: too few arguments to function call, expected 4, have 3
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
~~~~~~~~~~~~~~ ^
/usr/local/include/H5Rpublic.h:80:8: note: 'H5Rdereference2' declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^
mat73.c:2788:77: error: too few arguments to function call, expected 4, have 3
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
~~~~~~~~~~~~~~ ^
/usr/local/include/H5Rpublic.h:80:8: note: 'H5Rdereference2' declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^
4 errors generated.
make[2]: *** [mat73.lo] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1
The max default dimension for HDF5-based MAT variables was changed from stack allocation to heap allocation by dec7a79. A better idea is to have some kind of (configurable) max. dimension for stack allocation. e.g., if rank is less than or equal to 3 stack allocation is used, whereas higher-dimensional arrays utilize dynamic allocation (heap).
I have just created a matio-package for MSYS2, see https://github.com/Alexpux/MINGW-packages/pull/2296
So far my mileage is pretty good, but in order to make test test suite work, I have to use a strange
workaround: I have to touch test/testsuite.at
before executing make check
. Is thee some logic
that uses the file date for something related to the tests? If yes, is there a better, cleaner solution
then to touch the file?
Hi,
In the process of preparing the HDF5 1.10 transition in Debian, i've tried a build of Matio 1.5.8 against the HDF5 1.10 packages currently in Debian experimental. Matio builds fine but the test suite reports 2 failures :
Note: this instance of HDF5 1.10 was configured with --with-default-api-version=v18.
Thanks.
testsuite-dir.tar.gz
Hi, I am wondering how the perfomance of MATIO is compared to the standard matlab libraries. I need to read a lot of data from multiple .mat files and also write alot of data so it is of high importance that it can be done quickly. So I tried comparing MATIO with Matlab 2015b libraries. Unfortunately MATIO was much slower (when Matlab libs took 60s to read a bunch of files it took almost 400s for MATIO).
But I don't know if I that have compiled MATIO with settings that caused it to be much slower or if my benchmarking software has some bugs in it. Is there a big performance difference or have I just done something wrong on my end?
See https://travis-ci.org/tbeu/matio/jobs/169049417#L911-L914 for the GCC output with warnings on unused return value of fread.
Hello,
is it possible to read a complex valued Matrix from some .mat file using your software?
I couldn't figure out how this is done.
Considering this .mat file, I want to read the variable usol, which is a 2d complex double array of size 256x101.
The code looks like this:
#include <iostream>
#include "matio.h"
int main(void)
{
mat_t* matfp;
matvar_t* usol_var;
matfp = Mat_Open("burgers.mat", MAT_ACC_RDONLY);
usol_var = Mat_VarRead(matfp, "usol");
mat_complex_split_t* usol_data = (mat_complex_split_t*) usol_var->data;
// Gives correct results:
std::cout << *((double*)usol_data[0].Re) << "\n";
// Won't work:
// std::cout << *((double*)usol_data[1].Re) << "\n";
Mat_Close(matfp);
return 0;
}
I get correct results for the really first entry of the matrix. But moving forward the usol_var pointer, I get inconsistent results. On the other hand, reading standard double vectors (t,x) from data works fine.
I am using the matio library to write the contents of a std::vector<std::vector<std::vector<double>>>
and some other data to a MatLab file. I used a tutorial here: http://na-wiki.csc.kth.se/mediawiki/index.php/MatIO
However, Mat_Open
stays NULL
, meaning I can't write anything to the file. There are no error messages, linking/compilation is fine, Saving mat file
is never printed and the exit code of the program is 0
. Am I missing something obvious?
Contents of the function that should create the MatLab file:
mat_t *mat;
matvar_t *matvar;
unsigned long dims[3] = {100, 100, 3};
unsigned long single_dim[1] = {1};
unsigned long size = 100;
double number = 3.14;
mat = Mat_Open("/home/ruben/data.mat", MAT_ACC_RDWR);
if(mat != NULL) {
std::cout << "Saving mat file" << std::endl;
matvar = Mat_VarCreate("matrix", MAT_C_DOUBLE, MAT_T_DOUBLE, 3, dims, matrix, 0);
Mat_VarWrite(mat, matvar, MAT_COMPRESSION_ZLIB);
Mat_VarFree(matvar);
matvar = Mat_VarCreate("number", MAT_C_DOUBLE, MAT_T_DOUBLE, 1, single_dim, &number, 0);
Mat_VarWrite(mat, matvar, MAT_COMPRESSION_ZLIB);
Mat_VarFree(matvar);
Mat_Close(mat);
}
I include the library using #include <matio.h>
, and link it with -std=c++11 -L/usr/lib -lmatio -lz
. Any help is appreciated.
Hello,
I have found that Fortran interface for matio cannot be compiled.
I would be grateful if you could fix that.
There seems to be some strange interaction of the use of MATIO_NORETURN and EXTERN when building a C++ code on mac and linux.
If I use EXTERN MATIO_NORETURN void Mat_Error( const char *format, ... )
on the Mac it works. The reverse 'MATIO_NOTRETURN EXTERNfails as does having the
MATIO_NORETURN` on a separate line. This is with g++-5.3.0
This ordering also works on my linux system using g++-5.2.0.
A related issue is that g++-4.7.2 seems to define __cplusplus >= 201103L even though it doesn't support the [[noreturn]]. This wasn't supported until g++-4.8.
I think that one fix is to change the ordering to 'EXTERN MATIO_NORETURN' in matio.h
Not sure what the fix is for the [[noreturn]] issue; just opening it up for discussion.
I have been tracking an issue when reading a matlab matrix with matio. The matrix is a full_matrix of size 13446 by 45000. Here is the matvar content:
gdb p *matvar
$11 = {nbytes = 545592704, rank = 2, data_type = MAT_T_DOUBLE, data_size = 8, class_type = MAT_C_DOUBLE, isComplex = 0,
isGlobal = 0, isLogical = 0, dims = 0x64cd20, name = 0x64cd40 "linop", data = 0x0, mem_conserve = 0,
compression = MAT_COMPRESSION_ZLIB, internal = 0x643040}
gdb> p matvar->dims[0]@2
$8 = {13446, 45000}
In Read5, we have : matvar->nbytes = lenmatvar->data_size;
and len is computed as 1344645000.
The problem is that this number is bigger than 2^32 so it overflows, so that nbytes=545592704, which is
13446450008 modulo (2^32) instead of the proper value of 4840560000.
Silently having the overflow is bad. We should either emit an error or increase the size of the nbytes field. Note that matlab reads the matrix correctly, which may mean that such big matrices are allowed by matlab, which hints at correcting the bug by making nbytes a 64 bytes unsigned integer. If this is the proper correction, I can do it, but I'd first prefer getting an agreement that this is the proper solution....
After fixing #40 I noticed that writing of MAT files with MCOS results in MAT files that MATLAB cannot open though it looks good in matio.
Dump variables of original bugv6.mat MAT file (saved by MATLAB). The trailing UINT8 variable without a name is the MCOS data.
$ matdump -f whos bugv6.mat
Name Size Bytes Class
la 1x7 7306 mxSTRUCT_CLASS
1x88 88 mxUINT8_CLASS
Copy bugv6.mat to copy6.mat.
$ test_mat copy -v 5 bugv6.mat -o copy6.mat
Dump gives same output again.
$ matdump -f whos copy6.mat
Name Size Bytes Class
la 1x7 7306 mxSTRUCT_CLASS
1x88 88 mxUINT8_CLASS
However, MATLAB fails
>> load copy6.mat
??? Error using ==> load
Can't read file d:\temp\copy6.mat.
There has been significant reverse engineering work on MCOS by @mbauman and published at http://nbviewer.jupyter.org/gist/mbauman/9121961. Currently, I would be happy to know why copy6.mat is not as expected and what can be worked around to make it a valid MAT file again.
In case of compressed v5 MAT files the write problem is more apparent
$ test_mat copy -v 5 -z bugv7.mat -o copy7.mat
-E- : fields[1], Uncompressed type not MAT_T_MATRIX
-E- : fields[2], Uncompressed type not MAT_T_MATRIX
-E- : fields[3], Uncompressed type not MAT_T_MATRIX
-E- : fields[4], Uncompressed type not MAT_T_MATRIX
-E- : fields[5], Uncompressed type not MAT_T_MATRIX
-E- : fields[6], Uncompressed type not MAT_T_MATRIX
-E- : fields[12], Uncompressed type not MAT_T_MATRIX
-E- : fields[13], Uncompressed type not MAT_T_MATRIX
-E- : fields[14], Uncompressed type not MAT_T_MATRIX
-E- : fields[15], Uncompressed type not MAT_T_MATRIX
-E- : fields[16], Uncompressed type not MAT_T_MATRIX
-E- : fields[17], Uncompressed type not MAT_T_MATRIX
-E- : fields[18], Uncompressed type not MAT_T_MATRIX
-E- : fields[19], Uncompressed type not MAT_T_MATRIX
-E- : fields[20], Uncompressed type not MAT_T_MATRIX
Files are availabe from issue47.zip.
I create two Makefiles with configure.
The first Makefile is created with --enable-debug=no, the secound one with --enable-debug=yes
I diff the files and the only different is a space in the CFLAGS argument.
CFLAGS = -g -O2
CFLAGS = -g -O2
Is anyone interested in CMake support? I have a WIP now, I can make a pull request if there is demand.
It comes with a drawback, though. I might have to rearrange the files in the /src
folder, and possibly more. Existing Makefiles and VS projects will certainly break.
Hello,
currently I'm creating a matvar like this (simplified example)
std::string structName ("Structure");
std::string structFieldName ("CharArray");
std::string testString ("test");
size_t dims[2] = {1, 1};
matvar_t * mvMainStruct = nullptr;
matvar_t * mvCharArray = nullptr;
dims[0] = testString.size();
mvCharArray = Mat_VarCreate(nullptr, MAT_C_CHAR, MAT_T_UTF8, 2, dims, testString.data(), 0);
dims[0] = structName.size();
mvMainStruct = Mat_VarCreateStruct(structName.data(), 2, dims, nullptr, 0);
Mat_VarAddStructField(mvMainStruct, structFieldName.data());
Mat_VarSetStructFieldByName(mvMainStruct, structFieldName.data(), 0, mvCharArray);
This leeds to a structure with 1 field holding a char array. This is stored to a mat file and later read back from it. And there is the problem:
With 1.5.11, reading the CharArray back, its data_type was MAT_T_UTF8 as expected.
Now, having 1.5.12, it is detected as MAT_T_UTF16 type.
opening the mat file in matlab, matlab says
>> Structure.CharArray
ans =
4x1 char array
't'
'e'
's'
't'
Debugging show's, that CharArray content is hold in RAM as ['t', '', 'e', '', 's', '', 't', ''] after reading from mat file.
So I'm wondering, if (MAT_C_CHAR, MAT_T_UTF8) is always stored as 16 byte wide type and reading it back is correct now.
Or is this a bug in matio? (I would expect if MAT_T_UTF8 is written, MAT_T_UTF8 also comes back.)
Hello,
thank you for this great piece of software! I'm packaging matio for Void Linux (https://github.com/voidlinux/void-packages/pull/10554) and have come across
checking for va_copy() function... configure: error: in `/builddir/matio-1.5.11':
configure: error: cannot run test program while cross compiling
during cross compilation. Do you deem it possible to add a configure option to turn of running test programs? I tried turning off checking entirely using ./configure --disable-option-checking
to no avail.
Regards
Florian
Hi,
I'm facing an issue when reading matlab files of version 7.3 with Mat_VarRead(...). Although the file is closed, it cannot be deleted as the process still keeps a reference to the file. Simply calling Mat_VarReadInfo() works fine.
Example:
mat_t* matfp = Mat_Open("C:/temp/test/matlab.mat", MAT_ACC_RDONLY);
// matvar_t* matvar = Mat_VarReadInfo(matfp, "Name"); // -> works fine, file handle is released
matvar_t* matvar = Mat_VarRead(matfp, "name"); // -> fails -> file handle not released after Mat_Close()
Mat_VarPrint(matvar, 1);
Mat_VarFree(matvar);
Mat_Close(matfp);
Environment is windows 7, 64bit, visual studio 2012, hdf4-lib version 1.10.0-patch1. No issues when using matlab v4.0 format.
Can you please investigate and confirm. Thank you for your support.
cstaub
It would be nice to make a Prebuild lib available.
Maybe you can add .lib/.dll (artifacts) for windows on AppVeyor on the same way as the .pdf/.log here:
https://drone.io/github.com/tbeu/matio/files
Hi,
I use Mat_Open compared to Mat_CreateVer, as typically my mat files contain multiple variables.
In this context, I realized that on multiple executions of a code, its associated mat file grows, as if the implicated variables were appended instead of replaced by their new values.
Hope you understand what I mean. Please let me know otherwise, and I will try to clarify this further.
Very useful library btw, thanks!
As far as I know, it is easy to read a double matrix from Mat file using matio.
But, I am wondering How to read sparse matrix from a mat file ? Because my cases are saved in mat, some small matrices are saved in dense matrix, but some are in sparse due to the matrix size.
Please help.
all I can find the support of sparse begin with : matVar->class_type, what should I do next ?
Thanks in advance.
Is there any way to open file paths that are UTF-16 encoded, e.g. записано.mat? Mat_Open requires const char* as input, no way no use const wchar_t* or any other solution?
I don't know what changed, it was working before.
testsuite.log
With MATLAB R2017a, saving MAT v7.3 without compression was introduced:
https://www.mathworks.com/products/matlab/whatsnew.html
save Function: Save MAT v7.3 files without compression for improved performance on some storage devices
Is there additional effort necessary to support this in matio?
Just a friendly question: Is there a Java wrapper or a Java interface for matio
? I could not find one, only JNMatLib that uses the native Matlab dlls.
The empty struct tests that use test/matlab/test_write_cell_empty_struct.m (2753, 2827, 2903, and 2979) fail due to a typo in the equality check. The 'expdata1' on line 5 should be 'expdata'.
The other empty struct test that uses test/matlab/test_write_empty_struct.m (2708) is also failing, but for different reason. The file fails to load. Matlab responds with the error:
'Unable to read MAT-file /ldisk/matio/matio.git/test/testsuite.dir/2708/test_write_empty_struct.mat. File might be corrupt.'
Note that all 5 of these tests are passing the matio portion of the test.
This may be rushing things a little, but I am building on the latest Arch Linux update and I run into an error that I post below.
matio had been building previously, so I assume that there must have been a change in HDF5. I can confirm that if I downgrade to 1.8 it works fine.
I expect this to be an issue with matio being deployed on different systems, but would it be possible to get a new branch?
mat73.c: In function ‘Mat_H5ReadDatasetInfo’:
mat73.c:610:60: error: incompatible type for argument 3 of ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^~~~~~~
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:610:26: error: too few arguments to function ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
mat73.c: In function ‘Mat_H5ReadGroupInfo’:
mat73.c:909:49: error: incompatible type for argument 3 of ‘H5Rdereference2’
ref_ids+l);
^~~~~~~
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:908:34: error: too few arguments to function ‘H5Rdereference2’
ref_id = H5Rdereference(field_id,H5R_OBJECT,
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
mat73.c: In function ‘Mat_H5ReadNextReferenceInfo’:
mat73.c:1070:68: error: incompatible type for argument 3 of ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^~~~~~~
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:1070:34: error: too few arguments to function ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
mat73.c: In function ‘Mat_VarRead73’:
mat73.c:2271:60: error: incompatible type for argument 3 of ‘H5Rdereference2’
dset_id = H5Rdereference(ref_id,H5R_OBJECT,&matvar->internal->hdf5_ref);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:2271:27: error: too few arguments to function ‘H5Rdereference2’
dset_id = H5Rdereference(ref_id,H5R_OBJECT,&matvar->internal->hdf5_ref);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
mat73.c: In function ‘Mat_VarReadData73’:
mat73.c:2526:60: error: incompatible type for argument 3 of ‘H5Rdereference2’
dset_id = H5Rdereference(ref_id,H5R_OBJECT,&matvar->internal->hdf5_ref);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:2526:27: error: too few arguments to function ‘H5Rdereference2’
dset_id = H5Rdereference(ref_id,H5R_OBJECT,&matvar->internal->hdf5_ref);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
mat73.c: In function ‘Mat_VarReadNextInfoIterate’:
mat73.c:2842:68: error: incompatible type for argument 3 of ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^~~~~~~
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: expected ‘H5R_type_t {aka enum <anonymous>}’ but argument is of type ‘hobj_ref_t * {aka long unsigned int *}’
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
^~~~~~~~~~~~~~~
In file included from /usr/include/H5public.h:34:0,
from /usr/include/hdf5.h:24,
from matio_private.h:37,
from mat73.c:38:
mat73.c:2842:34: error: too few arguments to function ‘H5Rdereference2’
ref_id = H5Rdereference(dset_id,H5R_OBJECT,ref_ids+i);
^
In file included from /usr/include/hdf5.h:38:0,
from matio_private.h:37,
from mat73.c:38:
/usr/include/H5Rpublic.h:63:14: note: declared here
H5_DLL hid_t H5Rdereference2(hid_t obj_id, hid_t oapl_id, H5R_type_t ref_type, const void *ref);
Hello.
I found a NULL pointer dereference bug in matdump.
In my opinion,
810 line: It seems that the problem is caused by exceeding the array range.
732 line: There is code that performs NULL validation, but a crash occurs.
Please confirm.
Thanks.
OS: Ubuntu 17.10 64bit
PoC download : PoC
Lines 730 to 810 in aa7ffd3
=================================================================
==2742==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x00000050d188 bp 0x00000000002a sp 0x7ffd5d0f7cc0 T0)
==2742==The signal is caused by a READ memory access.
==2742==Hint: address points to the zero page.
#0 0x50d187 in print_default /home/karas/matio/tools/matdump.c:735:22
#1 0x50e2ea in print_default /home/karas/matio/tools/matdump.c:810:17
#2 0x50ea10 in print_default /home/karas/matio/tools/matdump.c:786:21
#3 0x50ca04 in main /home/karas/matio/tools/matdump.c:932:17
#4 0x7fc482d993f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
#5 0x41a589 in _start (/home/karas/matio/tools/.libs/matdump+0x41a589)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/karas/matio/tools/matdump.c:735:22 in print_default
==2742==ABORTING
when running ./configure
main-10-230-5-186:matio-1.5.2 aaron$ ./configure --with-default-api-version=v18
configure: WARNING: unrecognized options: --with-default-api-version
I'm trying to install PSPNet/caffe (https://github.com/hszhao/PSPNet), which uses matio, I've successfully installed the newest matio with the newest hdf5, however I'm getting these errors, which seem related to matio and hdf5
main-10-230-5-186:PSPNet aaron$ make -j8
[ 1%] Built target proto
Scanning dependencies of target caffe
[ 2%] Building CXX object src/caffe/CMakeFiles/caffe.dir/util/matio_io.cpp.o
[ 4%] Linking CXX shared library ../../lib/libcaffe.dylib
ld: warning: text-based stub file /System/Library/Frameworks//Accelerate.framework/Accelerate.tbd and library file /System/Library/Frameworks//Accelerate.framework/Accelerate are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks//vecLib.framework/Versions/A/vecLib.tbd and library file /System/Library/Frameworks//vecLib.framework/Versions/A/vecLib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib are out of sync. Falling back to library file for linking.
ld: warning: text-based stub file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.tbd and library file /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib are out of sync. Falling back to library file for linking.
Undefined symbols for architecture x86_64:
"_H5LTfind_dataset", referenced from:
caffe::SGDSolver<float>::RestoreSolverStateFromHDF5(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in sgd_solver.cpp.o
caffe::SGDSolver<double>::RestoreSolverStateFromHDF5(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in sgd_solver.cpp.o
void caffe::hdf5_load_nd_dataset_helper<float>(long long, char const*, int, int, caffe::Blob<float>*) in hdf5.cpp.o
void caffe::hdf5_load_nd_dataset_helper<double>(long long, char const*, int, int, caffe::Blob<double>*) in hdf5.cpp.o
"_H5LTget_dataset_info", referenced from:
void caffe::hdf5_load_nd_dataset_helper<float>(long long, char const*, int, int, caffe::Blob<float>*) in hdf5.cpp.o
void caffe::hdf5_load_nd_dataset_helper<double>(long long, char const*, int, int, caffe::Blob<double>*) in hdf5.cpp.o
caffe::hdf5_load_string(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in hdf5.cpp.o
"_H5LTget_dataset_ndims", referenced from:
void caffe::hdf5_load_nd_dataset_helper<float>(long long, char const*, int, int, caffe::Blob<float>*) in hdf5.cpp.o
void caffe::hdf5_load_nd_dataset_helper<double>(long long, char const*, int, int, caffe::Blob<double>*) in hdf5.cpp.o
"_H5LTmake_dataset_double", referenced from:
void caffe::hdf5_save_nd_dataset<double>(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, caffe::Blob<double> const&, bool) in hdf5.cpp.o
"_H5LTmake_dataset_float", referenced from:
void caffe::hdf5_save_nd_dataset<float>(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, caffe::Blob<float> const&, bool) in hdf5.cpp.o
"_H5LTmake_dataset_int", referenced from:
caffe::hdf5_save_int(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) in hdf5.cpp.o
"_H5LTmake_dataset_string", referenced from:
caffe::hdf5_save_string(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in hdf5.cpp.o
"_H5LTread_dataset_double", referenced from:
void caffe::hdf5_load_nd_dataset<double>(long long, char const*, int, int, caffe::Blob<double>*) in hdf5.cpp.o
"_H5LTread_dataset_float", referenced from:
void caffe::hdf5_load_nd_dataset<float>(long long, char const*, int, int, caffe::Blob<float>*) in hdf5.cpp.o
"_H5LTread_dataset_int", referenced from:
caffe::hdf5_load_int(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in hdf5.cpp.o
"_H5LTread_dataset_string", referenced from:
caffe::hdf5_load_string(long long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in hdf5.cpp.o
"_Mat_Close", referenced from:
void caffe::ReadBlobFromMat<float>(char const*, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<double>(char const*, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<int>(char const*, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<unsigned int>(char const*, caffe::Blob<unsigned int>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<float>(char const*, bool, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<double>(char const*, bool, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<int>(char const*, bool, caffe::Blob<int>*) in matio_io.cpp.o
...
"_Mat_CreateVer", referenced from:
void caffe::WriteBlobToMat<float>(char const*, bool, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<double>(char const*, bool, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<int>(char const*, bool, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<unsigned int>(char const*, bool, caffe::Blob<unsigned int>*) in matio_io.cpp.o
"_Mat_Open", referenced from:
void caffe::ReadBlobFromMat<float>(char const*, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<double>(char const*, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<int>(char const*, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<unsigned int>(char const*, caffe::Blob<unsigned int>*) in matio_io.cpp.o
"_Mat_VarCreate", referenced from:
void caffe::WriteBlobToMat<float>(char const*, bool, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<double>(char const*, bool, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<int>(char const*, bool, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<unsigned int>(char const*, bool, caffe::Blob<unsigned int>*) in matio_io.cpp.o
"_Mat_VarFree", referenced from:
void caffe::ReadBlobFromMat<float>(char const*, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<double>(char const*, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<int>(char const*, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<unsigned int>(char const*, caffe::Blob<unsigned int>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<float>(char const*, bool, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<double>(char const*, bool, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<int>(char const*, bool, caffe::Blob<int>*) in matio_io.cpp.o
...
"_Mat_VarReadDataLinear", referenced from:
void caffe::ReadBlobFromMat<float>(char const*, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<double>(char const*, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<int>(char const*, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<unsigned int>(char const*, caffe::Blob<unsigned int>*) in matio_io.cpp.o
"_Mat_VarReadInfo", referenced from:
void caffe::ReadBlobFromMat<float>(char const*, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<double>(char const*, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<int>(char const*, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::ReadBlobFromMat<unsigned int>(char const*, caffe::Blob<unsigned int>*) in matio_io.cpp.o
"_Mat_VarWrite", referenced from:
void caffe::WriteBlobToMat<float>(char const*, bool, caffe::Blob<float>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<double>(char const*, bool, caffe::Blob<double>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<int>(char const*, bool, caffe::Blob<int>*) in matio_io.cpp.o
void caffe::WriteBlobToMat<unsigned int>(char const*, bool, caffe::Blob<unsigned int>*) in matio_io.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [lib/libcaffe.1.0.0-rc3.dylib] Error 1
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
make: *** [all] Error 2
I'm using matio 1.5.10 and matio-ffi.torch. I have a 100Mb file that makes matio to allocate suspiciously a lot of memory:
du -h SelectiveSearchVOC2007trainval.mat.edgeboxes.mat
# 93M SelectiveSearchVOC2007trainval.mat.edgeboxes.mat
/usr/bin/time -f %M th -e '(require "matio").load("SelectiveSearchVOC2007trainval.mat.edgeboxes.mat")'
# 20407944 KiB
Probably I'm missing something obvious, but such memory consumption seems a little fishy to me. Doing the same with matdump
gives:
/usr/bin/time -f %M matdump SelectiveSearchVOC2007trainval.mat.edgeboxes.mat > log.txt
# 5102308
Does this discrepancy of 5Gb vs 20Gb mean matio-ffi.torch is using matio sub-optimally?
log.txt
contains substring Empty
many times:
grep Empty log.txt | wc -l
# 50210218
wc -l log.txt
# 50230273 log.txt
File uploaded to my OneDrive: https://1drv.ms/u/s!Apx8USiTtrYmprRlRQmgSbPJNcWzEw
I was checking the travis builds to check the mat73 tests for hdf5 1.10 and I'm not sure those are executed in the CI.
For this configuration "env": "ENABLE_EXTENDED_SPARSE=yes ENABLE_MAT73=yes WITH_ZLIB=$TRAVIS_BUILD_DIR/zlib HDF5_VERSION_DIR=hdf5_1_10_1 MAX_RANK=3"
that leeds to this configuration:
MATIO Configuration Summary
==============================================================
C Compiler: gcc-4.8
CFLAGS: -g -O2 -fprofile-arcs -ftest-coverage
Shared Libraries: yes
Static Libraries: yes
default MAT version: MAT_FT_MAT5
Features --------------------------------------------
MAT v7.3 file support: yes
Extended sparse support: yes
Packages --------------------------------------------
zlib: -L/home/travis/build/massich/matio/zlib/lib -lz
hdf5: -L/home/travis/build/massich/matio/hdf5_1_10_1/hdf5/lib -lhdf5
MATLAB:
the tests show as skipped in travis:
2830: Write 2D double array skipped (mat73_write.at:30)
2831: Write 2D single array skipped (mat73_write.at:41)
2832: Write 2D int64 array skipped (mat73_write.at:53)
2833: Write 2D uint64 array skipped (mat73_write.at:65)
2834: Write 2D int32 array skipped (mat73_write.at:76)
2835: Write 2D uint32 array skipped (mat73_write.at:87)
2836: Write 2D int16 array skipped (mat73_write.at:98)
2837: Write 2D uint16 array skipped (mat73_write.at:109)
(see here)
I've been trying to trigger them myself here but I didn't manage.
Hey, guys
I have found an issue that related to the higher version of hdf library (above 1.8.16). My code snipper
is like :
matvar_t * shrink = Mat_VarGetStructFieldByName(pChnsfield, "shrink", 0);
assert(shrink != NULL);
paraPyr.chnsParas.shrink = *((double *) shrink->data); //shrink->data becomes null with 1.8.19 or 1.8.20 hdf library
this code works perfectly with version1.8.16 of hdf library.
But it doesn't work with 1.8.19 or later version of hdf library.
I spent some time debugging but haven't found out the reason.
Hope you can give me some help.
Thanks.
Charles Tamz
Problem is wrong matvar->nbytes calculation when using utf8, because actual character datasize depends on utf code. So matvar->nbytes = nmemb*matvar->data_size is incorrect when using specific utf characters.
It seems lib have to be massively rewritten in order to fully support utf8. That's sad.
hi,
we've been trying with @massich to migrate our build system to the matio conda packages for the OpenMEEG project.
So far we build everything from source using CMake external project mechanism but it makes our CI really slow and makes us maintain a lot of code. Also we would like to make soon a conda package based on the matio one.
while we thought we were almost there as it was working on mac and linux, we hit test errors on windows that only appear when we allow support for matlab IO. So it's matio related. See CIs status at the bottom of : massich/findmkl_openmeeg#32
before digging into this we wanted to know if tests were run on windows by appveyor or some continuous integration? if someone tells us that all tests pass on windows then it's a bug in our code:
https://github.com/massich/findmkl_openmeeg/blob/master/OpenMEEGMaths/include/MatlabIO.H
that is specific to windows....
could there be any compiler definition / option that we may have missed on windows?
thanks a lot for any feedback or hint.
thanks
Hello,
Happy New Year.
I do not know if this information will help you.
I checked the source code using PVS-Studio.
I received the following report.
There may be a false here.
The documentation for all analyzer warnings is available here: www.viva64.com/en/w
Thanks.
/home/karas/check/matio/src/inflate.c 61 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 78 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 82 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 97 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 66 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 129 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 141 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 146 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 157 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 133 warn V507 Pointer to local array 'uncomp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 253 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 265 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 304 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 316 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 358 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 370 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 397 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 409 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 447 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 459 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 498 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 510 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 549 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 563 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 604 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 616 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 664 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 684 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 725 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 737 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 776 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 788 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 835 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/inflate.c 847 warn V507 Pointer to local array 'comp_buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/matvar_cell.c 106 warn V522 There might be dereferencing of a potential null pointer 'cells'. Check lines: 106, 103.
/home/karas/check/matio/src/matvar_cell.c 152 warn V522 There might be dereferencing of a potential null pointer 'cells'. Check lines: 152, 149.
/home/karas/check/matio/src/mat.c 846 warn V522 There might be dereferencing of a potential null pointer 'matvar->dims'. Check lines: 846, 844.
/home/karas/check/matio/src/mat.c 909 warn V522 There might be dereferencing of a potential null pointer 'matvar->internal->fieldnames'. Check lines: 909, 907.
/home/karas/check/matio/src/mat.c 1229 warn V522 There might be dereferencing of a potential null pointer 'out->internal->fieldnames'. Check lines: 1229, 1225.
/home/karas/check/matio/src/mat.c 1665 warn V522 There might be dereferencing of a potential null pointer 'subs'. Check lines: 1665, 1659.
/home/karas/check/matio/src/mat.c 1705 warn V522 There might be dereferencing of a potential null pointer 'subs'. Check lines: 1705, 1698.
/home/karas/check/matio/src/mat.c 1921 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/src/mat.c 1927 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/src/mat.c 1937 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/src/mat.c 1942 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/src/matvar_struct.c 66 warn V522 There might be dereferencing of a potential null pointer 'matvar->dims'. Check lines: 66, 64.
/home/karas/check/matio/src/matvar_struct.c 98 warn V522 There might be dereferencing of a potential null pointer 'field_vars'. Check lines: 98, 95.
/home/karas/check/matio/src/matvar_struct.c 459 warn V522 There might be dereferencing of a potential null pointer 'fields'. Check lines: 459, 451.
/home/karas/check/matio/tools/matdump.c 354 warn V522 There might be dereferencing of a potential null pointer 'start'. Check lines: 354, 350.
/home/karas/check/matio/tools/matdump.c 355 warn V522 There might be dereferencing of a potential null pointer 'stride'. Check lines: 355, 351.
/home/karas/check/matio/tools/matdump.c 356 warn V522 There might be dereferencing of a potential null pointer 'edge'. Check lines: 356, 352.
/home/karas/check/matio/tools/matdump.c 369 warn V522 There might be dereferencing of a potential null pointer 'z'. Check lines: 369, 367.
/home/karas/check/matio/tools/matdump.c 485 warn V522 There might be dereferencing of a potential null pointer 'start'. Check lines: 485, 481.
/home/karas/check/matio/tools/matdump.c 486 warn V522 There might be dereferencing of a potential null pointer 'stride'. Check lines: 486, 482.
/home/karas/check/matio/tools/matdump.c 487 warn V522 There might be dereferencing of a potential null pointer 'edge'. Check lines: 487, 483.
/home/karas/check/matio/tools/matdump.c 659 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/tools/matdump.c 671 err V767 Suspicious access to element of 'dims' array by a constant index inside a loop.
/home/karas/check/matio/test/test_snprintf.c 54 err V536 Be advised that the utilized constant value is represented by an octal form. Oct: 0203, Dec: 131.
/home/karas/check/matio/test/test_mat.c 3477 warn V614 Potentially uninitialized variable 'data_type' used. Consider checking the third actual argument of the 'Mat_VarCreate' function.
/home/karas/check/matio/test/test_mat.c 3644 warn V614 Potentially uninitialized variable 'data_type' used. Consider checking the third actual argument of the 'Mat_VarCreate' function.
/home/karas/check/matio/test/test_mat.c 2569 warn V666 Consider inspecting second argument of the function 'Mat_VarCreateStruct'. It is possible that the value does not correspond with the length of a string which was passed with the first argument.
/home/karas/check/matio/src/mat5.c 392 warn V575 The potential null pointer is passed into 'memset' function. Inspect the first argument. Check lines: 392, 390.
/home/karas/check/matio/src/mat5.c 403 warn V575 The potential null pointer is passed into 'memset' function. Inspect the first argument. Check lines: 403, 391.
/home/karas/check/matio/src/mat5.c 546 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 559 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 569 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 566 warn V507 Pointer to local array 'pad' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 590 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 608 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 620 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 617 warn V507 Pointer to local array 'pad' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 636 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 1195 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 1208 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 1218 warn V507 Pointer to local array 'buf' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 1215 warn V507 Pointer to local array 'pad' is stored outside the scope of this array. Such a pointer will become invalid.
/home/karas/check/matio/src/mat5.c 1332 warn V522 There might be dereferencing of a potential null pointer 'cells[i]->dims'. Check lines: 1332, 1329.
/home/karas/check/matio/src/mat5.c 1358 warn V522 There might be dereferencing of a potential null pointer 'cells[i]->name'. Check lines: 1358, 1355.
/home/karas/check/matio/src/mat5.c 1594 warn V522 There might be dereferencing of a potential null pointer 'matvar->internal->fieldnames'. Check lines: 1594, 1592.
/home/karas/check/matio/src/mat5.c 1596 warn V769 The 'ptr' pointer in the 'ptr + i * fieldname_size' expression could be nullptr. In such case, resulting value will be senseless and it should not be used. Check lines: 1596, 1588.
/home/karas/check/matio/src/mat5.c 1687 warn V522 There might be dereferencing of a potential null pointer 'fields[i]->dims'. Check lines: 1687, 1683.
/home/karas/check/matio/src/mat5.c 2208 warn V575 The potential null pointer is passed into 'fwrite' function. Inspect the first argument. Check lines: 2208, 2204.
/home/karas/check/matio/src/mat5.c 2580 warn V575 The potential null pointer is passed into 'memset' function. Inspect the first argument. Check lines: 2580, 2568.
/home/karas/check/matio/src/mat5.c 2955 err V512 A call of the 'memset' function will lead to underflow of the buffer 'uncomp_buf'.
/home/karas/check/matio/src/mat5.c 3208 warn V522 There might be dereferencing of a potential null pointer 'matvar->dims'. Check lines: 3208, 3207.
/home/karas/check/matio/src/mat5.c 5153 err V512 A call of the 'memset' function will lead to underflow of the buffer 'uncomp_buf'.
/home/karas/check/matio/src/mat5.c 5392 warn V575 The potential null pointer is passed into 'fwrite' function. Inspect the first argument. Check lines: 5392, 5388.
/home/karas/check/matio/src/mat5.c 5624 warn V575 The potential null pointer is passed into 'inflateInit_' function. Inspect the first argument. Check lines: 5624, 5623.
/home/karas/check/matio/src/mat5.c 5680 warn V522 There might be dereferencing of a potential null pointer 'matvar->dims'. Check lines: 5680, 5677.
/home/karas/check/matio/src/mat5.c 5712 warn V575 The potential null pointer is passed into 'memcpy' function. Inspect the first argument. Check lines: 5712, 5711.
Hi there! Is it possible to append data to an existing matlab variable? I'm creating the file using Mat_Open, creating a variable using Mat_VarCreate and finally writing data using Mat_VarWrite. Unfortunately I cannot find a way to resize the variable in order to add new data. Any suggestions?
This week I help my classmate to port an algorithm written by MATLAB to an ARM platform.
MATLAB do not have libraries on ARM version, so everything should be a C or C++ source.
With the help of matio
, finally I made this.
The code worked fine; 'matio' is a great project. :)
Anyway, I only resolved the link errors, so it is very incomplete and unsafe.
Hi, I'm reading a MatLab file with Torch wrapper for matio and getting a Segfault at https://github.com/tbeu/matio/blob/master/src/mat5.c#L4331.
Program received signal SIGSEGV, Segmentation fault.
Read5 (mat=0x722ef0, matvar=<optimized out>) at mat5.c:4331
4331 fields[i]->internal->fp = mat;
(gdb) bt
#0 Read5 (mat=0x722ef0, matvar=<optimized out>) at mat5.c:4331
#1 0x00007fffa425d3a0 in Read5 (mat=0x722ef0, matvar=<optimized out>) at mat5.c:4332
#2 0x00007fffa425d3a0 in Read5 (mat=0x722ef0, matvar=<optimized out>) at mat5.c:4332
#3 0x00007fffa4268714 in Mat_VarRead (mat=0x722ef0, name=<optimized out>) at mat.c:1987
#4 0x000000000048ce29 in lj_vm_ffi_call ()
#5 0x000000000045d4e0 in lj_ccall_func ()
#6 0x000000000045f7b6 in lj_cf_ffi_meta___call ()
#7 0x000000000048ae6a in lj_BC_FUNCC ()
#8 0x000000000047a6dd in lua_pcall ()
#9 0x000000000041131f in pmain ()
#10 0x000000000048ae6a in lj_BC_FUNCC ()
#11 0x000000000047a757 in lua_cpcall ()
#12 0x000000000040f234 in main ()
The file I'm loading is a few hundred megabytes, but I could provide it if needed. Thanks!
Hi,
I have been trying to package this program for conda but all tests fail after I compile this on travis osx:
https://travis-ci.org/bioidiap/bob.conda/jobs/112988995
Do you have any idea what goes wrong?
You can see my conda recipe here:
https://github.com/bioidiap/bob.conda/tree/master/libmatio
I ran this on my mac and here is the log:
testsuite.txt
The return values of fwrite, H5Awrite and H5Dwrite are currently ignored most of the time. This is bad coding and was requested to be improved in https://sourceforge.net/p/matio/bugs/37.
I have installed matio 1.5.2 and found
libmatio.a
libmatio.la
libmatio.so
libmatio.so.2
libmatio.so.2.0.2
under /usr/local/lib/ directory. However, I cannot compile any program as the following errors appear:
/tmp/ccI8aVfw.o: In function `main':
testMatio.cpp:(.text+0x39): undefined reference to `Mat_Open'
testMatio.cpp:(.text+0xad): undefined reference to `Mat_VarRead'
testMatio.cpp:(.text+0x204): undefined reference to `Mat_VarRead'
testMatio.cpp:(.text+0x360): undefined reference to `Mat_VarRead'
testMatio.cpp:(.text+0x43f): undefined reference to `Mat_VarRead'
testMatio.cpp:(.text+0x519): undefined reference to `Mat_Close'
collect2: error: ld returned 1 exit status
My program is from the internet and it's as follow:
#include <iostream>
#include <matio.h>
#define tS(x) std::cout<<"\t"<<(#x)<<" == "<<(x)<<"\n"
int main(int argc, char **argv)
{
const char *fileName = argc==1?"./S.mat":argv[1] ;
mat_t *mat = Mat_Open(fileName,MAT_ACC_RDONLY);
if(mat)
{
std::cout<<"Otwarto plik do odczytu\n\tmat == "<<mat<<"\n" ;
matvar_t *matVar=0 ;
std::cout<<"Wypisujemy dane\n\n" ;
std::cout<<"x:\n" ;
matVar = Mat_VarRead(mat, (char*)"x") ;
if(matVar)
{
unsigned xSize = matVar->nbytes/matVar->data_size ;
const double *xData = static_cast<const double*>(matVar->data) ;
for(int i=0; i<xSize; ++i)
{
std::cout<<"\tx["<<i<<"] = "<<xData[i]<<"\n" ;
}
std::cout<<"\n" ;
for(int i=0; i<matVar->rank; ++i)
{
std::cout<<"\tdim["<<i<<"] == "<<matVar->dims[i]<<"\n" ;
}
}
std::cout<<"y:\n" ;
matVar = Mat_VarRead(mat, (char*)"y") ;
unsigned ySize = matVar->nbytes/matVar->data_size ;
const double *yData = static_cast<const double*>(matVar->data) ;
for(int i=0; i<ySize; ++i)
{
double d = yData[i] ;
std::cout<<"\ty["<<i<<"] = "<<d<<"\n" ;
}
std::cout<<"\n" ;
for(int i=0; i<matVar->rank; ++i)
{
std::cout<<"\tdim["<<i<<"] == "<<matVar->dims[i]<<"\n" ;
}
std::cout<<"minX, step:\n";
matVar = Mat_VarRead(mat, (char*)"minX") ;
const double *minXData = static_cast<const double*>(matVar->data) ;
std::cout<<"\tminX = "<<minXData[0]<<"\n" ;
std::cout<<"\n" ;
for(int i=0; i<matVar->rank; ++i)
{
std::cout<<"\tdim["<<i<<"] == "<<matVar->dims[i]<<"\n" ;
}
matVar = Mat_VarRead(mat, (char*)"step") ;
const double *stepData = static_cast<const double*>(matVar->data) ;
std::cout<<"\tstep = "<<stepData[0]<<"\n" ;
std::cout<<"\n" ;
for(int i=0; i<matVar->rank; ++i)
{
std::cout<<"\tdim["<<i<<"] == "<<matVar->dims[i]<<"\n" ;
}
Mat_Close(mat);
}
else
{
std::cout<<"Nie można otworzyć pliku\n" ;
return 1;
}
return 0;
}
I compiled it with
g++ -I/usr/include -L/usr/local/lib -lmatio -lz testMatio.c -o testMatio
and with this from the source directory:
g++ -I./src/ -L./src/.libs/ -lmatio -lz testMatio.cpp -o testMatio
all give the same error.
Can you please help me to fix this? Thank you.
The following tests are reported as failed even though they really pass:
2665-2680
2684-2754
2758-2828
2834-2904
2911-2980
It looks like the generated 'testsuite' script is adding an extra newline to the Matlab output,
causing the script to think the test failed.
Editing the generated testsuite with the following sed script fixes the problem and all the above tests pass.
sed -i '/^echo >>"\$at_stdout"; \$as_echo "PASSED"/s/^echo/echo -n/' testsuite
Maybe there is an issue with my version of autoconf (version 2.69 on a RHEL 7 system)?
the short description of the function Mat_GetVersion
in the file documentation/Mat_GetVersion.3 is the same as the short description for Mat_GetFilename
but should not be.
Hello together. I have an issue w/ getting HDF5 support to work and would be happy for some help in that regard.
As of the HDF5 install itself, I followed the regular instructions on their webpage, resulting in an installation in /opt:
taco@Tuxedo ~ $ ls /opt/HDF_Group/HDF5/1.10.1
bin include lib share
I ran configure like this (with various variants):
./configure --with-default-api-version=v110 --enable-mat73=yes --with-hdf5=/opt/HDF_Group/HDF5/1.10.1/
But I always end up like this:
MATIO Configuration Summary
==============================================================
C Compiler: gcc
CFLAGS: -g -O2
Shared Libraries: yes
Static Libraries: yes
default MAT version: MAT_FT_MAT5
Features --------------------------------------------
MAT v7.3 file support: no
Extended sparse support: yes
Packages --------------------------------------------
zlib: -lz
hdf5:
MATLAB: /usr/local/bin/matlab
What am I doing wrong here?
Cheers!
HDF5 is built from source. When configure matio, it reported can't found HDF5:
./configure --enable-mat73=yes --with-hdf5=/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18 --with-default-file-ver=7.3
...
checking for HDF5 software... no
...
MATIO Configuration Summary
==============================================================
C Compiler: gcc
CFLAGS: -g -O2
Shared Libraries: yes
Static Libraries: yes
default MAT version: MAT_FT_MAT73
Features --------------------------------------------
MAT v7.3 file support: no
Extended sparse support: yes
Packages --------------------------------------------
zlib: -lz
hdf5:
MATLAB:
config.log:
...
configure:15844: checking for HDF5 software
configure:15881: gcc -o conftest -I/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/include -g -O2 conftest.c -L/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/lib -lhdf5 -lz -lm >&5
/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/lib/libhdf5.a(H5PL.c.o): In function `H5PL_term_interface':
H5PL.c:(.text+0x32c): undefined reference to `dlclose'
/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/lib/libhdf5.a(H5PL.c.o): In function `H5PL_load':
H5PL.c:(.text+0x5e8): undefined reference to `dlsym'
H5PL.c:(.text+0x7ac): undefined reference to `dlopen'
H5PL.c:(.text+0x7bc): undefined reference to `dlsym'
H5PL.c:(.text+0x934): undefined reference to `dlclose'
H5PL.c:(.text+0x99c): undefined reference to `dlerror'
H5PL.c:(.text+0xca4): undefined reference to `dlclose'
/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/lib/libhdf5.a(H5Z.c.o): In function `H5Z_init_interface':
H5Z.c:(.text+0x958): undefined reference to `SZ_encoder_enabled'
/home/pi/matio/CMake-hdf5-1.8.18/HDF5-1.8.18-Linux/HDF_Group/HDF5/1.8.18/lib/libhdf5.a(H5Zszip.c.o): In function `H5Z_filter_szip':
H5Zszip.c:(.text+0x104): undefined reference to `SZ_BufftoBuffDecompress'
H5Zszip.c:(.text+0x19c): undefined reference to `SZ_BufftoBuffCompress'
collect2: error: ld returned 1 exit status
...
Fix is
change HDF5_LIBS in matio_hdf5.m4
From "-L${HDF5_DIR}/lib -lhdf5"
To "-L${HDF5_DIR}/lib -lhdf5 -ldl -lszip"
When using scatteredInterpolant in MatLab, a variable of type scatteredInterpolant is created. As far as i know it behaves like a struct, but this type cannot be read by matio. All I could figure out is that class_type is MAT_C_OPAQUE and the rest of matvar_t is empty.
I build the matio-1.5.9 which downloaded from sourceforge on unbuntu14.04. When i run make check,Write Mat version 4 and Write Uncompressed Mat version 5 failed at writing 2D double/single/int64/uint64 array.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.