juliaio / fileio.jl Goto Github PK

Hello,

I understand JLD is distinguished from HDF5, which is great. I my own packages I rely on JLD to store specific types (e.g., a GaussianMixtures::GMM). I believe JLD already handles most of what I want, but sometimes I'd like to call a constructor-specific hook for loading or saving. How should I handle this in the FileIO infrastructure.

To be more concrete, suppose I have a type like:

type T1
  compact::Matrix
  derived::Vector{Matrix}
  function T1(c::Matrix)
    d = some_heavyish_precomputation(c)
    new(c, d)
  end
end

then I'd like to save only T1.c and upon loading call something like T1(load(stream)["t1.c"]). But it would be nice if this can be accomplished all from just loading a JLD-encoded file using FileIO magic.

How would I go about that?

---david

The tests of https://github.com/juliohm/GslibIO.jl were passing fine locally, I then did Pkg.checkout("FileIO") and the tests are now hanging.

Can you please help me understand what is happening?

Hi, what do you all think of this way of loading and registering file extensions?
https://github.com/JuliaIO/FileIO.jl/blob/master/src/core_prototype.jl
It's just a crude prototype by now:

# Example taken from MeshIO
#Registering a file format
#Mime(unique_identifier, file_ending, magic_number)
const ply_binary = Mime(:ply_binary, :ply, b"ply\nformat binary_little_endian 1.0\n")
const ply_ascii  = Mime(:ply_ascii,  :ply, b"ply\nformat ascii 1.0\n")

file = file"test_ascii.ply" #just wraps the path in the file type
obj = load(file) # tries to find mime and opens io stream and returns appropriate julia object
obj = load(file, GLNormalMesh) # tries to return a GLNormalMesh while loading
obj = GLNormalMesh(file) # just works if defined in the mesh package, as call(::Type{Any}, ::File) leads
# to millions of ambiguity warnings

this is heavily inspired by the approach taken in Images.jl ( I'm not sure who wrote this particular code, but probably @timholy can say something about this)

Best,
Simon

It seems we have Appveyor running on PRs, but it seems to consistently fail. We also need to add a badge for it in the README.

what do i have to do to add support for multi-page TIFs? i've tried both ImageMagick and QuartzImageIO and neither currently work. the former errors; and the latter quietly saves a 2D tiff. or perhaps there is a keyword option? thanks.

julia> using Images, ImageMagick, QuartzImageIO, ColorTypes, FileIO

julia> foo=Image(zeros(Gray{FixedPointNumbers.UFixed{UInt16,16}},(128,128,150)),
           spatialorder=["x", "y"], pixelspacing=[1, 1], timedim=3)
Gray Images.Image with:
  data: 128x128x150 Array{ColorTypes.Gray{FixedPointNumbers.UFixed{UInt16,16}},3}
  properties:
    timedim: 3
    spatialorder:  x y
    pixelspacing:  1 1

julia> ImageMagick.save(query("foo.tif"), foo)
ERROR: NoEncodeDelegateForThisImageFormat `MIFF' @ error/constitute.c/WriteImage/1167
 in error at /Applications/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
 in error at /Users/arthurb/.julia/v0.4/ImageMagick/src/libmagickwand.jl:153
 in writeimage at /Users/arthurb/.julia/v0.4/ImageMagick/src/libmagickwand.jl:250
 in save_ at /Users/arthurb/.julia/v0.4/ImageMagick/src/ImageMagick.jl:155
 in save at /Users/arthurb/.julia/v0.4/ImageMagick/src/ImageMagick.jl:65

julia> QuartzImageIO.save(query("foo.tif"), foo)

julia> load("foo.tif")
RGBA Images.Image with:
  data: 128x128 Array{ColorTypes.RGBA{FixedPointNumbers.UFixed{UInt8,8}},2}
  properties:
    imagedescription: <suppressed>
    spatialorder:  x y
    pixelspacing:  1 1

I wanted to add Object File formats, but ran into the problem that MachO can use several magic numbers, which doesn't currently seem supported. Here's what I tried:

# Object File Formats
add_format(format"ELF", UInt8['\177','E','L','F'], [".o",".a",".so",""])
add_loader(format"ELF", :ELF)
add_saver(format"ELF", :ELF)

macho_magics = [0xfeedface,0xfeedfacf,0xcafebabe]
append!(macho_magics,map(bswap,macho_magics))
add_format(format"MachO", map(x->reinterpret(UInt8, [x]),macho_magics), [".o",".a",".dylib",""])
add_loader(format"MachO", :MachO)
add_saver(format"MachO", :MachO)

add_format(format"ELF", UInt8['M','Z'], [".o",".a",".lib",".obj",".exe"])
add_loader(format"ELF", :COFF)
add_saver(format"ELF", :COFF)

I'm trying to implement a loader for an ASCII format known as Z2 and I can't seem to register the loader properly:

using FileIO

function FileIO.load(f::File{format"Z2"})
    open(f) do s
        println(readstring(s))
    end
end

add_format(format"Z2", (), ".#=Z2")
load("22\ Dec\ 2016_12-24-13_1.=#Z2")

yields

LoadError: FileIO.File{FileIO.DataFormat{:UNKNOWN}}("/Users/tamasnagy/Google Drive/notes/data/2016_12_22_osmoshock_50min_initial/22 Dec 2016_12-24-13_1.=#Z2") couldn't be recognized by FileIO.

while loading In[1], in expression starting on line 10

 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:UNKNOWN}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:75
 in load(::FileIO.File{FileIO.DataFormat{:UNKNOWN}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:73
 in #load#13(::Array{Any,1}, ::Function, ::String) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:45
 in load(::String) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:45

Any ideas on what I'm doing wrong here? I'm stumped.

In Meshes there are often binary and ASCII formats for each file type. There doesn't seem to be a way to resolve which one you would like easily. For example file"foo.ply creates a File{:ply}("foo.ply"), but the only dispatchable signatures are File{:binary_ply}, so I cannot use the m_str form.

What should be the behavior here?

Hello. I want to load an image whose format is jpg or png. But when I try:
julia> img=load("test.jpg")
The following warning shows up:
WARNING: FileIO.NotInstalledError(:ImageMagick,"")
in checked_import at C:\Users\XXX.julia\v0.4\FileIO\src\loadsave.jl:12
in load at C:\Users\XXX.julia\v0.4\FileIO\src\loadsave.jl:79
in load at C:\Users\XXX.julia\v0.4\FileIO\src\loadsave.jl:42
Library ImageMagick is not installed but can load format: FileIO.File{FileIO.DataFormat{:JPEG}}("test.jpg")
should we install ImageMagick for you? (y/n):

Now, independently from julia, I already installed the latest version of ImageMagick on my system at at C:\Program Files\ImageMagick-7.0.1-Q16. Why doesn't julia recognize this installed ImageMagick? If I type 'y' to the above prompt, along with ImageMagick.jl, all the ImageMagick library items are installed under my .julia\v0.4\ImageMagick\deps\ folder. I do not want to install another version of ImageMagick because it is a waste of space. How can I use the preinstalled version of ImageMagick library. Please help!

julia> using FileIO

julia> obj = load("M4L-prikooli.png")
RGB Images.Image with:
  data: 3000x3000 Array{ColorTypes.RGB{FixedPointNumbers.UFixed{UInt8,8}},2}
  properties:
    colorspace: sRGB
    spatialorder:  x y

julia> save("foo.png",obj)
ERROR: UndefVarError: save not defined

julia> Pkg.status("FileIO")
 - FileIO                        0.0.3

I guess I am missing something...? save() is mentioned in docs...

Say I have a custom plot type MyPlot. Lets also assume I already have defined a show method for the image/png mime type:

function Base.show(io::IO, m::MIME"image/png", v::MyPlot)

I would have assumed that save("filename.png", p) where p is a MyPlot would just work? Or how could that be integrated?

I guess in general, if a type has a show method for a given mime type, and that mime type corresponds directly to a given file extension, it should be straightforward to just save things to a file with the save function?

I would like to inform that I am working towards making Libz compliant with FileIO.

Apologies if this question is out of scope for the project, but is there currently a way to save a PNG mosaic image? That is, I have a 3D array with type:

Array{ColorTypes.Gray{FixedPointNumbers.UFixed{UInt8,8}},3}

and would like to save all dimensions as a PNG. Currently it only saves the first dimension to the file.

Many thanks for any help and keep up the great Julia work.

Right now you can't supply a URL for a filename; you have to download and then load. What do people think about having that be more transparent?

If folks like that: since I'm pretty net-naive, what would be the best thing to search for? Should we require that the string begins with "http://" or "https://"?

CC @alanedelman

Can we get a new release for FLAC.jl and WAV.jl? We've had releases on both of those packages with the new-and-improved FileIO support.

It might be better to use magic numbers to detect what kind of file you're trying to access. This may cause problems with internally-formatted text files (like CSV), but at that point perhaps the file suffix makes sense.

As discussed in JuliaImages/Images.jl@2762928#commitcomment-13530108 and https://github.com/timholy/Images.jl/pull/374/files#r40807839

Solutions:

• put Important packages in the Require
• list of recommended packages
• when it's detected that a package could load a file, but it's not installed, prompt the user to install it (maybe even offering to install it automatically)

I'm in favor of the last option, since it deals with the problem of only installing the minimal set of needed packages while still keeping usability up high.

I can prepare a PR when you all agree!

Best,
Simon

First, let me apologize for this kinda difficult to reproduce issue report but it depends on a interaction with a C library that I just fixed for this to work. The background is that explained in this thread

What happens is the following. Somehow the order by which I load an image is important.
This works well:

using FixedPointNumbers, Images, FileIO, GMT

julia> img1=load("lixa.JPG")
4×3 Array{RGB{N0f8},2}:
 RGB{N0f8}(0.282,0.263,0.239)  RGB{N0f8}(0.514,0.486,0.373)  RGB{N0f8}(0.89,0.843,0.694)
 RGB{N0f8}(0.216,0.176,0.137)  RGB{N0f8}(0.871,0.835,0.714)  RGB{N0f8}(0.278,0.227,0.09)
 RGB{N0f8}(0.333,0.267,0.204)  RGB{N0f8}(0.969,0.91,0.78)    RGB{N0f8}(0.298,0.239,0.11)
 RGB{N0f8}(0.596,0.502,0.408)  RGB{N0f8}(0.671,0.6,0.459)    RGB{N0f8}(0.239,0.18,0.067)

julia> I = gmt("read lixa.JPG -Ti"); imf8 = reinterpret(N0f8, I.image);

julia> img2 = colorview(RGB, imf8)
4×3 Array{RGB{N0f8},2}:
 RGB{N0f8}(0.282,0.263,0.239)  RGB{N0f8}(0.514,0.486,0.373)  RGB{N0f8}(0.89,0.843,0.694)
 RGB{N0f8}(0.216,0.176,0.137)  RGB{N0f8}(0.871,0.835,0.714)  RGB{N0f8}(0.278,0.227,0.09)
 RGB{N0f8}(0.333,0.267,0.204)  RGB{N0f8}(0.969,0.91,0.78)    RGB{N0f8}(0.298,0.239,0.11)
 RGB{N0f8}(0.596,0.502,0.408)  RGB{N0f8}(0.671,0.6,0.459)    RGB{N0f8}(0.239,0.18,0.067)

But if I just revert the order of the operations

using FixedPointNumbers, Images, FileIO, GMT

julia> I = gmt("read lixa.JPG -Ti"); imf8 = reinterpret(N0f8, I.image);

julia> img2 = colorview(RGB, imf8)
4×3 Array{RGB{N0f8},2}:
 RGB{N0f8}(0.282,0.263,0.239)  RGB{N0f8}(0.514,0.486,0.373)  RGB{N0f8}(0.89,0.843,0.694)
 RGB{N0f8}(0.216,0.176,0.137)  RGB{N0f8}(0.871,0.835,0.714)  RGB{N0f8}(0.278,0.227,0.09)
 RGB{N0f8}(0.333,0.267,0.204)  RGB{N0f8}(0.969,0.91,0.78)    RGB{N0f8}(0.298,0.239,0.11)
 RGB{N0f8}(0.596,0.502,0.408)  RGB{N0f8}(0.671,0.6,0.459)    RGB{N0f8}(0.239,0.18,0.067)

julia> img1=load("lixa.JPG")
Error encountered while loading "v:\lixa.JPG".
Fatal error:
ERROR: RegistryKeyLookupFailed `CoderModulesPath' @ error/module.c/GetMagickModulePath/666
 in error(::ImageMagick.MagickWand) at c:\j\.julia\v0.5\ImageMagick\src\libmagickwand.jl:198
 in readimage(::ImageMagick.MagickWand, ::String) at c:\j\.julia\v0.5\ImageMagick\src\libmagickwand.jl:277
 in #load_#20(::Type{T}, ::String, ::Void, ::Function, ::String) at c:\j\.julia\v0.5\ImageMagick\src\ImageMagick.jl:72
 in #load#13(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JPEG}}) at c:\j\.julia\v0.5\ImageMagick\src\ImageMagick.jl:51
 in anonymous at .\<missing>:?
 in eval(::Module, ::Any) at .\boot.jl:234
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:JPEG}}) at c:\j\.julia\v0.5\FileIO\src\loadsave.jl:87
 in load(::FileIO.File{FileIO.DataFormat{:JPEG}}) at c:\j\.julia\v0.5\FileIO\src\loadsave.jl:75
 in #load#13(::Array{Any,1}, ::Function, ::String) at c:\j\.julia\v0.5\FileIO\src\loadsave.jl:45
 in load(::String) at c:\j\.julia\v0.5\FileIO\src\loadsave.jl:45

I'm attaching the (tiny tiny) test image for the case you find it useful

Might be worth taking a look at CSVReaders API to see whether there is some inspiration there.

I'd be happy to integrate ExcelReaders into this framework at some point. You might take a quick look at it to get a sense whether the APIs will match or not.

Recent FileIO changes have broken Gadfly tests:

WARNING: FileIO.LoaderError("RData","load not defined")
 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:RData}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:83
 in load(::FileIO.File{FileIO.DataFormat{:RData}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:73
 in #load#13(::Array{Any,1}, ::Function, ::String) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:45
 in dataset(::String, ::String) at /Users/tamasnagy/.julia/v0.5/RDatasets/src/dataset.jl:6
 in include_from_node1(::String) at ./loading.jl:488
 in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in evalfile(::String, ::Array{String,1}) at ./loading.jl:504
 in run_tests(::String) at /Users/tamasnagy/.julia/v0.5/Gadfly/test/runtests.jl:142
 in include_from_node1(::String) at ./loading.jl:488
 in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in process_options(::Base.JLOptions) at ./client.jl:262
 in _start() at ./client.jl:318
 in _start() at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
FAILED!
ERROR: LoadError: LoadError: RData load error: load not defined
  Will try next loader.

 in #load#17(::Array{Any,1}, ::Function, ::FileIO.File{FileIO.DataFormat{:RData}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:83
 in load(::FileIO.File{FileIO.DataFormat{:RData}}) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:73
 in #load#13(::Array{Any,1}, ::Function, ::String) at /Users/tamasnagy/.julia/v0.5/FileIO/src/loadsave.jl:45
 in dataset(::String, ::String) at /Users/tamasnagy/.julia/v0.5/RDatasets/src/dataset.jl:6
 in include_from_node1(::String) at ./loading.jl:488
 in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in evalfile(::String, ::Array{String,1}) at ./loading.jl:504
 in run_tests(::String) at /Users/tamasnagy/.julia/v0.5/Gadfly/test/runtests.jl:142
 in include_from_node1(::String) at ./loading.jl:488
 in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in process_options(::Base.JLOptions) at ./client.jl:262
 in _start() at ./client.jl:318
 in _start() at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
while loading /Users/tamasnagy/.julia/v0.5/Gadfly/test/points.jl, in expression starting on line 4
while loading /Users/tamasnagy/.julia/v0.5/Gadfly/test/runtests.jl, in expression starting on line 187

see https://travis-ci.org/dcjones/Gadfly.jl/jobs/164972396. RData hasn't changed recently so I doubt it is a culprit.

I previously posted this on Images.jl but i was able to narrow it down to FileIO.jl. I am having trouble to find the source of this bug since my computer hangs when running this, but I have been able to narrow it down to this package, so any help on what logs I can submit here would be helpful.

When I run the following on Julia Version 0.5.0-rc1+0

julia> using FileIO

I get almost 100% (of 32GB) memory usage and then my computer basically freezes as it starts eating up swap. I am able to pkill julia, but I'm not sure on where I can find an appropriate log. I'm on the latest tagged version of FileIO.jl (0.1.0). I tried to do a Pkg.rm("FileIO"); Pkg.add("FileIO") but it didn't help. I don't seem to have problems with other packages.

I'm running Fedora 23.

seems frozen, @SimonDanisch please look into what's going on with your account when you can

In the Jupyter notebook, a cell with

foo = "bar"

is fine, but

using FileIO
foo = "bar"

gives

MethodError: `start` has no method matching start(::Symbol)

 in writemime at /Users/rrock/.julia/v0.4/FileIO/src/FileIO.jl:105
 in base64encode at base64.jl:160
 in display_dict at /Users/rrock/.julia/v0.4/IJulia/src/execute_request.jl:32

I first saw this here at the REPL, because my startup file includes TerminalExtensions.jl:

❯ julia --startup-file=no
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.4.0-rc3 (2015-09-27 20:34 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-apple-darwin15.0.0

julia> using FileIO

julia> foo = "bar"
"bar"

julia> using TerminalExtensions

julia> foo = "bar"
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
WARNING: Base.Base64Pipe is deprecated, use Base.Base64.Base64EncodePipe instead.
1mERROR: MethodError: `start` has no method matching start(::Symbol)
 in writemime at /Users/rrock/.julia/v0.4/FileIO/src/FileIO.jl:105
 in display at /Users/rrock/.julia/v0.4/TerminalExtensions/src/TerminalExtensions.jl:169
 in display at /Users/rrock/.julia/v0.4/TerminalExtensions/src/TerminalExtensions.jl:179
 in display at multimedia.jl:151
 in print_response at REPL.jl:135
 in print_response at REPL.jl:122
 in anonymous at REPL.jl:625
 in anonymous at /Users/rrock/.julia/v0.4/TerminalExtensions/src/TerminalExtensions.jl:227
 in run_interface at ./LineEdit.jl:1610
 in run_frontend at ./REPL.jl:864
 in run_repl at ./REPL.jl:168
 in _start at ./client.jl:453

When running the tests with 0.3 or 0.4 from the shell (julia ./test/runtests.jl), I get the following error. Pkg.test("FileIO") from the REPL works fine however. I've been digging at this one for a while and I don't understand this behavior, so I am looking for an explanation. The error is identical on 0.4 and 0.3 as well.

steve:~/.julia/v0.3/FileIO(master)$ julia3 ./test/runtests.jl 
DataFormat
17 facts verified.
streams
5 facts verified.
query
12 facts verified.
Load
  Error   :: (line:-1)
    load(joinpath("files","file1.pbm")) --> PBMText
no file for check of magic bytes and multiple extensions possible: [:PBMText,:PBMBinary]
 in error at error.jl:21
 in query at /home/steve/.julia/v0.3/FileIO/src/query.jl:260
 in load at /home/steve/.julia/v0.3/FileIO/src/FileIO.jl:48
 in anonymous at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:218
 in do_fact at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:277
 in anonymous at /home/steve/.julia/v0.3/FileIO/test/loadsave.jl:218
 in facts at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:391
 in anonymous at no file:18
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
  Error   :: (line:-1)
    load(joinpath("files","file2.pbm")) --> PBMBinary
no file for check of magic bytes and multiple extensions possible: [:PBMText,:PBMBinary]
 in error at error.jl:21
 in query at /home/steve/.julia/v0.3/FileIO/src/query.jl:260
 in load at /home/steve/.julia/v0.3/FileIO/src/FileIO.jl:48
 in anonymous at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:218
 in do_fact at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:277
 in anonymous at /home/steve/.julia/v0.3/FileIO/test/loadsave.jl:218
 in facts at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:391
 in anonymous at no file:18
 in include at ./boot.jl:245
 in include_from_node1 at ./loading.jl:128
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
Out of 6 total facts:
  Verified: 4
  Errored:  2
Save
12 facts verified.
ERROR: FactCheck finished with 2 non-successful tests.
 in error at error.jl:21
 in exitstatus at /home/steve/.julia/v0.3/FactCheck/src/FactCheck.jl:500
 in include at ./boot.jl:245
 in include_from_node1 at loading.jl:128
 in process_options at ./client.jl:285
 in _start at ./client.jl:354
while loading /home/steve/.julia/v0.3/FileIO/test/runtests.jl, in expression starting on line 8

As compressed files come as an e.g. zip, which is its own data format, but contain some other file, it might make sense to give them special treatment.
I tried to integrate NPZ, which is basically a compressed NPY file, so it first has the magic bytes of a zip file, but you wouldn't really like to get a zip filereader returned but rather the uncompressed python array.
So should we deeply integrate all the compression libraries, uncompress files first and then do the "real" loading?
CC: @fhs, @hustf

Best,
Simon

Right now this happens:

f = "test.nrrd" #empty file
load(f) 
#ERROR: No load function defined for format FileIO.DataFormat{:UNKNOWN} with filename test.nrrd
save(f) 
#ERROR: No save function defined for format FileIO.DataFormat{:UNKNOWN} with filename test.nrrd

For load we should probably fall back to the file ending or have something like DataFormat{:NRRD_CORRUPTED}.
For save it should probably just work and overwrite it. Or do we want to warn the user and make them use e.g. save! for overwriting?!

Both .avi and .wav files use RIFF as their magic bytes, followed by 4 bytes of size information and then WAVE or AVI.

What's the general strategy for nested file types like this? Should there be a format"RIFF" loader that then dispatches to separate format"WAV" and format"AVI" loaders? Another option would be to fall back to file type if the magic bytes conflict, but that doesn't help you e.g. if you have a generic container type that can have different contained file types.

I'm trying to go through and tag some packages I've been working on and I'm currently depending on FileIO master to get #69. If there's nothing specific holding up a release it would improve my life if you could tag a new version with that fix. :) Thanks!

I tried to find as much extensions and magic bytes as possible for ImageMagick, but am still not there.
As I dislike the idea to just let ImageMagick load everything without extension, I would suggest I add all the formats I could determine and the rest will get added on demand.
Like this, the work will be spread over a bigger amount of time, but we will still be able to read the most important formats.

How/where would I document the keyword arguments of some specific loader, ideally using Julia's help system? Just adding a doc-string to each load method doesn't work, as that would make for a super long concatenated doc-string. Maybe add it to the format-string somehow?

I read some tif files, but there result has some artifacts.

notice the grid like things are artifacts. the values were changed after reading. The wired part is that only part of them are incorrect! And for multi-page tif, only the first page has this artifact.

code:

using FileIO
using ImageView
image = load("test.tif")
view(image)

with this simple code, you have to zoom in the error part and look at it carefully.

test data:

test.tif.zip

my versioninfo:

ubuntu 14.04

versioninfo()
Julia Version 0.4.6
Commit 2e358ce (2016-06-19 17:16 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

ImageMagick.libversion
v"6.7.7"

FactCheck.jl is a very nice package that would help us write a simple and efficient test suite. On the other hand, it would be a dependency for FileIO.jl. What do you think about using it?

Maybe this organization should be announced on mailing lists like julia-dev or julia-users (I haven't found nothing on my searches). This would bring more people to the project, new ideas and speed things up.

I first thought about creating this sort of package because of the save method collisions in DataFrames and HDF5 (JuliaIO/HDF5.jl#101). The words load and save have a higher-level connotation to me than read/write, and packages seem to be already using that sort of terminology. I'm not sure if there's a particularly compelling reason to use one or the other, but I wanted to open the discussion.

We have several packages that might handle a specific type of file (and will return an object based on those file's contents). Does the presence of the centralized registry mean that only one package can "register" a dependency for that type of file for use in FileIO?

To use an existing example, it looks like PNG requires ImageMagick. But if I'm working on my own package that uses PNGs, and don't use ImageMagick myself in my package, does this mean that if I want to use FileIO for loading PNGs, I'm stuck with requiring my users to download (if not use) ImageMagick?

Sorry for the basic questions. My use case is with graph and matrix file formats, and there are many packages out there that use them in completely different ways.

We need to document how to extend FileIO to new filetypes.

IIUC load and save are generally one-off operations, i.e. load("somefile.wav") loads the full wave file into memory.

Sometimes though, you have huge files that you might not want to load into memory all at once. In my LibSndFile package I also export loadstream and savestream functions that instead return special stream types that you can read/write chunks of audio from.

@jongwook's MP3 package also has the same functionality, but it's not clear who should own those functions. It seems like it would make sense to have them in FileIO as they use all the same query tooling. In fact, we implement load and save as wrappers around the streaming versions.

Do these functions seem like they could fit into the FileIO architecture? What are the video packages currently doing for this sort of thing?

Consider the simplest saver:

function save(file::File{format"GSLIB"}, A::AbstractArray)
  open(file, "w") do f
    writedlm(f, A[:])
  end
end

It doesn't work because writedlm does not have a method for this package's' type. Do you consider adding a method for it?

Hi @SimonDanisch and @timholy,

In general terms, what does it mean to make Libz compliant with FileIO? That's my next goal.
Libz can read from a GZ and write to another GZ. FileIO just detects the GZ format and dispatches it to Libz, which I believe I've already done with add_format(format"GZIP", [0x1f, 0x8b], ".gz", [:Libz]).

Is there any more compliance to be done between the two?

Cheers, HP

I was following the README and wrote a simple saver:

function save{T<:AbstractFloat}(fname::File{format"GSLIB"}, A::AbstractArray{T,3})
  nx, ny, nz = size(A)

  open(fname, "w") do f
    # write header with grid dimensions
    write(f, @sprintf("%i %i %i\n1\nreal\n", nx, ny, nz))

    # flatten array and write to disk
    writedlm(f, A[:])
  end
end

There is no way to use it as save("foo.gslib", ones(3,3,3))? What is my misunderstanding here?

For saving some file formats like HTML, markdown, and images, it may make sense to fallback to the display hierarchy with MIME types for writing. This would, for example, allow one to save any plot with a simple write(File("plot.png"), plt) without any new code on the part of the plotting package. It'd be essential to pass through keyword arguments to allow optional formatting specifications (but that's another can of worms).

This would also mean that string IO would simply fall back to print with any .txt file.

If I add the following:

diff --git a/test/query.jl b/test/query.jl
index 3ab8ce8..449caff 100644
--- a/test/query.jl
+++ b/test/query.jl
@@ -37,6 +37,9 @@ try
         @fact FileIO.ext2sym[".jnk"]  --> :JUNK
         @fact FileIO.ext2sym[".junk"] --> :JUNK
         @fact FileIO.magic_list --> [Pair((0x4a,0x55,0x4e,0x4b),:JUNK)]
+
+
+        add_format(format"FOO", UInt8[], ".foo")
     end

     facts("query") do

I get:

ERROR: LoadError: LoadError: magic bytes () are already registered
 in add_format at /home/steve/.julia/v0.4/FileIO/src/query.jl:41
 in anonymous at /home/steve/.julia/v0.4/FileIO/test/query.jl:80
 in facts at /home/steve/.julia/v0.4/FactCheck/src/FactCheck.jl:391
 in anonymous at no file:45
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:200
 in include at ./boot.jl:254
 in include_from_node1 at ./loading.jl:200
 in process_options at ./client.jl:308
 in _start at ./client.jl:411
while loading /home/steve/.julia/v0.4/FileIO/test/query.jl, in expression starting on line 14
while loading /home/steve/.julia/v0.4/FileIO/test/runtests.jl, in expression starting on line 4

cc: @timholy

I'm not sure if this is a bug in FileIO, or in the underlying Netpbm (or even a bug at all - I only noticed it because of a misplaced parenthesis). Anyway, one can induce an infinite recursion with the following code:

using Images
save("test.pgm")

It seems that in the absence of any data to save, the call at loadsave.jl line 92 is recursing infinitely.

You might want to look at ZipFile.jl to see how you might include this in the overall package. ZipFile is nice because it will read any text file format, compressed or not, and perform necessary uncompression on the fly.

I can't seem to find any support for SVG format here. Has anyone thought about this? I'm happy to pitch in to get it working

It's hard to know how much of this is expected noise and how much I should be worried about:

julia> versioninfo()
Julia Version 0.4.3-pre+6
Commit adffe19 (2015-12-11 00:38 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin14.5.0)
  CPU: Intel(R) Core(TM) M-5Y71 CPU @ 1.20GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

julia> Pkg.test("FileIO")
INFO: Computing test dependencies for FileIO...
INFO: No packages to install, update or remove
INFO: Testing FileIO
INFO: Recompiling stale cache file /Users/stefan/.julia/lib/v0.4/FileIO.ji for module FileIO.
FileIO
  > OS
  > DataFormat
  > streams
  > query
  > multiple libs
WARNING: ErrorException("")
 in load at /Users/stefan/.julia/v0.4/FileIO/test/query.jl:38
 in load at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:77
 in load at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:42
 in anonymous at /Users/stefan/.julia/v0.4/FileIO/test/query.jl:244
 in context at /Users/stefan/.julia/v0.4/FactCheck/src/FactCheck.jl:474
 [inlined code] from /Users/stefan/.julia/v0.4/FileIO/test/query.jl:227
 in anonymous at no file:0
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in anonymous at /Users/stefan/.julia/v0.4/FileIO/test/runtests.jl:4
 in facts at /Users/stefan/.julia/v0.4/FactCheck/src/FactCheck.jl:448
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in process_options at ./client.jl:280
 in _start at ./client.jl:378
Trying next loading library! Please report this issue on Github
  > STL detection
  > PLY detection
  > Multiple Magic bytes
  > Load
  > Save
WARNING: could not import FileIO.Streamm into AmbigExt
  > Ambiguous extension
  > Not installed
WARNING: FileIO.NotInstalledError(:NotInstalled,"")
 in checked_import at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:12
 in save at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:91
 in save at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:51
 in anonymous at task.jl:447
Library NotInstalled is not installed but can load format: FileIO.File{FileIO.DataFormat{:NotInstalled}}("test.not_installed")
should we install NotInstalled for you? (y/n):
INFO: Start installing NotInstalled...
WARNING: FileIO.NotInstalledError(:NotInstalled,"")
 in checked_import at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:12
 in save at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:91
 in save at /Users/stefan/.julia/v0.4/FileIO/src/loadsave.jl:51
 in anonymous at task.jl:447
Library NotInstalled is not installed but can load format: FileIO.File{FileIO.DataFormat{:NotInstalled}}("test.not_installed")
should we install NotInstalled for you? (y/n):
invalid is not a valid choice. Try typing y or n
should we install NotInstalled for you? (y/n):
INFO: Not installing NotInstalled
90 facts verified.
INFO: FileIO tests passed
INFO: No packages to install, update or remove

If this is all expected, perhaps there should be messages indicating that?

The magic number for PCX format is set as UInt8[0x0a,0x05,0x01,0x01] in registry.jl. The fourth element represents the number of bits per pixel, so that load(filename) can fail for images that will correctly load with load(File(format"PCX", filename)).
The attached 256-colour PCX file (zipped, so that it can be attached), with 0x08 as the fourth byte, is a simple example.