GithubHelp home page GithubHelp logo

npy's Introduction

JetBrains Research Tests Status Maven Central

npy

npy allows to read and write files in NPY npy and [NPZ] npy formats on the JVM.

Installation

The latest version of npy is available on [Maven Central] maven-central. If you're using Gradle just add the following to your build.gradle:

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.jetbrains.bio:npy:0.3.5'
}

With Maven, specify the following in your pom.xml:

<dependency>
  <groupId>org.jetbrains.bio</groupId>
  <artifactId>npy</artifactId>
  <version>0.3.5</version>
</dependency>

The previous versions were published on Bintray. They can be downloaded from GitHub Releases.

Examples

NPY

val values = intArrayOf(1, 2, 3, 4, 5, 6)
val path = Paths.get("sample.npy")
NpyFile.write(path, values, shape = intArrayOf(2, 3))

println(NpyFile.read(path))
// => NpyArray{data=[1, 2, 3, 4, 5, 6], shape=[2, 3]}

NPZ

val values1 = intArrayOf(1, 2, 3, 4, 5, 6)
val values2 = booleanArrayOf(true, false)
val path = Paths.get("sample.npz")

NpzFile.write(path).use {
    it.write("xs", values1, shape = intArrayOf(2, 3))
    it.write("mask", values2)
}

NpzFile.read(path).use {
    println(it.introspect())
    // => [NpzEntry{name=xs, type=int, shape=[2, 3]},
    //     NpzEntry{name=mask, type=boolean, shape=[2]}]

    println("xs   = ${it["xs"]}")
    println("mask = ${it["mask"]}")
    // => xs   = NpyArray{data=[1, 2, 3, 4, 5, 6], shape=[2, 3]}
    //    mask = NpyArray{data=[true, false], shape=[2]}
}

Limitations

The implementation is rather minimal at the moment. Specifically it does not support the following types:

  • unsigned integral types (treated as signed),
  • bit field,
  • complex,
  • object,
  • Unicode
  • void*
  • intersections aka types for structured arrays.

Building from source

The build process is as simple as

$ ./gradlew jar

Note: don't use ./gradlew assemble, since it includes the signing of the artifacts and will fail if the correct credentials are not provided.

Testing

No extra configuration is required for running the tests from Gradle

$ ./gradlew test

However, some tests require Python and NumPy to run and will be skipped unless you have these.

npy's People

Contributors

dievsky avatar iromeo avatar olegs avatar superbobry avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

npy's Issues

NegativeArraySizeException for very large arrays

When trying to load a very large matrix (8312274035 elements) stored in npy format, I get a "NegativeArraySizeException".
It seems, that the operation to calculate the size being int32 limits the maximal number of elements (line 184 in NpyFile.class, int32(8312274035) = -277660557).

Maybe, there is a possibility to make this fit for large arrays?

Numpy cannot load created file.

import numpy as np
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
npz = np.load('/Users/oleg/Desktop/zinbra_peaks_200_1.0E-4.npz')
npz.files
Out[31]:
['H4K20me1',
 'H3K4me1',
 'H3K9ac',
 'H3K4me3',
 'H3K4me2',
 'H3K27ac',
 'Input_Sonicated',
 'CTCF',
 'H3K36me3',
 'H3K27me3']
In [32]:

npz['CTCF']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-32-56170b6f17ad> in <module>()
----> 1 npz['CTCF']

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/npyio.py in __getitem__(self, key)
    222                 return format.read_array(bytes,
    223                                          allow_pickle=self.allow_pickle,
--> 224                                          pickle_kwargs=self.pickle_kwargs)
    225             else:
    226                 return self.zip.read(key)

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
    621     version = read_magic(fp)
    622     _check_version(version)
--> 623     shape, fortran_order, dtype = _read_array_header(fp, version)
    624     if len(shape) == 0:
    625         count = 1

/Users/oleg/miniconda3/lib/python3.5/site-packages/numpy/lib/format.py in _read_array_header(fp, version)
    504             not numpy.all([isinstance(x, (int, long)) for x in d['shape']])):
    505         msg = "shape is not valid: %r"
--> 506         raise ValueError(msg % (d['shape'],))
    507     if not isinstance(d['fortran_order'], bool):
    508         msg = "fortran_order is not a valid bool: %r"

ValueError: shape is not valid: 550581

Migrate to Maven Central

JFrog is sunsetting Bintray, so we need to move.

We already have credentials and deploy permissions to org.jetbrains.bio, so all that remains is:

  • add Javadoc JAR to artifacts
  • implement artifact signing
  • provide sufficient metadata
  • configure Gradle to upload to OSSRH
  • configure a TeamCity build to upload to OSSRH with signing and deployment credentials

Impossible to use with Maven?

It looks like the library can't be used with Maven at the moment.

The Kotlin dependency is declared using Gradle-style wildcards (1.0.+), which Maven doesn't support. This results in Maven trying to find a version that is literally named 1.0.+.

What's even worse, this makes the downstream dependencies of npy (e.g. viktor) unusable with Maven.

Proposed solution: replace the wildcard dependency with a fixed one or a range.

Kotlin glitch

java.lang.IllegalAccessError: tried to access method kotlin.LazyKt.getValue(Lkotlin/Lazy;Ljava/lang/Object;Lkotlin/reflect/KProperty;)Ljava/lang/Object; from class org.jetbrains.bio.npy.NpyFile$Header
    at org.jetbrains.bio.npy.NpyFile$Header.getMeta(Npy.kt)
    at org.jetbrains.bio.npy.NpyFile$Header.allocate(Npy.kt:78)
    at org.jetbrains.bio.npy.NpyFile$Companion.allocate$npy_compileKotlin(Npy.kt:256)
    at org.jetbrains.bio.npy.NpzFile$Writer.write(Npz.kt:103)
    at org.jetbrains.bio.histones.GenomeCoverage.save(GenomeCoverage.kt:79)
    at org.jetbrains.bio.histones.GenomeCoverageTest.testPartialLoading(GenomeCoverageTest.kt:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
    at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
    at org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
    at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.