hubject / aws-s3-io

Additional I/O classes for AWS S3

AWS S3 IO Additions

This library contains additions to the AWS S3 SDK that help with I/O tasks.

Setup

Add the dependency com.hubject:aws-s3-io from Maven central to your project:

<dependecy>
    <groupId>com.hubject</groupId>
    <artifactId>aws-s3-io</groupId>
    <version>${hubject.aws-s3-io.version}</version>
</dependency>

Development

This project uses maven for dependency management and the build process. The typical commands work:

# compile and run unit tests
mvn clean test 

# install into local maven repository
mvn clean install

Usage

The manuals and examples for individual components are on separate pages:

• S3OutputStream - Write to S3 like to regular files using streams.

aws-s3-io's Issues

NoClassDefFoundError for BASE64Encoder with useChecksums = true

I am using Java 11 and getting the following exception if I use MD5KT/useChecksums=true when closing the S3OutputStream.kt:

java.lang.NoClassDefFoundError: sun/misc/BASE64Encoder
at com.hubject.aws.s3.io.Md5Kt.toBase64(md5.kt:42) ~[task/:na]
at com.hubject.aws.s3.io.S3OutputStream.close(S3OutputStream.kt:187)

The JDK 11 replacement for sun.misc.BASE64Encoder is java.util.Base64.Encoder

Reduce allocations in SimpleByteBufferPool

Currently, if SimpleByteBufferPool receives two pop() calls with small size, followed only by calls with larger size, the smaller ones will not be re-used and larger ones will be allocated for every call to pop() and then de-allocated when free()d. This is excatly the behaviour ByteBufferPools try to avoid.

To enhance this, i suggest the following solution:
Keep a running average in of the minSize parameter for pop() invocations. When a buffer is free()d and there are buffers in reserve which are smaller than the average, drop the smallest in reserve and add the free()d buffer to the reserve (as opposed to keeping the smaller one and dropping the larger).

@JvmOverloads on constructor?

I saw this:
https://stackoverflow.com/a/34518024/988346
Think it would be a good idea to add @jvmoverloads to S3OutputStream?

why use @JvmName("default")?

Doesn't this just make it harder to use from java code where default is a keyword?
b3bc64d