kotlin / kotlindl

High-level Deep Learning Framework written in Kotlin and inspired by Keras

KotlinDL: High-level Deep Learning API in Kotlin

KotlinDL is a high-level Deep Learning API written in Kotlin and inspired by Keras. Under the hood, it uses TensorFlow Java API and ONNX Runtime API for Java. KotlinDL offers simple APIs for training deep learning models from scratch, importing existing Keras and ONNX models for inference, and leveraging transfer learning for tailoring existing pre-trained models to your tasks.

This project aims to make Deep Learning easier for JVM and Android developers and simplify deploying deep learning models in production environments.

Here's an example of what a classic convolutional neural network LeNet would look like in KotlinDL:

private const val EPOCHS = 3
private const val TRAINING_BATCH_SIZE = 1000
private const val NUM_CHANNELS = 1L
private const val IMAGE_SIZE = 28L
private const val SEED = 12L
private const val TEST_BATCH_SIZE = 1000

private val lenet5Classic = Sequential.of(
    Input(
        IMAGE_SIZE,
        IMAGE_SIZE,
        NUM_CHANNELS
    ),
    Conv2D(
        filters = 6,
        kernelSize = intArrayOf(5, 5),
        strides = intArrayOf(1, 1, 1, 1),
        activation = Activations.Tanh,
        kernelInitializer = GlorotNormal(SEED),
        biasInitializer = Zeros(),
        padding = ConvPadding.SAME
    ),
    AvgPool2D(
        poolSize = intArrayOf(1, 2, 2, 1),
        strides = intArrayOf(1, 2, 2, 1),
        padding = ConvPadding.VALID
    ),
    Conv2D(
        filters = 16,
        kernelSize = intArrayOf(5, 5),
        strides = intArrayOf(1, 1, 1, 1),
        activation = Activations.Tanh,
        kernelInitializer = GlorotNormal(SEED),
        biasInitializer = Zeros(),
        padding = ConvPadding.SAME
    ),
    AvgPool2D(
        poolSize = intArrayOf(1, 2, 2, 1),
        strides = intArrayOf(1, 2, 2, 1),
        padding = ConvPadding.VALID
    ),
    Flatten(), // 3136
    Dense(
        outputSize = 120,
        activation = Activations.Tanh,
        kernelInitializer = GlorotNormal(SEED),
        biasInitializer = Constant(0.1f)
    ),
    Dense(
        outputSize = 84,
        activation = Activations.Tanh,
        kernelInitializer = GlorotNormal(SEED),
        biasInitializer = Constant(0.1f)
    ),
    Dense(
        outputSize = 10,
        activation = Activations.Linear,
        kernelInitializer = GlorotNormal(SEED),
        biasInitializer = Constant(0.1f)
    )
)


fun main() {
    val (train, test) = mnist()
    
    lenet5Classic.use {
        it.compile(
            optimizer = Adam(clipGradient = ClipGradientByValue(0.1f)),
            loss = Losses.SOFT_MAX_CROSS_ENTROPY_WITH_LOGITS,
            metric = Metrics.ACCURACY
        )
    
        it.logSummary()
    
        it.fit(dataset = train, epochs = EPOCHS, batchSize = TRAINING_BATCH_SIZE)
    
        val accuracy = it.evaluate(dataset = test, batchSize = TEST_BATCH_SIZE).metrics[Metrics.ACCURACY]
    
        println("Accuracy: $accuracy")
    }
}

Table of Contents

• Library Structure
• How to configure KotlinDL in your project
  • Working with KotlinDL in Android projects
  • Working with KotlinDL in Jupyter Notebook
• KotlinDL, ONNX Runtime, Android, and JDK versions
• Documentation
• Examples and tutorials
• Running KotlinDL on GPU
• Logging
• Fat Jar issue
• Limitations
• Contributing
• Reporting issues/Support
• Code of Conduct
• License

Library Structure

KotlinDL consists of several modules:

• kotlin-deeplearning-api api interfaces and classes
• kotlin-deeplearning-impl implementation classes and utilities
• kotlin-deeplearning-onnx inference with ONNX Runtime
• kotlin-deeplearning-tensorflow learning and inference with TensorFlow
• kotlin-deeplearning-visualization visualization utilities
• kotlin-deeplearning-dataset dataset classes

Modules kotlin-deeplearning-tensorflow and kotlin-deeplearning-dataset are only available for desktop JVM, while other artifacts could also be used on Android.

How to configure KotlinDL in your project

To use KotlinDL in your project, ensure that mavenCentral is added to the repositories list:

repositories {
    mavenCentral()
}

Then add the necessary dependencies to your build.gradle file.

To start with creating simple neural networks or downloading pre-trained models, just add the following dependency:

// build.gradle
dependencies {
    implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-tensorflow:[KOTLIN-DL-VERSION]'
}

// build.gradle.kts
dependencies {
    implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-tensorflow:[KOTLIN-DL-VERSION]")
}

Use kotlin-deeplearning-onnx module for inference with ONNX Runtime:

// build.gradle
dependencies {
    implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]'
}

// build.gradle.kts
dependencies {
  implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]")
}

To use the full power of KotlinDL in your project for JVM, add the following dependencies to your build.gradle file:

// build.gradle
dependencies {
    implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-tensorflow:[KOTLIN-DL-VERSION]'
    implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]'
    implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-visualization:[KOTLIN-DL-VERSION]'
}

// build.gradle.kts
dependencies {
  implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-tensorflow:[KOTLIN-DL-VERSION]")
  implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]")
  implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-visualization:[KOTLIN-DL-VERSION]")
}

The latest stable KotlinDL version is 0.5.2, latest unstable version is 0.6.0-alpha-1.

For more details, as well as for pom.xml and build.gradle.kts examples, please refer to the Quick Start Guide.

Working with KotlinDL in Jupyter Notebook

You can work with KotlinDL interactively in Jupyter Notebook with the Kotlin kernel. To do so, add the required dependencies in your notebook:

@file:DependsOn("org.jetbrains.kotlinx:kotlin-deeplearning-tensorflow:[KOTLIN-DL-VERSION]")

For more details on installing Jupyter Notebook and adding the Kotlin kernel, check out the Quick Start Guide.

Working with KotlinDL in Android projects

KotlinDL supports an inference of ONNX models on the Android platform. To use KotlinDL in your Android project, add the following dependency to your build.gradle file:

// build.gradle
implementation 'org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]'

// build.gradle.kts
implementation ("org.jetbrains.kotlinx:kotlin-deeplearning-onnx:[KOTLIN-DL-VERSION]")

For more details, please refer to the Quick Start Guide.

KotlinDL, ONNX Runtime, Android, and JDK versions

This table shows the mapping between KotlinDL, TensorFlow, ONNX Runtime, Compile SDK for Android and minimum supported Java versions.

KotlinDL Version	Minimum Java Version	ONNX Runtime Version	TensorFlow Version	Android: Compile SDK Version
0.1.*	8		1.15
0.2.0	8		1.15
0.3.0	8	1.8.1	1.15
0.4.0	8	1.11.0	1.15
0.5.0-0.5.1	11	1.12.1	1.15	31
0.5.2	11	1.14.0	1.15	31
0.6.*	11	1.16.0	1.15	31

Documentation

• Presentations and videos:
  • Deep Learning with KotlinDL (Zinoviev Alexey at Huawei Developer Group HDG UK 2021, slides)
  • Introduction to Deep Learning with KotlinDL (Zinoviev Alexey at Kotlin Budapest User Group 2021, slides)
• Change log for KotlinDL
• Full KotlinDL API reference

Examples and tutorials

You do not need prior experience with Deep Learning to use KotlinDL.

We are working on including extensive documentation to help you get started. At this point, please feel free to check out the following tutorials we have prepared:

• Quick Start Guide
• Creating your first neural network
• Importing a Keras model
• Transfer learning
• Transfer learning with Functional API
• Running inference with ONNX models on JVM
• Running inference with ONNX models on Android

For more inspiration, take a look at the code examples in this repository and Sample Android App.

Running KotlinDL on GPU

To enable the training and inference on a GPU, please read this TensorFlow GPU Support page and install the CUDA framework to allow calculations on a GPU device.

Note that only NVIDIA devices are supported.

You will also need to add the following dependencies in your project if you wish to leverage a GPU:

// build.gradle
implementation 'org.tensorflow:libtensorflow:1.15.0'
implementation 'org.tensorflow:libtensorflow_jni_gpu:1.15.0'

// build.gradle.kts
implementation ("org.tensorflow:libtensorflow:1.15.0")
implementation ("org.tensorflow:libtensorflow_jni_gpu:1.15.0")

On Windows, the following distributions are required:

• CUDA cuda_10.0.130_411.31_win10
• cudnn-7.6.3
• C++ redistributable parts

For inference of ONNX models on a CUDA device, you will also need to add the following dependencies to your project:

// build.gradle
api 'com.microsoft.onnxruntime:onnxruntime_gpu:1.16.0'

// build.gradle.kts
api("com.microsoft.onnxruntime:onnxruntime_gpu:1.16.0")

To find more info about ONNXRuntime and CUDA version compatibility, please refer to the ONNXRuntime CUDA Execution Provider page.

Logging

By default, the API module uses the kotlin-logging library to organize the logging process separately from the specific logger implementation.

You could use any widely known JVM logging library with a Simple Logging Facade for Java (SLF4J) implementation such as Logback or Log4j/Log4j2.

You will also need to add the following dependencies and configuration file log4j2.xml to the src/resource folder in your project if you wish to use log4j2:

// build.gradle
implementation 'org.apache.logging.log4j:log4j-api:2.17.2'
implementation 'org.apache.logging.log4j:log4j-core:2.17.2'
implementation 'org.apache.logging.log4j:log4j-slf4j-impl:2.17.2'

// build.gradle.kts
implementation("org.apache.logging.log4j:log4j-api:2.17.2")
implementation("org.apache.logging.log4j:log4j-core:2.17.2")
implementation("org.apache.logging.log4j:log4j-slf4j-impl:2.17.2")

<Configuration status="WARN">
    <Appenders>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n"/>
        </Console>
    </Appenders>

    <Loggers>
        <Root level="debug">
            <AppenderRef ref="STDOUT" level="DEBUG"/>
        </Root>
        <Logger name="io.jhdf" level="off" additivity="true">
            <appender-ref ref="STDOUT" />
        </Logger>
    </Loggers>
</Configuration>

If you wish to use Logback, include the following dependency and configuration file logback.xml to src/resource folder in your project

// build.gradle
implementation 'ch.qos.logback:logback-classic:1.4.5'

// build.gradle.kts
implementation("ch.qos.logback:logback-classic:1.4.5")

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

These configuration files can be found in the examples module.

Fat Jar issue

There is a known Stack Overflow question and TensorFlow issue with Fat Jar creation and execution on Amazon EC2 instances.

java.lang.UnsatisfiedLinkError: /tmp/tensorflow_native_libraries-1562914806051-0/libtensorflow_jni.so: libtensorflow_framework.so.1: cannot open shared object file: No such file or directory

Despite the fact that the bug describing this problem was closed in the release of TensorFlow 1.14, it was not fully fixed and required an additional line in the build script.

One simple solution is to add a TensorFlow version specification to the Jar's Manifest. Below is an example of a Gradle build task for Fat Jar creation.

// build.gradle

task fatJar(type: Jar) {
    manifest {
        attributes 'Implementation-Version': '1.15'
    }
    classifier = 'all'
    from { configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) } }
    with jar
}

// build.gradle.kts

plugins {
    kotlin("jvm") version "1.5.31"
    id("com.github.johnrengelman.shadow") version "7.0.0"
}

tasks{
    shadowJar {
        manifest {
            attributes(Pair("Main-Class", "MainKt"))
            attributes(Pair("Implementation-Version", "1.15"))
        }
    }
}

Limitations

Currently, only a limited set of deep learning architectures are supported. Here's the list of available layers:

• Core layers:
  • Input, Dense, Flatten, Reshape, Dropout, BatchNorm.
• Convolutional layers:
  • Conv1D, Conv2D, Conv3D;
  • Conv1DTranspose, Conv2DTranspose, Conv3DTranspose;
  • DepthwiseConv2D;
  • SeparableConv2D.
• Pooling layers:
  • MaxPool1D, MaxPool2D, MaxPooling3D;
  • AvgPool1D, AvgPool2D, AvgPool3D;
  • GlobalMaxPool1D, GlobalMaxPool2D, GlobalMaxPool3D;
  • GlobalAvgPool1D, GlobalAvgPool2D, GlobalAvgPool3D.
• Merge layers:
  • Add, Subtract, Multiply;
  • Average, Maximum, Minimum;
  • Dot;
  • Concatenate.
• Activation layers:
  • ELU, LeakyReLU, PReLU, ReLU, Softmax, ThresholdedReLU;
  • ActivationLayer.
• Cropping layers:
  • Cropping1D, Cropping2D, Cropping3D.
• Upsampling layers:
  • UpSampling1D, UpSampling2D, UpSampling3D.
• Zero padding layers:
  • ZeroPadding1D, ZeroPadding2D, ZeroPadding3D.
• Other layers:
  • Permute, RepeatVector.

TensorFlow 1.15 Java API is currently used for layer implementation, but this project will be switching to TensorFlow 2.+ in the nearest future. This, however, does not affect the high-level API. Inference with TensorFlow models is currently supported only on desktops.

Contributing

Read the Contributing Guidelines.

Reporting issues/Support

Please use GitHub issues for filing feature requests and bug reports. You are also welcome to join the #kotlindl channel in Kotlin Slack.

Code of Conduct

This project and the corresponding community are governed by the JetBrains Open Source and Community Code of Conduct. Please make sure you read it.

License

KotlinDL is licensed under the Apache 2.0 License.

kotlindl's Issues

[Feature Request] Clone model

As far as I can tell the API does not allow you to directly clone a model? Specifically I want an identical model with all the same layers, weights, optimizer settings etc.

The only current solution I can find is to save and load the model from disk, but it would be nice to have the ability to copy without spending time on all the IO required for saving and loading from disk.

[Feature Request] Support batch processing for predictSoftly

Sequential.predictSoftly should also have a batch version, similar to Sequential.predict.

This came up in #25:

c.)
There should be a means to call predictSoftly on a batch of inputs. At the moment this is only possible for predict.

That would be awesome as well. My current pet project to try out this library is a board game AI that very crudely learns through self play, where the neural net is the search heuristic. The AI plays a semi-random game sequence and for each pair of successive moves I create a training data point. (needs 1 predictSoftly per data point) Building these self-play datasets would probably greatly benefit from batch soft-predicting 😄

Originally posted by @LostMekka in #25 (comment)

[Feature] Add Activation layer

We need a separate Activation layer because of #34. In many cases, the BatchNorm layer sits between the previous layer with the Linear Activation function and its own non-linear Activation function.

Also, it's important for Transfer Learning on Keras models, due to the widely used separate Activation layer.

Create API for Saving and Loading Model Snapshot

I want to try out different training mechanisms using save-restore-replay. Could you tell me if it's possible to make API to support such use-cases?

Add PReLU layer

We are missing some activation layers to support the export of models from Keras fully. One of them is the PReLU layer.

Add an activation layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the ReLU activation layer could be used, but feel free to improve it!

Also, support for export and import of activation layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the activation layer can be found here

[Feature] Add GlobalAveragePooling2D layer

Add ThresholdedReLU layer

We are missing some activation layers to support the export of models from Keras fully. One of them is the LeakyReLU layer.

Add an activation layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the ReLU activation layer could be used, but feel free to improve it!

Also, support for export and import of activation layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the activation layer can be found here

Metrics logs with `fit` are visible with verbose=0

I am not sure if it is a really an issue, but it has confused me a little bit. Calling fit on model for training with verbose=false produces logs with metrics. As far as I understood, INFO log level was set properly on beginning, but for metrics is logger called as debug.

It is a different to behaviour with Keras, at least with 2.x If I wanted more granular informations in loggs regarding training, I would implemented a callback for that, so I believe true/false for complete silence is fine.

[Audio] Add GlobalAveragePooling1D layer

We are missing some layers to support the export of models from Keras fully. One of them is the GlobalMaxPooling1D layer.

Add a layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the GlovalAvgPool2D layer could be used, but feel free to improve it!

If some refactoring to pooling layers could be applied, please, try to do it!

Also, support for export and import of layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the layer can be found here

[Video] Add MaxPooling3D layer

We are missing some layers to support the export of models from Keras fully. One of them is the MaxPooling3D layer. A detailed description of the layer can be found here.

• add layer class to the package using MaxPool2D as a reference implementation
• add documentation
• add test
• add export function (see ModelSaver.kt)
• add import function (see ModelLoader.kt)
• (optional) create an example

The initializer name schema is violated by Zero and Ones ops

If you will load a graph from Keras in hdf5 format it could keep Zeros or Ones as a default initializer for biases.
But the standard schema of loading weights via initializers names doesn't work well in this way. Should be used another initializer (maybe special) for handling this issue.

Name: Init_conv2d_25_conv2d_bias/Zero; Type: Const; Out #tensors: 1
Name: Init_conv2d_25_conv2d_bias/Fill; Type: Fill; Out #tensors: 1

Data and Models separation from core API library

Current API jar contains resources with data sets and serialized models, which are used in examples. It makes jar file huge and brings a lot of questions regardings maintainance, data sources legal licencing etc.

Supported use cases:

• ML "hello world" with MNIST data set should a few lines both in regular java application and so in Jupyter
• API deliverable itself is slick jar to be usable in small docker containers (even serverless lambdas ???)

Ideas:

• plugable data set loader as a separate library (support for various sources http, git .... ??? )
• gradle plugin ( not usefull with jupyter ... )

[Video] Add AveragePooling3D layer

We are missing some layers to support the export of models from Keras fully. One of them is the AveragePooling3D layer. A detailed description of the layer can be found here.

• add layer class to the package using AvgPool2D as a reference implementation
• add documentation
• add test
• add export function (see ModelSaver.kt)
• add import function (see ModelLoader.kt)
• (optional) create an example

Automated tests (CI) for PRs and merges

I was wondering if there is a plan to setup continuous integration for running the unit and integration tests automatically on each pull request as well as on merges to master branch. I think as the project and the number of contributors grow, this would be a useful addition to get a quick feedback and keep the project stable. Of course, each contributor/reviewer can independently run the tests locally on their own machine before submitting a PR/merge; however it may not be efficient and is kind of error-prone.

[Audio] Add AveragePooling1D layer

We are missing some layers to support the export of models from Keras fully. One of them is the AveragePooling1D layer.

Add a layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the AvgPool2D layer could be used, but feel free to improve it!

If some refactoring to pooling layers could be applied, please, try to do it!

Also, support for export and import of layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the layer can be found here

P.S. There are no-ops for tf.nn.avgPool1d in Java API, so you could try to implement it from scratch via available ops or use tf.nn.avgPool like in TensorFlowJS with reshape in tfjs-core/src/ops/avg_pool.ts

Add Softmax layer

We are missing some activation layers to support the export of models from Keras fully. One of them is the Softmax layer.

Add an activation layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the ReLU activation layer could be used, but feel free to improve it!

Also, support for export and import of activation layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the activation layer can be found here

Convert functions to properties in public API

Convert function to property
public abstract fun getOptimizerName(): String
=>
public abstract val optimizerName: String
in Optimimizer.kt

Exception in combination with Log4J

My project uses Log4J in combination with kotlin-logging:

dependencies {
   ...
   implementation("io.github.microutils", "kotlin-logging-jvm", "2.0.3")
   implementation("org.apache.logging.log4j", "log4j-slf4j-impl", "2.13.0")
}

This library is obviously hard-coded to expect logback loggers:

Exception in thread "main" java.lang.ClassCastException: org.apache.logging.slf4j.Log4jLogger cannot be cast to ch.qos.logback.classic.Logger
	at org.jetbrains.kotlinx.dl.api.inference.InferenceModel.setLevel(InferenceModel.kt:64)
	at org.jetbrains.kotlinx.dl.api.core.Sequential.internalFit(Sequential.kt:364)
	at org.jetbrains.kotlinx.dl.api.core.Sequential.fit(Sequential.kt:313)

Create Website with API Documentation from KDoc

It would be really great to have the website with a search ability from all documentation we already have in KDoc. Here is the link to get started: https://kotlin.github.io/dokka/1.4.32/user_guide/gradle/usage/

Add missed loaders for the ReLU and ELU activation layers

Each layer should have an implementation for export/import from/to JSON configuration (Keras compatible format) (see ModelLoader.kt and ModelSaver.kt)

The saving functions for the ReLU and ELU activation layer are missed

As an integration example, add a convolutional neural network to the examples package (in CNN package for example), it could be based on improved LeNet model, train it on "mnist" or "fashion mnist" and add it as an integration test to the "examples" module tests

• add saving function for ReLU activation layer
• add saving function for ELU activation layer
• add an example
• convert this example to the integration test
• train network, tune it to achieve accuracy more than >70%

Add Identity initializer

We are missing some initializers to fully support the export of models from Keras. One of them is the Identity initializer.

Add an initializer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link.

Also, support for export and import of initializer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the initializer can be found here

[Bug] Suspiciously slow calls to `Sequential.predictSoftly`

Calls to Sequential.predictSoftly start out fairly slow and each call takes a bit more time than the last one.

My example model is quite small, but the first call already takes about 25ms on my machine. (for comparison: fitting 1.000 random data points with 10 epochs takes 415ms) After calling predictSoftly 10.000 times, each successive call takes a third of a second.

Code to reproduce:

@OptIn(ExperimentalTime::class)
fun main() {
    Sequential.of(
        Input(36),
        Dense(36),
        Dense(36),
        Dense(36),
        Dense(36),
        Dense(36),
        Dense(16),
        Dense(8),
        Dense(3),
    ).use { model ->
        model.compile(
            optimizer = Adam(),
            loss = Losses.SOFT_MAX_CROSS_ENTROPY_WITH_LOGITS,
            metric = Metrics.MSE,
        )
        model.init()
        val features = FloatArray(36) { Random.nextFloat() }
        var predictionCalls = 0
        var predictionTimeOfBatch = Duration.ZERO
        val predictionTimes = mutableListOf<Double>()
        repeat(100_000) {
            val timing = measureTimedValue { model.predict(features) }
            predictionCalls++
            predictionTimeOfBatch += timing.duration
            predictionTimes += timing.duration.inMilliseconds
            if (predictionCalls % 100 == 0) {
                val csv = predictionTimes
                    .withIndex()
                    .joinToString("\n") { (i,t)->"${i + 1},$t" }
                File("timing.csv").writeText(csv)
                println("$predictionCalls calls done. (${predictionTimeOfBatch / 100} per call)")
                predictionTimeOfBatch = Duration.ZERO
            }
        }
    }
}

[Audio] Add Conv1D layer

We are missing some layers to support the export of models from Keras fully. One of them is the Conv1D layer.

Add a layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the Conv2D layer could be used, but feel free to improve it!

If some refactoring to convolutional layers could be applied, please, try to do it!

Also, support for export and import of layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the layer can be found here

P.S. There are no-ops for tf.nn.conv1d in Java API, so you could try to implement it from scratch via available ops or use tf.nn.conv2d like in TensorFlowJS with reshape in tfjs-core/src/ops/conv1d.ts

Add NFResNet model to models package

The new family and approach of BatchNorm-free NN architectures look very perspective due to the lack of BatchNorm training support.

In the paper "High-Performance Large-Scale Image Recognition Without Normalization" the new approach is proposed.

We should apply this approach to the traditional ResNet architecture to prevent gradient exploding without BatchNorm layers.

The pre-trained weights will be a plus but are not nessesary.

Add regularization to the layers

Each layer in Keras have internal regularization implemented with

• L1
• L2
• L1_L2

regularization functions.

Currently. these functions are not applied on the KotlinDL side, it should be added to the export/import model

Add Orthogonal initializer

We are missing some initializers to fully support the export of models from Keras. One of them is the Orthogonal initializer.

Add an initializer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link.

Also, support for export and import of initializer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the initializer can be found here

[Audio] Add GlobalMaxPooling1D layer

We are missing some layers to support the export of models from Keras fully. One of them is the GlobalMaxPooling1D layer.

Add a layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the GlovalAvgPool2D layer could be used, but feel free to improve it!

If some refactoring to pooling layers could be applied, please, try to do it!

Also, support for export and import of layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the layer can be found here

[Feature] Image preprocessing and new Dataset API

Current Dataset API is very limited and has a few sufficient limitations. We have no goal to copy-paste Keras and TensorFlow Dataset API now, but a few ideas could be adopted here.

First of all, we need in

1. Basic image preprocessing (resize, crop)
2. Currently the whole dataset is located in heap memory. We need to implement an alternative with batch-by-batch loading into RAM
3. Ability to write custom data loaders (it's now implemented and should not be broken in this PR)

Error during evalution in the example LeNetCifar10.kt

Exception in thread "main" java.lang.IllegalArgumentException: buffer with 1536000 elements is not compatible with a Tensor with shape [1000, 32, 32, 3]
at org.tensorflow.Tensor.incompatibleBuffer(Tensor.java:580)
at org.tensorflow.Tensor.allocateForBuffer(Tensor.java:295)
at org.tensorflow.Tensor.create(Tensor.java:188)
at api.keras.Sequential.evaluate(Sequential.kt:382)
at examples.keras.cifar10.LeNetCifar10Kt.main(LeNetCifar10.kt:97)
at examples.keras.cifar10.LeNetCifar10Kt.main(LeNetCifar10.kt)

[Feature Request] Expand current Dataset API

Currently the dataset api allows only for simple X -> Y data sets. So one input and one output. There are many models that require more input and/or output data to be fed into the training loop.

Since other API's in KotlinDL rely on the dataset API, would it make sense to already come up with a more generic API?

Typically in the Python world the output it is either arbitrary length Tuple (so perhaps a List in Kotlin) or a Dict (which is nicer IMHO since the key gives some indication what the individual tensors are used for).

[Video] Add GlobalAveragePooling3D layer

We are missing some layers to support the export of models from Keras fully. One of them is the GlobalAveragePooling3D layer. A detailed description of the layer can be found here.

• add layer class to the package using GlobalAveragePooling3D as a reference implementation
• add documentation
• add test
• add export function (see ModelSaver.kt)
• add import function (see ModelLoader.kt)
• (optional) create an example

Automated linting and style check

I was wondering if there is a plan to have automated style check and linting for contributions. This would be really useful to keep the codebase clean and consistent. Additionally, some pre-commit hooks for running style checks and linting could be set up to address this locally before pushing commits.

Add LeakyReLU layer

We are missing some activation layers to support the export of models from Keras fully. One of them is the LeakyReLU layer.

Add an activation layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the ReLU activation layer could be used, but feel free to improve it!

Also, support for export and import of activation layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the activation layer can be found here

Loss value returns NaN in LeNetMnistWithCustomCallbacks example

When trained, the loss of each batch is NaN. This is also on test data. As a result, the accuracy is 0.098.
This does not happen on every run.

[Visualisation] Find a best way for activation (feature) map visualisation

Currently, we have a very poor example of feature map visualization in the examples module for LeNet-5.

Hope these examples could be extended and rewritten in a more clear, Kotlin-idiomatic, and universal manner to visualize it on Swing.

But let's don't limit yourself to good old Swing, what is about modern approaches, HTML + JS tracer, or maybe Compose or you could suggest the better options here.

Let's start our research here and implement the best practices

Add ELU layer

We are missing some activation layers to support the export of models from Keras fully. One of them is the ELU layer.

Add an activation layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the ReLU activation layer could be used, but feel free to improve it!

Also, support for export and import of activation layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the activation layer can be found here

Find a best way to import weigths to the Keras

We have the ability to load the model configuration from Keras JSON format and pre-trained weights stored in h5 format.
But how to export the model pre-trained in KotlinDL to the Keras or pure TensorFlow?

Currently, we support the weights export to the txt files. It's a baseline solution.
We need research here to find the best way to export weights.

There are a few open questions:

1. What part of weights loading should be implemented on the KotlinDL side or maybe we need a special loader to the Keras in Python (published as a separate pip package)?
2. Should we export to txt or another popular format or maybe to NumPy arrays to easily parse it on the Python side?
3. Is it easy to write h5 writer to store them in Keras weight format?
4. Will it be enough of a universal approach which could be reused for weights export to ONNX or PyTorch? Or we will eat the elephant in parts and develop an individual approach to exporting weights to different systems.

Let's discuss here, in this issue the possible approaches, their advantages, and disadvantages?

P.S Read the Contributing Guidelines.

Get rid of Activations enum

Activations enum duplicates Activation interface and makes adding new activation function harder for library contributors and impossible for library users.

Get rid of enum and add convenient "provider" service which stores activation functions objects.

Add model export for a few layers, missed in ModelSaver.kt

Each layer should be exported in JSON, Keras compatible format
The following layers are missed in ModelSaver.kt

1. Concatenate
2. ZeroPadding2D
3. DepthwiseConv2D
4. SeparableConv2D

NOTE: some another layers' export, like Activation or merge layers like Subtract and so on are parts of other PRs now

Klaxon dependency

Setting up a dependency to org.jetbrains.kotlin.kotlin-dl:api:0.0.5 does not seem to be enough.
I got:
Could not find com.beust:klaxon:5.0.1.
Required by:
project : > org.jetbrains.kotlin.kotlin-dl:api:0.0.5

Adding explicit dependency on klaxon helps. However, it's not ideal.

Model save/load compatibility with Keras

@zaleslaw, I was wondering if we are aiming for a two-way compatibility with Keras in terms of saving and loading of the models. In particular, should both of the following conditions satisfy?

• Models trained and saved with Keras could be fully loaded in KotlinDL.
• Models trained and saved with KotlinDL could be fully loaded in Keras.

[Feature Request] Support for functional API

As of now, only Sequential models are supported it would be great to allow models to take inspiration from Keras Functional APIs and allow building such models. This would allow creating models that are more flexible with KotlinDL and also open up more possibilities with the kind of models one could make with KotlinDL including branching, multiple-inputs, multiple-outputs and so on

[Audio] Add MaxPooling1D layer

We are missing some layers to support the export of models from Keras fully. One of them is the MaxPooling1D layer.

Add a layer class, write documentation for it, write a test for it, try, if possible, create a small trainable network with it (in your own GitHub) and attach a link here in the comments.

The layer should be placed here

As a reference implementation, the MaxPool2D layer could be used, but feel free to improve it!

If some refactoring to pooling layers could be applied, please, try to do it!

Also, support for export and import of layer in JSON format should be added (see ModelLoader.kt and ModelSaver.kt)

A detailed description of the layer can be found here

P.S. There are no-ops for tf.nn.maxPool1d in Java API, so you could try to implement it from scratch via available ops or use tf.nn.maxPool like in TensorFlowJS with reshape in tfjs-core/src/ops/max_pool.ts

reshapeFunction not initialized after model load

I store a model with

val model = Sequential.of(
            Input(...),
            Dense(...),
            Dropout(...),
            Dense(...),
            Dense(1)
        )
model.compile(...)
model.fit(...)
model.save(dir, TF_GRAPH_CUSTOM_VARIABLES, true, FAIL_IF_EXISTS)

then I reload and use it with

val model = InferenceModel.load(dir)
model.predictSoftly(...)

But I get an exception:

Exception in thread "main" kotlin.UninitializedPropertyAccessException: lateinit property reshapeFunction has not been initialized
	at org.jetbrains.kotlinx.dl.api.inference.InferenceModel.predictSoftly(InferenceModel.kt:128)
	at org.jetbrains.kotlinx.dl.api.inference.InferenceModel.predictSoftly$default(InferenceModel.kt:127)

My fault or is this a bug?
Not sure about SavingFormat because two entries have same description.

Questions about design goals, roadmap, PyTorch integration, DL4j, etc.

I am currently learning Kotlin and in order to enhance my learning experience I was looking for a project accepting contributions. So I am excited to find this library, especially considering the fact that I have a relatively fair amount of experience working with Keras/TF in Python. However, I have a few questions about some of the high-level aspects of the project and I would appreciate if the maintainers could answer them:

• Is the goal to mimic the Python API of Keras as close as possible, or instead, should it focus on "what's the idiomatic way to implement XYZ feature in Kotlin?" regardless of how the that feature has been implemented in Python API of Keras (i.e. kotlinic vs. pythonic)? As I see, currently there is a rather strong focus to follow Keras API.
• What's the plan for switching to TF 2+ Java APIs?
• Is TF the only backend to be supported, or is there a plan to add support for more backends (e.g. PyTorch)?
• How is this going to differ from Deeplearning4J?
• Is there a long-term goal to get rid of the backend(s) in future and replace them with Kotlin implementations and libraries (i.e. similar to what DL4J has done, though from the beginning)?

Thanks!

[Video] Add Conv3D layer

We are missing some layers to support the export of models from Keras fully. One of them is the Conv3D layer. A detailed description of the layer can be found here.

• add layer class to the package using Conv2D as a reference implementation
• add documentation
• add test
• add export function (see ModelSaver.kt)
• add import function (see ModelLoader.kt)
• (optional) create an example

This issue depends on the results of the #59

Incorrect work of LogLoss function

Exception in thread "main" java.lang.IllegalStateException: Tensor is not a scalar
for it.compile(optimizer = SGD(LEARNING_RATE), loss = LossFunctions.LogLoss) in LeNetMnist example

[Feature] Add Merge layers (Add, Subtract, Multiply, Concatenate, Maximum, Minimum)

[Feature] Add BatchNorm layer

Keras framework has a BatchNorm layer, and it's a significant and widely used layer in modern CNN architectures.

Unfortunately, the TF Java API 1.15 and TF Java API 2.x wraps outdated BatchNorm operand; alternative FusedBatchNorm is a real operand at all and could not be used as part of the trained graph.

The best option here creates our own BatchNorm operand based on another low-level TF operation with the ability to participate in backward and forward calculations.

Multiplatform support

I think multiplatform support would quite useful. I assume this has been discussed already but there is no Github issue for it.

Use cases:

• Some Kotlin code targets only certain native platforms or only JS (in the future I think only WASM will be quite common) and it would be nice to be able to use this library from those projects.
• For multiplatform programs that do any kind of on device ML.

Since Tensforflow already has APIs for these other platforms it seems like implementing multiplatform support should be fairly straight forward.