• Verify that vulkan.hpp can be used on Android.
• Create android specific CMakeList files (platforms folder?)
• Test application.

Publish the python package to Pypi

Use Github actions

• Bazel cache for Linux and Windows.
• Publish python packages as release artifacts for both OSs.
• Try with Github Actions

Currently all objects that are created by the Session class hold a reference to it. Instead, hold a reference to the Vulkan resources needed by that object for maintaining its life-cycle. In particual, the vk::Device reference.

• C++ library and executables
• C++ dependencies
• Cython
• Python library
• Tests

• Instead of adding specific containers for buffers, images, etc, use a single container of Object to store them all.
• Change the JSON format to include a single objects list. Distinguish each object with a type attribute.

Currently, the glsl folder in the repository contains shader code for testing purposes. In the future, this folder will contain general (library) code for the GPU.

Task

• Move the test shaders to a core/test/glsl folder.
• Adapt the CMake scripts to support shader compilation from several places.

• Add Doxyfile.
• Define the documentation template for classes and methods.

The Python library needs more convenience functions for repetitive tasks such as:

• Creation of Buffers and Images. This includes creation from Numpy arrays.
• Copy operations (Buffer-Buffer, Image-Buffer). The following is the current way to do it:

  cmdBuffer = session.createCommandBuffer()
  cmdBuffer.begin()
  cmdBuffer.changeImageLayout(img, 'TransferDstOptimal')
  cmdBuffer.copyBufferToImage(stageBuffer, img)
  cmdBuffer.changeImageLayout(img, 'General')
  cmdBuffer.end()
  session.run(cmdBuffer)

• Give the API a more "dynamic" flavour. For instance, running a compute node directly from the object:

  node = session.readComputeNode(...)
  node.run()

• Create ComputeNode objects from GLSL code.

Code the first examples using the existing version of the engine to discover how to design the interface for developing real-life compute graphs.

Examples:

• Image pyramid.
• Image gradient with separable convolution masks.

• Check if Travis CI supports running the core tests.
• Build C++ library.
• Build Python wrappers.
• Build Doxygen.
• Build Python doc (Sphinx?).
• Run test (C++, Python).

• OS-specific compile flags in the repo.
• Include jobs in Github Actions to validate.
• Build the C++ library and Python wheel.

• Create ComputeNodeDescriptor from JSON feed.
• Build ComputeNode objects
• Bind buffer objects to node.

• RGB 2 HSV

• HSV 2 RGB

• RGB 2 Lab*

• Lab* 2 RGB

• RGB 2 Gray

• RGB 2 XYZ

• XYZ 2 RGB

Code the optical flow algorithm in https://github.com/jadarve/optical-flow-filter using Lluvia.

Define the compute graph API for building complex pipelines.

Inspirations:

• GStreamer
• TensorFlow

Create unit test cases for the Python wrappers.

• Use Pytest as testing framework
• Check if it is possible to run the tests through the setup.py script.

Check that the memory type in memoryTypeBits returned by device.getBufferMemoryRequirements(vkBuffer); is supported by the memory trying to allocate the buffer in Memory::createBuffer

See: https://www.khronos.org/registry/vulkan/specs/1.0-extensions/html/vkspec.html#VkMemoryRequirements

• Consider blocking the memory page.
• Do the mapping from the buffer object, not the memory one.
• Return some type of smart pointer of the mapped region.
• Validate if the buffer is mappable.

• Use Cython.
• Check possibility to automate parts of the generation. Like creating the Cython header files.

Code example

import lluvia.core as ll
import numpy as np

session = ll.createSession()

memory = session.createMemory(ll.MemoryFlags.HostVisible | ll.MemoryFlags.HostCoherent, 4096, False)

buf1 = memory.createBuffer(2048)
buf2 = memory.createBuffer(2048)

program = session.createProgram('add.spv')

# setting all this is boring! maybe could be read from a JSON
nodeDesc = ll.ComputeNodeDescriptor(program, 'main', [32, 1, 1], [32, 1, 1])
nodeDesc.parameters.append(ll.ParameterType.Buffer)
nodeDesc.parameters.append(ll.ParameterType.Buffer)

node = session.createComputeNode(nodeDesc)
node.bind(0, buf1)
node.bind(1, buf2)


# alternative way, read a JSON with all the information
"""
{
    "function": "main",
    "grid_x": 1,
    "grid_y": 1,
    "grid_z": 1,
    "local_x": 1024,
    "local_y": 1,
    "local_z": 1,
    "parameters": [
        "BUFFER",
        "BUFFER"
    ],
    "spirv": "gADABsAAAAZAAAA/QABADgAAQA="
}
"""
node2 = session.readComputeNode('pathToNodeDesc.json', localGroup=[16, 1, 1])
node2.grid = [32, 1, 1] # local workgroup cannot be changed from here
                        # as the node has been compiled.
node2.bind(0, buf1)
node2.bind(1, buf2)

# node2 can be saved to JSON. It contains all the relevant information
ll.writeComputeNode(node2, 'pathToFile.json')


cmdBuffer = session.createCommandBuffer()
cmdBuffer.run(node)

session.run(cmdBuffer)

# convert to numpy array
bufCopy = buf1.toHost([1024], np.dtype.float32)

buf2.fromHost(bufCopy)

• Check which image formats are usable in compute shaders.
• Supported image formats. Number of channels and type.
• Image class.

Create the read the docs page :)

This will allow me to control which flags are exposed to users. Some of the Vulkan flags I don't really use.

Add Bazel rules to create node libraries. A node library is a collection of Lua and SPIR-V files packed together and ready to be used within a session.

Compile the library to Android. Check if the wrappers should be done manually or using something like JavaCpp https://github.com/bytedeco/javacpp

Also, check how to connect the camera2 API with Vulkan. A starting point could be this: https://stackoverflow.com/questions/43507536/how-to-connect-android-mediacodec-surface-to-vulkan

• https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VK_ANDROID_external_memory_android_hardware_buffer
• https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#AHardwareBuffer

Hi @jadarve,

I would like to try lluvia. I tried compiling using CMake without success as some files cannot be found.

Is there any plan on adding build and installation instruction?

Thank you!

• Inclue Lluvia as a dependency on mediapipe's bazel scripts.
• Create calculators to run graphs within mediapipe.

Implement the optical flow filter algorithm in https://github.com/jadarve/optical-flow-filter.

• Build compute pipeline depending on filter parameters (pyramid levels, maxflow).
• Shared/Static library?
• WebCam demo with OpenCV.

See in which cases it is necessary to pass std::shared_ptr to methods by value or by reference.

Modify the build scripts to support compilation using the MacOS Vulkan SDK.

See more: https://www.lunarg.com/lunarg-releases-vulkan-sdk-1-0-69-0-for-mac/

See the implementation of objectTypeToString and stringToObjectType coded in #14.

See https://clang.llvm.org/docs/ClangFormatStyleOptions.html

• This will eliminate depending on Python for building pure C++ projects.
• Consider using https://github.com/pantor/inja

Create a first version of the project's wiki.

• About the project
• Getting started:
  • Building C++ library.
  • Building Python wrappers.
  • Basic usage C++ and Python

Print the output of ll-info in JSON format instead of plain text.

#################################################

SOURCE FILES SETTINGS

#################################################
add_subdirectory(core)--------------------------->core is not exsit, you means : cpp/core ?
add_subdirectory(tools)
add_subdirectory(samples)

creates the shaders custom target

include (cmake/shaders.cmake)

Now, there are some compilation errors with cmake...............

Build a Python wheel compatible with Google's colab.

• Add more properties to the port descriptor in order to validate proper binding. Examples include:
  • Image channels
  • Image channel type
  • Image view sampled
  • Image view normalized coordinates

For instance, take RGBA2Gray node:

function builder.newDescriptor()

    local desc = ll.ComputeNodeDescriptor.new()

    desc:init(builder.name, ll.ComputeDimension.D2)

    -- TOTHINK: increased port contracts by checking internal attributes of the PortType
    -- For ImageView, check all image attributes + image view attributes.
    desc:addPort(ll.PortDescriptor.new(0, 'in_rgba', ll.PortDirection.In, ll.PortType.ImageView))
    desc:addPort(ll.PortDescriptor.new(1, 'out_gray', ll.PortDirection.Out, ll.PortType.ImageView))

    return desc
end

function builder.onNodeInit(node)

    local in_rgba = node:getPort('in_rgba')

    -- validate in_rgba is actually a rgba8ui image
    -- TODO: remove once port-contracts are implemented
    local err = ll.isValidImage(in_rgba, ll.ChannelCount.C4, ll.ChannelType.Uint8)
    if err ~= nil then
        error(builder.name .. ': error validating in_rgba: ' .. err)
    end

    -- ...
end

The code in the builder, as well as the likelihood of errors, can be reduced if the node internally validates if in_rgba has 4 channels and it's of the correct channel type.

⚠️ Consider also the parameterization of nodes according to channel type and count. For instance, ImagePyramid_r8ui is tailored to 8-bit single-channel images. Any other combination would require a new node definition.

Check how to use Sphinx to generate the documentation pages.

Use bazel to compile the Python wrappers :)

Some useful stuff:

• https://github.com/horia141/bazel-pypi-package

• Compile the engine for the Linux for Tegra distribution.
• Run the optical-flow example in one board: TX2 or TX nano.

The script below does not work:

local builder = ll.class(ll.ComputeNodeBuilder)

function builder.newDescriptor() 
    
    local desc = ll.ComputeNodeDescriptor.new()
    
    desc.builderName  = 'Square'
    desc.localShape   = ll.vec3ui.new(32, 1, 1)
    desc.gridShape    = ll.vec3ui.new(1, 1, 1)
    desc.program      = ll.getProgram('Square')
    desc.functionName = 'main'

    desc:addPort(ll.PortDescriptor.new(0, 'in_buffer', ll.PortDirection.In, ll.PortType.Buffer))
    desc:addPort(ll.PortDescriptor.new(1, 'out_buffer', ll.PortDirection.Out, ll.PortType.Buffer))

    return desc
end

function builder.onNodeInit(node)
    
    local in_buffer = node:getPort('in_buffer')

    -- Allocate out_buffer in the same Memory as in_buffer
    local memory = in_buffer.memory
    
    -- TODO: call createBuffer using default usage flags
    -- TODO: overload of createBuffer with in_buffer.usageFlags does not work. Need unsafe version
    local out_buffer = memory:createBuffer(in_buffer.size)
    
    -- bind the output
    node:bind('out_buffer', out_buffer)
    
    -- configure the dispath shapes according to the size of in_buffer
    node:configureGridShape(ll.vec3ui.new(in_buffer.size / 4, 1, 1))

end

ll.registerNodeBuilder('Square', builder)

• Build Python packages for Linux and Windows.
• Create releases.

A new convenience function to reduce the number of lines of code needed to create an ImageView.

Currently, I need to do this:

# GPU memory where the images will be allocated
memory = session.createMemory()
 
# reads the image and transfer it to a ll.Image object (in the GPU)
mouse = ll_util.readSampleImage('mouse')
mouse_gpu = memory.createImageFromHost(mouse)
 
# creates an ImageView used to read the pixels in the GPU shader
in_rgba = mouse_gpu.createImageView(ll.ImageFilterMode.Nearest, ll.ImageAddressMode.ClampToEdge, False, False)

A more convenient way would be:

# GPU memory where the images will be allocated
memory = session.createMemory()
 
# reads the image
mouse = ll_util.readSampleImage('mouse')
 
# creates an ImageView used to read the pixels in the GPU shader
in_rgba = memory.createImageViewFromHost(mouse, ll.ImageFilterMode.Nearest, ll.ImageAddressMode.ClampToEdge, False, False)