cocoa-xu / tflite_beam

TensorFlow Lite Erlang bindings with optional EdgeTPU support.

TFLiteBEAM

TensorFlow Lite BEAM bindings with optional EdgeTPU support.

OS	Arch	ABI	Build Status	Has Precompiled Library
Ubuntu 20.04	x86_64	gnu		Yes
Ubuntu 20.04	arm64	gnu		Yes
Ubuntu 20.04	armv7l	gnueabihf		Yes
Ubuntu 20.04	armv6	gnueabihf		Yes
Ubuntu 20.04	riscv64	gnu		Yes
macOS 11 Big Sur	x86_64	darwin		Yes
macOS 11 Big Sur	arm64	darwin		Yes

Coral Support

Dependencies

For macOS

# only required if not using precompiled binaries
# for compiling libusb
brew install autoconf automake

For some Linux OSes you need to manually execute the following command to update udev rules, otherwise, libedgetpu will fail to initialize Coral devices.

bash "3rd_party/cache/${TFLITE_BEAM_CORAL_LIBEDGETPU_RUNTIME}/edgetpu_runtime/install.sh"

Compile-Time Environment Variable

• TFLITE_BEAM_PREFER_PRECOMPILED

  Use precompiled binaries when TFLITE_BEAM_PREFER_PRECOMPILED is true. Otherwise, this library will compile from source.

  Defaults to true.

• TFLITE_BEAM_CORAL_SUPPORT

  Enable Coral Support.

  Defaults to true.

• TFLITE_BEAM_CORAL_USB_THROTTLE

  Throttling USB Coral Devices. Please see the official warning here, google-coral/libedgetpu.

  Defaults to true.

  Note that only when TFLITE_BEAM_CORAL_USB_THROTTLE is set to NO, :tflite_beam will use the non-throttled libedgetpu libraries.

• TFLITE_BEAM_CORAL_LIBEDGETPU_LIBRARIES

  Choose which ones of the libedgetpu libraries to copy to the priv directory of the :tflite_beam app.

  Default value is native - only native libraries will be downloaded and copied. native corresponds to the host OS and CPU architecture when compiling this library.

  When set to a specific value, e.g, darwin_arm64 or darwin_x86_64, then the corresponding one will be downloaded and copied. This option is expected to be used for cross-compiling, like with nerves.

  Available values for this option are:

  Value	OS/CPU
  aarch64	Linux arm64
  armv7l	Linux armv7
  armv6	Linux armv6
  k8	Linux x86_64
  x86_64	Linux x86_64
  riscv64	Linux riscv64
  darwin_arm64	macOS Apple Silicon
  darwin_x86_64	macOS x86_64

Installation

If available in Hex, the package can be installed by adding tflite_beam to your list of dependencies in rebar.config:

{deps, [
  {tflite_beam, "0.3.2"}
]}

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/tflite_beam.

tflite_beam's Issues

Missing module/function docs

Here are a list of blank documents as of 2023-03-14

module docs

• TFLiteElixir (top-level module)
• TFLiteElixir.Ops.Builtin.BuiltinResolver
• TFLiteElixir.Coral

function docs

• TFLiteElixir.reset_variable_tensor/1
• TFLiteElixir.Coral.contains_edge_tpu_custom_op?/1
• TFLiteElixir.Coral.dequantize_tensor/2
• TFLiteElixir.Coral.edge_tpu_devices/0
• TFLiteElixir.Coral.get_edge_tpu_context/1
• TFLiteElixir.Coral.get_edge_tpu_context/2
• TFLiteElixir.Coral.make_edge_tpu_interpreter/2
• TFLiteElixir.FlatBufferModel.initialized/1
• TFLiteElixir.Interpreter.get_signature_defs/1
• TFLiteElixir.Interpreter.predict/2

Example: BERT Question and Answer

Use a TensorFlow Lite model to answer questions based on the content of a given passage.

• #56
  • Implement BasicTokenizer
  • Implement WordpieceTokenizer
  • Implement FullTokenizer
• Load the model and get results

related link: https://www.tensorflow.org/lite/examples/bert_qa/overview

Erlang Bindings

• #48
• #49
• #50

precompile for armv6?

any reason they are not precompiled presently?

tflite_elixir seems to have armv6 support..

[deps] add support for TensorFlow 2.12.0

Segfault

It's rare but I found that it sometimes causes segfaults in the GitHub CI.

https://github.com/cocoa-xu/tflite_elixir/actions/runs/4405411900/jobs/7716261476

Port all existing modules to Erlang.

Convert from `lib/tflite_beam_nif.ex` to `tflite_beam_nif.erl`

Add a wrapper layer `TFLite.Backend` for `TfLiteTensor`

Right now we have something like

Nx.tensor([1, 2, 3])
|> Nx.to_binary()
|> then(&TFTensor.set_data!(input_tensor, &1))

the goal is to do the same thing without explicitly calling to_binary/from_binary (maybe also include some other functions).

tensor = Nx.tensor([1, 2, 3])
tf_tensor = Nx.backend_transfer(tensor, TFLite.Backend)

TFLite.run(model, tf_tensor)
#=> This will also return a tensor allocated on TFLite.Backend

tensor = Nx.tensor([1, 2, 3])
TFLite.run(model, tensor)

TFLiteTensor.dims vs out_tensor.shape

This is rather a question about accessing tensor fields.

As I play with the Super Resolution example, I notice there are two ways to access shape and type of a tensor and they seem to give the same result at least in the context of the example. I am wondering if there is a case where we have to use the functions. Depending on it, we might need to explain in the doc or to consider deprecating the functions.

out_tensor.shape
# vs
TFLite.TFLiteTensor.dims(out_tensor)

out_tensor.type
# vs
type = TFLite.TFLiteTensor.type(out_tensor)

Change name to `tflite_beam`

cryptic `TFLiteTensor.to_nx/2` error message when no Nx backend provided

When TFLiteTensor.to_nx/2 is called without Nx backend provided, a cryptic error can occur. I am wondering if there is a way to make this error less cryptic and/or to print a human friendly error message.

https://hexdocs.pm/tflite_elixir/TFLiteElixir.TFLiteTensor.html#to_nx/2

The error seems to occur when Nx.from_binary/3 is called.

https://hexdocs.pm/nx/Nx.html#from_binary/3

** (UndefinedFunctionError) function nil.from_binary/3 is undefined
    nil.from_binary(#Inspect.Error<
  got KeyError with message:

      """
      key :__struct__ not found in: nil. If you are using the dot syntax, such as map.field, make sure the left-hand side of the dot is a map
      """

  while inspecting:

      %{__struct__: Nx.Tensor, data: nil, names: [nil], shape: {100}, type: {:f, 32}}

  Stacktrace:

    (nx 0.5.1) lib/nx/tensor.ex:165: Inspect.Nx.Tensor.inspect/2
    (elixir 1.14.3) lib/inspect/algebra.ex:341: Inspect.Algebra.to_doc/2
    (elixir 1.14.3) lib/kernel.ex:2254: Kernel.inspect/2
    (elixir 1.14.3) lib/exception.ex:717: anonymous fn/2 in Exception.format_arity/1
    (elixir 1.14.3) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (elixir 1.14.3) lib/exception.ex:717: Exception.format_arity/1
    (elixir 1.14.3) lib/exception.ex:712: Exception.format_mfa/3
    (elixir 1.14.3) lib/exception.ex:623: Exception.format_stacktrace_entry/1

>, <<105, 9, 17, 64, 151, 171, 24, 191, 192, 69, 116, 62, 25, 75, 60, 63, 192, 10, 3, 192, 114, 80, 4, 63, 114, 80, 4, 64, 16, 4, 201, 63, 36, 209, 246, 63, 154, 234, 95, 191, 79, 52, 183, 190, 79, 52, 183, 190, 7, 189, ...>>, [])
    (tflite_elixir 0.1.5) lib/tflite_elixir/tflite_tensor.ex:124: TFLiteElixir.TFLiteTensor.to_nx/2

This error can be recreated just by removing :backend option in the examples/artistic_style_transfer.livemd notebook.

Parse `TFLITE_METADATA`

Create some high-level modules

• ImageClassification
• ObjectDetection

Getting "Runtime node closed unexpectedly - no connection" on example tpu notebook

Hi. I wanted to try it with coral but my livebook dies on the last cell in the tpu.livemd
I tried to run it as mix task but also getting

[1]    55100 segmentation fault (core dumped)  mix classify_image -m /tmp/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflit

I have the coredump but it's 175MB in size

Implement BERT Tokenizer

rebar3: Add support for using precompiled binaries.