cwarcup / wtf_is_this

Use Tensorflowjs to determine if an image contains a banana.

Is this a banana? 🍌

No. Maybe? I don't know. Use this app to find out.

Use the MobileNet model to confirm your suspicions about bananas 🍌🤔

Getting Started

Visit the live demo here on mobile or desktop and upload an image. After a few seconds, you'll get a prediction.

Feel free to test it out with these images: Banana, Hamburger, Apple

Demo

Built With

Motivation

I wanted to learn more about machine learning and how to use TensorFlow.js. I explored the TensorFlow.js website and found a pre-trained image classification model called MobileNet. I used this model to classify images of bananas and other objects.

Challenges

Image Upload and Tensor

I first had to learn how to use the URL.createObjectURL method to create a URL for the image file. After reading the TensorFlow docs, I learned that the tf.browser.fromPixels method can be used to convert an image to a tensor. The fromPixels method takes in either an ImageData, HTMLImageElement, HTMLCanvasElement, or HTMLVideoElement. I learnt the image data contained the imageElement which could be used to convert the image to a tensor.

Using the tensor, I could then pass it into the classify method of the MobileNet model. The classify method returns a promise which resolves to an array of predictions. I then used the map method to get the top predictions and determined if the predictions included the string banana.