code4sac / trash-ai

Web based trash image classification

Trash AI: Web application for serverless image classification of trash

Project Information

• Sponsor: Win Cowger, Moore Institute for Plastic Pollution Research
• Meeting Times: Wednesdays at 6:30pm PT Weekly Brigade Meetings
• Publication Pending: paper.md

Project Summary

Trash AI is a web application where users can upload photos of litter, which will be labeled using computer vision to detect and categorize litter in the image by type. Early inspiration from WADE AI streamlined this development. Trash AI will enhance the abilities of researchers to quickly label trash in photos.

Demo

Deployment

You can simply go to www.trashai.org to start using the tool or deploy it yourself. Current self-deployment options are local deployment with docker to remote on Amazon Web Services (AWS).

Run Local Docker Instance

docker run -p 5150:5150 -it code4sac/trashai:latest

Navigate to http://localhost:5150

If you are attempting to run on an Apple Silicon device, you might get the following error:

docker: no matching manifest for linux/arm64/v8 in the manifest list entries.

This can be remedied by specifying the build platform.

docker run --platform linux/x86_64 -p 5150:5150 -it code4sac/trashai:latest

Deploy to Any Webserver

If you want to deploy this to a static web directory and serve it using apache or nginx, you can do so with the following command using /var/www/html as an example destination directory.

# create container from latest public trash ai docker image
id=$(docker create code4sac/trashai:latest)

# copy the static files
docker cp $id:/usr/share/nginx/html /var/www/html

# remove created container
docker rm -v $id

Local Development

• Run the environment live with localstack and docker.

AWS Deployment

• Instructions on bringing up a new AWS deployment.

Continuous Integration and Continuous Delivery (CI/CD) - Github Actions

• Mostly CD at this point.

Github Actions AWS Deployment Role

• Runs the complex stuff so you don't have to.

Tests

Instructions for automated and manual tests here.

Functionality

Documentation on typical workflow and functionality of the tool can be found here

Contribute

We welcome contributions of all kinds.

To get started, look at the Start Here section of the project board

You can open an issue or pull request.

Here are some ideas on How to Contribute.

Please adhere to this project's Code of Conduct.

trash-ai's Issues

Add test image on upload tab

I think we should have a test image for people to upload to the tool if they don't have one they feel comfortable using right away since we are requiring all images to get saved currently. That way they can see a good example and be incentivized to use the tool.

[Feature]: Build models in Cloud to support model development

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

Currently, any models we build must be trained on someone's local computer. This can be an issue as not everyone who contributes to this project may have a modern GPU and will have the resources to train the model.

Ideally, we would be able to train models in AWS to support model development and deployment.

What can we use in AWS or other cloud-based solutions to support model development (Sagemaker maybe)? How do we provide access to train models? Can we build Github Action scripts to support model building on merges to specific branches like staging and production?

Problem

Access to model training hardware can be difficult.

Proposed Solution

Use Cloud based solutions to train models for project members.

Alternatives Considered

We could not support cloud solutions and require a project member to train any models that get developed.

Analytics - track processed counts

Submit events to google analytics to track how many images are processed

If possible also track:

• Classified
• No classification
• Mapping/EXIF data

Fix code scanning alert - Polynomial regular expression used on uncontrolled data

Tracking issue for:

• https://github.com/code4sac/trash-ai/security/code-scanning/1

Import tensorflow model to the application and demonstrate image invocation

Once we have a serialized model from #7, import the model into NodeJs and create a simple demo script for invoking the model with an image and getting the results.

[Feature]: Better deployment and use instructions

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

Reviewer 1: While the video walkthrough is great, it would probably be best to have a bit more written instruction on usage in the README/on the website. I think the setup documentation could also be improved a bit. The local setup seems to refer to specifics of WSL a lot where it could be more system-agnostic, and while the usage of GH workflows is novel, the AWS deployment instructions are a bit nonstandard in that regard plus seem to assume that you're deploying it from the repo itself/as a repo owner.

Problem

Instructions are overly specific and not enough information on the website.

Proposed Solution

System agnostic deployment instructions and more information on the website.

Alternatives Considered

NA

Add EXIF properties to metadata

Add additional metadata from exif around geography in addition to the bounding box and label data.

Add feedback form to about page

https://forms.gle/nSPKf6tWkPo8Cftp6

Implement github actions automated tests

The journal we are submitting to requires that there are some automated tests in the repo that ensure that the code follows some standard logic. I can't find the tests currently but it is likely here. Could someone point me to it? I will add a link to it to the readme so that it is obvious for the reviewers.

Add Confirmation Message / Progress Bar for Uploads

We need to add a confirmation message or some visual way for users to know that the pictures they are uploading have uploaded successfully.

Change UI and backend to allow multiple files uploaded at once

We're using Multer and multers3 which we should be able to change to allow multiple files at once to be uploaded for user convenience

Invoke model when a file is uploaded and return results to client

Use the demonstrated method from #8 to invoke the model when an image is received on the /upload route. Return the results to the client (or any errors).

Improve UI for upload tab in the header

The upload link in the header looks very bad. Make it look better.

Get application demo ready

Get the app in a demo ready state so that we can present it to stakeholders and gather feedback on the future direction of the project.

Add CDK code for deploying backend

This should include:

1. Provisioning and running the express api (possibly in lambda, or a container on an ec2 instance).
2. Provisioning an S3 bucket which is used when uploading photos.
3. IAM roles necessary for the application to function

[Feature]: Instructions to verify behavior and Tests

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

There doesn't seem to be any automated tests. While the desired behavior here is relatively clear/trivial, there's no formal instructions to verify behavior.

Problem

NO

Proposed Solution

Need to add formal instructions to verify behavior and add clarity on the automated tests currently implemented.

Alternatives Considered

NA

Fix code scanning alert - Clear-text logging of sensitive information

Tracking issue for:

• https://github.com/code4sac/trash-ai/security/code-scanning/2

Allow user to download metadata long with the classified image

Fix zone logic for deploy role

When updating the deploy_map zone it doesn't pick up the change and adjust the
iam deploy permissions.

Change SSM parameter setup for the zone id variable

Drag and Drop the same image multiple times generates Vue Duplicate Key warning

The duplicate key warning is triggered in the console when draging and dropping the same image file in two or more separate drag-drop operations.

The attached log file contains the full stack trace.

vue.runtime.esm.js:619 [Vue warn]: Duplicate keys detected: 'Screen Shot 2022-02-07 at 10.38.15 AM.png'. This may cause an update error.

found in

---> <TestUpload> at src/components/test/upload.vue
       <VCard>
         <Src/pages/index.vue> at src/pages/index.vue
           <Nuxt>
             <VMain>
               <VApp>
                 <NavBase> at src/components/nav/base.vue
                   <Default> at src/layouts/innertab.vue
                     <Root>

localhost-1647059099920.log

Add issue template

@createHernandez recommended we add issue templates, love the idea! saw this on another repo recently and was really impressed by the experience and also the data they got in the end from the issue was much higher quality.

I initially signed up for this but I don't have access to change the settings and would need access to be able to get this going. I ended up deploying it on another repo in just about 5 min using these instructions and mostly the vanilla settings. If someone wants to give me settings access I would be happy to set it up, just let me know.

Consolidate all local development in Docker

There should be a single docker-compose file in the root of the project that manages the backend and front-end web apps. This should also support hot reloading when actively developing.

Create Project Overview for Phase 2

• annotation automation
• ML auto build process / batch / schedule?

Need beginner installation instructions for Local Deployment

The journal submission requires that we make sure there are pretty detailed deployment instructions for someone who is at least familiar with the coding language we are using. I think they will only review the local deployment as AWS deployment is a little outside the scope of the journal. Could we go through the local deployment https://github.com/code4sac/trash-ai/blob/production/docs/localdev.md and make sure that someone starting from having nothing downloaded on their device could get this up and running there? Maybe this already has been done, if so let me know!

Create more helpful issue label system

Dependencies

• Elizabeth needs permission to create new labels/milestones within this repo (check with Brianda)

Overview

We need to create a more useful/informative labeling system for issues. This will allow existing and new contributors to have more direction/autonomy as they work on and complete issues.

Action Items

• Add labels for size: 1pt = less than 1 hour, 2pt = 1-3 hours, 3pt = 3-6 hours, 4pt = 6-12 hours, 5pt = 12+ hours (should be broken down into smaller issues!)
• Add labels for role: front end, back end, marketing, product, UI/UX, ML, leads
• Add labels for features: this will need some input... different enhancements?
• Get feedback on labels from other team members
• Create issue label guide for anyone who is new to the project or wants to create their own issues.

Resources/Instructions

Here's a link to how Hack for LA is already doing something like this:

• https://github.com/hackforla/website/issues (their website repo)
• https://github.com/hackforla/internship/issues (their product/research-heavy internship repo)

Add batch upload that doesn't display images

The page should be able to process at least 100 images at a time. After processing, the user should be presented with the metadata for the images. The browser should not display the processed images.

Review Manuscript

@wincowgerDEV would like to have the manuscript reviewed by 11/15/2022. If you have been assigned, please comment after you have reviewed the manuscript.

Default to about page

I think we should default trashai.org to open the about tab first (not the upload tab) so that people can't say they didn't see the information about the tool e.g. upload saving or that it wasn't obvious to them what the tool did.

Add CI/CD for deploying app to AWS

Use a CI/CD platform (preferably github actions?) to manage the deployment of the trash-ai web application to AWS when there is a commit on the main branch.

Build a better mobile experience

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

After uploading a file on mobile, it is not clear that the user must navigate to the hamburger menu in the top left of the app in order to access the upload page and the summary page. I think we can design a better user experience by making these options more explicit on the initial page that after you upload, these are the beneficial pages to navigate too in order to access your insights.

Problem

a

Proposed Solution

a

Alternatives Considered

a

Testing Local Deployment Issues

This comes from our coauthor Kris Haamer who tested out the local deployment on his mac and was able to get it to run but had a few challenges.

1. The local branch mentioned in the docs doesn't exist (I used the production branch) - or I guess the docs mean to say: create your own local branch
2. Maybe mention the need for a Google Maps API key in the docs (the compiler complains VITE_GOOGLE_MAPS_API_KEY is missing) - AWS Amplify seems to require it but not sure where and for what it's going to be used

[Feature]: consolidate deployment scripts

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

Reviewer 1 says: Even with the current structure, I'll also note that the deployment scripts and such could probably be simplified and consolidated a bit - when taking a brief look at it, it felt like I had to jump around a bit to find all the different bits and pieces and had a lot of different calls between different tools (eg, why use makefiles vs just having a single docker-compose in the root and use docker-compose up?).

Problem

Challenging deployment

Proposed Solution

Make deployment scripts easier to follow.

Alternatives Considered

NA

Update README with production workflow instructions DEV -> Staging -> Production

Add template code to respository

Add both front-end and back-end into project

Add a tool to annotate images

We'd like to improve the model over time.

One way we can do that is to have a way to look at images that have no annotations and be able to draw and label trash in the images in the browser. We could also have a way to modify existing bounding boxes and label.

Ideally we would also upload the image and the new annotations to an s3 bucket so that we can use that data for future model refinement, but this can be addressed in a future issue to reduce scope of this one. (if not addressed in this ticket, open a new one when this is closed)

Add meta repo files

We should document some of the following to encourage and guide contributions:

• Contributor file
• Issue creation templates and advice

[Feature]: Disable Classification piece (until wired in) that exists in staging and to enable new image layout

Guidelines

• I agree to follow this project's Contributing Guidelines.

Description

The new image layout is nice, but we don't have the classification piece wired in. Should be simple enough to make a switch to enable / disable classification.

Problem

The new image layout is nice, but we don't have the classification piece wired in. Should be simple enough to make a switch to enable / disable classification.

Proposed Solution

The new image layout is nice, but we don't have the classification piece wired in. Should be simple enough to make a switch to enable / disable classification.

Alternatives Considered

The new image layout is nice, but we don't have the classification piece wired in. Should be simple enough to make a switch to enable / disable classification.

Ensure merge requests to main squash commits

Incorporate TACO dataset labels into computer vision model

[Recruitment] Recruit someone with Machine Learning expertise for making model improvements.

• Tensorflow or Yolo V5 experience
• What we need them to be able to do WRT TrashAI's model:
  • be able to create way for the model to incorporate feedback (e.g., we need to be able to relabel incorrectly labeled trash and the model should be able to learn from this)
  • create new bounding boxes around trash that was missed (not labeled at all)
  • build some kind of interface for the user to provide this feedback (this might be a front-end responsibility)

From the recruitment standpoint:

• social media
• personal connections?

Add disclaimer that uploaded images and classifications are stored

Currently we store all uploaded images and their classifications so that we have data to improve the model over time. It should be clear as a user that this is the case.

Add google analytics tracking

We want to track how many people use TrashAI.

Drink Soda

[Bug]: References

Guidelines

• I agree to follow this project's Contributing Guidelines.

Project Version

No response

Platform and OS Version

No response

Existing Issues

No response

What happened?

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

• 10.5281/zenodo.4154370 is OK
• 10.48550/ARXIV.2003.06975 is OK

MISSING DOIs

• 10.1186/s43591-022-00035-1 may be a valid DOI for title: Trash Taxonomy Tool: harmonizing classification systems used to describe trash in environments
• 10.1186/s40965-018-0050-y may be a valid DOI for title: OpenLitterMap.com – Open Data on Plastic Pollution with Blockchain Rewards (Littercoin)

INVALID DOIs

• https://doi.org/10.1029/2019EA000960 is INVALID because of 'https://doi.org/' prefix
• https://doi.org/10.1016/j.wasman.2021.12.001 is INVALID because of 'https://doi.org/' prefix

Steps to reproduce

Automated JOSS Bot.

Expected behavior

DOI's to be valid.

Attachments

No response

Screenshots or Videos

No response

Additional Information

No response

[BUG] Non-www trashai.org does not resolve

Visting https://trashai.org/ does not resolve -- you have to hit the www version https://www.trashai.org/.

We should either redirect non-www to www OR host the app at non-www and redirect www to it.

Fill out "About" page in webapp

frontend/src/components/about.vue

About Tab Updates

Recommending some updates to the About tab:

About Trash AI
Welcome to Trash AI!
This is an open source project developed and maintained by Code For Sacramento in partnership with Win Cowger from The Moore Institute for Plastic Pollution Research and Walter Yu from CALTRANS. There have also been many contributors to the code base.

What is it?
Trash AI allows you to upload an image containing trash and get back data about the trash in the image, including the classification of trash and bounding box of where the trash is in the image.

How does it work?
Trash AI builds a model using the YoloV5 toolset trained on the TACO dataset. The model takes an image containing trash and returns a list of annotations and bounding boxes of trash within the image. The model is imported into the front-end Vue.js application where it is invoked when an image is uploaded. The Vue application then displayes the results of the model on the image.

How can I use Trash AI?
Trash AI is open source and free to use however you see fit. You may classify images and download the data. You may copy and modify the code for your own use. If you are using the tool for research purposes, we recommend you QAQC the output and we make no guarantee about the accuracy of the model for your specific images. Our overall goal is a generalizable AI, but there is still more work to do to get us there.

Disclaimer about uploaded images
The current version of Trash AI and the model we are using is just a start! When you upload an image, we are storing the image and the classification in an effort to expand the trash dataset and improve the model over time.

Reporting issues and improvements
If you would like to report an issue or request a feature, please open a Github Issue in our repository.

How to contribute
Open a Github Pull Request.

How to cite
We are working on a manuscript for the software, but in the meantime please cite as:
"Code for Sacramento, Trash AI, www.trashai.org, 2022.)

Design/UX pass

Now that we are getting close to a functional web application that can classify images, we should do a pass on the look and feel of the application to improve the user experience.

We can perhaps reach out to other local groups if we need help with this task.

[SPIKE] Explore alternative training frameworks

Currently we use YoloV5.

Can we try training the model on a different framework?

How does it impact accuracy?

Are there other ways to improve accuracy of the model (i.e. more epochs)?

Might it be useful to have a model selector when using TrashAI?

Acceptance Criteria:

• After exploring these models, document the findings in a readme file inside of the repo in the model folder. It would be great to see which models were tested, what the findings were, and ideally a way to reproduce the findings.

Set log group expire times to something reasonable