freesoft / detox_bot2

University of Illinois@Urbana-Champaign MCS-DS CS498 Cloud Computing Applications project

Hi there 👋

• 🔭 I work for Blizzard Entertainment, B&OP(old Battle.net), as a Senior Software Engineer.
• 🌱 Graduated from B.S in Computer Engineering at Dong-A University, South Korea
• 🌱 Graduated from Master's in Computer Science - Data Science(MCS-DS) at University of Illinois - Urbana Champaign
• 🌱 Current Georgia Tech's OMSCS grad program student as of August 2023, focusing on Interactive Intelligence.
• 💬 Ask me about volunteer chances for profit or non-profit projects, especially for eduTech.
• 📫 How to reach me: https://www.linkedin.com/in/wonheejung/

detox_bot2's Issues

[High] Current detox_bot uses MultinomialNB that allows out-of-core training ( means you can keep training it without starting over ) through partial_fit(). However, entire feature is already decided at the first then when training. I see there is a few way, including using word2vec on Keras or etc, and wondering if we can use it.

OR

If we can significantly boost the training speed, it doesn't matter that much. scikit-learn with MultinomialNB takes too much memory while it's running and it ends up using all my MacBook pro's memory so I had to partial run several times. Maybe we can improve it by just cherrypick those words to training and reduce the # of features.

[Low] Update TwitchBot (written in python) and make it runnable and deployable to multiple channel easily.

Current TwitchBot is just console command based, you give it a twitch channel name and run it. Would be great if we can somehow make it as web app or service and easily deploy to multiple TwitchTV channel, grab those chat information and show up what's toxic chat or not on web page instead of console.

submit the project proposal

This need to be done on both Coursera and Piazza. #4 needs to be done first.

Decide what progress we are going to make and address it to individual member(s)

Collects what we have done so far to prepare progress report which is due very soon.

write the project proposal

Clean up. some of the extra TODO and stuff from current markdown file, run some fancy decoration, generate PDF from it, and check-in to the repo.

Ask teammates for review before submission.

[High] Research if we can use Spark( + MLlib) and replace current scikit-learn based toxic chat classifier. Where to run it(AWS, GCP, etc?)

Research Kubernetes services for detox application

Research if we can run current detox_bot in Kubernetes cluster and can take more incoming request than 1 instance and How. ***Conrad is researching this ***

[HIigh] Writing final report and submit

Summarize what progress we made

Need opinion and things you've done so we. can add those items in progress report.
So far what I can remember is,

Implemented working prototype ( by copying existing detox_bot ... )
Investigated IaaS/PaaS for running the service on k8s. (need more detail from @harley3 if you have documentation for the investigation )
Made a progress and now prototype app runs on Google Cloud Platform's k8s cluster. (means it's dockerized as well )
Investigated Apache Spark MLlib and TensorFlow if it's good enough to replace current scikit-learn implementation and also scalable ( need more detail from @kevinmackie and @noya for this )

[Low] Adding more stop words. detox_bot uses stopwords which is pre-defined set of words plus a few custom words that I added. However, there are more lot of stopword that's recurring too much or pretty much useless when detecting toxic chat. I would like to decent amount of stop words in there ( not just a, the, this, that, etc ) so that we can expect better accuracy.

Depends on which ML library we are going to use(Spark+Mllib or Tensorflow?), those stopwords need to be regenerated and/or migrated as well as adding more stopwords in it.