manimcommunity / manim-website-api Goto Github PK

We are about to deploy the initial version of the API which currently provides data for the awesome-manim feed on the website (corresponding PR for the website ManimCommunity/manim-website#73)

The scraper currently:

• fetches the list of all YouTube channel links from the README file
• scrapes the publicly available RSS feeds that are available from https://www.youtube.com/feeds/videos.xml?channel_id=xxxxxxx.
• searches for the substrings Manim (case insensitive), #some (case insensitive), SoME (case sensitive) in the video title or description. When found, the videos are marked as "being manim videos".
• The scraper then puts them on a MySQL database and serves videos chronologically on a paginated endpoint /videos/n, 30 videos at a time.

This issue records some ideas we could implement in the future based on feedback.

• A deeper scrape of all the channels (RSS feeds just return the latest 15 videos)
• An algorithmic feed that prioritizes videos with higher engagement, but still retains the chronological ordering to some degree
• ...

Feel free to discuss these and propose any other ideas.

I think this app can be rewritten using a serverless architecture using cloud functions (and all) while also using a hosted version of MySQL (I got to know about planetscale recently). For updating the videos we can use GitHub Actions and modify the DB.

Also, we could apply for some sponsorship, say from Vercel, if we are exceeding their free limits, see https://vercel.com/guides/can-vercel-sponsor-my-open-source-project. Though first, we would need to decide on the stack we are going to use.

It would be great to hear some thoughts about this.

Why?

Currently, this app is hosted in Oracle Cloud's free tier. Since we are only using the free tier sometimes the resource is automatically deallocated which is making things really difficult. The app was down for a few days (not sure how long) this month and I haven't noticed it until I was trying to do regular server maintenance. It was quite tedious setting up the server from scratch again (though most of it is automated with scripts).

Doing some analysis with logs I have from the server (from the 5th of this month), there are around 6k requests and only 2.5k of them are legit requests. Others seem to be spam or other tools trying to do something. All these can be avoided if we were using serverless architecture.

Either we should host it elsewhere, or rewrite this using a completely different architecture. That's why I created this issue to discuss.

cc @prism0x @behackl

Without cache headers, the browser will request each time the page reloads, which isn't really expensive. Add corresponding cache headers to avoid this.

api.manim.community is down.