nagasaki45 / krihelinator

"Trendiness of open source software should be assessed by contribution rate, not by stars" - Meir Kriheli

The Krihelinator

⚠️ This project reached its end-of-life! The info here is only for documentation. The project was shut down on April 2020. On July 2020 the krihelinator.xyz domain will stop working and this page will only be available on nagasaki45.github.io/krihelinator.

About

"Trendiness of open source software should be assessed by contribution rate, not by stars"

- Meir Kriheli

This project proposes an alternative to github's trending page, by exposing projects with high contribution rate, instead of daily stars. The krihelimeter of each repository is calculated using the commits, pull requests, and issues of that project, from the past week (based on github's pulse page).

Krihelimeter =	20	* authors +
	8	* merged and proposed pull requests +
	8	* new and closed issues +
	1	* commits

During the development of this project I found out that people use github as a backup service, automating hundreds of commits per week. Therefor, to filter these projects out, only projects with more than one author enters the Krihelinator DB.

Drop me a line at [email protected] if you do somethig interesting with this project. Will be happy to hear about it and might be able to help.

About the shutdown

• On January 2019 github changed the way repo's pulse page is loading. Instead of having the entire HTML available at once, some info was fetched in subsequent calls. Specifically, the number of commits and authors were missing from the pulse page. This broke the calculation of the krihelimeter. Note that although the krihelimeter calculation was now different than the one intended, the information presented on the krihelinator was still relevant because it affected all projects / languages in the same way.
• By the end of March 2020 github started to block scrapers like the krihelinator, returning HTTP error code 429 (Too Many Requests).
• On early April 2020 I've decided to shutdown the project. The krihelinator.xyz domain now points to a page saying that the project is down. Links to badges return an end-of-life (EOL) badge.
• On July 2020 the domain shut down. The end-of-life remained available on nagasaki45.github.io/krihelinator.

Similar projects

• GitHut and GitHut 2
• IsItMaintained
• GitHub profiler

krihelinator's Issues

Use redis to persist the poller state between restarts

Filter out projects with number of authors below threshold

Expose the krihelimeter calculation to frontend

Automate deployment

Measure load with loader.io

• Should I need to add nginx with caching?
• Should I use cloudflare?

Move most of the "about" page to the homepage

Add krihelimeter badge to README

Better krihelimeter calculation

Take the DB, apply PCA to 1 dimension, use coefficients.

Find a way to filter out uninteresting projects

Examples:

• https://github.com/mostafa-z/Gabriel_MM
• https://github.com/linux-scraping/linux-grsecurity
• https://github.com/masahir0y/linux-yamada

Maybe by filtering out projects with 0 forks.

Fix commits scraper to handle comma (numbers larger than 1000)

Save the krihelimeter in the model

Add a task that update the krihelimeter on every row.

The DBCleaner should facilitate ecto / DB to do queries and deletes

Improve the docs

• Document the Periodic process.
• Document the DB structure (GithubRepo / Language / LanguageHistory).
• Document the web routes / controllers (PageController / BadgeController endpoints).
• Embed README.
• Hide modules that are not interesting.

Add "trending" to the model. Mark repos scraped by the trending poller as trending and present on frontend

New feature: Most trending language

Find a way to penalize these non interesting projects on the top of the list

The poller fail repeatedly on timeout when trying to scrape github page with server error (status code 500)

I started to hit an error on github, which stopped the polling completely. The following page returns a server error (status code 500) after ~10 seconds:
https://github.com/cobr123/cobr123.github.com/pulse
Therefore, the poller times out on poolboy timeout, and repeat, forever:

12:23:44.192 [info] Starting to poll from 'repositories?since=5108814'
12:23:44.196 [error] Task #PID<0.13194.0> started from #PID<0.357.0> terminating
** (stop) exited in: GenServer.call(#PID<0.394.0>, {:process, %{description: "", name: "cobr123/cobr123.github.com"}}, 5000)
    ** (EXIT) time out
    (elixir) lib/gen_server.ex:604: GenServer.call/3
    (poolboy) src/poolboy.erl:76: :poolboy.transaction/3
    (elixir) lib/enum.ex:657: anonymous fn/3 in Enum.each/2
    (elixir) lib/enum.ex:1628: anonymous fn/3 in Enum.reduce/3
    (elixir) lib/stream.ex:1264: anonymous fn/3 in Enumerable.Stream.reduce/3
    (elixir) lib/enum.ex:2843: Enumerable.List.reduce/3
    (elixir) lib/stream.ex:1284: Enumerable.Stream.do_each/4
    (elixir) lib/enum.ex:1627: Enum.reduce/3
Function: &Krihelinator.Background.Poller.process_repos/1
    Args: [:url_from_stash]
12:23:48.266 [warn] Failed to scrape cobr123/cobr123.github.com: %HTTPoison.Error{id: nil, reason: :timeout}

I must separate the scrapers from the poller, so scraper failures will pass almost silently. It might be fixed by #19.

Meanwhile, try to enter the process and set another URL in the PollerStash.

All deletes should be done by the DataHandler

Make the DBCleaner call the DataHandler to perform the deletes

If I got a redirect the periodic process fails

Handle redirects as :dmca_takedown - delete the repo.

Add the repo language to the model

Present the "updated_at" field

Refactor the background system to use GenStage based pipeline

Add support for badges

I think it would be nice to be able to have your own badge, which would also help to improve to grow the pool of processes to scale.
(As far as I understand those get pulled from the list on GitHub at the moment only?!)

Present the number of contributors

Depends on #15.

Nicer presentation of stats. More similar to pulse page

Dont render template on server side. Use API and vue.js from client side.

Depends on #37.

Use task pool for the scraper

Limit the output to 50 repos

Change the model to have only repo name instead of user/repo

A way to disable the language filter

Do not work on the repo directly in periodic nor in pipeline

Try to pass a tuple of {repo, params}, it will allow better utilization of the changeset later.

Implement a SelfPoller that will run over all repos in DB and update stats periodically

Fix leaping back to the beginning of the loop

Filter by language

Depends on #17.

Add JSON API

Gzip statics (a flag in endpoint.ex)

Allow the poller to fail by saving it's state and resume from last state

Remove vendor assets from repo. Manage them with npm

"fork me on github"

Allow line breaks in repo names. Mobile view is too wide

Update README

Memory leak causes the OS to kill the docker container

I found it while searching for the periodic kicks ins in the log. They are not there. Later, I found this. The kill is presented in /var/log/kern.log.

There is a chance that it's this known issue (which I'm not fully understand). If so, it might be fixed by bfb890d. It's already in production, so wait to see if the failure repeats.

Move the periodic process out of it's init call

A GenServer.init is called synchronously. Move the work outside by calling send_after in init.

Generate docs

Filter out forks

Write a script to clean them from the DB

Add about page

Add analytics

Present the repo language on frontend

Depends on #17.

Move aliases in the controller to web.ex

Add the number of contributors to the stats / calculation