Javascript data accurate? about githut

[madnight]

I think there are two reasons. One is the rise of typescript, which mostly gets it's user base from javascript. And the second thing is that I started filtering events like Pull Request from bot accounts like dependabot. There's a massive amounts of bot events in the JS world.

[derwaldgeist]

It would be nice if one could see the numbers of JavaScript and TypeScript combined in the graphics somehow, as it is basically the same ecosystem.

[jamesmurdza]

@madnight That makes sense. My hunch is that the change in counting method is the main difference. By other metrics, JavaScript activity is not declining that suddenly.

I guess there is no way to retroactively get trend data?

[madnight]

The Problem is that I haven't yet removed all bot accounts for the whole history, because that requires quite a bit of BigQuery query volume. I think as soon as I apply it to the whole history the trend data should be correct again.

[madnight]

We also have a problem with pushes regarding Python.

The following filter:

WHERE NOT LOWER(actor.login) LIKE "%bot%")

is not good enough, because there are scripted actors that generate huge amount of pushes without "bot" in their name. I updated the query again and try a new approach. Any individual actor.login (github user) can only generate 1000 pushes per quarter per language, otherwise it will not be counted for the this statistic, since we are interested in language popularity among the 99.9% user base and not in the popularity of pushes of bots and scripts.

But for this month I've used all my quota:

Error while querying the BigQuery Google API Error: Quota exceeded: Your project exceeded quota for free query bytes scanned. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas

[madnight]

Okay, I hade enough quota this time and was able to fix the issue.

Before:

After:

The data is now much more accurate in terms of reflecting popularity, since it does not take bots into account. As we can see JS Pull Requests have been inflated for quite some times, due to bots like dependabot.