wasp-lang / wasp-bot Goto Github PK

This seems to have happened a couple times so far, here are a few instances that I detected:

• Only events for last day were fetched (~100 of them): https://discord.com/channels/686873244791210014/835130205928030279/1209016517518430268 .
• Only some events were fetched (~3000 of them): https://discord.com/channels/686873244791210014/835130205928030279/1208291741686562826 .
• Only some events were fetched (~ 3000 of them): https://discord.com/channels/686873244791210014/835130205928030279/1205030251285979147 .

It must be connected to fetching events from PostHog and how we cache them.
Maybe these are instances when fetching them from PostHog failed and then when we resumed fetching them, things went wrong (although it should work fine and continue from the last downloaded events)? Or maybe the mistakes is completely on the PostHog side (I wouldn't expect it, but we are using their old API which maybe they are not testing well)?

Although we have detection for CI events and are handling them separately in our analytics, it seems we still get some of those come through without picking them.
Last case we had was a bunch of builds on one day coming from DigitalOcean droplet.

Here are things we can do to improve our detection:

1. If there are multiple events from same IP in a short time span, we can "group" them together and look at it as if it is one event -> maybe easiest to just have a step where we delete the redundant ones. We need to be careful about build vs non-build events here, we should not mix those. And I am not sure how to determine the time span -> maybe best to go for a day? I am not sure if we can do this in linear complexity though, this might be quite slow hm, have to give it a try. But this would also work retroactively which is good.
2. Make our logic for checking if in CI stronger.
   Here is simple suggestion by ChatGPT:

  travis <- lookupEnv "TRAVIS"
  jenkins <- lookupEnv "JENKINS_HOME"
  githubActions <- lookupEnv "GITHUB_ACTIONS"
  gitlabCI <- lookupEnv "CI_SERVER"
  aws <- lookupEnv "AWS_EXECUTION_ENV"
  gcp <- lookupEnv "GOOGLE_CLOUD_PROJECT"
  azure <- lookupEnv "AZURE_PIPELINE_ID"
  digitalOcean <- lookupEnv "DOCKER_HOST"

so let's see if we are not checking some of these vars and we can add checks for them if we don't have them.

We are using quite old version, we should update it, so we get support for threads also, but quite bit changed in the versions of discord.js so this update will take some time and reading a Changelog.

Right now we cache fetched posthog events on disk, in a .json file.
Problem is, in production, when deployed to Heroku, this file is lost whenever a dyno (Heroku instance) is restarted, which can happen quite often on Heroku, possibly even daily (Heroku restarts dynos often to keep them "healthy").

Solution is to store the file somewhere on persistent storage -> most likely on S3.

Since this PR got merged https://github.com/gitpod-io/template-wasp/pull/1/files , we should now be able to differentiate between activity coming from Gitpod and the rest of the activity.

Since on Gitpod each new workspace is considered as a new project / user, we want to track those activities separately, so they don't bloat our metrics.

We should filter by WASP_TELEMETRY_CONTEXT: gitpod when doing main analytics, but then also report gitpod analytics separately.

In wasp-lang/wasp#1586 we added new "NON_INTERACTIVE_SHELL" value to our telemetry. We think it should directly correspond with if Wasp is running in CI or not, but we are not yet 100% sure, so we want to first see how it behaves in wild.

We should look at which events in PostHog have this value, and if it indeed corresponds to what I wrote above. If that behaves as it should, then we should start using it in our analytics here in wasp-bot to categorize all events that have this value in their context to be recognized as CI events!

When calculating age for users from previuos periods (week/month/...), we calc their age at the present moment, but instead we should be calculating their age as it was at that period of time (I guess at the end of it?).

So let's say I am doing report for last 4 weeks. When calculating the age of unique users in week num 2, I should calc their age by doing the difference between the last event in week num 2 and their first event they ever had, not between the last even they ever had and first event they ever had.