Improve flaky test reporting in Sentry about rspecq

[glampr]

Makes sense thanks @agis.
I will create a new issue to track this per your suggestion.

[kpelelis]

I will work on this one

[kpelelis]

Since this topic is open ended, we can discuss different ideas about it. To answer the questions, I think we need to leverage the way Sentry groups reports. In my opinion, it would be better if we could group reports per file, and each time a flaky job is raised, append it to the event list.

1. when was this particular flaky job introduced?

Check the timestamp of the first event

2. which file has the most flaky jobs?

Sort by events

3. Which are the files currently that contain flaky jobs?

Unresolved events

[glampr]

It would be nice to also include instructions on how to reproduce the execution order that lead to the error.

[agis]

Since this topic is open ended, we can discuss different ideas about it. To answer the questions, I think we need to leverage the way Sentry groups reports. In my opinion, it would be better if we could group reports per file, and each time a flaky job is raised, append it to the event list.

1. when was this particular flaky job introduced?

Check the timestamp of the first event

1. which file has the most flaky jobs?

Sort by events

1. Which are the files currently that contain flaky jobs?

Unresolved events

@kpelelis "Reports per file" is the most straightforward and simple thing to do and I believe provides some immediate benefits (everything you mentioned). Sentry uses what it calls "fingerprint" to decide how to group events. Depending on the default behavior, we might need to explicitly change the fingerprint to achieve this, or the event title change might be sufficient.

It would be nice to also include instructions on how to reproduce the execution order that lead to the error.

@glampr We could do that, but that could potentially be a list of 500 spec files that were executed in the same worker, prior to the flaky one, and that would be a huge payload to submit to Sentry. We could perhaps submit the N (5-10) jobs that run prior to the flaky one, as a best-effort approach. That said, this requires some effort and it's not straightforward to implement, so I suggest to track it in a new issue.

[glampr]

If I understand correctly, we don't need all possible combinations that lead to the error, for most cases one would be sufficient, to be able to reproduce the error locally and fix it.

[agis]

@glampr RSpecQ workers run in a loop, continuously popping tests of the queue and executing them, until the queue is empty. As a result, prior to encountering a flaky test, a worker might have executed a lot of other jobs (i.e. spec files), and you can't know which one is the one that caused the flakiness (if at all).

So we'd have to keep a list of all the jobs that the worker executed prior executing the flaky one, during the current build.

One thing we could also do, but this too is not so straightforward, is to emit the RSpec seed to Sentry. We could do these in next iterations.