Vertical white column: usually means "install failed". That's because we can't run tests if the installer didn't complete, and test grid omits squares. Almost every "infrastructure" flake related to the process of CI will be in this category - if there is any green in the column, odds are the problem is either in the test or in the cluster, not in the CI cluster or the job itself (very rarely will this be something network related).
Rows with red interspersed with green: almost always a flaky test, but sometimes a core bug across. multiple tests. You can guesstimate the frequency by counting the red squares and then visually estimating how many runs showed up over that interval (the boxes are regular, so I usually just hold up two fingers for 20 results and then see how many fall into that. If you look at this page frequently, you can also narrow down the day when the flake was introduced because it'll start flaking at some point
Rows with solid red chunks: almost always a regression either in the test or the product. If you see it on test-grid, it usually means it's post merge, so either someone force merged (bad!) or the test behaves differently between PR and release jobs (for instance, auth and whether it's on quay).
Row with solid red chunk and white to the right: a new test was added that is failing when run in the release job. In the picture below, that's the storage test we added that worked in PR but didn't work against the older RHCoS image
Repeating vertical red bars: If you see a set of rows that all fail together on the same runs, that usually means a subsystem has a bug. Previously, we were seeing that on quota, so every 5-10 runs all of the quota tests would fail in a given run because the kube-apiserver stopped handling quota and all tests would fail.
Failure waterfall: If you see a meandering line moving from bottom to top, right to left, this almost always means "core control plane (cluster infra) flake during the run". This is because the sort order of the grid prioritizes failed runs, and so you can see different tests hitting the flake each time. e2e tests are run in random order, so the same test is unlikely to fail twice if the test runs at different times on each run. Also, in e2e we re-run up to a limited number of failures at the end of the test to see whether these are reproducible failures or just a flaky tests. If the test passes the second time we record it here (red square) but the run itself is allowed to pass if the limit is low.
In the picture below, the line is caused by the kube-apiserver doing a rolling restart after the e2e tests are started (which shouldn't be happening) but if graceful restart is working correctly, the tests shouldn't fail (the point of graceful restart is to drain all short lived requests before we stop accepting new connections). It impacts different tests each time, and some tests are more impacted than others.
-- Clayton