Loki logs are missing in some regions about catalyst

[emranemran]

Searching on Loki using a search like this shows no logs coming in:

{namespace="canary", container="catalyst", region="nyc"}

Even this shows nothing:

{region="lon"}

This seems to be something that broke in the last 2-3 days.

[emranemran]

@hjpotter92 and I just reran some tests and logs in various regions like lax for example are still missing.
We just tried running:
{namespace="canary", container="catalyst", region="lax"} on loki at ~10AM PST and we're getting absolutely nothing. However region=fra works.

cc @victorges

[victorges]

@emranemran @hjpotter92 I think what Eli did to fix some of these was restarting the monitoring pods in the clusters. Have you tried doing that already?

I'm not sure what's the root cause for this tho, but restarting should hopefully fix once it happens.

[emranemran]

Most regions are good after @ya7ya's fix but sao/mdw are still missing logs.