[App]: Data lost for first several training steps about wandb

[ArtsiomWB]

Hi @YK-YoungK! Thank you for writing in.

Could you please send a link to your workspace where you are experiencing the first part of your graph not appearing in the UI?

[YK-YoungK]

Hi @YK-YoungK! Thank you for writing in.

Could you please send a link to your workspace where you are experiencing the first part of your graph not appearing in the UI?

Hi, here is the link: https://wandb.ai/yk-youngk/grok-mlp-default-2?workspace=user-yk-youngk.

[ArtsiomWB]

Two quick follow-up questions:

• could you please point me to the graph where you are seeing this concern? not having much luck locating it.
• Are you logging data every single wandb step?

[YK-YoungK]

Two quick follow-up questions:

• could you please point me to the graph where you are seeing this concern? not having much luck locating it.
• Are you logging data every single wandb step?

I see this concern in the figure train_accuracy, val_accuracy in project train_pct=70. Besides, if the epoch x is logged, then the next logging epoch is max(1.01x, x+1).

[ArtsiomWB]

Could you please add that figure to a report and then share it with me? Still no luck on my side.

I think what could be happening is our Sampling behavior. After we have logged >1500 points our chart sampling kicks in for performance reasons. If more than 1500 points are being displayed at once, some of the points get combined to prevent slowing down the UI. To confirm this what you can do is simply zoom in to the part of the graph that is missing in your UI and see if those points start appearing again.

[ArtsiomWB]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[YK-YoungK]

Could you please add that figure to a report and then share it with me? Still no luck on my side.

I think what could be happening is our Sampling behavior. After we have logged >1500 points our chart sampling kicks in for performance reasons. If more than 1500 points are being displayed at once, some of the points get combined to prevent slowing down the UI. To confirm this what you can do is simply zoom in to the part of the graph that is missing in your UI and see if those points start appearing again.

Hi, here is one report: https://api.wandb.ai/links/yk-youngk/0s97madx. Besides, when I zoom into the part of the missing curve the data seems to appear. Sorry for late response.

[ArtsiomWB]

Thank you so much for sending it over! It does look like sampling. Our engineers are aware of the issue and are looking into a fix. Thank you so much for bringing this up!