[App]: Deleting some media files, crashes the panels about wandb

[kkoutini]

Hi, I fixed the issue by keeping the last two steps. Apparently, the problem was that some of the runs crashed before uploading the last step. Thanks again for the help I'll close the issue.

[ArtsiomWB]

Hey @kkoutini , Thank you for writing in!

Could you please provide a link to a workspace where you are experiencing this behavior?

[ArtsiomWB]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[kkoutini]

Hi, Thank you, it seems to be working now. I feel like something was cached, which caused the failures. Most of the runs are working now, but here is one example I found that is still not working, although the media files from the last steps still exist.

[ArtsiomWB]

Thank you for the update! Could you please send me a link to a run where it is working for you now - for me to compare both of the workspaces?

[kkoutini]

Hi thank you!
Here is a run that works for me, I can choose the steps 429 and 430, these are the only steps that still have the audio files, and I can play them in the webapp.
PS: Sometimes, I may need to edit the panels to choose Columns=Example index in advanced to get the steps slider, to be able to slide it to the end.

Here is also a minimal code snippet for what I do to delete the intermediate files:

files_gt = defaultdict(list)
for file in  run.files():
   if file.name.endswith(extension) and "gt" in file.name:
      f = file.name.rsplit("/",1)[-1] # get the file name
      k= int(f.rsplit("_",2)[-2])  # this should be  analogous to the step num
      files_gt[k].append(file)
for k in sorted(files_gt)[:-1]: # exclude the last step
   file_to_delete.extend(files_gt[k]) # add all the other indices and steps to the delete list

[ArtsiomWB]

Thank you for providing the code snippet above.
When do you delete your files, as well as how often do you log them? Do you log them every wandb step?

Do you delete your files when the training is over and when wandb.finish() has been called, or do you do it in the middle of the experiment?

[ArtsiomWB]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[kkoutini]

Hi, I log the files after each epoch but I don't set a step, during the epoch I log other metrics. Therefore, I think it's not every wandb step.
After the traning is done and wandb.finish is called. I do some anlysis and in some cases, I don't need the files anymore (in most cases). I connect to wandb api and clean the intermediate files and leave only the last logged files. I use the code posted above.
I don't know which was the last step so sort by the file name and delete all but the last step.
Thanks for the help!