[Q] Offline currently running jobs shows as "Finished" in the UI, am I doing it wrong? about wandb

[anmolmann]

Hi @ankile, this does seem like a bug on our end where a live run is being marked as finished upon being synced. We did fix this issue a few years ago and it seems like this is a regression. Here's the condition in our code which handles this, we'll investigate further internally and identify the root cause.

I was able to reproduce this with the following script as well:

import wandb
import time
import numpy as np
import pandas as pd

run = wandb.init(project="<project_name>")
for i in range(0, 100):
    table = wandb.Table(data=pd.DataFrame(np.random.randint(0, 1001, size=(3, 3)),
                                          columns=['a', 'b', 'c']),
                        columns=['a', 'b', 'c'])
    wandb.log({"table": table})
    time.sleep(5)

wandb.finish()

On a side note, W&B recommends syncing an offline run once it finishes (after run.finish() is called) to avoid running into any error states. The reason is if one tries to sync an active run, the config of the run and some other files are not up-to-date at the point of syncing and only partial data is uploaded. You should see a warning such as:

 WARNING .wandb file is incomplete (record checksum is invalid, data may be corrupt), be sure to sync this run again once it's finished

in your console when syncing an active run.