Bug: Race case in temporary directory creation about docker-tar-pusher

[karolyp]

Hi @a3957273 ,

Thanks for letting me know about your issue. May I ask you to provide the driver code you used (e.g. how you implemented parallel exeuction), so that I can reproduce the exact issue you encountered with?

Thank you!

[a3957273]

It's quite hard to reliably reproduce, due to the very quick speed of:

      const manifest = this.workDirUtils.readManifest();

I don't have a reliable script to reproduce. But logically, with two separate processes you could run into the following situation:

A: this.workDirUtils.createTempDir() (mkdir /tmp/dtp-)
B: this.workDirUtils.createTempDir() (mkdir /tmp/dtp-) # this is fine, recreating an existing folder is a no-op
A: this.workDirUtils.extract(this.config.tarball) 
B: this.workDirUtils.extract(this.config.tarball) # this is bad, as we've just overwritten `manifest.json`
A: const manifest = this.workDirUtils.readManifest() # !!! refers to the B manifest

Due to your use of sync, this is only an issue with our setup where we run multiple Node processes on the same host. It's possible to make this easier to reproduce by adding a sleep 10s into the readManifest() function.

[karolyp]

As far as I understood, this race condition would be correct, if we would have only one DockerTarPusher object whose pushToRegistry could be re-used. Since you have to instantiate a new DockerTarPusher every time you want to upload a new image, it will instantiate a new WorkDirUtils which will eventually solve this concurrency issue.

I tried to reproduce your issue with some images and I also added some extra info to the logs (see here). As you can see on the timestamps, all of them were running concurrently, without an issue.

Here is the driver code I used:

const tarballs = [...];
const dtpUploads = tarballs 
  .map((tarball) => getDTP(tarball))
  .map(
    (dtp) =>
      new Promise((res) => {
        dtp.pushToRegistry().then(res);
      })
  );
(async () => {
  await Promise.all(dtpUploads);
})();

[a3957273]

You have to be running in different processes as all your calls are synchronous, so your above example will actually be running them one at a time (despite them looking to be promises).

[a3957273]

const fs = require('fs')

// run:
//   node race.js & node race.js && fg

function synchronousWait(ms) {
  let start = Date.now()
  let now = start
  while (now - start < ms) {
    now = Date.now()
  }
}

function fake_workDirUtils_extract(name) {
  // fake opening a tar file and extracting the metadata.json
  fs.writeFileSync('./metadata.json', JSON.stringify({ name }))
}

function fake_workDirUtils_readManifest() {
  // make this function take some time
  synchronousWait(150)

  return JSON.parse(fs.readFileSync('./metadata.json'))
}

function main() {
  const name = String(Math.random())

  fake_workDirUtils_extract(name)
  const metadata = fake_workDirUtils_readManifest()

  console.assert(name === metadata.name, JSON.stringify({ name, metadataName: metadata.name }))
  console.assert(name !== metadata.name, 'Program saw no race case')
}

main()

Save this to a file race.js and run it with node race.js to see that it succeeds. Switch the main() call out to do many of them, e.g:

for (let i = 0; i < 20; i++) {
    new Promise(() => main())
}

// expected output:
// Assertion failed: Program saw no race case
// Assertion failed: Program saw no race case
// Assertion failed: Program saw no race case
// Assertion failed: Program saw no race case
// Assertion failed: Program saw no race case
// Assertion failed: Program saw no race case

And it still succeeds. Run multiple processes at once and you can reliably get a race case:

// go back to just running `main()`, instead of the loop.
node race.js & node race.js && fg

// expected output:
// [1] 30057
// Assertion failed: {"name":"0.3444892547688003","metadataName":"0.15235109223203924"}
// Assertion failed: Program saw no race case
// [1]+  Done                    node race.js

Observe that one thread didn't see the same file read as it wrote, because we're running multiple simultaneously. This doesn't happen even if you put them all inside promises inside one process because under the hood v8 still processes all synchronous jobs (as is the case with your readFileSync and extract({ sync: true })) one at a time.