Potential race condition? about marian-dev

[afaji]

addition:
after some tests with 2 GPUs, I got:

with stream sync : Ep. 151 Up. 4500 : Sen. 256 : Cost 74.35
without sync : Ep. 151 : Up. 4500 : Sen. 256 : Cost 106.45

[emjotde]

This is some toymodel? I was training with normal models and did not see any difference from single gpu model. And wouldn't that synchronize automatically as copyFrom is called from stream 0 and the following updates as well? I'll check with real models.

[emjotde]

There is a cudaStreamSynchronize(0) in copyFrom. So that can't be it. Also removing does not seem to change anything for me. Most probably because of the effect I described in the previous message.

[afaji]

because you use single GPU, (there is if-statement to handle single GPU). Try with more than one

yes, there is synchronize in copyFrom, but not in parameter update.

[emjotde]

I was testing with 3 GPUs, but misunderstood which update you mean. OK, let's check again.

[afaji]

Try this

void pushGradients(Tensor newGrads) {
      if(graphs_.size() < 2) {
        opt_->update(graphs_[0]);
      }
      else {
        std::lock_guard<std::mutex> guard(sync_);
        // add instead of copy?
        grads_->copyFrom(newGrads);
        opt_->update(params_, grads_);
      }
    }

versus this

   void pushGradients(Tensor newGrads) {
      if(graphs_.size() < 2) {
        opt_->update(graphs_[0]);
      }
      else {
        std::lock_guard<std::mutex> guard(sync_);
        // add instead of copy?
        grads_->copyFrom(newGrads);
        opt_->update(params_, grads_);
        cudaStreamSynchronize(0);
      }
    }

Also make sure use more than 1 GPU because the if-statement above.

Edit: the code messed up, not sure how to fix them
mjd: three apostrophes

[emjotde]

OK, I think you are right. It also seems that the speed of your implementation matches now my implementation without sharding. To fix this issue I will probably just pull your pull request.

[afaji]

have you check the non-sharding speed with synchronize? I believe that might slow it down a bit.

I still need to verify the correctness of my PR in terms of convergence.

[emjotde]

I will post stats in a moment under the pull request