Error while initializing multiple models about deepspeed

[1Konny]

Hi all, sorry for the late response.

this initialization issue has been fixed in my case as I commented. I set dist_init_required=True when initializing the first model engine, and set dist_init_required=False for the rest. Now it works like a charm.

I modified my code as below and the issue does not appear now, but I wonder this is fine.

training_data = load_dataset()
encoder_params = filter(lambda p: p.requires_grad, encoder.parameters())
decoder_params = filter(lambda p: p.requires_grad, decoder.parameters())
self.encoder, self.encoder_optim, train_loader, _ = deepspeed.initialize(opt, model=encoder, model_parameters=encoder_params, training_data=training_data, dist_init_required=True)
self.decoder, self.decoder_optim, _, _ = deepspeed.initialize(opt, model=decoder, model_parameters=decoder_params, dist_init_required=False)

when I set dist_init_requires=False in all deepspeed.initialize() calls, raise RuntimeError("trying to initialize the default process group " occurs since I conjecture there is no any other deepspeed.initialize() calls in my codes.

[1Konny]

I modified my code as below and the issue does not appear now, but I wonder this is fine.

training_data = load_dataset()
encoder_params = filter(lambda p: p.requires_grad, encoder.parameters())
decoder_params = filter(lambda p: p.requires_grad, decoder.parameters())
self.encoder, self.encoder_optim, train_loader, _ = deepspeed.initialize(opt, model=encoder, model_parameters=encoder_params, training_data=training_data, dist_init_required=True)
self.decoder, self.decoder_optim, _, _ = deepspeed.initialize(opt, model=decoder, model_parameters=decoder_params, dist_init_required=False)

Also, following this tutorial, I changed my training code as below, which also throws an error: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

loss = calculate_loss(output, target)
self.encoder.backward(loss)
self.decoder.backward(loss)
self.encoder.step()
self.decoder.step()

I couldn't find a way to set an option like retrain_graph=True, so now I changed my code as below and do not see any errors, but still wonder if this is the right way to use deepspeed.

self.encoder.zero_grad()
self.decoder.zero_grad()
loss = calculate_loss(output, target)
loss.backward()
self.encoder.step()
self.decoder.step()

So I would greatly appreciate if you could provide us a tutorial when there are multiple models to be trained. Thanks!

[tjruwase]

raise RuntimeError("trying to initialize the default process group "

RuntimeError: trying to initialize the default process group twice!

This error message suggests to me that nccl initialization had occurred earlier in your code before the first call to deepspeed.initialize(). Can you confirm that? If this is the case, can you try setting dist_init_required=False in all the deepspeed.initialize() calls?

[jeffra]

Hi @1Konny, have you tried anything further with this issue?