Missing dict.txt? about metaseq

[stephenroller]

To unblock this, here's the dict.txt file. Put it in the same folder as the .pt files.

https://gist.github.com/stephenroller/fbc74423445091531aa6b0452f5efaa2

[stephenroller]

Thanks, Stephen! That definitely gets me farther. Now I'm stuck on the next issue:

  File "/home/hlang/Megatron-LM/megatron/mpu/layers.py", line 190, in __init__
    self.tensor_model_parallel_size = get_tensor_model_parallel_world_size()
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 258, in get_tensor_model_parallel_world_size
    return torch.distributed.get_world_size(group=get_tensor_model_parallel_group())
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 215, in get_tensor_model_parallel_group
    assert _TENSOR_MODEL_PARALLEL_GROUP is not None, \
AssertionError: intra_layer_model parallel group is not initialized

Which seems like the same problem Patrick is running in to here (I'm also trying with the smallest model for now).
But closing this issue. Thanks for the help.

HI Hunter, were you able to solve this problem? I have been getting the same error warning. If so, would you mind sharing how you solved it? Thanks

It should be that you need to set --model-parallel 8

[hunterlang]

Thanks, Stephen! That definitely gets me farther. Now I'm stuck on the next issue:

  File "/home/hlang/Megatron-LM/megatron/mpu/layers.py", line 190, in __init__
    self.tensor_model_parallel_size = get_tensor_model_parallel_world_size()
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 258, in get_tensor_model_parallel_world_size
    return torch.distributed.get_world_size(group=get_tensor_model_parallel_group())
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 215, in get_tensor_model_parallel_group
    assert _TENSOR_MODEL_PARALLEL_GROUP is not None, \
AssertionError: intra_layer_model parallel group is not initialized

Which seems like the same problem Patrick is running in to here (I'm also trying with the smallest model for now).

But closing this issue. Thanks for the help.

[stephenroller]

You need to specify --model-parallel N based on the settings of the particular model. (2 for 30B, 8 for 175B.)

[ParadoxZW]

About the problem dict.txt, I found that when point LOCAL_SSD to a different path with that of MODEL_SHARED_FOLDER, the program will automatically generate a dict.txt in the folder of LOCAL_SSD. But this would not happen if LOCAL_SSD = None or LOCAL_SSD = MODEL_SHARED_FOLDER. Is this strange behavior a bug?

[guialfaro053]

Thanks, Stephen! That definitely gets me farther. Now I'm stuck on the next issue:

  File "/home/hlang/Megatron-LM/megatron/mpu/layers.py", line 190, in __init__
    self.tensor_model_parallel_size = get_tensor_model_parallel_world_size()
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 258, in get_tensor_model_parallel_world_size
    return torch.distributed.get_world_size(group=get_tensor_model_parallel_group())
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 215, in get_tensor_model_parallel_group
    assert _TENSOR_MODEL_PARALLEL_GROUP is not None, \
AssertionError: intra_layer_model parallel group is not initialized

Which seems like the same problem Patrick is running in to here (I'm also trying with the smallest model for now).

But closing this issue. Thanks for the help.

HI Hunter, were you able to solve this problem? I have been getting the same error warning. If so, would you mind sharing how you solved it? Thanks

[aarush7]

Thanks, Stephen! That definitely gets me farther. Now I'm stuck on the next issue:

  File "/home/hlang/Megatron-LM/megatron/mpu/layers.py", line 190, in __init__
    self.tensor_model_parallel_size = get_tensor_model_parallel_world_size()
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 258, in get_tensor_model_parallel_world_size
    return torch.distributed.get_world_size(group=get_tensor_model_parallel_group())
  File "/home/hlang/Megatron-LM/megatron/mpu/initialize.py", line 215, in get_tensor_model_parallel_group
    assert _TENSOR_MODEL_PARALLEL_GROUP is not None, \
AssertionError: intra_layer_model parallel group is not initialized

Which seems like the same problem Patrick is running in to here (I'm also trying with the smallest model for now).
But closing this issue. Thanks for the help.

HI Hunter, were you able to solve this problem? I have been getting the same error warning. If so, would you mind sharing how you solved it? Thanks

It should be that you need to set --model-parallel 8

Where should we set this parameter?

[Dev-hestabit]

@hunterlang, @aarush7 , @stephenroller , @guialfaro053, @ParadoxZW hey, can you able to solve this issue?
i am trying to load 1.3B model but getting (AssertionError: intra_layer_model parallel group is not initialized) this error

[guialfaro053]

@hunterlang, @aarush7 , @stephenroller , @guialfaro053, @ParadoxZW hey, can you able to solve this issue?
i am trying to load 1.3B model but getting (AssertionError: intra_layer_model parallel group is not initialized) this error

I actually managed to run it after create a new conda env and installing everything again.
I remember I had some conflicts with the Apex library and Fairseq

[Dev-hestabit]

i have tried creating conda env two times
can you please suggest some steps that you follow to create your conda env if you remember.

[Dev-hestabit]

actually i am trying to run it on single GPU, if that is the issue then please suggest some solution.

[guialfaro053]

I installed all needed libraries from here. If there is an error while installing them, you won't be able to run BB3. It can take a while to install everything.
Also, when I ran BB3 30B (it's a big model), I used two (A100 40Gb) GPUs so I really doubt you can run it with one single GPU without high specs.

[Dev-hestabit]

Actually i am trying to load 2.7B or 6.7B and i think it will work on A10G 24GB
i have followed the same setup guide but it is not working for me

[xyjigsaw]

Any update? I encounter the same problem.