[Bug]: 无法重现LLama的example about spu

[anakinxc]

看起来用的 python 3.10? 试一下 3.8？

@Ye-D

[gystar]

我重新安装了一个3.8的环境，一样的错误 @anakinxc

[tpppppub]

这个错误是 easylm 转 llama 本身有点问题，建议换个环境重新再跑一下 easylm 转换的过程看看

[gystar]

@tpppppub 我用3.8的环境重新跑了一遍easylm还是这个问题

[gystar]

Traceback (most recent call last):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 442, in load_flax_weights
state = from_bytes(cls, state_f.read())
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/flax/serialization.py", line 425, in from_bytes
state_dict = msgpack_restore(encoded_bytes)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/flax/serialization.py", line 407, in msgpack_restore
state_dict = msgpack.unpackb(
File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
msgpack.exceptions.ExtraData: unpack(b) received extra data.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 446, in load_flax_weights
if f.read().startswith("version"):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "flax_llama7b/flax_llama7b.py", line 58, in
pretrained_model = FlaxLLaMAForCausalLM.from_pretrained(model_path, config=config)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 910, in from_pretrained
state = cls.load_flax_weights(resolved_archive_file)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 455, in load_flax_weights
raise EnvironmentError(f"Unable to convert {resolved_archive_file} to Flax deserializable object. ")
OSError: Unable to convert /xxx/projects/open_llama_7b/flax_model.msgpack to Flax deserializable object.

[gystar]

@tpppppub 重新跑的easylm：
python ./convert_hf_to_easylm.py --checkpoint_dir /home/xxx/projects/open_llama_7b/ --output_file /home/xxx/projects/open_llama_7b/flax_model.msgpack --model_size 7b --streaming
checkpoint_dir: /home/xxx/projects/open_llama_7b/
output_file: /home/xxx/projects/open_llama_7b/flax_model.msgpack
model_size: 7b
streaming: True
Start convert weight to easylm format...
Convert weight to easylm format finished...
Start to save...
Save finished!!! take time: 118.73943781852722 save path: /home/xxx/projects/open_llama_7b/flax_model.msgpack

[Ye-D]

@gystar streaming: True 改为False试一下

[gystar]

@Ye-D ，改了之后好像可以加载模型，但是会报错：
Run on SPU
Traceback (most recent call last):
File "flax_llama7b/flax_llama7b.py", line 179, in
outputs_ids = run_on_spu()
File "flax_llama7b/flax_llama7b.py", line 169, in run_on_spu
outputs_ids = ppd.device("SPU")(text_generation, copts=copts)(input_ids, params)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 656, in call
args, kwargs = self.device._place_arguments(*args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 429, in _place_arguments
return tree_map(place, (args, kwargs))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/jax/_src/tree_util.py", line 210, in tree_map
return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/jax/_src/tree_util.py", line 210, in
return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 427, in place
return Device.move(obj, self)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 450, in move
return move_fn(dest, obj)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1240, in PYU2SPU
share_objs = obj.device(make_shares)(
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 499, in call
self.device.node_client.run(server_fn, *args, **kwargs),
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in _call
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in rebuild_messages
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 540, in next
return self._next()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 966, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Socket closed", grpc_status:14, created_time:"2023-12-14T16:38:41.904485087+08:00"}"

这个错误的相关3pc.json的内容如下：
{
"id": "outsourcing.3pc",
"nodes": {
"node:0": "127.0.0.1:81920",
"node:1": "127.0.0.1:81921",
"node:2": "127.0.0.1:81922",
"node:3": "127.0.0.1:81923",
"node:4": "127.0.0.1:81924"
},
"devices": {
"SPU": {
"kind": "SPU",
"config": {
"node_ids": [
"node:0",
"node:1",
"node:2"
],
"spu_internal_addrs": [
"127.0.0.1:81930",
"127.0.0.1:81930",
"127.0.0.1:81930"
],
"runtime_config": {
"protocol": "ABY3",
"field": "FM64",
"enable_pphlo_profile": true,
"enable_hal_profile": true,
"fxp_exp_mode": 0,
"fxp_exp_iters": 5
}
}
},
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
}
}
}

因为这次使用的不是llama example里原始的配置，因此我改为原始配置，改之后pdd的init也不行了
Traceback (most recent call last):
File "./flax_llama7b.py", line 45, in
ppd.init(conf["nodes"], conf["devices"])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1110, in init
_CONTEXT = HostContext(nodes_def, devices_def)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1037, in init
self.devices[name] = SPU(
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 952, in init
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 952, in
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 242, in _call
raise Exception("remote exception", result)
Exception: ('remote exception', Exception('Traceback (most recent call last):\n File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 324, in Run\n ret_objs = fn(self, *args, **kwargs)\n File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 543, in builtin_spu_init\n link = libspu.link.create_brpc(desc, my_rank)\nRuntimeError: what: \n\t[external/yacl/yacl/link/transport/brpc_link.cc:131] Fail to initialize channel, host=127.0.0.1:81930, err_code=-1\nstacktrace: \n#0 yacl::link::FactoryBrpc::CreateContext()+0x7f264b2b922e\n#1 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7f2649e1a524\n#2 pybind11::cpp_function::dispatcher()+0x7f2649df989e\n#3 PyCFunction_Call+0x4f5572\n\n\n'))

[anakinxc]

"node:0": "127.0.0.1:81920",
"node:1": "127.0.0.1:81921",
"node:2": "127.0.0.1:81922",
"node:3": "127.0.0.1:81923",
"node:4": "127.0.0.1:81924"

这些已经不是有效端口了

[gystar]

@anakinxc @Ye-D
我修改了配置如下：
{
"id": "outsourcing.3pc",
"nodes": {
"node:0": "127.0.0.1:7920",
"node:1": "127.0.0.1:7921",
"node:2": "127.0.0.1:7922",
"node:3": "127.0.0.1:7923",
"node:4": "127.0.0.1:7924"
},
"devices": {
"SPU": {
"kind": "SPU",
"config": {
"node_ids": [
"node:0",
"node:1",
"node:2"
],
"spu_internal_addrs": [
"127.0.0.1:7930",
"127.0.0.1:7931",
"127.0.0.1:7932"
],
"runtime_config": {
"protocol": "ABY3",
"field": "FM64",
"enable_pphlo_profile": true,
"enable_hal_profile": true,
"fxp_exp_mode": 0,
"fxp_exp_iters": 5,
"experimental_enable_inter_op_par": true,
"experimental_enable_intra_op_par": true
}
}
},
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
}
}
}

然后大约运行了5分钟左右，报错：
Run on SPU
Traceback (most recent call last):
File "./flax_llama7b.py", line 179, in
outputs_ids = run_on_spu()
File "./flax_llama7b.py", line 169, in run_on_spu
outputs_ids = ppd.device("SPU")(text_generation, copts=copts)(input_ids, params)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 679, in call
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 679, in
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in _call
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in rebuild_messages
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 540, in next
return self._next()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 966, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-12-14T17:17:40.609971717+08:00", grpc_status:14, grpc_message:"Socket closed"}"

[Ye-D]

@gystar 可能3pc.json写的有点问题。应该是
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:3"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:4"
}
}

[gystar]

@Ye-D @anakinxc 感谢。 跑成功了，感觉Llama这个example还是有坑啊，提个pr吗

[anakinxc]

@gystar 欢迎提 pr 啊

[gystar]

@anakinxc 稍等两天忙完了再提