Comments (14)
看起来用的 python 3.10? 试一下 3.8?
from spu.
我重新安装了一个3.8的环境,一样的错误 @anakinxc
from spu.
这个错误是 easylm 转 llama 本身有点问题,建议换个环境重新再跑一下 easylm 转换的过程看看
from spu.
@tpppppub 我用3.8的环境重新跑了一遍easylm还是这个问题
from spu.
Traceback (most recent call last):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 442, in load_flax_weights
state = from_bytes(cls, state_f.read())
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/flax/serialization.py", line 425, in from_bytes
state_dict = msgpack_restore(encoded_bytes)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/flax/serialization.py", line 407, in msgpack_restore
state_dict = msgpack.unpackb(
File "msgpack/_unpacker.pyx", line 201, in msgpack._cmsgpack.unpackb
msgpack.exceptions.ExtraData: unpack(b) received extra data.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 446, in load_flax_weights
if f.read().startswith("version"):
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "flax_llama7b/flax_llama7b.py", line 58, in
pretrained_model = FlaxLLaMAForCausalLM.from_pretrained(model_path, config=config)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 910, in from_pretrained
state = cls.load_flax_weights(resolved_archive_file)
File "/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 455, in load_flax_weights
raise EnvironmentError(f"Unable to convert {resolved_archive_file} to Flax deserializable object. ")
OSError: Unable to convert /xxx/projects/open_llama_7b/flax_model.msgpack to Flax deserializable object.
from spu.
@tpppppub 重新跑的easylm:
python ./convert_hf_to_easylm.py --checkpoint_dir /home/xxx/projects/open_llama_7b/ --output_file /home/xxx/projects/open_llama_7b/flax_model.msgpack --model_size 7b --streaming
checkpoint_dir: /home/xxx/projects/open_llama_7b/
output_file: /home/xxx/projects/open_llama_7b/flax_model.msgpack
model_size: 7b
streaming: True
Start convert weight to easylm format...
Convert weight to easylm format finished...
Start to save...
Save finished!!! take time: 118.73943781852722 save path: /home/xxx/projects/open_llama_7b/flax_model.msgpack
from spu.
@gystar streaming: True 改为False试一下
from spu.
@Ye-D ,改了之后好像可以加载模型,但是会报错:
Run on SPU
Traceback (most recent call last):
File "flax_llama7b/flax_llama7b.py", line 179, in
outputs_ids = run_on_spu()
File "flax_llama7b/flax_llama7b.py", line 169, in run_on_spu
outputs_ids = ppd.device("SPU")(text_generation, copts=copts)(input_ids, params)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 656, in call
args, kwargs = self.device._place_arguments(*args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 429, in _place_arguments
return tree_map(place, (args, kwargs))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/jax/_src/tree_util.py", line 210, in tree_map
return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/jax/_src/tree_util.py", line 210, in
return treedef.unflatten(f(*xs) for xs in zip(*all_leaves))
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 427, in place
return Device.move(obj, self)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 450, in move
return move_fn(dest, obj)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1240, in PYU2SPU
share_objs = obj.device(make_shares)(
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 499, in call
self.device.node_client.run(server_fn, *args, **kwargs),
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in _call
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in rebuild_messages
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 540, in next
return self._next()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 966, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Socket closed", grpc_status:14, created_time:"2023-12-14T16:38:41.904485087+08:00"}"
这个错误的相关3pc.json的内容如下:
{
"id": "outsourcing.3pc",
"nodes": {
"node:0": "127.0.0.1:81920",
"node:1": "127.0.0.1:81921",
"node:2": "127.0.0.1:81922",
"node:3": "127.0.0.1:81923",
"node:4": "127.0.0.1:81924"
},
"devices": {
"SPU": {
"kind": "SPU",
"config": {
"node_ids": [
"node:0",
"node:1",
"node:2"
],
"spu_internal_addrs": [
"127.0.0.1:81930",
"127.0.0.1:81930",
"127.0.0.1:81930"
],
"runtime_config": {
"protocol": "ABY3",
"field": "FM64",
"enable_pphlo_profile": true,
"enable_hal_profile": true,
"fxp_exp_mode": 0,
"fxp_exp_iters": 5
}
}
},
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
}
}
}
因为这次使用的不是llama example里原始的配置,因此我改为原始配置,改之后pdd的init也不行了
Traceback (most recent call last):
File "./flax_llama7b.py", line 45, in
ppd.init(conf["nodes"], conf["devices"])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1110, in init
_CONTEXT = HostContext(nodes_def, devices_def)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 1037, in init
self.devices[name] = SPU(
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 952, in init
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 952, in
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 242, in _call
raise Exception("remote exception", result)
Exception: ('remote exception', Exception('Traceback (most recent call last):\n File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 324, in Run\n ret_objs = fn(self, *args, **kwargs)\n File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 543, in builtin_spu_init\n link = libspu.link.create_brpc(desc, my_rank)\nRuntimeError: what: \n\t[external/yacl/yacl/link/transport/brpc_link.cc:131] Fail to initialize channel, host=127.0.0.1:81930, err_code=-1\nstacktrace: \n#0 yacl::link::FactoryBrpc::CreateContext()+0x7f264b2b922e\n#1 pybind11::cpp_function::initialize<>()::{lambda()#3}::_FUN()+0x7f2649e1a524\n#2 pybind11::cpp_function::dispatcher()+0x7f2649df989e\n#3 PyCFunction_Call+0x4f5572\n\n\n'))
from spu.
"node:0": "127.0.0.1:81920",
"node:1": "127.0.0.1:81921",
"node:2": "127.0.0.1:81922",
"node:3": "127.0.0.1:81923",
"node:4": "127.0.0.1:81924"
这些已经不是有效端口了
from spu.
@anakinxc @Ye-D
我修改了配置如下:
{
"id": "outsourcing.3pc",
"nodes": {
"node:0": "127.0.0.1:7920",
"node:1": "127.0.0.1:7921",
"node:2": "127.0.0.1:7922",
"node:3": "127.0.0.1:7923",
"node:4": "127.0.0.1:7924"
},
"devices": {
"SPU": {
"kind": "SPU",
"config": {
"node_ids": [
"node:0",
"node:1",
"node:2"
],
"spu_internal_addrs": [
"127.0.0.1:7930",
"127.0.0.1:7931",
"127.0.0.1:7932"
],
"runtime_config": {
"protocol": "ABY3",
"field": "FM64",
"enable_pphlo_profile": true,
"enable_hal_profile": true,
"fxp_exp_mode": 0,
"fxp_exp_iters": 5,
"experimental_enable_inter_op_par": true,
"experimental_enable_intra_op_par": true
}
}
},
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:0"
}
}
}
}
然后大约运行了5分钟左右,报错:
Run on SPU
Traceback (most recent call last):
File "./flax_llama7b.py", line 179, in
outputs_ids = run_on_spu()
File "./flax_llama7b.py", line 169, in run_on_spu
outputs_ids = ppd.device("SPU")(text_generation, copts=copts)(input_ids, params)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 679, in call
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 679, in
results = [future.result() for future in futures]
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 249, in run
return self._call(self._stub.Run, fn, *args, **kwargs)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in _call
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in rebuild_messages
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 218, in
return b''.join([msg for msg in msgs])
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/spu/utils/distributed.py", line 238, in
rsp_data = rebuild_messages(rsp_itr.data for rsp_itr in rsp_gen)
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 540, in next
return self._next()
File "/home/xxx/miniconda3/envs/spu_py38/lib/python3.8/site-packages/grpc/_channel.py", line 966, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Socket closed"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-12-14T17:17:40.609971717+08:00", grpc_status:14, grpc_message:"Socket closed"}"
from spu.
@gystar 可能3pc.json写的有点问题。应该是
"P1": {
"kind": "PYU",
"config": {
"node_id": "node:3"
}
},
"P2": {
"kind": "PYU",
"config": {
"node_id": "node:4"
}
}
from spu.
@Ye-D @anakinxc 感谢。 跑成功了,感觉Llama这个example还是有坑啊,提个pr吗
from spu.
@gystar 欢迎提 pr 啊
from spu.
@anakinxc 稍等两天忙完了再提
from spu.
Related Issues (20)
- How to use SPU to evaluate private models in 2PC setting with only one machine? HOT 16
- [Bug]: One more minus sign HOT 2
- [Bug]: Package 'examples/python/ml/my_custom_file' contains errors HOT 10
- [Bug]: The critical condition judgment is wrong HOT 5
- [Bug]: Error when trying to benchmark SPU latency in 2PC setting. HOT 3
- another case where secret indexing doesn't seem to work HOT 2
- [Bug]: 8x communication compared to reported in Cheetah HOT 5
- [Operation Question] How to separate truncation and matmul operations HOT 9
- [Question]: Are there any files building correspondence between the kernels and their dispatching functions? HOT 3
- [Bug]: bitintl_b in ab_api.cc is wrong HOT 3
- [Question]: The number of convolutional multiplication decreases but the communication cost increases in SPU HOT 4
- [Bug]: gRPC Socket Shutting Down After Many Runs HOT 7
- [Bug]: gcc 11.2下的编译问题 HOT 12
- [Question]: 能否不重复编译外部库,加速编译速度? HOT 4
- [Question]: stub_method方法实现将函数交由server执行,请问如何调试server中函数的具体执行过程? HOT 4
- [Question]: Common type of Ashare and Bshare HOT 1
- [Question]: Evaluating the model with significantly reduced number of ReLUs receives limited communication savings HOT 17
- [Question]: Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.0KiB (rounded to 3072) HOT 9
- [Usability]: 运行LLaMa时报错'Config' object has no attribute 'define_bool_state' HOT 10
- 我想请问一下bumblebee的复现配置 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spu.