smerkyg / gptcore Goto Github PK

Hello, can you add the deepseek-moe component?
https://arxiv.org/abs/2401.06066
I want to train Mixture-of-Expert RWKV as well as Mixture-of-Expert gptalpha.

will this work (albeit slowly) on cpu-only systems?
or is cuda a hard requirement?

So I can get about 10min into training before it crashes, its consistant for multiple attempts, internet connect is 600mbit.

1. Is there a way to cache the files in advance or add auth to huggingface?
2. Would I need to download the the 800gb pile dataset as a fix?

'(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: fd98ddf
2-4690-4322-b018-8226d93d2e7d)')' thrown while requesting GET https://huggingface.co/datasets/monology/pile-uncopyrighted/resolve/main/train/14.jsonl.zst 

Can I train a model on 7900XT 4 cards?

I am getting the following error when trying the train a model following the readme.

Traceback (most recent call last):
File "/usr/lib/python3.10/pydoc.py", line 443, in safeimport
module = import(path)
File "/content/gptcore/dataset/init.py", line 35, in
class PipedDatasetWrapper(typing.Generic[T_co], torch.utils.data.datapipes.datapipe.IterDataPipe[T_co]):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/datapipes/_typing.py", line 373, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/datapipes/_typing.py", line 260, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/lib/python3.10/abc.py", line 106, in new
cls = super().new(mcls, name, bases, namespace, **kwargs)
TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/gptcore/util/config.py", line 479, in process
located = locate(fullid, Missing)
File "/content/gptcore/util/locate.py", line 57, in locate
nextmodule = pydoc.safeimport('.'.join(parts[:n+1]), forceload)
File "/usr/lib/python3.10/pydoc.py", line 458, in safeimport
raise ErrorDuringImport(path, sys.exc_info())
pydoc.ErrorDuringImport: problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/gptcore/cli.py", line 176, in
cli()
File "/content/gptcore/cli.py", line 89, in cli
disk_cfg = util.config.eval_first_expr(disk_cfg_str, macros)
File "/content/gptcore/util/config.py", line 630, in eval_first_expr
return ConfigParser().eval_first_expr(unparsed_input, incoming_macros)
File "/content/gptcore/util/config.py", line 399, in eval_first_expr
return self.process(node.value)
File "/content/gptcore/util/config.py", line 552, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=True)
File "/content/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/content/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/content/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/content/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/content/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/content/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/content/gptcore/util/config.py", line 607, in create_factory
func_ident = self.process(func_node)
File "/content/gptcore/util/config.py", line 560, in process
raise ConfigParseError(node, self.unparsed_input, msg="Internal exception during configuration parsing " + str(e))
util.config.ConfigParseError: Internal exception during configuration parsing problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe at line 70, col 36
datamodule_factory=lambda: dataset.DM(
^^^^^^^^^^^

Here is a google colab where its reproducable is.

7900x4:~/gptcore$ python cli.py train -c configs/gptalpha.cfg.py
Traceback (most recent call last):
File "/usr/lib/python3.10/pydoc.py", line 443, in safeimport
module = import(path)
File "/home/user/gptcore/dataset/init.py", line 35, in
class PipedDatasetWrapper(typing.Generic[T_co], torch.utils.data.datapipes.datapipe.IterDataPipe[T_co]):
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_typing.py", line 373, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_typing.py", line 260, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/lib/python3.10/abc.py", line 106, in new
cls = super().new(mcls, name, bases, namespace, **kwargs)
TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/gptcore/util/config.py", line 479, in process
located = locate(fullid, Missing)
File "/home/user/gptcore/util/locate.py", line 57, in locate
nextmodule = pydoc.safeimport('.'.join(parts[:n+1]), forceload)
File "/usr/lib/python3.10/pydoc.py", line 458, in safeimport
raise ErrorDuringImport(path, sys.exc_info())
pydoc.ErrorDuringImport: problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/user/gptcore/cli.py", line 176, in
cli()
File "/home/user/gptcore/cli.py", line 89, in cli
disk_cfg = util.config.eval_first_expr(disk_cfg_str, macros)
File "/home/user/gptcore/util/config.py", line 630, in eval_first_expr
return ConfigParser().eval_first_expr(unparsed_input, incoming_macros)
File "/home/user/gptcore/util/config.py", line 399, in eval_first_expr
return self.process(node.value)
File "/home/user/gptcore/util/config.py", line 552, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=True)
File "/home/user/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/home/user/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/home/user/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/home/user/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/home/user/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/home/user/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/home/user/gptcore/util/config.py", line 607, in create_factory
func_ident = self.process(func_node)
File "/home/user/gptcore/util/config.py", line 560, in process
raise ConfigParseError(node, self.unparsed_input, msg="Internal exception during configuration parsing " + str(e))
util.config.ConfigParseError: Internal exception during configuration parsing problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe at line 70, col 36
datamodule_factory=lambda: dataset.DM(
^^^^^^^^^^^

It would be good to have a section in the top level read.me on initializing a new pre-train model of various common sizes (up to 70B) for each architecture.

ie:
For RWKV to initialize a pre-train model of size 70B set flags:
example.py --n_embd XXXXX etc
For transformer to initialize a pre-train model of size 70B set flags:
example2.py --some_flag etc

Do this for each arch at 3B, 7B, 34B, 70B (or just a few small and large examples each)