smerkyg / gptcore Goto Github PK
View Code? Open in Web Editor NEWFast modular code to create and train cutting edge LLMs
License: Apache License 2.0
Fast modular code to create and train cutting edge LLMs
License: Apache License 2.0
Hello, can you add the deepseek-moe component?
https://arxiv.org/abs/2401.06066
I want to train Mixture-of-Expert RWKV as well as Mixture-of-Expert gptalpha.
will this work (albeit slowly) on cpu-only systems?
or is cuda a hard requirement?
So I can get about 10min into training before it crashes, its consistant for multiple attempts, internet connect is 600mbit.
'(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')), '(Request ID: fd98ddf
2-4690-4322-b018-8226d93d2e7d)')' thrown while requesting GET https://huggingface.co/datasets/monology/pile-uncopyrighted/resolve/main/train/14.jsonl.zst
Can I train a model on 7900XT 4 cards?
I am getting the following error when trying the train a model following the readme.
Traceback (most recent call last):
File "/usr/lib/python3.10/pydoc.py", line 443, in safeimport
module = import(path)
File "/content/gptcore/dataset/init.py", line 35, in
class PipedDatasetWrapper(typing.Generic[T_co], torch.utils.data.datapipes.datapipe.IterDataPipe[T_co]):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/datapipes/_typing.py", line 373, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/datapipes/_typing.py", line 260, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/lib/python3.10/abc.py", line 106, in new
cls = super().new(mcls, name, bases, namespace, **kwargs)
TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/gptcore/util/config.py", line 479, in process
located = locate(fullid, Missing)
File "/content/gptcore/util/locate.py", line 57, in locate
nextmodule = pydoc.safeimport('.'.join(parts[:n+1]), forceload)
File "/usr/lib/python3.10/pydoc.py", line 458, in safeimport
raise ErrorDuringImport(path, sys.exc_info())
pydoc.ErrorDuringImport: problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/content/gptcore/cli.py", line 176, in
cli()
File "/content/gptcore/cli.py", line 89, in cli
disk_cfg = util.config.eval_first_expr(disk_cfg_str, macros)
File "/content/gptcore/util/config.py", line 630, in eval_first_expr
return ConfigParser().eval_first_expr(unparsed_input, incoming_macros)
File "/content/gptcore/util/config.py", line 399, in eval_first_expr
return self.process(node.value)
File "/content/gptcore/util/config.py", line 552, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=True)
File "/content/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/content/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/content/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/content/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/content/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/content/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/content/gptcore/util/config.py", line 607, in create_factory
func_ident = self.process(func_node)
File "/content/gptcore/util/config.py", line 560, in process
raise ConfigParseError(node, self.unparsed_input, msg="Internal exception during configuration parsing " + str(e))
util.config.ConfigParseError: Internal exception during configuration parsing problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe at line 70, col 36
datamodule_factory=lambda: dataset.DM(
^^^^^^^^^^^
Here is a google colab where its reproducable is.
7900x4:~/gptcore$ python cli.py train -c configs/gptalpha.cfg.py
Traceback (most recent call last):
File "/usr/lib/python3.10/pydoc.py", line 443, in safeimport
module = import(path)
File "/home/user/gptcore/dataset/init.py", line 35, in
class PipedDatasetWrapper(typing.Generic[T_co], torch.utils.data.datapipes.datapipe.IterDataPipe[T_co]):
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_typing.py", line 373, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/home/user/.local/lib/python3.10/site-packages/torch/utils/data/datapipes/_typing.py", line 260, in new
return super().new(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
File "/usr/lib/python3.10/abc.py", line 106, in new
cls = super().new(mcls, name, bases, namespace, **kwargs)
TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/gptcore/util/config.py", line 479, in process
located = locate(fullid, Missing)
File "/home/user/gptcore/util/locate.py", line 57, in locate
nextmodule = pydoc.safeimport('.'.join(parts[:n+1]), forceload)
File "/usr/lib/python3.10/pydoc.py", line 458, in safeimport
raise ErrorDuringImport(path, sys.exc_info())
pydoc.ErrorDuringImport: problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/gptcore/cli.py", line 176, in
cli()
File "/home/user/gptcore/cli.py", line 89, in cli
disk_cfg = util.config.eval_first_expr(disk_cfg_str, macros)
File "/home/user/gptcore/util/config.py", line 630, in eval_first_expr
return ConfigParser().eval_first_expr(unparsed_input, incoming_macros)
File "/home/user/gptcore/util/config.py", line 399, in eval_first_expr
return self.process(node.value)
File "/home/user/gptcore/util/config.py", line 552, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=True)
File "/home/user/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/home/user/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/home/user/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/home/user/gptcore/util/config.py", line 618, in create_factory
positional_placeholders_count, placeholders, args, kwargs = self.process_args_and_keywords(node_args=node_args, node_keywords=node_keywords)
File "/home/user/gptcore/util/config.py", line 588, in process_args_and_keywords
value = self.process(kw.value)
File "/home/user/gptcore/util/config.py", line 522, in process
rv = self.create_factory(node, node.args, node.keywords, immediate=False)
File "/home/user/gptcore/util/config.py", line 607, in create_factory
func_ident = self.process(func_node)
File "/home/user/gptcore/util/config.py", line 560, in process
raise ConfigParseError(node, self.unparsed_input, msg="Internal exception during configuration parsing " + str(e))
util.config.ConfigParseError: Internal exception during configuration parsing problem in dataset - TypeError: Cannot create a consistent method resolution
order (MRO) for bases Generic, IterDataPipe at line 70, col 36
datamodule_factory=lambda: dataset.DM(
^^^^^^^^^^^
It would be good to have a section in the top level read.me on initializing a new pre-train model of various common sizes (up to 70B) for each architecture.
ie:
For RWKV to initialize a pre-train model of size 70B set flags:
example.py --n_embd XXXXX etc
For transformer to initialize a pre-train model of size 70B set flags:
example2.py --some_flag etc
Do this for each arch at 3B, 7B, 34B, 70B (or just a few small and large examples each)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.