wayveai / driving-with-llms Goto Github PK

Hi, I tried to install the requirement file as the author listed but it first raised the error of bitsandbytes not supporting GPU, I solved this by installing bitsandbytes==0.43.3. After that there is an error of 'Parameter' object has no attribute 'CB'. Does anyone have solution?

thanks for your great work!
I want to view the results after evaluation, where to find the WandB project "llm-driver"?

How to Use Multi-GPU Training

Can you provide the code and training command of the first stage: vector representation pre-training stage?

Hi， the error ‘requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/decapoda-research/llama-7b-hf/resolve/main/config.json’ occurred wen I run train.py . Whether or not I need add my username and password in the train.py script ?

Could you explain what the model's input and the corresponding labels are during the pre-training stage to help me better understand this process?

File "/home/~~/miniconda3/envs/drive_llm/lib/python3.8/site-packages/transformers/models/llama/modeling_llama.py", line 90, in forward
return self.weight * hidden_states.to(input_dtype)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Hi,Thanks for your great work! I would like to inquire, based on your experience and research, do you believe it is feasible to deploy this work onto real vehicles? Particularly, considering our current computational resources are 2 A800 GPUs (80G). In this scenario, how long do you think it might take us to achieve this goal? Are there any key technical challenges or issues that need to be addressed?

decapoda-research/llama-7b-hf is no longer accessible on HuggingFace。
I have a bug with “baffo32/decapoda-research-llama-7B-hf”：
ValueError: The device_map provided does not give any device for the following parameters: base_model.model.weighted_mask

Thank you for your very nice work!
I was trying to run the program and realized that the pre-trained model is not downloadable from hugging face
decapoda-research/llama-7b-hf/
Can you provide an alternative link for that model?

I'm grateful for your contributing work. Could you please introduce your setup like CUDA and cudnn? I have tried many times to run the code 'pip install -r requirements.txt.lock' but there are many different problems and errors.

Hi,

I see that the unspecified signals are not used for caption generation. But looks like they were still inputs for the full pipeline.

Would you be able to give the documentation about description of these unspecified signals ?

e.g.https://github.com/wayveai/Driving-with-LLMs/blob/main/utils/vector_utils.py#L111

Hello, your work is very exciting. First of all, I would like to know if your model contains a control signal output. Does the dataset contain a control signal? Second, if I want to replace the LLM with the pre-trained model of the first stage, how can I do it?

Hello, congrats on your work!

I was curious to know if you thought about including the real-world datasets as part of fine-tuning the LLM for decision making?

It is very related to your paper being ICRA 2024 cited, I hope you can have more and better results, but I don't seem to find the citations of this paper on the Internet, all of them are arxiv, do you have a citation in the bib file format of the ICRA2024，Thanks again！

How to tested in the real environment?

The inference process is currently quite slow. Are there any methods available to accelerate it?
For action task, it costs about 9s for a sample.

OSError: decapoda-research/llama-7b-hf is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

@olegsinavski @jltorresm Thanks for sharing the code base can we have the scripts for visualization code also to validate the results

Thanks in advance

Hello! Amazing work, really.
I was wondering whether this work and the consecutive one (LingoQA) are comparable and if it a real improvement of this work or just another approach.
Since it seems more of a driving scenario commentator and less as a control model that predicts future control signals.
Thank you!

Have you ever tested in real-world driving environment?

Hello, when I evaluate for Perception and Action Prediction, I got this error for decapoda-research/llama-7b-hf.
How can I fix this? Thanks!

@olegsinavski @melights @jltorresm @eltociear can u please share thr pretrained model ??