Comments (16)
from deepracer-core.
interestingly the log from the console of sagemaker stopped there without further output. Any idea I can set anything to get a more verbose output?
p.s.: Thanks a lot for sharing this project
from deepracer-core.
I am suspecting the script is terminated once it get passed 60 minutes. I just can't find the parameter that control this.
from deepracer-core.
There is a method which controls exiting in the simulation, I don't recall if it's also checked in the sagemaker part.
def is_training_done(self):
if ((self.target_number_of_episodes > 0) and (self.target_number_of_episodes == self.episodes)) or \
((self.is_number(self.target_reward_score)) and (self.target_reward_score <= self.reward_in_episode)):
self.is_simulation_done = True
return self.is_simulation_done
Is there anything further up in the sagemaker container output? You can try using tee
to capture the log and see it at the same time
from deepracer-core.
I'm getting the same error whenever it finishes 20 episodes and sagemaker needs to update the model - can't find any error message beyond the one above.
Were you are able to get any more info on the issue? Anywhere else I can try to get logs from?
from deepracer-core.
I am not sure why but the issue is gone.
I did two things:
- I moved to an ubuntu desktop
- I am using the latest code of this repo (probably the updated docker image is the reason)
from deepracer-core.
Ah, I've been using it on mac - might have to try ubuntu instead.
Oddly enough I've switched from the Tokyo track to the re:Invent and don't seem to be getting the same problem.
from deepracer-core.
I'm using the longer track. Had the same problems on my mac pro at home. On my other computer I made sure to update the robomaker.env with the correct local IP Address. (not the 127 one but the one I get from the router.) I'm thinking this could've been it? I'll verify when I get home. Makes sense that the error happens when it tries to talk back to sagemaker to start the training epochs.
from deepracer-core.
In my case it's not the IP address (have that part done already) - I can get it to work now when training a new model, but whenever starting a job using an existing model checkpoint (i.e. continuing to train an existing) it does this on the first model update... the lack of error message is making this hard to track down!
from deepracer-core.
I trudged through every theory from permissions to virus scan when I got home. I ended up deleting all my folders and tried starting from scratch. Now I'm getting errors about tempfiles in robo/containers not existing.
from deepracer-core.
There could be any reason for the error. That error message is just saying that python encountered an error. There is normally a stack trace further up.
from deepracer-core.
Beats me what's causing my error now. Looks like it's a temp folder creation issue but it's got all the permissions it needs. I still have everything working fine on one computer though.
S03734-MBPR:rl_coach josephn$ ipython rl_deepracer_coach_robomaker.py
Looking for config file: /Users/josephn/.sagemaker/config.yaml
Model checkpoints and other metadata will be stored at: s3://bucket/rl-deepracer-sagemaker
Uploading to s3://bucket/rl-deepracer-sagemaker
WARNING:sagemaker:Parameter `image_name` is specified, `toolkit`, `toolkit_version`, `framework` are going to be ignored when choosing the image.
s3.ServiceResource()
Using provided s3_client
INFO:sagemaker:Creating training-job with name: rl-deepracer-sagemaker
Starting training job
...
FileNotFoundError: [Errno 2] No such file or directory: '/Users/josephn/Documents/deepracer/robo/container/tmpz16mdufb'`
from deepracer-core.
from deepracer-core.
Yeah the error was not enough sleep. I had created "containers" instead of "container". And fortunately I am happy to say that everything is working after deleting all repo folders and then re-running through the steps. The only thing that I didn't test before removing and re-setting up, was if some of my changes to the hyperparameters had screwed things up. I'll also say that it took a long time before the policy training started (couple minutes) and it ran the epochs much slower than my other computer. I changed my cpus for docker from 8 to 6 after this and that made VNC run a little better but little else. But either way it's a success. Thanks again. If you're ever in NY before AWS NY Summit, hit me up and I'll show you the Slalom office in the new World Trade Center. (I'm building a custom track in there as well.)
from deepracer-core.
Finally got round to sorting mine out - seems that increasing the max memory \ cpu in docker preferences fixed it - not getting that particular error anymore.
from deepracer-core.
@joezen777 @Nicolas-Kuhl are we happy to keep this closed? I am closing this and if we need to re-open it I can
from deepracer-core.
Related Issues (20)
- rl_coach startup error (ContextualVersionConflict) HOT 1
- Clone fails HOT 3
- Can't get running on GCP nvidia since this commit https://github.com/crr0004/deepracer/commit/9ba1fa2fc54336b2a9a99b32df7183933e275bfc4336b2a9a99b32df7183933e275bfc HOT 5
- Expecting checkpoint >= 1. Waiting. | After one lap in the simulator HOT 2
- The access key ID you provided does not exist in our records. HOT 2
- No such file or directory: 'DeepRacer/robo/container/tmp95am0kpa' HOT 2
- agent9 process has died. Initiating shutdown! HOT 1
- Updated sagemaker nvidia image does not work HOT 1
- Training stops due to "botocore.exceptions.ClientError: An error occurred (RequestTimeTooSkewed)" HOT 1
- error gathering device information while adding custom device "/dev/kfd" HOT 1
- Failure in robomaker on new install on Ubuntu 18.04 LTS HOT 1
- unexpected 's3_client' HOT 1
- S3UploadFailedError HOT 1
- ModuleNotFoundError: No module named 'rospy' HOT 1
- Evaluation job status: Failed HOT 3
- how do i confirm that my geforce 1070 gtx gpu is being used? HOT 1
- Stuck at ## Creating graph - name: BasicRLGraphManager on robomaker in MacOS HOT 1
- How to download pretrained model from AWS to local machine and clone. HOT 1
- TypeError: __init__() got an unexpected keyword argument 's3_client' HOT 1
- Model load failed Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepracer-core.