ExecuteUserScriptError during training about deepracer-core

[crr0004]

You should see the direct error further up in the log. Python should have printed a stack trace. The current setup is still untested in longer form training as I just haven't had time to use it. I will be doing a lot of this weekend and I'll be fixing any issues.

…

On Thu., 6 Jun. 2019, 9:54 pm sctse999, ***@***.***> wrote: Training lasted for 5x minutes and throw a ExecuteUserScriptError algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=73, Total reward=127.31, Steps=5404, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=74, Total reward=124.84, Steps=5571, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=75, Total reward=106.79, Steps=5748, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=76, Total reward=59.27, Steps=5833, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=77, Total reward=74.12, Steps=5928, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=78, Total reward=8.67, Steps=5960, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=79, Total reward=11.85, Steps=6000, Training iteration=3 algo-1-71gs6_1 | Training> Name=main_level/agent, Worker=0, Episode=80, Total reward=10.17, Steps=6030, Training iteration=3 2019-06-06 10:44:23,909 sagemaker-containers ERROR ExecuteUserScriptError: algo-1-71gs6_1 | Command "/usr/bin/python training_worker.py --RLCOACH_PRESET deepracer --aws_region us-east-1 --loss_type mean squared error --model_metadata_s3_key s3://bucket/custom_files/model_metadata.json --s3_bucket bucket --s3_prefix rl-deepracer-sagemaker" I encountered this error on my mac and also ubuntu. Anyone encountered the same issue? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#14>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPNBILTXBQBJGRDYVYURX3PZD3F5ANCNFSM4HVBG3DA> .

[sctse999]

interestingly the log from the console of sagemaker stopped there without further output. Any idea I can set anything to get a more verbose output?

p.s.: Thanks a lot for sharing this project

[sctse999]

I am suspecting the script is terminated once it get passed 60 minutes. I just can't find the parameter that control this.

[crr0004]

There is a method which controls exiting in the simulation, I don't recall if it's also checked in the sagemaker part.

    def is_training_done(self):
        if ((self.target_number_of_episodes > 0) and (self.target_number_of_episodes == self.episodes)) or \
           ((self.is_number(self.target_reward_score)) and (self.target_reward_score <= self.reward_in_episode)):
            self.is_simulation_done = True
        return self.is_simulation_done

Is there anything further up in the sagemaker container output? You can try using tee to capture the log and see it at the same time

[Nicolas-Kuhl]

I'm getting the same error whenever it finishes 20 episodes and sagemaker needs to update the model - can't find any error message beyond the one above.

Were you are able to get any more info on the issue? Anywhere else I can try to get logs from?

[sctse999]

I am not sure why but the issue is gone.
I did two things:

1. I moved to an ubuntu desktop
2. I am using the latest code of this repo (probably the updated docker image is the reason)

[Nicolas-Kuhl]

Ah, I've been using it on mac - might have to try ubuntu instead.

Oddly enough I've switched from the Tokyo track to the re:Invent and don't seem to be getting the same problem.

[joezen777]

I'm using the longer track. Had the same problems on my mac pro at home. On my other computer I made sure to update the robomaker.env with the correct local IP Address. (not the 127 one but the one I get from the router.) I'm thinking this could've been it? I'll verify when I get home. Makes sense that the error happens when it tries to talk back to sagemaker to start the training epochs.

[Nicolas-Kuhl]

In my case it's not the IP address (have that part done already) - I can get it to work now when training a new model, but whenever starting a job using an existing model checkpoint (i.e. continuing to train an existing) it does this on the first model update... the lack of error message is making this hard to track down!

[joezen777]

I trudged through every theory from permissions to virus scan when I got home. I ended up deleting all my folders and tried starting from scratch. Now I'm getting errors about tempfiles in robo/containers not existing.

[crr0004]

There could be any reason for the error. That error message is just saying that python encountered an error. There is normally a stack trace further up.

[joezen777]

Beats me what's causing my error now. Looks like it's a temp folder creation issue but it's got all the permissions it needs. I still have everything working fine on one computer though.

S03734-MBPR:rl_coach josephn$ ipython rl_deepracer_coach_robomaker.py 
Looking for config file: /Users/josephn/.sagemaker/config.yaml
Model checkpoints and other metadata will be stored at: s3://bucket/rl-deepracer-sagemaker
Uploading to s3://bucket/rl-deepracer-sagemaker
WARNING:sagemaker:Parameter `image_name` is specified, `toolkit`, `toolkit_version`, `framework` are going to be ignored when choosing the image.
s3.ServiceResource()
Using provided s3_client
INFO:sagemaker:Creating training-job with name: rl-deepracer-sagemaker
Starting training job
...

FileNotFoundError: [Errno 2] No such file or directory: '/Users/josephn/Documents/deepracer/robo/container/tmpz16mdufb'`

[crr0004]

That directory is configured in the ~/. sagemaker/config.yaml file. It's so it doesn't try to create them in weird temp directories. Have a look at the FAQ, #2. It's a different error but same cause. I'll add the error message to the FAQ when I get a chance.

…

[joezen777]

Yeah the error was not enough sleep. I had created "containers" instead of "container". And fortunately I am happy to say that everything is working after deleting all repo folders and then re-running through the steps. The only thing that I didn't test before removing and re-setting up, was if some of my changes to the hyperparameters had screwed things up. I'll also say that it took a long time before the policy training started (couple minutes) and it ran the epochs much slower than my other computer. I changed my cpus for docker from 8 to 6 after this and that made VNC run a little better but little else. But either way it's a success. Thanks again. If you're ever in NY before AWS NY Summit, hit me up and I'll show you the Slalom office in the new World Trade Center. (I'm building a custom track in there as well.)

[Nicolas-Kuhl]

Finally got round to sorting mine out - seems that increasing the max memory \ cpu in docker preferences fixed it - not getting that particular error anymore.

[crr0004]

@joezen777 @Nicolas-Kuhl are we happy to keep this closed? I am closing this and if we need to re-open it I can