memoryerror with numpy about pygta5

[CT83]

My PC Specifications:
Lower-Mid Spec PC
AMD A8-6600K APU with Radeon(tm) HD Graphics 3.90 GHz
8.0 GB DDR3 1333 MHz
WD5000AZRX HDD

Problem: MemoryError while saving Pickle, same as OP @marius851000
Solution: Changed line 96 if len(training_data) == 500: to if len(training_data) == 200:
Result: Pickle Saved Successfully

Issue has been resolved! @Sentdex
Thanks to @germanmarky

[germanmarky]

@marius851000 can you give us more info?

• How much RAM does your machine have?
• Are you resizing the image?
• Are they RGB images?
• How much free space do you have on the drive you are saving them?

I see that there is already a file saved, was that captured with no changes to 1. collect_data.py?

[frossaren]

Have you changed where it should save the training data?

[marius851000]

1. I have 3,9 Go of ram

2. I have used the script without modification

3. I have approximatively 300 Go of free space

4. Yes, I've alrealy lauched the script once, It write a file of 60o, but it have also say an error

5. I've havent changed the default place of the training data

[frossaren]

You should change where the training data is supposed to be saved. That is probably the reason for the error

[CT83]

Having the same issue, almost the same configuration. I got my resources free, RAM 8 GB, 3 gigs free. Unable to record any training data. Any solutions? Did the OP find solution to his problem?
Btw, I am saving my training data, in the directory where the script is located, my D drive. Do I need to move it to C, for some reason?

[germanmarky]

@CT83, when running the capture script run Task Manager. If your RAM gets to about 80-90% allocated during the save process, the ndarray is to large to save. Make sure to close out all other unneeded applications that are RAM intensive which can be checked in Task Manager as well.

What you can do is change the max size of the array from 500 to 300 or even less. The actual size is arbitrary, you can increase or decrease the max size to fit the resources better in your system.

For example, I have 32GB of RAM and can capture a 2GB output file before worrying about running out of memory.

[CT83]

Oh Thanks. Yes, @germanmarky, I was researching the problem myself later and I thought of the same thing. Stackoverflow suggested using python64, I think I should switch to it for performance. I haven't looked over the training script yet. Do I need to make any changes to it? Btw what do you think about Hickle?

[germanmarky]

I attempted to use HDF5 for saving and loading as it's much more memory efficient but had issues matching up the containers on load and validating that the data was formatted correctly so just stayed with np.save().

I am also using Anaconda 4.1 64-bit and python 3.5 for my virtual env.

If you don't have the headroom in RAM to save the file, even 64-bit python won't help, but definitely won't hurt. I would start by dropping line 96 to 200 or 100 and keeping an eye on your RAM usage. You could also set it to 100 and comment out line 99 and see at what amount it actually fails, then just use that number.

[CT83]

Yes, I will try that, report here. Write a proper answer with my actual specs and what value I used. And we can close this Issue. 👍🏽 Hoping that @marius851000 is okay with it. 😇

Actually if anybody has another idea to solve this problem, because even by changing it to 100 doesn't help me, maybe by switching the append with something else ?
With 8GB DDR4, i can't go further than 15000 frames

[birdrecluse]

I had the same problem, thinking maybe it's because of data size I tried to save all at once is too big. Therefore I made it that it saves every 2000 frames, then it worked 50% of the time, meaning it could save successfully one time and would fail another time.
And then I just made changes to my virtual memory settings, adjusting it to be 2GB instead of 1GB, and then it finally worked perfectly fine.
You can find the settings for virtual memory in Computer attribute -> Advance System Settings -> Advance tab -> Performance -> Advance tab