Can't create project from big JSON file about google-refine

[GoogleCodeExporter]

Original comment by iainsproat on 17 Nov 2010 at 3:51

• Changed state: Accepted
• Added labels: Component-Logic

[GoogleCodeExporter]

How much physical memory do you have on your system?  You need to have enough 
to hold the entire database in memory, plus leave some room left over for 
Refine code, system processes, I/O buffers, etc.

I haven't looked at how the JSON importer is structured, but you may need X * 
original file size + Y * result data size, which both X & Y are > 1.  ie for a 
30 MB json file resulting in, say, 15 MB of data, I could easily see you 
needing 100 MB or more.  Having said that, 100 MB isn't very much these days 
and if it's actually needing more than 500 MB or so, then something definitely 
needs optimization.

Original comment by tfmorris on 17 Nov 2010 at 6:05

[GoogleCodeExporter]

3 GB of physical memory. Even tried with all programs closed.

Original comment by [email protected] on 17 Nov 2010 at 6:15

[GoogleCodeExporter]

That should be plenty for a data file of this size.  How big did you make your 
heap?  I'd increase it to at least 1 GB (-Xmx1024m) and you can probably go to 
1.5 or 2 GB without crowding the rest of your system.

Any chance you can provide the full file to test with?

Original comment by tfmorris on 17 Nov 2010 at 6:25