Can safe module avoid building an AST? about repy_v2

[choksi81]

Author: justinc
I'm starting to look at the memory issues described here. I'm using Zenodotus as a test application that may use a fair amount of memory...

Here is what we see in memory size given our current code (using the fork-then-safety-check model):

justincappos   10328 python repy.py restrictions.default zenodotus.mix -p 12345

By changing safe.safe_check(code) to call safe._check_ast(code) instead we see:

justincappos   21648 python repy.py restrictions.default zenodotus.mix -p 12345

[choksi81]

Author: justinc
Some more notes:

Disabling the check altogether has essentially the same memory use as our current code...

Calling parser.suite(code) has little impact on memory use...

The real memory use comes from this step: parser.st2tuple( parser.suite(code)). I've looked and this is a pure C module that hooks into the compiler. Unfortunately, there doesn't seem to be an easy way to read the elements from the parser one at a time (which is what we want).

For completeness, I've used strace to look at what the parser module does at this point and it seems normal. I'm going to look at the code now.

[choksi81]

Author: justinc
It isn't really possible to interpose with a legacy version of parser.so.

I'm almost out of ideas. About the only thing I think might work is to break the code up into separate pieces and check each separately. I need to think if this is sane, but it's clearly suboptimal. I would think that even just breaking it into a few pieces would have a huge impact. However, this wouldn't always be possible and would certainly be a pain. Is it worth the bother?

[choksi81]

Author: justinc
I've thought about this and the use of virtual namespaces to do separate imports will essentially do the same thing as I was saying in the last posting. We really should be able to remove this in Repy V2 once this is done. Let's retest then!

[choksi81]

Author: mkaplan
A major problem with safe_check (from a memory standpoint) was that it was using dictionary methods that create a copy of the dictionary it is iterating through. These include dict.keys() and dict.items(). This was especially problematic given the amount of recursive calls to _check_node().

I have only done minor benchmarking, but there seems to be a non-negligible difference in the memory used. Further benchmarking might be worthwhile, as it is possible that this was the cause of the large memory consumption.

These changes were implemented in r5700.

[choksi81]

Author: justinc
Replying to mkaplan:

I have only done minor benchmarking, but there seems to be a non-negligible difference in the memory used.

Can you do a quick test that quantifies this? My testing before (see comments above) showed that almost all of the memory cost comes from the parser.st2tuple call.

[choksi81]

Author: mkaplan
I was unfortunately unable to replicate the benefit that the initial testing had given me. It appears that I was incorrect. I apologize for the inconvenience.

My second testing was with the following code that I ran in the 'preparetest' directory:

import safe2 as safe #safe2 is the new version of safe. Replace with 'safe' to test the older version
import time
fh = open('librepysocket.repy','rb')
data = fh.read()
fh.close()

safe.safe_check(data)
time.sleep(5)