memory usage of everything in process tree about superlance

[CMCDragonkai]

This is what I was looking for. I have a situation where I may run "xvfb-run" which would run xulrunner. I was wondering if memmon would track the memory usage of xvfb-run or xulrunner or both. Does do what I'm thinking it does? If so why hasn't it been merged!

[CMCDragonkai]

@rgarcia I'm running this currently, but it doesn't get the full process tree. It seems to get the biggest process's RSS though.

Here's the command: xvfb-run --auto-servernum --server-num=1 --server-args="-ac +extension GLX +extension RANDR +render -noreset" slimerjs ./robot.js

I was outputting the RSS inside memmon.py to track what was going on.

Only 5 RSSs were being outputted. This makes sense since I started 4 numprocs of the xvfb-run command and plus the memmon.

But the RSS didn't come close to matching the overall tree of memory usage. Furthermore, even when it showed that RSS was greater than the limit, no restarts happened.

In fact this part never ran at all:

                for n in name, pname:
                    if n in self.programs:
                        self.stderr.write('RSS of %s is %s\n' % (pname, rss))
                        if  rss > self.programs[name]:
                            self.restart(pname, rss)
                            continue

[rgarcia]

@CMCDragonkai is there a simpler command you could run that I could use to reproduce? Not sure what robot.js does.

[CMCDragonkai]

Hey rgarcia, its just a slimerjs script that sets up a web server. http://slimerjs.org/

Heres the webserver documentation: http://docs.slimerjs.org/current/api/webserver.html

Here's an example webserver script, the command would be modified to:

command=xvfb-run --auto-servernum --server-num=1 --server-args="-ac +extension GLX +render -noreset" slimerjs ./robot.js port=%(process_num)s

var server = require('webserver').create();
var system = require('system');

var defaultConfig = {
    ipaddress: '127.0.0.1',
    port: 8500,
};

//filling up the options
var args = system.args;

args.forEach(function(value, index){

    //skip the first arg (the script name)
    if(index === 0){
        return;
    }

    var key = value.substr(0, value.indexOf('='));
    var propValue = value.substr(value.indexOf('=') + 1);

    if (!key || !propValue) {
        console.log('Incorrect parameters format, use key=value');
        phantom.exit();
    }

    if (key === 'ipaddress') defaultConfig.ipaddress = propValue;
    if (key === 'port') defaultConfig.port = propValue;

});

var service = server.listen(defaultConfig.ipaddress + ':' + defaultConfig.port, function(request, response){

    console.log('Robot received a task');

});

[rgarcia]

fixed in 7b6e1ab

I repro'd with a simple bash script: https://gist.github.com/rgarcia/7270421

[CMCDragonkai]

Thanks. Hopefully it works for many forked processes. BTW what does "repo'd" mean?

[CMCDragonkai]

Since this currently isn't merged, is it possible to run memmon by itself outside of superlance. I see that your bash script just directly runs the memmon.py via the python command.

[CMCDragonkai]

Hey @rgarcia You still have one more bug. I said before that this part doesn't run:

                for n in name, pname:
                    if n in self.programs:
                        self.stderr.write('RSS of %s is %s\n' % (pname, rss))
                        if  rss > self.programs[name]:
                            self.restart(pname, rss)
                            continue

This is because my programs are based off numbers.

I printed out what is "n" and what is "self.programs" at each iteration of Memmon.

In the self.programs array I get: [robot_slimer]

However inside "n" I get 2 types: robot_slimer_8501 and robot_slimer:robot_slimer_8501.

As you can see the condition for:

 if n in self.programs:

Never evaluates to true.

This is all happening because the memmon command which is:

command=memmon -p robot_slimer=130mb -t

Is the command that setsup the robot_slimer program name. But the actual program name is in my supervisord.conf:

process_name=%(program_name)s_%(process_num)s

How to fix this? I think this actually affects the other process groups... etc. But your script needs to be able to track the real name.

Perhaps it's as easy as changing the memmon command to have include the process numbers but I'm not sure if it would work. Something like this, but I don't like its non-flexibility. Perhaps if there was a way to refer to other section's variables.

command=memmon -p robot_slimer_8500=130mb -p robot_slimer_8501=130mb -t

Another method may be to use wildcards like https://github.com/aleszoulek/supervisor-wildcards or Supervisor/supervisor#302

At any case, the script is still not restarting processes that run above the RSS and does not work when using num_procs.

[CMCDragonkai]

I resolved this problem by using fnmatch.

First you need to import fnmatch

import fnmatch

Then change the section to:

                for n in name, pname:
                    for watched_program in self.programs:
                        if fnmatch.fnmatch(n, watched_program):
                            self.stderr.write('RSS of %s is %s\n' % (n, rss))
                            if  rss > self.programs[watched_program]:
                                self.restart(pname, rss)
                            break

Now you can use wildcards like:

command=memmon -p robot_slimer*=130mb -t

robot_slimer* would be matched to robot_slimer_8501...etc.

It checks twice due to name and pname. Not sure how to break the loop twice.

[CMCDragonkai]

I think this needs some tests to show whether it actually restarts services? I'm worried that other bugs may exist.

[mnaberez]

Fixed in 20b0733.