Remote tests ending early for an unknown reason. about multi-mechanize

[GoogleCodeExporter]

Upon further investication, I ran a 300 second test, where the results summary 
only considered the first 170 seconds.  Upon checking the output of the test, I 
noticed this at the 170 second mark.  

[=================56%=                 ]  167s/300s   transactions: 60193  
timer[=================56%=                 ]  168s/300s   transactions: 61488  
timer[=================56%=                 ]  169s/300s   transactions: 62014  
timer[=================57%==                ]  170s/300s   transactions: 62480  
timers: 62451  errors: 62480
Process UserGroup-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "multi-mechanize.py", line 205, in run
    agent_thread.start()
  File "/usr/lib/python2.6/threading.py", line 474, in start
    _start_new_thread(self.__bootstrap, ())
[=================57%==                ]  171s/300s   transactions: 62481  
timer[=================57%==                ]  172s/300s   transactions: 62481  
timer

Any thoughts?

Original comment by [email protected] on 21 Jan 2011 at 12:02

[GoogleCodeExporter]

Attempts to resolve this using ulimit -n 99999, and increasing the 
kernel.max_threads value in sysctl.conf didn't work properly.   I'll be looking 
into how the results are parsed, and see if I can't get the program to ignore 
thrown exceptions.

Original comment by [email protected] on 21 Jan 2011 at 1:11

[GoogleCodeExporter]

Hi.. just saw this issue.  sorry for the late reply.

do you only see this when launching 400 threads, or does it ever occur at lower 
levels?  It looks like it can't spawn new threads after a certain amount.  
(I've definitely launched > 400 threads, but not on EC2)

how much RAM does your EC2 instance have?  have you watched memory on the box?

Original comment by cgoldberg on 2 Feb 2011 at 2:59

[GoogleCodeExporter]

Sorry about the late feedback.  The ec2 instances were basic... not very beefy. 
 I've since started using bigger boxes with more processors and more ram.

Original comment by [email protected] on 11 Feb 2011 at 12:38

[GoogleCodeExporter]

Original comment by cgoldberg on 1 Jul 2011 at 1:26

• Changed state: Invalid