memory leak in copy_ticks on linux about chaco

[brycehendrix]

I've never seen this particular error. Can you provide a test case where we can reproduce it?

[HelmutFedder]

Great you are responding so quickly!

Please apologize the delay. Below please find pasted code with which I can produce the error. Normally I run it in ipython -wthread using the run magic command.

After a while different tracebacks should pop up and eventually crash the interpreter.

You can stop the thread by

b.stop()

When you uncomment the lines

b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6

the bug is gone.

Many thanks and many greetings,

Helmut

issue_9.py:

import threading
import numpy

from enthought.traits.api import SingletonHasTraits, Array, Instance
from enthought.traits.ui.api import View, Item
from enthought.enable.api import Component, ComponentEditor
from enthought.chaco.api import ArrayPlotData, Plot

class Bug( SingletonHasTraits ):

"""Produces a bug in chaco."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
    self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
    return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
    plot = Plot(self.plot_data, resizable='hv')
    plot.plot(('x','y'), style='line', color='blue')
    return plot

def start(self):
    self.stop_request = threading.Event()
    threading.Thread(target=self.run).start()

def stop(self):
    self.stop_request.set()

def run(self):
    while True:
        self.stop_request.wait(0.001)
        if self.stop_request.isSet():
            break
        self.data = self.data + numpy.random.random_integers(10, 100, 100)

traits_view = View(Item('plot', editor=ComponentEditor(), show_label=False, resizable=True),)     

if name == 'main':
b = Bug()
b.edit_traits()
"""
b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6
"""
b.start()

[brycehendrix]

Can you try to run it from plain-old python? I occasionally get the feeling
the ipython interweaving of the wx event loop causes weirdness with event
ordering.

Bryce

On Tue, May 24, 2011 at 1:27 PM, HelmutFedder <
[email protected]>wrote:

Great you are responding so quickly!

Please apologize the delay. Below please find pasted code with which I can
produce the error. Normally I run it in ipython -wthread using the run magic
command.

After a while different tracebacks should pop up and eventually crash the
interpreter.

You can stop the thread by

b.stop()

When you uncomment the lines

b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6

the bug is gone.

Many thanks and many greetings,

Helmut

issue_9.py:

import threading
import numpy

from enthought.traits.api import SingletonHasTraits, Array, Instance
from enthought.traits.ui.api import View, Item
from enthought.enable.api import Component, ComponentEditor
from enthought.chaco.api import ArrayPlotData, Plot

class Bug( SingletonHasTraits ):

"""Produces a bug in chaco."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
plot = Plot(self.plot_data, resizable='hv')
plot.plot(('x','y'), style='line', color='blue')
return plot

def start(self):
self.stop_request = threading.Event()
threading.Thread(target=self.run).start()

def stop(self):
self.stop_request.set()

def run(self):
while True:
self.stop_request.wait(0.001)
if self.stop_request.isSet():
break
self.data = self.data + numpy.random.random_integers(10, 100,
100)

traits_view = View(Item('plot', editor=ComponentEditor(),
show_label=False, resizable=True),)

if name == 'main':
b = Bug()
b.edit_traits()
"""
b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6
"""
b.start()

Reply to this email directly or view it on GitHub:
#9 (comment)

[HelmutFedder]

Dear Bryce,

did you have time to look into this?

That would be of big help!

Many greetings,
Helmut

[HelmutFedder]

Dear Bryce,

since I haven't heard back from you, I want to raise my voice again and stress the importance of this issue.

We are a team of physicists who is currently developing a set of python modules for experiment control in quantum computing applications (Rabi, Ramsey, Coherent control, etc.). The modules contain python backends for a large number of equipment, amongst, fast time sequence generators, single photon time tagging, NI based analog and digital IO, microwave synthesizers, etc.. The measurements include confocal microscopy, Electron spin resonance Nuclear spin resonance, and coherent control of single and multiple Quantum bits.

http://code.google.com/p/pi3diamond/

As it stands now the damaged regions issue is the major show stopper (it frequently kills our measurements, sometimes with fatal loss of expensive samples).

Therefore we would greatly appreciate if someone can take care about this issue.

The project is spearheaded by the university of Stuttgart. By now more than five other university groups have joined the project amongst, Harvard, Berlin and Vienna.

I should mention. Altogether there are about 60 scientists working with our code. Moreover, as you know, the the turnover of students and postdocs is large. Thus, indeed we bring a large number of young people in touch with your packages.

Your support would be very much appreciated.

Sincerely,

Dr. Helmut Fedder

[dpinte]

hi Helmut,

It seems the error is tightly linked to the refresh speed. Do you really need 1ms ? With a refresh rate of 25Hz, you do not get the error. You could keep the refresh rate for the data but trigger a graph update only at a human eye rate if needed.

[HelmutFedder]

Dear Didrik,

indeed, in our real application this issue occurs even with much lower refresh rates > 1s. I think this is because the application is much more busy with other things (several threads, computationally expensive numerical routines for on the fly data analysis, etc.). This seems to increase the chance for a race condition (but this is really over my head).

I should also point out, in our real application the error occurs maybe only once per hour. However, our data acquisition is often running over several days. Typically within 24h, the application crashes sue to the bug.

Indeed, he mini example code that I provided seems to behave differently. When the refresh rate is low, the issue seems to occur very rarely. Maybe if you run the example code for several hours the error will also show up. I must confess, I haven't checked that. Indeed, I simply set the refresh rate to this high value, so that you guys can see the error quickly.

I am very much hoping for a solution.

Many thanx and greetings, Helmut

[HelmutFedder]

Tested the following to resolve the issue (suggestion by Didrik).

In grid.py change the following lines

def _reset_cache(self):
""" Clears the cached tick positions.
"""
self._tick_positions = zeros((1,2), dtype=float) <--
#self._tick_positions = array([], dtype=float)
self._tick_extents = zeros((1, 2), dtype=float) <--
#self._tick_extents = array([], dtype=float)
self._cache_valid = False
return

On Linux, (Ubuntu natty) this does not solve the issue, however, the traceback is now different. See below.

ValueError Traceback (most recent call last)

/usr/lib/python2.7/dist-packages/enthought/enable/abstract_window.pyc in _paint(self, event)
348 self.component.do_layout()
349 gc = self._gc
--> 350 self.component.draw(gc, view_bounds=(0, 0, size[0], size[1]))
351
352 # damaged_regions = draw_result['damaged_regions']

/usr/lib/python2.7/dist-packages/enthought/enable/component.pyc in draw(self, gc, view_bounds, mode)
423 self.do_layout()
424
--> 425 self._draw(gc, view_bounds, mode)
426 return
427

/usr/lib/python2.7/dist-packages/enthought/enable/component.pyc in _draw(self, gc, view_bounds, mode)
759 else:
760 for layer in self.draw_order:
--> 761 self._dispatch_draw(layer, gc, view_bounds, mode)
762
763 return

/usr/lib/python2.7/dist-packages/enthought/enable/container.pyc in _dispatch_draw(self, layer, gc, view_bounds, mode)
308 my_handler = getattr(self, "draw_container" + layer, None)
309 if my_handler:
--> 310 my_handler(gc, view_bounds, mode)
311
312 # Now transform coordinates and draw the children

/usr/lib/python2.7/dist-packages/enthought/enable/container.pyc in _draw_container_underlay(self, gc, view_bounds, mode)
350
351 def _draw_container_underlay(self, gc, view_bounds=None, mode="normal"):
--> 352 self._draw_underlay(gc, view_bounds, mode)
353
354 def _draw_container_border(self, gc, view_bounds=None, mode="normal"):

/usr/lib/python2.7/dist-packages/enthought/enable/component.pyc in _draw_underlay(self, gc, view_bounds, mode)
876 # just overlays drawn at a different time in the rendering loop.

877             if underlay.visible:

--> 878 underlay.overlay(self, gc, view_bounds, mode)
879 return
880

/usr/lib/python2.7/dist-packages/enthought/chaco/grid.py in overlay(self, other_component, gc, view_bounds, mode)
304 return
305 self._compute_ticks(other_component)
--> 306 self._draw_component(gc, view_bounds, mode)
307 self._cache_valid = False
308 return

/usr/lib/python2.7/dist-packages/enthought/chaco/grid.py in _draw_component(self, gc, view_bounds, mode)
341 if self.orientation == "horizontal":
342 starts = self._tick_positions.copy()
--> 343 starts[:,0] = self._tick_extents[:,0]
344 ends = self._tick_positions.copy()
345 ends[:,0] = self._tick_extents[:,1]

ValueError: array dimensions are not compatible for copy

[MichaelKlas]

I today tested the Bugfix under Windows 7 Ultimate SP1 (64bit) installed with the latest enthought python distribution and received the following traceback of the error when running the code issue_9.py:
Die genaue Beschreibung der pythonversion ist:
Python 2-7.1 :EPD 7.0-2 (32-bit): (r271:86832,Dec 2 2010,10:35:02) [MSC v.1500 32 bit (Intel)]
I also tested the same version of python under Windows 7 (32 bit) and the same error/traceback occured.

---

ValueError Traceback (most recent call last)

C:\Python27\lib\site-packages\enthought\enable\abstract_window.py in _paint(self, event)
366 self.component.do_layout()
367 gc = self._gc
--> 368 self.component.draw(gc, view_bounds=(0, 0, size[0], size[1]))
369
370 # damaged_regions = draw_result['damaged_regions']

C:\Python27\lib\site-packages\enthought\enable\component.py in draw(self, gc, view_bounds, mode)
423 self.do_layout()
424
--> 425 self._draw(gc, view_bounds, mode)
426 return
427

C:\Python27\lib\site-packages\enthought\enable\component.py in _draw(self, gc, view_bounds, mode)
761 else:
762 for layer in self.draw_order:
--> 763 self._dispatch_draw(layer, gc, view_bounds, mode)
764
765 return

C:\Python27\lib\site-packages\enthought\enable\container.py in _dispatch_draw(self, layer, gc, view_bounds, mode)
308 my_handler = getattr(self, "draw_container" + layer, None)
309 if my_handler:
--> 310 my_handler(gc, view_bounds, mode)
311
312 # Now transform coordinates and draw the children

C:\Python27\lib\site-packages\enthought\enable\container.py in _draw_container_underlay(self, gc, view_bounds, mode)
350
351 def _draw_container_underlay(self, gc, view_bounds=None, mode="normal"):
--> 352 self._draw_underlay(gc, view_bounds, mode)
353
354 def _draw_container_border(self, gc, view_bounds=None, mode="normal"):

C:\Python27\lib\site-packages\enthought\enable\component.py in _draw_underlay(self, gc, view_bounds, mode)
878 # just overlays drawn at a different time in the rendering loop.
879 if underlay.visible:
--> 880 underlay.overlay(self, gc, view_bounds, mode)
881 return
882

C:\Python27\lib\site-packages\enthought\chaco\grid.py in overlay(self, other_component, gc, view_bounds, mode)
306 return
307 self._compute_ticks(other_component)
--> 308 self._draw_component(gc, view_bounds, mode)
309 self._cache_valid = False
310 return

C:\Python27\lib\site-packages\enthought\chaco\grid.py in _draw_component(self, gc, view_bounds, mode)
342 if self.orientation == "horizontal":
343 starts = self._tick_positions.copy()
--> 344 starts[:,0] = self._tick_extents[:,0]
345 ends = self._tick_positions.copy()
346 ends[:,0] = self._tick_extents[:,1]

ValueError: array dimensions are not compatible for copy

[HelmutFedder]

Today, I tested whether the issue still persists when using WXTimer instead of a thread.

The bug does not occur when using WXTimer (both ipython and bare python with wx app). A code snippet of the test is pasted below. This supports the previous idea that this bug is tightly related to multi threading. It suggests that setting the data of an ArrayPlotData from within a thread is not safe. The chain of events that is triggered by setting the data, is not finished before the data is set again. I think this is usually called a race condition, isn't it?

import threading
import numpy

from enthought.traits.api import SingletonHasTraits, Array, Instance
from enthought.traits.ui.api import View, Item, VSplit
from enthought.enable.api import Component, ComponentEditor
from enthought.chaco.api import ArrayPlotData, Plot
from enthought.pyface.timer.api import Timer

class IssueNineThread( SingletonHasTraits ):

"""Produces a bug in chaco."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
    self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
    return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
    plot = Plot(self.plot_data, resizable='hv')
    plot.plot(('x','y'), style='line', color='blue')
    return plot

def start(self):
    self.stop_request = threading.Event()
    threading.Thread( target=self.run ).start()

def stop(self):
    self.stop_request.set()

def run(self):
    while True:
        #self.stop_request.wait(0.001)
        if self.stop_request.isSet():
            break
        #self.trait_set(True, data=self.data + numpy.random.random_integers(10, 100, 100))
        self.data = self.data + numpy.random.random_integers(10, 100, 100)

traits_view = View(VSplit(Item('plot', editor=ComponentEditor(), show_label=False, resizable=True),
                          Item('none'),
                   ),
              )     

class IssueNineWXTimer( SingletonHasTraits ):

"""No error is produced when using WX Timer."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
    self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
    return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
    plot = Plot(self.plot_data, resizable='hv')
    plot.plot(('x','y'), style='line', color='blue')
    return plot

timer = Instance( Timer )

def start(self):
    self.timer = Timer(0, self.update_data)
    self.timer.Start()

def stop(self):
    self.timer.Stop()

def update_data(self):
    self.data = self.data + numpy.random.random_integers(10, 100, 100)

traits_view = View(Item('plot', editor=ComponentEditor(), show_label=False, resizable=True),)

if name == 'main':
import wx
#b = IssueNineWXTimer()
b = IssueNineThread()
b.edit_traits()
#"""
b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6
#"""
b.start()
wx.PySimpleApp().MainLoop()

[brycehendrix]

Good find. Setting the ArrayPlotData causes a cascade of traits events which
could take some time. If you want to use threads, you should definitely use
locks.

Bryce

On Thu, Jun 30, 2011 at 1:47 PM, HelmutFedder <
[email protected]>wrote:

Today, I tested whether the issue still persists when using WXTimer instead
of a thread.

The bug does not occur when using WXTimer (both ipython and bare python
with wx app). A code snippet of the test is pasted below. This supports the
previous idea that this bug is tightly related to multi threading. It
suggests that setting the data of an ArrayPlotData from within a thread is
not safe. The chain of events that is triggered by setting the data, is not
finished before the data is set again. I think this is usually called a race
condition, isn't it?

import threading
import numpy

from enthought.traits.api import SingletonHasTraits, Array, Instance
from enthought.traits.ui.api import View, Item, VSplit
from enthought.enable.api import Component, ComponentEditor
from enthought.chaco.api import ArrayPlotData, Plot
from enthought.pyface.timer.api import Timer

class IssueNineThread( SingletonHasTraits ):

"""Produces a bug in chaco."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
plot = Plot(self.plot_data, resizable='hv')
plot.plot(('x','y'), style='line', color='blue')
return plot

def start(self):
self.stop_request = threading.Event()
threading.Thread( target=self.run ).start()

def stop(self):
self.stop_request.set()

def run(self):
while True:
#self.stop_request.wait(0.001)
if self.stop_request.isSet():
break
#self.trait_set(True, data=self.data +
numpy.random.random_integers(10, 100, 100))
self.data = self.data + numpy.random.random_integers(10, 100,
100)

traits_view = View(VSplit(Item('plot', editor=ComponentEditor(),
show_label=False, resizable=True),
Item('none'),
),
)

class IssueNineWXTimer( SingletonHasTraits ):

"""No error is produced when using WX Timer."""

data = Array( value=numpy.zeros(100) )

def _data_changed(self):
self.plot_data.set_data('y', self.data)

plot_data = Instance( ArrayPlotData )
def _plot_data_default(self):
return ArrayPlotData(x=numpy.arange(100), y=self.data)

plot = Instance( Plot )

def _plot_default(self):
plot = Plot(self.plot_data, resizable='hv')
plot.plot(('x','y'), style='line', color='blue')
return plot

timer = Instance( Timer )

def start(self):
self.timer = Timer(0, self.update_data)
self.timer.Start()

def stop(self):
self.timer.Stop()

def update_data(self):
self.data = self.data + numpy.random.random_integers(10, 100, 100)

traits_view = View(Item('plot', editor=ComponentEditor(),
show_label=False, resizable=True),)

if name == 'main':
import wx
#b = IssueNineWXTimer()
b = IssueNineThread()
b.edit_traits()
#"""
b.plot.value_range.low_setting = 0.0
b.plot.value_range.high_setting = 1e6
#"""
b.start()
wx.PySimpleApp().MainLoop()

Reply to this email directly or view it on GitHub:
#9 (comment)

[HelmutFedder]

Dear Bryce,

Many thanks for your insight.

please apologize the late response - too much work.

OK, I am confused now. I believe 'traits' should somehow take care about this and be in a sense 'thread safe', because: if updating a plot can result in a race condition, can't any setting of a trait attribute, say a large array, do so too? Or is a race condition usually just so unlikely that it won't happen in a lifetime? If there can be problems under other circumstances, too, how can I judge whether it is safe to set a trait attribute out of a thread?

Yes, I also thought to use locks, previously, but how? I can acquire a lock before I set the array and release it afterwards. But then, to do something useful, I guess the event handler in the main thread / GUI thread has to acquire the lock as well. How to do that?

Does that mean, every application that has a plot and a thread that delivers the data should use locks?

Cheers, Helmut

[brycehendrix]

Traits isn't threadsafe by default. You can optionally set a value in the UI
thread, which may be enough for you if you always set the chaco data in the
UI thread.

https://svn.enthought.com/enthought/wiki/OnTraitChangeDispatch

Bryce

On Wed, Jul 6, 2011 at 1:49 AM, HelmutFedder <
[email protected]>wrote:

Dear Bryce,

Many thanks for your insight.

please apologize the late response - too much work.

OK, I am confused now. I believe 'traits' should somehow take care about
this and be in a sense 'thread safe', because: if updating a plot can result
in a race condition, can't any setting of a trait attribute, say a large
array, do so too? Or is a race condition usually just so unlikely that it
won't happen in a lifetime? If there can be problems under other
circumstances, too, how can I judge whether it is safe to set a trait
attribute out of a thread?

Yes, I also thought to use locks, previously, but how? I can acquire a lock
before I set the array and release it afterwards. But then, to do something
useful, I guess the event handler in the main thread / GUI thread has to
acquire the lock as well. How to do that?

Does that mean, every application that has a plot and a thread that
delivers the data should use locks?

Cheers, Helmut

Reply to this email directly or view it on GitHub:
#9 (comment)

[HelmutFedder]

Ah, this looks promising!

I did a first test and it seems it solves the problem

It is a bit tricky to test, because in the sample code, the refresh rate needs to be very high to produce the bug. With high refresh rate and dispatch='ui', the ui can freeze. So you need to play a bit with the refresh rate, until you find a value hiigh enough such that the bug is produced with dispatch='same' and low enough such that the ui stays active with dispatch='ui'.

We will do more testing on the real code within the next days.

Tentatively, this solves the issue.

I think the dispatch parameter definitely deserves more explanation in the documentation. Maybe, for now, you can just put a note with a link to the site that you sent me.

Thanks and many greetings,

Helmut

[HelmutFedder]

We have done some more testing.

Our application has now been running for >24h under windows XP as well as ubuntu without a crash. It looks like this indeed resolves the issue as well as issue #8.

Please give us some time for testing and go ahead and close this issue if you don't hear from us within the next few weeks.

Since threading is so important in many scientific applications, I do think the documentation deserves a chapter 'traits and multi threading' or something similar.

Thanks and many greetings,

Helmut

[brycehendrix]

Yup, I agree completely. Now we just have to find someone willing to write
that chapter :) I have brought up the issue of porting the old wiki docs
into the Traits documentation, hopefully someone with some free time can
address it.

Bryce

On Mon, Jul 11, 2011 at 3:36 PM, HelmutFedder <
[email protected]>wrote:

We have done some more testing.

Our application has now been running for >24h under windows XP as well as
ubuntu without a crash. It looks like this indeed resolves the issue as well
as issue #8.

Please give us some time for testing and go ahead and close this issue if
you don't hear from us within the next few weeks.

Since threading is so important in many scientific applications, I do think
the documentation deserves a chapter 'traits and multi threading' or
something similar.

Thanks and many greetings,

Helmut

Reply to this email directly or view it on GitHub:
#9 (comment)

[fnoble]

Just stumbled upon this issue too, as well as related issue #8. I was already vaguely aware that TraitsUI wasn't completely thread safe, but I feel that simply setting an ArrayPlotData really should be otherwise the abstraction is getting a bit too leaky.