Jupyterlab kernel crashes about py-eddy-tracker

[AntSimi]

wow! i never try with this resolution with unregular grid
maybe code need some update to manage well this amount of data.
I could take a look if you share data and script

[marimpacheco]

This is one time step of the data (203MB). I cut the data, so now I have one file for each time step.
This worked for the first step, but as it went to the second, the kernel crashed again.
Then, after initializing the kernel, I could not even detect for a single step (even when I try outside the function)...

My code:

month_days = pd.date_range(start='1/3/2001', end='12/29/2001', freq='5D')

def detection(year, n, run=1, version_res=6):
    
    # get_path gives the file_name
    data = get_path(run, version_res, year, n)
    g = UnRegularGridDataset(data, "geolon_t", "geolat_t", centered=True, indexs = dict(time=0))

    # since my data starts in the year 181. Get every 5th day
    date = datetime(year, month_days.month[n-1], month_days.day[n-1])

    g.high_filter('sea_level', 700)

    a, c = g.eddy_identification("sea_level", "u", "v", date,  # Date of identification
    0.004,  # step between two isolines of detection (m)
    pixel_limit=None,  # Min and max pixel count for valid contour
    shape_error=70,  # Error max (%) between ratio of circle fit and contour
    )

    with Dataset(date.strftime(datapath + 'detection' + "/Anticyclonic_%Y%m%d.nc"), "w") as g: # NEED TO CHANGE THIS, DATAPATH NOW TO FORCED 2.6
        a.to_netcdf(g)
    with Dataset(date.strftime(datapath + 'detection' + "/Cyclonic_%Y%m%d.nc"), "w") as g:
        c.to_netcdf(g)

for i in np.arange(1,74):
        detection(year=181,i)

The link to the file.
Thanks!

[AntSimi]

Did you check how many level of isolines are produce? you could see if you activate DEBUG in log

from py_eddy_tracker import start_logger
start_logger().setLevel("DEBUG")  # Available options: ERROR, WARNING, INFO, DEBUG

[marimpacheco]

Are these values?

INFO 2023-02-03 18:31:26,941 eddy_feature.__init__ :
   Start computing iso lines with 421 levels from -0.816000 to 0.864000 ...
INFO 2023-02-03 18:31:47,768 eddy_feature.__init__ :
   Finish computing iso lines
INFO 2023-02-03 18:31:49,285 eddy_feature.__init__ :
   Repair 15527 closed contours and 347 almost closed contours / 108252 contours

[AntSimi]

Yes values are OK, I will try to run my computer

[marimpacheco]

Thanks!

[AntSimi]

When i run script for one day with your data, i use no more 2 Go of RAM, ...
Could you list version of following package?:

• matplotlib
• opencv-python
• polygon3
• scipy
• numpy
• numba
  Try also to run a top in parallel of your script to see how many RAM are consume at each step.

[marimpacheco]

# Name                    Version                   Build  Channel
numba                     0.55.2                   pypi_0    pypi
numpy                     1.22.4          py310h4ef5377_0    conda-forge
scipy                     1.9.3           py310heeff2f4_0
polygon3                  3.0.9.1                  pypi_0    pypi
opencv-python             4.7.0.68                 pypi_0    pypi
matplotlib                3.6.2           py310h06a4308_0  

Is there a way of running PET in parallel? Or do you mean running the loop in parallel? Today it ran fine for one day of the data, but it crashed again when trying doing another day, not doing it on the loop.

[marimpacheco]

I loaded dask and now it works, although in the dashboard it seems not much is happening. Sorry I am a bit new to running things in parallel. It only works though if I do something like this (and with dask):

for i in np.arange(1,74):
    detection(181,i)
    gc.collect(generation=2)

[AntSimi]

Parallel detection is a good solution. There is no implemented method to use multicore for on detection step.