Comments (10)
wow! i never try with this resolution with unregular grid
maybe code need some update to manage well this amount of data.
I could take a look if you share data and script
from py-eddy-tracker.
This is one time step of the data (203MB). I cut the data, so now I have one file for each time step.
This worked for the first step, but as it went to the second, the kernel crashed again.
Then, after initializing the kernel, I could not even detect for a single step (even when I try outside the function)...
My code:
month_days = pd.date_range(start='1/3/2001', end='12/29/2001', freq='5D')
def detection(year, n, run=1, version_res=6):
# get_path gives the file_name
data = get_path(run, version_res, year, n)
g = UnRegularGridDataset(data, "geolon_t", "geolat_t", centered=True, indexs = dict(time=0))
# since my data starts in the year 181. Get every 5th day
date = datetime(year, month_days.month[n-1], month_days.day[n-1])
g.high_filter('sea_level', 700)
a, c = g.eddy_identification("sea_level", "u", "v", date, # Date of identification
0.004, # step between two isolines of detection (m)
pixel_limit=None, # Min and max pixel count for valid contour
shape_error=70, # Error max (%) between ratio of circle fit and contour
)
with Dataset(date.strftime(datapath + 'detection' + "/Anticyclonic_%Y%m%d.nc"), "w") as g: # NEED TO CHANGE THIS, DATAPATH NOW TO FORCED 2.6
a.to_netcdf(g)
with Dataset(date.strftime(datapath + 'detection' + "/Cyclonic_%Y%m%d.nc"), "w") as g:
c.to_netcdf(g)
for i in np.arange(1,74):
detection(year=181,i)
The link to the file.
Thanks!
from py-eddy-tracker.
Did you check how many level of isolines are produce? you could see if you activate DEBUG in log
from py_eddy_tracker import start_logger
start_logger().setLevel("DEBUG") # Available options: ERROR, WARNING, INFO, DEBUG
from py-eddy-tracker.
Are these values?
INFO 2023-02-03 18:31:26,941 eddy_feature.__init__ :
Start computing iso lines with 421 levels from -0.816000 to 0.864000 ...
INFO 2023-02-03 18:31:47,768 eddy_feature.__init__ :
Finish computing iso lines
INFO 2023-02-03 18:31:49,285 eddy_feature.__init__ :
Repair 15527 closed contours and 347 almost closed contours / 108252 contours
from py-eddy-tracker.
Yes values are OK, I will try to run my computer
from py-eddy-tracker.
Thanks!
from py-eddy-tracker.
When i run script for one day with your data, i use no more 2 Go of RAM, ...
Could you list version of following package?:
- matplotlib
- opencv-python
- polygon3
- scipy
- numpy
- numba
Try also to run a top in parallel of your script to see how many RAM are consume at each step.
from py-eddy-tracker.
# Name Version Build Channel
numba 0.55.2 pypi_0 pypi
numpy 1.22.4 py310h4ef5377_0 conda-forge
scipy 1.9.3 py310heeff2f4_0
polygon3 3.0.9.1 pypi_0 pypi
opencv-python 4.7.0.68 pypi_0 pypi
matplotlib 3.6.2 py310h06a4308_0
Is there a way of running PET in parallel? Or do you mean running the loop in parallel? Today it ran fine for one day of the data, but it crashed again when trying doing another day, not doing it on the loop.
from py-eddy-tracker.
I loaded dask and now it works, although in the dashboard it seems not much is happening. Sorry I am a bit new to running things in parallel. It only works though if I do something like this (and with dask):
for i in np.arange(1,74):
detection(181,i)
gc.collect(generation=2)
from py-eddy-tracker.
Parallel detection is a good solution. There is no implemented method to use multicore for on detection step.
from py-eddy-tracker.
Related Issues (20)
- Request for help HOT 3
- Bug with matplotlib 3.8
- py_eddy_tracker.dataset.grid issue HOT 6
- pixel position of grid HOT 4
- Lifetime average HOT 3
- The function in the library tried to use Numba to speed up the computation, but received an array of type numpy.ma.MaskedArray that is not supported by Numba. HOT 2
- error about eddy_identification function HOT 2
- Eddies detected on land HOT 6
- Eddy kinetic energy of different types of eddies and its calculation HOT 1
- about from py_eddy_tracker.dataset.grid import RegularGridDataset HOT 1
- Eddy detection issue HOT 3
- batch identification of eddies over multiple days from a single netCDF file HOT 14
- Request for help: No extrema found in contour of xxx pixels in level xxx HOT 6
- Issue reading NEMO UnRegularGrid HOT 2
- grid_count and grid_stat error HOT 2
- The boundary cannot be closed normally, and the longitude and latitude of some points are wrongγ HOT 7
- How to store the eddy tracking results HOT 4
- Data is Empty
- Data is Empty HOT 1
- Cartopy projection HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from py-eddy-tracker.