ldesousa / hex-utils Goto Github PK
View Code? Open in Web Editor NEWUtilities for ASCII Hexagonal grids
hex-utils's People
Contributors
Stargazers
Watchers
hex-utils's Issues
Refactor surface scripts
surface2hasc and surface2asc have similar code setting up arguments.
Create commonParser
Similar input arguments across various scripts:
. csv2hasc
. surface2hasc
. surface2asc
Promote surfaceParser to something more general.
saveAsGeoJSON converts all values to strings
the resulting geojson stores hasc values as strings instead of original numerical values
Add licence notice to each code file
Implement angle rotation
Parse and apply the angle parameter when converting from HASC to GML.
Paper example on how to use surface2hasc doesn't match code class names in surface directory anymore
I've been trying to run some surface generator classes without success.
Can you please provide a brief example on how to use them?
Thank you
Last test
$ python3.6 /usr/local/bin/surface2hasc -x 0 -y 0 -X 2000 -Y 2000 -s 19.6 -m /usr/local/lib/python3.6/dist-packages/surfaces/pitGaussian.py -f Gaussian -o gaussian.hasc
TypeError: '<' not supported between instances of 'float' and 'Gaussian'
Check for nodata
Check for nadata values and dump them correctly into the output file.
wrong None check in asc.py causes last row not being loaded and asc2hasc fail
when nodata header line is missing line:
self._loadHeaderLine(self._nextLine, self._key_nodata, type("a"), True)
returns None but you check self._nodata != "" that would always return True, so first line of data is skipped and this causes last row not being loaded (full of None), then "-m nn" fails to run with misleading error:
def _loadHeader(self):
# Mandatory header
self._set_ncols(self._loadHeaderLine(self._file.readline(), self._key_ncols, type(1)))
self._set_nrows(self._loadHeaderLine(self._file.readline(), self._key_nrows, type(1)))
self._xll = self._loadHeaderLine(self._file.readline(), self._key_xll, type(1.0))
self._yll = self._loadHeaderLine(self._file.readline(), self._key_yll, type(1.0))
self._set_size( self._loadHeaderLine(self._file.readline(), self._key_size, type(1.0)))
# Optional headers
self._nextLine = self._file.readline()
self._nodata = self._loadHeaderLine(self._nextLine, self._key_nodata, type("a"), True)
if self._nodata != "" :
self._nextLine = self._file.readline()
surface2hasc must take full module path as input
Ritgh now it only works with relative paths, but in most cases this is not realistic.
Check if side and size are greater than zero
None of these properties can be 0 or negative.
prerequisite of GIS packages for the hex2GML
error:
Leaded input HASC, converting...
Traceback (most recent call last):
File "C:\Prog\GITS\hex-utils\hex_utils\hasc.py", line 88, in saveAsGML
from osgeo import ogr
ImportError: No module named 'osgeo'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Prog\GITS\hex-utils\hex_utils\hasc2gml.py", line 52, in
main()
File "C:\Prog\GITS\hex-utils\hex_utils\hasc2gml.py", line 49, in main
hexGrid.saveAsGML(outputFile)
File "C:\Prog\GITS\hex-utils\hex_utils\hasc.py", line 92, in saveAsGML
apt install python-gdal""")
TypeError: exceptions must derive from BaseException
Obviously I need the osgeo packages, which I can surmise are 'geo' related, i.e. GIS packages.
Should note the prerequisite for software in the readme ^_^
csv2hasc
Creates a new HASC raster from a traditional CSV file containing as columns (x,y,value). It can use a simple interpolation method, e.g. IDW.
Investigate if there is something already available in Python for splines, polinomi, etc.
asc2hasc calculating top right coordinates wrong
The lower left coordinates must be added to the computation of rows/columns times cell side.
Files no longer need to import from hex_utils package, as they all reside in there
Seems to be, since I'm running the asc2hasc, hasc2gml files inside the hex_utils folder, and all the other files are there too, they no longer need to import from hex_utils.***
Rename class Grid to Mesh or Raster
Since grid can not be used with hexagons.
Print out raster geometries in surface2hasc
Both asc2hasc and csv2hasc already do this.
asc2hasc generating zeros with multiquadratic method
The multiquadratic method is producing 'None' results for the southern most cells, that are then wrongly coded as zeros.
csv2asc
Creates a new ESRI ASCII raster from a traditional CSV file containing as columns (x,y,value). It can use a simple interpolation method, e.g. IDW.
Investigate if there is something already available in Python for splines, polinomi, etc.
Add same resolution transformation to asc2hasc
Same resolution should be calculated with:
hexagon_area = square_area * 1.13
asc2hasc fails with rasters with CRS EPSG:4326 and/or with non-square pixels
With the following image I get "numpy.linalg.linalg.LinAlgError: Matrix is singular" error
Driver: GTiff/GeoTIFF
Files: sagama20160824_ros.tif
sagama20160824_ros.tif.aux.xml
Size is 139, 115
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (8.520520469873661,40.293932069181139)
Pixel Size = (0.000898315284120,-0.000898315284120)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 8.5205205, 40.2939321) ( 8d31'13.87"E, 40d17'38.16"N)
Lower Left ( 8.5205205, 40.1906258) ( 8d31'13.87"E, 40d11'26.25"N)
Upper Right ( 8.6453863, 40.2939321) ( 8d38'43.39"E, 40d17'38.16"N)
Lower Right ( 8.6453863, 40.1906258) ( 8d38'43.39"E, 40d11'26.25"N)
Center ( 8.5829534, 40.2422789) ( 8d34'58.63"E, 40d14'32.20"N)
Input square cell area : 8.069703494680006e-07
Hexagon cell area : 9.151043762967126e-07
Hexagon side length : 0.0005934844889809739
Hexagon perpendicular : 0.0005139726442095491
Number of rows in mesh : 101
Number of columns in mesh : 141
Converting with Multi-quadratic interpolation ...
/usr/lib/python3/dist-packages/scipy/linalg/basic.py:223: RuntimeWarning: scipy.linalg.solve
Ill-conditioned matrix detected. Result is not guaranteed to be accurate.
Reciprocal condition number: 9.251858537656306e-18
' condition number: {}'.format(rcond), RuntimeWarning)
Traceback (most recent call last):
File "/usr/local/bin/asc2hasc", line 7, in <module>
from hex_utils.asc2hasc import main
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc2hasc.py", line 109, in <module>
main()
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc2hasc.py", line 100, in main
hexRaster.set(i, j, interpol(x, y))
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc.py", line 137, in interpolMultiquadratic
f = interpolate.Rbf(xx, yy, vals, epsilon=epsilon)
File "/usr/lib/python3/dist-packages/scipy/interpolate/rbf.py", line 222, in __init__
self.nodes = linalg.solve(self.A, self.di)
File "/usr/lib/python3/dist-packages/scipy/linalg/basic.py", line 219, in solve
raise LinAlgError('Matrix is singular.')
numpy.linalg.linalg.LinAlgError: Matrix is singular.
With the following image (that is the EPSG:32632 reprojection of the one above) I have "TypeError: '<=' not supported between instances of 'NoneType' and 'int'" error
Driver: GTiff/GeoTIFF
Files: sagama20160824_ros_32632.tif
Size is 139, 116
Coordinate System is:
PROJCS["WGS 84 / UTM zone 32N",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",9],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AXIS["Easting",EAST],
AXIS["Northing",NORTH],
AUTHORITY["EPSG","32632"]]
Origin = (459185.728300000017043,4460491.910400000400841)
Pixel Size = (76.796454676259017,-99.279828448282103)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 459185.728, 4460491.910) ( 8d31'11.25"E, 40d17'38.14"N)
Lower Left ( 459185.728, 4448975.450) ( 8d31'13.89"E, 40d11'24.63"N)
Upper Right ( 469860.436, 4460491.910) ( 8d38'43.38"E, 40d17'39.78"N)
Lower Right ( 469860.436, 4448975.450) ( 8d38'45.33"E, 40d11'26.26"N)
Center ( 464523.082, 4454733.680) ( 8d34'58.46"E, 40d14'32.27"N)
Error, malformed file. Expected cellsize but read dx
Traceback (most recent call last):
File "/usr/local/bin/asc2hasc", line 7, in <module>
from hex_utils.asc2hasc import main
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc2hasc.py", line 109, in <module>
main()
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc2hasc.py", line 56, in main
esriGrid.loadFromFile(args.inputFile)
File "/usr/local/lib/python3.6/dist-packages/hex_utils/raster.py", line 86, in loadFromFile
self._loadHeader()
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc.py", line 148, in _loadHeader
self._set_size( self._loadHeaderLine(self._file.readline(), self._key_size, type(1.0)))
File "/usr/local/lib/python3.6/dist-packages/hex_utils/asc.py", line 36, in _set_size
if (size <= 0):
TypeError: '<=' not supported between instances of 'NoneType' and 'int'
I have to reproject the EPSG:4326 image into a map projection like EPSG:32632 and then warp it to make square pixels to be able to run asc2hasc with both interpolations successfully.
I've used:
gdalwarp -tr 10.0 -10.0 -r bilinear sagama20160824_ros_32632.tif sagama20160824_ros_32632_squared.tif
Driver: GTiff/GeoTIFF
Files: sagama20160824_ros_32632_squared.tif
Size is 1067, 1154
Coordinate System is:
PROJCS["WGS 84 / UTM zone 32N",
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.257223563,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0,
AUTHORITY["EPSG","8901"]],
UNIT["degree",0.0174532925199433,
AUTHORITY["EPSG","9122"]],
AUTHORITY["EPSG","4326"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",9],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AXIS["Easting",EAST],
AXIS["Northing",NORTH],
AUTHORITY["EPSG","32632"]]
Origin = (459185.728300000017043,4460491.910400000400841)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 459185.728, 4460491.910) ( 8d31'11.25"E, 40d17'38.14"N)
Lower Left ( 459185.728, 4448951.910) ( 8d31'13.89"E, 40d11'23.87"N)
Upper Right ( 469855.728, 4460491.910) ( 8d38'43.18"E, 40d17'39.78"N)
Lower Right ( 469855.728, 4448951.910) ( 8d38'45.13"E, 40d11'25.49"N)
Center ( 464520.728, 4454721.910) ( 8d34'58.36"E, 40d14'31.88"N)
Geometries:
Input square cell area : 100.0
Hexagon cell area : 113.39999999999999
Hexagon side length : 6.606639111585838
Hexagon perpendicular : 5.72151730426919
Number of rows in mesh : 1009
Number of columns in mesh : 1077
Converting with Multi-quadratic interpolation ...
Finished successfully.
asc2hasc determining interpolation method incorrectly
Switch in the if multiquadratic and nearest neighbour.
surface2hasc: adapt module import to Python 3
Aa it stands it only works on Python 2
Expand copyright to 2017
asc2hasc: check for exceptions when saving output
There is already a try block for the input, but not the output.
Provide error message when file not found
Check if the files exist before opening and provide a useful error message.
all the utils gets executed twice overwriting it's own result and taking twice the time
I've tested it with ubuntu 18.04 system python3.6 and two different miniconda python 3.6 & python 3.7 environments
➜ asc2hasc -r -m mq -i process sagama20160824_ros_32632_squared.asc -o sagama20160824_ros_32632_squared.hasc
Geometries:
Input square cell area : 100.0
Hexagon cell area : 113.39999999999999
Hexagon side length : 6.606639111585838
Hexagon perpendicular : 5.72151730426919
Number of rows in mesh : 1009
Number of columns in mesh : 1077
Converting with Multi-quadratic interpolation ...
Finished successfully.
Geometries:
Input square cell area : 100.0
Hexagon cell area : 113.39999999999999
Hexagon side length : 6.606639111585838
Hexagon perpendicular : 5.72151730426919
Number of rows in mesh : 1009
Number of columns in mesh : 1077
Converting with Multi-quadratic interpolation ...
Finished successfully.
➜ ros_hex hasc2gml sagama20160824_ros_32632_squared_test.hasc sagama20160824_ros_32632_squared_test.gml
Loaded input HASC, converting...
Conversion successfully completed.
Loaded input HASC, converting...
Conversion successfully completed.
Update variables and methods named 'grid'
Apply PyDoc comments
Commments on properties and methods of the API.
Reposition first hexagon computed in asc2hasc
The first hexagon (0,0) computed in asc2hasc can be positioned more efficiently, overlapping more of the first square and less of the empty space.
Use initWithExtent in all scripts that create grid geometry
This calculation is spread around in various scripts and must be refactored.
math.ceil returns float on Python 2.7
It functions properly on Python 3, but not on 2.7. It must be corrected to work with QGis.
Top row is not being read if angle is not specified
Try with this raster:
ncols 3
nrows 3
xll 2.5
yll -2.5
side 1.5
no_data 9999
23 24 23.2
24 24.2 23.7
24.1 23.8 23.3
Refactor Gaussian and Hubbert surfaces
These surface models have a good deal of code in common.
surface2* scripts must take inputs as command line arguments
Use a scheme like:
surface2hex -m module -f fun -x 0 -y 0 -X 1000 -Y 1000 -o output.hasc
Accept input and output files as command line arguments
Check out this tutorial:
http://www.tutorialspoint.com/python/python_command_line_arguments.htm
Sample issue
Debugging GitHub automatic issue closure.
Make the hexPerp property available with a getter method.
Add unit tests
A milestone perhaps?
Replace OGR for Shapely
This needs a bit more research, but it might be a wiser choice.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.