M values are not optional for shape types with Z geom about pyshp

[micahcochran]

I looked at the ESRI 1998 Shapefile Spec. for guidance about this matter.

It appears that if you have an any Point, Line, Polygon (or other geometry type) with X, Y, and Z coordinates that M (measure) coordinates are included. But there is a way to make the M coordinate optional.

From the ESRI 1998 Shapefile Spec. electronic page 6, paper page 2:

Floating point numbers must be numeric values. Positive infinity, negative infinity, and
Not-a-Number (NaN) values are not allowed in shapefiles. Nevertheless, shapefiles
support the concept of "no data" values, but they are currently used only for measures. Any floating point number smaller than –10^38 is considered by a shapefile reader to represent a "no data" value.

It looks like the referenced issue in BlenderGIS has some example data.

[micahcochran]

I walked through the example shapefile with the Reader.__shapeheader method.

This isn't verbatim, but I think you get the point.

In [22]: measure = unpack("<2d", fp.read(16))

In [23]: measure
Out[23]: (-1e+38, -9.999999680285692e+37)

Table 1 of the spec (electronic page 8, paper page 4), has a note: * Unused, with value 0.0, if not Measured or Z type

There is a line that checks for the self.measure in the shapefile's header to have a 0.0 in Reader.__shape function (https://github.com/GeospatialPython/pyshp/blob/master/shapefile.py#L355). I think this code is slightly incorrect, because you could start a measure at 0.0 and end at a higher number (100.0). 0.0 would be the minM, 100.0 would be the maxM. This implementation would ignore it. The Shapefile Writer could anticipate that by setting the minM to -0.1. Or the reader could read both values. Getting back on topic.

Either one seems like it is reasonably a valid value. I'm a little curious what the shapelib's behavior is in GDAL for reading shapefiles. I just need to find or convert some data with XYZs.

[karimbahgat]

That's how I understood the spec as well, that m values are always included. So am I understanding correctly that this means the error was caused by a faulty file where the m values were not included, adn we can close the issue?

Nevertheless, the issue that you brought up @micahcochran remains. The lib skips reading shape mbox and m values if the total shapefile mbox includes a 0.0. Here is the line you reference along with the original comment:

# Read m extremes and values if header m values do not equal 0.0
if shapeType in (13,15,18,23,25,28,31) and not 0.0 in self.measure:

Only thing I don't get is why would a 0.0 value indicate that we shouldn't calculate the mbox and m values? Isn't it the "nodata" value we should be looking for, values lower than –10^38? And then we could simply test if the total shapefile mbox has only nodatavalues, since that would mean every mbox and m values must also be nodata and there is no point in reading. If so, this should be an easy fix by instead testing for ...and not (self.measure[0] <= -10e38 and self.measure[1] <= -10e38).

[karimbahgat]

The original issue was about reading the m bounding box for shapes when there was none to be found. After some digging this seems to have been caused by a bug where PyShp used the file header m range to indicate whether it should read m values from shapes. But the code misinterpreted the ESRI spec on the difference between the meaning of 0.0 (can be valid, but also can indicate non-m shapetype) and -10e38 (if all m values in the file were set to nodata). In addition, using the m range values to decide whether to read shape m values is a bit misleading, since a file can be written with faulty m range values, and this should rather be determined directly from the shapetype.

Changed the code behavior on this in 4f89beb.

[karimbahgat]

See also a8bd8dd.

[karimbahgat]

After additional user feedback and re-reading of the specs, seems this problem may also have been caused by m-values that were missing entirely (fields themselves are marked as "optional" in the spec). Have now updated so the code will respect the shaperecord length and only read mranges and mvalues if there is enough data left. See #174 .