Comments (10)
This seems to happen when you add attempt to add a float value to an integer field (ie decimal=0), so it expects but fails to format it as an integer. In v1.2.10 this went unnoticed since we just forced the value to string.
In one way the error is doing its job, but the lib should also do some autocorrecting, so I think we can add force conversion of floats to ints. Will look at adding a fix.
from pyshp.
I'm experiencing a similar issue, after upgrade from 1.2.10 to 1.2.11, writing a shapefile:
Unknown format code 'd' for object of type 'str'
using python 2.7.13
from pyshp.
pyshp 1.2.11 seems to introduce incompatibility with the prior documented method of specifying the shapefile attributes. While this version does make more sense (using numeric for field length), it is not compatible with the previously documented specification, string for the length, numeric for the precision.
This probably should have been a major release rather than a bug fix.
Or at least maintain compatibility with the previous field specification.
from pyshp.
@jesegal, for your first comment could you share the code that produces the error? As in my previous answer, the error you mention seems to be caused by adding a string value to a field defined as "N" with decimal=0. The fact that this went unnoticed in the previous version was a bug, this simply fixes it, and notifies the user as it should. Since several have now raised this issue, do you think perhaps we should explicitly test for the value mismatch and raise a more informative exception?
As for you second comment, you seem to be referring to another issue unrelated to this one: a change in how we define fields, that the README used to specify the "size" arg of the field()
method with a string-number, whereas now it is specified with a number, i.e. field("Field1", "C", "30")
VS field("Field1", "C", 30)
? It's true that the documentation changed, but both in the previous version and the current version it was acceptable to use either a string-number or number for the "size" arg, since internally its value is forced to an int. So there shouldn't be any backwards incompatibility, it's just a change in the docs. Did I understand you correctly on this point?
Regardless, your point about major version incrementing for backwards incompatible changes is well taken, and the upcoming version will indeed increment the major version due to some other changes in the code.
from pyshp.
@karimbahgat, Using shapefiles with field specifications, some of them as shown below.
It seems the Numeric fields are causing the issue, where it was possible to specify the field width using a string until version 1.2.11.
Yes, it makes more sense to specify this as a number, but the documentation clearly specified it as a string, until version 1.2.11.
('LOCN_TYPE', 'C', '10'),
('LAT', 'N', '14', 8),
('LON', 'N', '14', 8),
This field is failing: ('ID', 'N', '8')
It's failing at the following statement:
w.save(sf)
...and stepping into the library, it's failing at:
value = format(value, "d")[:size].rjust(size) # caps the size if exceeds the field size
in my scenario, at this line of code
value is a str: '0'
which causes this to fail.
This worked in 1.2.10
from pyshp.
Ok, so it seems you indeed are encountering the same issue as @etiennethomassen. The issue is with the value you are supplying to the field when you call record(*values)
. The error says it all: You are trying to insert a string value (value is a str: '0'
) into your ID field which is defined as numeric. Instead of inserting the number 0 you are inserting it as a '0' string.
To make it more concrete, this does not work:
w = shapefile.Writer()
w.field('ID', 'N', '8')
w.record("0")
w.null()
w.save("test.shp")
But changing w.record("0")
to w.record(0)
works.
So your error has nothing to do with how you define the field. As I said previously, the field width change was only in the docs, the library accepts it as either a string or a number. You can write it like this w.field('ID', 'N', 8)
, or like this w.field('ID', 'N', '8')
, both works.
It worked in 1.2.10, but only because it was a bug. Passing strings to a numeric field really shouldn't be allowed, a bug that was fixed in 1.2.11. I am will probably end up loosening this criteria a little, only throw an error if the string cannot be forced to a number.
from pyshp.
Since the issue seems common enough and does break with the more lenient approach of <=1.2.10, it has now been fixed both in the 1.2.x branch and the master branch, and will be included in release 1.2.12 and the upcoming major version 2.0.0.
This should make it compatible with <1.2.10 again and make it easier to write slightly wrong value types, such as float to int, string number to int or float, etc. But attempting to write non-numeric strings to numeric fields will still raise an exception, as it should.
from pyshp.
Version 1.2.12 which fixes this issue is now up on PyPI.
from pyshp.
Hi Karimbahgat,
I just wanted to leave a quick comment regarding this issue. I think there is a bug that caused this which still exists. Specifically, with the 'field' method when using floats. For example, in the un-fixed version I create float field like so:
shp.field('lat', 'F')
shp.field('lon', 'F')
lat = 60.47
shp.point(lon,lat)
shp.record(lat, lon)
shp.save('out.shp')
In this example with the previous version I got the same error code as the others mentioned, even though I know this was a float field and float number.
In the new version 1.2.12 I did the same and it writes the field as 60. (Thus dropping the decimal). I believe this is what is causing the error for people. However, of course I can fix the problem like such:
shp.field('lat','F', 5,5)
Thus, it seems users need to explicitly specify the length of floats fields or else it may change it to integer.
from pyshp.
Hi klasko2, you raise a good point. This is actually the reason I originally didn't want to force value type conversion to the field type, and instead raise an exception.
As you say, the reason here is that the default decimal length is 0, which is stated in the Readme. This has always been the case, but previous versions of pyshp simply ignored the decimal arg, and wrote the entire floating nr regardless, leading to possibly incorrect files.
So the behavior is correct, but I agree with your point that since the 'F' type is float by definition it should default to using decimals>0. Also with the 'N' numeric type I think a case can be made as you suggest that most users would expect it to default to decimals rather than int. Defaulting to decimals is also less damaging because if people forget to specify field type then at least no information is lost.
I am going to raise this as a separate improvement issue, and we can discuss there whether the default decimal should be changed to >0 and if so how many decimals. Also maybe a discussion of the default field size, and if that should be different for different data types.
from pyshp.
Related Issues (20)
- shapes and records are not pickable HOT 3
- ValueError when reading shapefile from ZIP archive HOT 3
- logging should use named logger rather than root logger HOT 1
- Document the reason for first field always being a DeletionFlag HOT 2
- "README: Testing" section seems out of date HOT 1
- add/modify test routine for running pytest / ship test file with installation HOT 2
- pyshp 2.3 closing bytesio objects with Writer HOT 5
- Add read geojson capability HOT 1
- Support for getting encoding from .cpg files HOT 2
- Enhance combination with shapely HOT 2
- Why was Writer.save() removed? HOT 1
- How to modify the shape value HOT 1
- struct.error: unpack requires a buffer of 32 bytes HOT 2
- How to get the field name for data? HOT 1
- Inconsistent bounding box filtering HOT 1
- check if polygon is closed fails HOT 1
- Build marked as failing. Tests are not running in CI. Github /actions/setup-python no longer supports Python 2.7. HOT 1
- PyShp's own Example from README.md fails under Doctest with Python 2.7. .__geo_interface__ = behaves differently in Python 2? HOT 1
- Doc test in README.md fails due to undefined variable HOT 1
- Run tests on Python 3.10 and higher versions (as well as 2.7, 3.5, ..., 3.9)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyshp.