mapbox / vector-tile-base Goto Github PK

I noticed add_ring is not treating my closed ring like a closed ring:

ipdb> pt1
(19188372.37109346, -5402212.477493137)
ipdb> pt2
(19188372.37109346, -5402212.477493137)
ipdb> n
> /Users/cdestigter/checkout/sno/sno/pyvendor/vector_tile_base/engine.py(314)_points_equal()
    313         if pt1[0] is not pt2[0] or pt1[1] is not pt2[1] or (self._has_elevation and pt1[2] is not pt2[2]):
--> 314             return False
    315         return True

ipdb> pt1[0] == pt2[0]
True
ipdb> pt1[0] is pt2[0]
False

is not isn't the appropriate comparator here, it shoudl be !=. python doesn't guarantee two equal floats are the same float.

There is convinient fromGeojsonVt method at https://github.com/mapbox/vt-pbf. Could you consider adding the same function here?

They must be a list or a delta-encoded list, but if they are not, they are silently dropped instead.

Every time I create the tests data using create_test_data.py I get a different mvt file. I believe this is because before Python 3.6 dictionaries are not ordered.

This is probably the problematic place:
https://github.com/mapbox/vector-tile-base/blob/vt3/tests/create_test_data.py#L137-L151

The _is_in_values function is performing a type & value check which could be more efficient if It would be done inversely (type & value).

I noticed this when adding several (thousands) feature attributes to a layer.

👋 @flippmoke, first thanks for your work on this great module.

I'm using it to run simple tests on some vector tiles, but still in version 2. Latest commits seem to broke my tests (change from properties to attributes) I'm assuming this is a change to support vt3 ?

To prevent other breaks would it be possible to create some tags/release or some stale branches (e.g vt2) ?

Thanks

Is it possible to have a pulished PyPI package of this library? It would be better than cloning this repository with pip.

I try to decode the google vector tile ,but it throw an error.
code as follow:

error:

Hello.
I want to delete a layer from *.pbf , such as 'road' layer. How to do this？ Thanks for any help.

Add the ability to add repeated keys to a property by exposing a list option for Properties.

agri_data/tests/imperviousness/test_views.py:5: in <module>
    from vector_tile_base import VectorTile
/usr/local/lib/python3.9/site-packages/vector_tile_base/__init__.py:1: in <module>
    from . import engine
/usr/local/lib/python3.9/site-packages/vector_tile_base/engine.py:3: in <module>
    from . import vector_tile_pb2
/usr/local/lib/python3.9/site-packages/vector_tile_base/vector_tile_pb2.py:33: in <module>
    _descriptor.EnumValueDescriptor(
/usr/local/lib/python3.9/site-packages/google/protobuf/descriptor.py:755: in __new__
    _message.Message._CheckCalledFromGeneratedFile()
E   TypeError: Descriptors cannot not be created directly.
E   If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
E   If you cannot immediately regenerate your protos, some other possible workarounds are:
E    1. Downgrade the protobuf package to 3.20.x or lower.
E    2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
E   
E   More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

The protobuf changelog says the following:

We made some changes in Python language support in Protocol Buffers. Version 4.21.0 is a new major version, following 3.20.1. The new version is based on the upb library, and offers significantly better parsing performance than previous releases, especially for large payloads.
[...]

• Python upb requires generated code that has been generated from protoc 3.19.0 or newer.

@flippmoke
in vector_tile_base/engine.py, function: _add_legacy_attributes you check to see if the value was already inserted to self_values by:
if not (v in self._values and type(self._values[self._values.index(v)]) == type(v)):

Scenario:
when self._values contains same value with different variable type then self._values.index(v) will always find the first one so all duplicates of the second value will be considered as unseen values and will be added as duplicates to the protobuf message.
e.g. [0, 0.0, 0.0, 0.0, 0.0, ....]

proposed solution: check all matching values in self._values:
e.g.
`

def _is_in_values(self, val):

    found = False

    start_index = -1

    try:

        while not found:

            start_index = self._values.index(val, start_index + 1)

            if type(self._values[start_index]) == type(val):

                found = True

    except ValueError:

        pass

    return found

`

@flippmoke

in vector_tile_base/engine.py:

you have the Float wrapper class and the code that check for duplicates float values:

and
`

class Float(float):

def __new__(self, *args, **kwargs):

    return float.__new__(self, *args, **kwargs)

def __init__(self, *args, **kwargs):

    float.__init__(*args, **kwargs)

def _add_inline_value(self, v):

...

    elif isinstance(v, Float):

        try:

            index = self._float_values.index(v)

            return complex_value_integer(CV_TYPE_FLOAT, index)

        except ValueError:

            self._float_values.append(v)

            self._layer.float_values.append(v)

            return complex_value_integer(CV_TYPE_FLOAT, len(self._float_values) - 1)

`

The Float wrapper class preserve the double precision. When checking to find if Float value already exists in self._float_values list, you check against the double precision value while the float precision value is the one going inside the vector tile protobuf message thus all double values which maps to the same float value will be duplicated.

e.g. all the below numbers
Float(293.045998633665)
Float(293.045998634)
Float(293.045998633748)
will turn to
293.04598999
and this value will be duplicated 3 times inside the protobuf message.

proposed solution: convert the number inside the Float constructor to float precision
and pass it to the super class (float) constructor

i.e.

`
import struct

...

class Float(float):

def init(self, *args, **kwargs):

    x = float.__init__(*args, **kwargs)

    new_float = struct.unpack('f', struct.pack('f', x))[0]

    super(Float, self).__init__(x=new_float)

`