tilezen / tapalcatl-py Goto Github PK

Eg to zoom 17 for a 256 sized tile (which would normally stop at 16).

I deployed using zappa and have rawr tiles and metatiles in a s3 bucket

when I got to /preview.html to see the tiles, each tile request is responded with 403

Why is the request looking like
https://some_aws_domain/tilezen/vector/v1/512/all/1/1/0.mvt?api_key=&

why does it have tilezen/vector/v1? should it be in s3 as well?
Should there be other configurations that zappa doesn't do (or perhaps specific configurations that must exist in the zappa configs?)

Like transit instead of all

Right now 256 or 512 tile_size needs to be specified or it'll 404 without clear explanation as to why.

Bonus points for better error messages around tile size.

See tilezen/tilequeue#344.

Should be an (app startup) config to determine which hash pattern to use.

Using this template:
https://github.com/tilezen/vector-datasource/blob/master/tilejson/tilejson.json.erb#L14-L22

We documented this before over at https://mapzen.com/documentation/vector-tiles/use-service/#tilejson which amounts to http://tile.mapzen.com/mapzen/vector/v1/tilejson/mapbox.json in the old way.

So adapt to the new domain and URL pattern setup, for instance: https://tile.nextzen.org/tilezen/vector/v1/tilejson/mapbox.json and https://tile.nextzen.org/tilezen/vector/v1/512/tilejson/mapbox.json (the URL pattern inside the tilejson file will be adapted per tile size).

A web mercator tile z/x/y can only exist if x or y are at least 0 and less than 2**z. We can use this information to quickly respond to requests outside of that range for either coordinate (i.e: without needing to check S3). We can also respond to requests where z is negative or larger than some configurable value.

We should still respond with a 404, but can include a helpful message so that clients get more information about why the tile isn't present.

Hi future self and others that might run into this problem.

I was trying to install Zappa after having upgraded to Python 3.7 and not having any luck. When installing Zappa, I ran into errors while compiling the pyyaml package that looked like this:

    /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
        PyObject *curexc_traceback;
                  ^
    ext/_yaml.c:24212:24: error: no member named 'exc_type' in 'struct _ts'
        tmp_type = tstate->exc_type;
                   ~~~~~~  ^
    ext/_yaml.c:24213:25: error: no member named 'exc_value' in 'struct _ts'; did you mean 'curexc_value'?
        tmp_value = tstate->exc_value;
                            ^~~~~~~~~
                            curexc_value
    /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
        PyObject *curexc_value;
                  ^
    ext/_yaml.c:24214:22: error: no member named 'exc_traceback' in 'struct _ts'; did you mean 'curexc_traceback'?
        tmp_tb = tstate->exc_traceback;
                         ^~~~~~~~~~~~~
                         curexc_traceback
    /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
        PyObject *curexc_traceback;
                  ^
    ext/_yaml.c:24215:13: error: no member named 'exc_type' in 'struct _ts'
        tstate->exc_type = local_type;
        ~~~~~~  ^
    ext/_yaml.c:24216:13: error: no member named 'exc_value' in 'struct _ts'; did you mean 'curexc_value'?
        tstate->exc_value = local_value;
                ^~~~~~~~~
                curexc_value
    /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
        PyObject *curexc_value;
                  ^
    ext/_yaml.c:24217:13: error: no member named 'exc_traceback' in 'struct _ts'; did you mean 'curexc_traceback'?
        tstate->exc_traceback = local_tb;
                ^~~~~~~~~~~~~
                curexc_traceback
    /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
        PyObject *curexc_traceback;
                  ^
    51 warnings and 15 errors generated.
    error: command 'clang' failed with exit status 1

    ----------------------------------------

  Failed building wheel for pyyaml
Command "/Users/ian.dees/.local/share/virtualenvs/tapalcatl-py-NSq2EmQ3/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/g1/sdx4zczd78q3x8y9jzqjcdhc0000gp/T/pip-install-7hxh7ba4/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/g1/sdx4zczd78q3x8y9jzqjcdhc0000gp/T/pip-record-yyt616j2/install-record.txt --single-version-externally-managed --compile --install-headers /Users/ian.dees/.local/share/virtualenvs/tapalcatl-py-NSq2EmQ3/bin/../include/site/python3.7/pyyaml" failed with error code 1 in /private/var/folders/g1/sdx4zczd78q3x8y9jzqjcdhc0000gp/T/pip-install-7hxh7ba4/pyyaml/

It looks like Python 3.7 breaks PyYAML's build, so I installed pyenv and used pyenv to install Python 3.6.6, switch to that version and re-install dependencies for tapalcatl-py with pipenv --rm && pipenv install --dev.

I'm really interested in trying this tool out, but can't quite figure out what the source "metatiles" directory structures should look like, or how they can be generated from a .mbtiles file. Is there any tool to do that? My expected workflow would be: shapefiles -> geojson -> mbtiles (tippecanoe) -> zip (via ???) -> S3.

A longer version of this question here: https://gis.stackexchange.com/questions/271670/how-to-generate-metatiles-for-use-with-tapalcatl

Looking at a debug build of tapalcatl-py, I see that 80-90% of the runtime is spent waiting around for S3 to do its job:

We already use cachetools for an in-memory cache to reduce the time spent waiting for S3. @mojodna reminded me a while back that Lambda functions can store stuff to /tmp and it might still be there for the next execution. It might be a good idea to use a slightly more sophisticated caching system like flask-caching to store to /tmp so that we would get a better hit rate on those S3 objects.

This is also why we should build bigger metatiles 😄 .

Thanks for this awesome repository and your great work. I just had a look at this repo and wanted it to set it up to work with a requester pays bucket on S3.

I think this would be fixable by changing:

    get_params = {
        "Bucket": s3_bucket,
        "Key": s3_key,
    }

to:

    get_params = {
        "Bucket": s3_bucket,
        "Key": s3_key,
        "RequestPayer": "requester",
    }

Obviously not everyone needs this feature, so it would make sense to make this configurable :).

We're getting errors while trying to access the S3 zip files. Is the S3 bucket still valid?

server.UnknownMetatileException: AccessDenied at s3://vector-tiles-prod/20171221/42ec2/all/11/328/791.zip

Flask might already be doing this for us, but we should make sure that tapalcatl-py sends a Connection: close header on all server (i.e: 5xx) error responses. This is so that when we have region-wide failures (e.g: S3 goes down) the client will reconnect and be routed to a non-failing region.