mapbox / mapbox-cli-py Goto Github PK

Continued from mapbox/mapbox-sdk-py#13 (comment)

Here's a proof-of-concept asyncio version of the batch_geocode CLI: https://gist.github.com/perrygeo/60633de1b2cd90b07b3d

Some issues with it

• This only works on Python 3.4 (would need to port to Trollius to get python < 3.4 support).
• no error handling
• no match thresholds or resolving ambiguous results, it just takes the first feature
• needs to respect rate limits
• not clear if this would integrate with, replace or sit along side the current mapbox geocode

The SDK PR is in the works. Meanwhile let's talk about the implications for the CLI...

As noted in the SDK planning ticket the Datasets API is a bit trickier as it involves two resources types: Datasets which are collections of Features. Then there are all of the standard "CRUD" operations on those resources.

How do we best represent these actions in the CLI? My initial thought is sub-sub-commands like:

mapbox dataset list-features
               read-features
               update-features
               delete-features
               list-datasets
               create-datasets
               read-datasets          

A starting point for talking about how we're going to package and distribute this thing.

@tmcw mentioned that homebrew and apt-get should be the two primary targets.

I'd add that packaging a windows installer would really expand the potential audience. Whether it's worth the added maintenance burden (I don't even have access to a windows computer at the moment) is debatable.

• identify target platforms
• identify contributors with experience in those packaging systems
• ...
• everyone in the world has access to a one-line/one-click installer for command line access to Mapbox APIs

We should add a --worldfile option the staticmaps command which would write a GDAL-recognizable world file alongside the image. This would help users to quickly grab a portion of their tilesets and load them into a local GIS viewer to compare to local data.

We'd need to determine the worldfile extension based on requested image.

Because the exact extent would only be know if bounds were provided, we could make it invalid to specify --worldfile with --auto. Or is the --auto behavior defined well enough to predict it?

We should include an option to use the mapbox.places-permanent dataset. The SDK supports it already, just a matter of specifying it in the service constructor.

For compatibility with mapbox-upload, the current upload CLI arguments follow suit. However, we might want to consider tweaking the design a bit

1. mapbox upload DESTINATION-tileset and SOURCE-file - the current ordering is opposite what one would expect from unix tools like cp which generally follow the cmd source destination pattern. Switch 'em?
2. we should consider making --name default to the basename of the file rather than None. This avoids having a bunch of "Untitled" tilesets in studio should you forget to supply the optional name.
3. we should also consider making the username in the tileset id optional. We can infer the username from the access token and prepend it if there is no username supplied. This would help to write more reusable bash scripts intended to be run with different accounts.

Should we implement these changes? If so, do they belong here or could some be baked into SDK behavior?

continuation of mapbox/mapbox-sdk-py#110

The problem is in the CLI. Using the python SDK gives be similar results to the bites

>>> import mapbox
>>> result = mapbox.Geocoder().forward("Florianópolis, Brazil").json()
>>> for x in result['features']:
...     print x['text']
Florianópolis
Jardim Florianópolis
Rua Florianopólis
Rua Florianópolis
Rua Florianopolis

But using the CLI, I get an error

$ mapbox geocoding "Florianópolis, Brazil"
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py:1303: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return ''.join(map(quoter, s))
Traceback (most recent call last):
  File "/Users/mperry/env/mapbox/bin/mapbox", line 9, in <module>
    load_entry_point('mapboxcli==0.2.0', 'console_scripts', 'mapbox')()
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/mperry/work/mapbox-cli-py/mapboxcli/scripts/geocoding.py", line 101, in geocoding
    q, types=place_type, lat=lat, lon=lon, country=country)
  File "/Users/mperry/work/mapbox-sdk-py/mapbox/services/geocoding.py", line 51, in forward
    dataset=self.name, query=address)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/uritemplate/template.py", line 127, in expand
    return self._expand(var_dict, False)
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/uritemplate/template.py", line 89, in _expand
    expanded.update(v.expand(expansion))
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/uritemplate/variable.py", line 333, in expand
    expanded = expansion(name, value, opts['explode'], opts['prefix'])
  File "/Users/mperry/env/mapbox/lib/python2.7/site-packages/uritemplate/variable.py", line 284, in _string_expansion
    return quote(value, self.safe)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 1303, in quote
    return ''.join(map(quoter, s))
KeyError: u'\xf3'

Failing test case pushed to the geounicode branch.

/cc @ingalls @sgillies

@planemad reports

$ mapbox upload username.tileset output.geojson 
Usage: mapbox upload [OPTIONS]
Error: Got unexpected extra arguments (username.tileset output.geojson)

... with instructions for development setup and running tests

cc @perrygeo

        result = runner.invoke(
            main_group,
            ['geocoding', '--reverse'],
            input='{0},{1}'.format(lon, lat),
            env={'MapboxAccessToken': 'bogus'})
>       assert result.exit_code == 0
E       assert -1 == 0
E        +  where -1 = <Result ConnectionError(u'Connection refused: GET https://api.mapbox.com/geoco...123xyz',)>.exit_code

tests/test_geocoding.py:93: AssertionError

where MAPBOX_ACCESS_TOKEN=123xyz is set in my environment. If I unset MAPBOX_ACCESS_TOKEN it works.

@sgillies is this just a matter of MAPBOX_ACCESS_TOKEN taking precedence over MapboxAccessToken?

We're developing this project outside of the mapbox-sdk-py project for product reasons, but want to keep it well synchronized. For 0.1, I set mapbox==0.5.0, which was a mistake because it prevents CLI users from getting mapbox-sdk-py bug fixes. I'll change it to mapbox>=0.5 and we'll continue like this, setting a new minimum version for every new mapbox-cli-py release.

• close issues
• release mapbox-sdk-py 0.5: mapbox/mapbox-sdk-py#90.
• pin mapbox package version
• tag (+ release)
• grab a glass and pop a cork

First draft:

$ mbx distance --help
Usage: mbx distance [OPTIONS] WAYPOINTS...

  The Distance API returns all travel times between many points (also known
  as Distance Matrix). This is often used as input for solving routing
  optimization problems.

    $ mbx distance "[-122.681, 45.528]" "[-122.716, 45.525]" "[-122.691, 45.518]"

  The output is a json object with a "durations" key containing a 2D array
  of travel times between waypoints.

  An access token is required, see `mbx --help`.

Options:
  --profile [driving|walking|cycling]
                                  Mapbox direction profile id
  -o, --output TEXT               Save output to a file.
  --help                          Show this message and exit.

It should be possible to accept an s3 uri as an upload object. With a file we transfer the data to s3 with stage and then calling create to start the processing. With an s3 uri we skip the transfer and call create directly

Questions

• Does this work across accounts?
• If not, there may be an intermediate option where we can use boto to transfer between s3 buckets (advantage: no need to pull data locally) then call create.

cc @rclark

First draft

$ mbx surface --help
Usage: mbx surface [OPTIONS] MAPID LAYER FIELDS WAYPOINTS...

  Mapbox Surface API enables flexible querying of data stored in vector
  tiles on Mapbox, to create results like elevation profiles.

        $ mbx surface mapbox.mapbox-terrain-v1 contour ele \
        "[-122.681, 45.528]" "[-122.716, 45.525]"

  An access token is required, see `mbx --help`.

Options:
  -z, --zoom INTEGER              Zoom level to query (default: 14)
  --interpolate / --no-interpolate
                                  Weighted average interpolation (default:
                                  True)
  --geojson / --no-geojson        Return geojson feature collection (default:
                                  full response json)
  -o, --output TEXT               Save output to a file.
  --help                          Show this message and exit.

I'd like to make this possible:

$ mapbox datasets create
$ fio cat a.shp b.shp c.shp | mapbox datasets update-features $id

reading a GeoJSON feature sequence from stdin, or reading from a GeoJSON feature collection file.

$ mapbox datasets create
$ fio cat a.shp b.shp c.shp | fio collect > lolwut.json
$ mapbox datasets update-features $id lolwut.json

I get this error when running mapbox:

pkg_resources.DistributionNotFound: The 'futures==2.2.0' distribution was not found and is required by boto3

Full dump of error output: https://gist.github.com/lxbarth/5559bec90ebccdad16ce

My python version:

$ python --version
Python 2.7.11

Currently, using upload still require user to manually go into Studio and set new tileset rights to public (default is private). Would be great to have a function added that sets privacy via CLI, especially in cases where upload replaces an existing tileset that is already exposed via styles.

maybe something like: mapbox upload <TILESET> <INFILE> (public,private)

First draft

$ mbx uploads --help
Usage: mbx uploads [OPTIONS] USERNAME INPUT TILESET

  Upload data to Mapbox accounts. All endpoints require authentication.
  Uploaded data lands at https://www.mapbox.com/data/ and can be used in new
  or existing projects.

    $ mbx uploads username data.geojson username.data

  An access token with upload scope is required, see `mbx --help`.

Options:
  --name TEXT  Name for the data upload
  --help       Show this message and exit.

Featuring new commands compatible with fio-cat and similar programs (#36).

Hello,

I'm a junior developer and I'm playing around with the Mapbox CLI. I installed through Homebrew with no problems, but when I tried setting the token as per the instructions here on Github:

$ mapbox --access-token tokenfromstudiohomepagegoeshere12323342423442

I'm getting

Usage: mapbox [OPTIONS] COMMAND [ARGS]...
Error: Missing command.

Is there something basic I'm missing here?

The directions, distance and surface APIs all require input waypoints. Let's specify this in more detail...

At the python SDK level, all of the services take an iterable/list of GeoJSON-like feature mappings. At the command line, it's not clear to me exactly what the inputs will be.

The CLI will take x and normalize it to a list of features where x could potentially be

• line-delimited GeoJSON features on the stdin stream (output of fio cat)
• path to a GeoJSON file containing a featurecollection of points (probably the most common case?)
• paths to GeoJSON files each containing point features (makes interop with geocoding easier since you're likely to get one feature per call)
• stream or file of GeoJSON geometries
• stream or multiple args with JSON arrays of coordinate pairs "[lat, lon]" (ala the input to reverse geocoder)
• ... others? ...

So the question is, What types of inputs will we support for waypoints? On one hand I want to support the most common workflows even if that means accepting a bunch of different input formats. On the other hand, unix design principles - let other tools do the lifting to get data into our expected format.

And on a related note, if we're accepting GeoJSON and the service requires points, how should we handle other geometry types? Automatically explode multipoints and lines to points? Skip them? Be strict? Most of these validation decisions will likely be implemented at the SDK level but it might be useful to discuss them in this context.

cc @sgillies

First draft of proposed interface:

$ mbx directions --help
Usage: mbx directions [OPTIONS] WAYPOINTS...

  Calculate optimal route with turn-by-turn directions between up to 25
  waypoints.

    $ mbx directions "[-122.681032, 45.528334]" "[-122.71679, 45.525135]"

  An access token is required, see `mbx --help`.

Options:
  --profile [mapbox.driving|mapbox.walking|mapbox.cycling]
                                  Mapbox direction profile id
  --alternatives / --no-alternatives
                                  Generate alternative routes?
  --instructions [text|html]      Format for route instructions
  --geometry [geojson|polyline|false]
                                  Geometry encoding
  --steps / --no-steps            Include steps in the response
  --geojson / --no-geojson        Return geojson feature collection (default:
                                  full response json)
  -o, --output TEXT               Save output to a file.
  --help                          Show this message and exit.

@sgillies let's coordinate on a new release, it's been a while and I want to make sure our release process is ironed out, esp. with homebrew builds.

I've started a 0.4 milestone with @mattficke's work on geocoding bbox support and my upload arg change PR.

https://github.com/mapbox/mapbox-cli-py/pulls?q=is%3Apr+is%3Aopen+milestone%3A0.4

Anything else we want to get it for 0.4?

Once mapbox/cligj#9 is merged

Just got this somewhat-cryptic message on trying to upload this valid demo geojson file

| => mapbox upload --name landplanner.demopoints demo.geojson
Usage: mapbox upload [OPTIONS] TILESET [INFILE]

Error: Invalid value: Object `None` has no .read method, a file-like object is required

My key is properly configured (I gave it all 12 scopes just in case) and exported to $MAPBOX_ACCESS_TOKEN, and the file is valid, so I'm not sure what's going on here. Is my account missing some authorization globally?

Replaces mapbox/mapbox-sdk-py#10.

We're developing a simple and easy to use CLI for the mapbox client module both as a service to customers and as a check on the usability of the module.

The CLI should be 99% about arg/opt definition and interoperation with other programs in the shell.

The CLI will have first class support for JSON and GeoJSON.

We'll rely on other programs like sed, awk, jq for finer parsing of responses from APIs at first, absorbing some of this when it makes sense.

A config file is something quality tools have. It would allow developers to have per-directory access tokens among other things. .ini style is (see Python's distutils or https://pip.pypa.io/en/latest/user_guide.html#config-file) one way to go.

Replaces mapbox/mapbox-sdk-py#25.

Moar APIs and moar features.

See mapbox/mapbox-sdk-py#107

Instead of getting a featurecollection with the first 5 results, many use cases at the command line will require selecting a single result, usually the first feature.

I've been using this pattern to get a GeoJSON Feature

mapbox geocoding "place" | jq -c .features[0]

Useful for geocoding multiple places and making a GeoJSON FeatureCollection

cat places.txt | parallel mapbox geocoding {} | jq -c .features[0] | fio collect

But jq isn't installed everywhere (it should be, it's awesome) and the arguments are not immediately clear.

Proposal

A --first-feature option to output the first, highest-ranked Feature (without line-breaks to facilitate streaming) rather than a FeatureCollection.

The parallel geocoding of multiple addresses becomes a little cleaner:

cat places.txt | parallel mapbox geocoding --first-feature {} | fio collect

The only reason I'd hesitate to implement this would be our design principles #10:

We'll rely on other programs like sed, awk, jq for finer parsing of responses from APIs at first, absorbing some of this when it makes sense.

So, does --first-feature make sense? Or do we just rely on jq?

-help says its MAPBOX_ACCESS_TOKEN but using this gave me the error

Before running this command you must set a Mapbox access token to an environment
variable named MapboxAccessToken. The access token must include the "uploads:write"
scope. Learn more about access tokens at https://www.mapbox.com/developers/api/#access-tokens.

So I tried export MapboxAccessToken= and this did the trick. I'm running v0.2.0.

cc @perrygeo

In closed issues I've found similar reports, but it seems that subsequent updates—I'm using v0.3.1—haven't dealt with the problem. I'm occasionally getting error messages (see screenshot below); in this case mapbox-cli-py is part of a shell script that runs within BitBar.

• Full API support
• High quality documentation
• Easy installation on Linux, OS X, Windows

I installed with sudo pip install mapboxcli and I get the following when running mapbox. I haven't made much progress debugging myself. Any idea what's going on?

$ mapbox
Usage: mapbox [OPTIONS] COMMAND [ARGS]...

  This is the command line interface to Mapbox web services.

  Mapbox web services require an access token. Your token is shown on the
  https://www.mapbox.com/developers/api/ page when you are logged in. The
  token can be provided on the command line

    $ mapbox --access-token MY_TOKEN ...

  as an environment variable named MAPBOX_ACCESS_TOKEN (higher precedence)
  or MapboxAccessToken (lower precedence).

    $ export MAPBOX_ACCESS_TOKEN=MY_TOKEN
    $ mapbox ...

  or in a config file

    ; configuration file mapbox.ini
    [mapbox]
    access-token = MY_TOKEN

  The OS-dependent default config file path is something like

    ~/Library/Application Support/mapbox/mapbox.ini
    ~/.config/mapbox/mapbox.ini
    ~/.mapbox/mapbox.ini

Options:
  --version            Show the version and exit.
  -v, --verbose        Increase verbosity.
  -q, --quiet          Decrease verbosity.
  --access-token TEXT  Your Mapbox access token.
  -c, --config PATH    Config file
  --help               Show this message and exit.

Commands:
  config       † Warning: could not load plugin. See `mapbox config --help`.
  dataset      † Warning: could not load plugin. See `mapbox dataset --help`.
  directions   † Warning: could not load plugin. See `mapbox directions
               --help`.
  distance     † Warning: could not load plugin. See `mapbox distance --help`.
  geocoding    † Warning: could not load plugin. See `mapbox geocoding
               --help`.
  mapmatching  † Warning: could not load plugin. See `mapbox mapmatching
               --help`.
  staticmap    † Warning: could not load plugin. See `mapbox staticmap
               --help`.
  surface      † Warning: could not load plugin. See `mapbox surface --help`.
  upload       † Warning: could not load plugin. See `mapbox upload --help`.

Sort of like npm-config. It would tell the user

• active access token
• output verbosity

and whether these values are coming from environment variables or config file. It's also going to be handy for configuration testing.

A general design question: What is our strategy for dealing with validation of input parameters?

The way it works now is

• some known bad parameters are caught by the SDK and raise a python exception (the stacktrace is a bit intimidating and exposes the python underbelly to the CLI world)
• other bad parameters are passed through to the API and the error message is conveyed (actually much cleaner errors, for example the static map size error in #19 returns Error: {"message":"Height must be between 1-1280."}) This puts a burden on the API when we could easily catch it before we hit the request.

I'm thinking the best approach is:

1. do as much validation as possible in the SDK and raise a consistent exception type (ValueError seems too general)
2. catch that exception at the CLI and re-raise using either a click.UsageError or a click.BadParameter

Alternatively, we could do all the validation at the CLI end and let the Python SDK play fast and loose, relying on the API response.

Doing the validation in both spots seems like a bad plan.

/cc @sgillies

Currently the mapmatching API is limited to a single Feature with LineString geometry.

But that doesn't mean we have to limit the CLI to a single feature. What about taking multiple features and just calling the API multiple times (possibly asynchronously)?

We're using Click's app directory, but need to tell users exactly where this is. We could surface it in the help. We could surface it in mapbox-config.

There's an implicit 1280x1280 pixel limit on the size.

For example, creating wallpapers for my Thinkpad X1 Carbon fails with an API error:

mapbox staticmap --zoom 12 --size 2560 1440 --lat 52.500385 --lon 13.419779 mapbox.emerald out.png

It would be great, to either document that limit and catch the error already at the argparser level or provide transparent stitching, like @zugaldia does in his wallmapper (this may be out of scope, though).

This is the only blocker for Python 3.5 support

With warnings.filterwarnings('error')

  File "/Users/mperry/work/mapbox-cli-py/mapboxcli/scripts/geocoding.py", line 114, in geocoding
    for lon, lat in map(coords_from_query, iter_query(query)):
  File "/Users/mperry/work/mapbox-cli-py/mapboxcli/scripts/geocoding.py", line 30, in coords_from_query
    vals = re.split(r"\,*\s*", query.strip())
  File "/Users/mperry/env/mapbox35/lib/python3.5/re.py", line 203, in split
    return _compile(pattern, flags).split(string, maxsplit)
FutureWarning: split() requires a non-empty pattern match.

Why is a future warning a blocker? Because for whatever reason (probably a bug in python 3.5 click), the future warning is captured in the output which messes up the CLI tests.

>       assert result.output.strip() == body
E       assert '/Users/mperr...71, 37.5227]}' == '{"query": [-7...71, 37.5227]}'
E         - /Users/mperry/env/mapbox35/lib/python3.5/re.py:203: FutureWarning: split() requires a non-empty pattern match.
E         -   return _compile(pattern, flags).split(string, maxsplit)
E           {"query": [-77.4371, 37.5227]}

tests/test_geocoding.py:70: AssertionError

Catching up with the SDK.

In mapbox/mapbox-sdk-py#100 I proposed that we leave caching up to users of the SDK.

A command (mapbox cache?) for inspecting and clearing the cache would round this out.

Just add a little blurb to its short_help after we merge.

I'm not sure if this is the way to go, but I've been curious about cx_freeze.

See also #18.

In anticipation of someday being able to change the name of the main command to mapbox.

I'm currently testing mapbox results for walking directions from one location to a public transportation stop/station in the Berlin area (Germany). However, mapbox returns only driving directions. The question is, if I'm doing something wrong, or if mapbox is simply incapable of handling such a request.

These are the commands:

MB_DIRECTIONS=$(mapbox directions "[$CURRENT_LONG, $CURRENT_LAT]" "[$STATION_LONG, $STATION_LAT]" --profile mapbox.walking --alternatives)
MB_INSTRUCTIONS=$(echo "$MB_DIRECTIONS" | jq '.routes[0].steps[] .maneuver.instruction' -M -r)
MB_DISTANCES=$(echo "$MB_DIRECTIONS" | jq '.routes[0].steps[] .distance' -M -r)

Variables for current lat/long and station lat/long are defined earlier.

This is the mapbox result (note the difference between linear and "walking" distance):

From: Herbert-von-Karajan-Straße 1, Berlin, Berlin 10785, Germany [52.510032,13.369693]
To: Potsdamer Straße, Berlin, Berlin 10785, Germany [52.507568,13.369186]

109     Head right
40      Turn right onto Tiergartenstraße (L 1137) at the end of the road
216     Turn right onto Ben-Gurion-Straße
153     Turn right onto Potsdamer Straße (B 1)
21      Turn left onto Potsdamer Straße
115     Turn right onto Potsdamer Straße (B 1) at the end of the road
0       You have arrived at your destination, on the right

Linear Distance: 275.9384 Meters
Walking Distance: 654 Meters
Approximate Walking Time: 8 Minutes

This is the (accurate) result in Apple Maps:
http://i.imgur.com/zLhpMQI.png

So Mapbox sends me the completely wrong way, north first, then east, then south, then onto the actual destination street, but not without a u-turn later on. These are clearly driving directions, and not even those are the fastest ones possible.

What's wrong here?