mapbox / untiler

Stitch image tiles into larger composite TIFs

untiler

Utility to take a directory of {z}/{x}/{y}.(jpg|png) tiles, and stitch into a scenetiff (tif w/ exact merc tile bounds). Future versions will support fast indexed reading directly from tar archives.

Install

make a virtual env + activate, then:

pip install untiler

Dev installation

git clone [email protected]:mapbox/untiler.git

cd untiler

pip install -e .

Usage

Usage: untiler [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  inspectdir
  streamdir
  streammbtiles

streamdir

Given a directory of tiles + a read template, mosaic into tifs at a lower parent "composite" zoom extent

untiler streamdir [OPTIONS] INPUT_DIR OUTPUT_DIR

-c, --compositezoom INTEGER  Tile size to mosaic into [default=13]
-z, --maxzoom INTEGER        Force a maxzom [default=max in each
                           compositezoom area]
-l, --logdir TEXT            Location for log files [default=None]
-t, --readtemplate TEXT      File path template
                           [default='jpg/{z}/{x}/{y}.jpg']
-s, --scenetemplate TEXT     Template for output scenetif filenames
                           [default='{z}-{x}-{y}-tile.tif']
-w, --workers INTEGER        Number of workers in the processing pool
                           [default=4]
-x, --no-fill                Don't fill in with lower zooms
-r, --tile-resolution       Size of input tiles for eg 256, 512 etc
--help                       Show this message and exit.

streammbtiles

Mosaic an mbtiles into tifs of "composite" zoom extent

untiler streammbtiles [OPTIONS] MBTILES OUTPUT_DIR

Options:
  --co NAME=VALUE              Driver specific creation options.See the
                               documentation for the selected output driver
                               for more information.
  -c, --compositezoom INTEGER  Tile size to mosaic into [default=13]
  -z, --maxzoom INTEGER        Force a maxzom [default=max in each
                               compositezoom area]
  -s, --scenetemplate TEXT     Template for output scenetif filenames
                               [default='{z}-{x}-{y}-tile.tif']
  -w, --workers INTEGER        Number of workers in the processing pool
                               [default=4]
  -x, --no-fill                Don't fill in with lower zooms
  --help                       Show this message and exit.

inspectdir

Stream [x, y, z]s of a directory

untiler inspectdir [OPTIONS] INPUT_DIR

Options:
-z, --zoom INTEGER  Zoom to inspect [default = all]
--help              Show this message and exit.

Outputs a line-delimited stream of tile [x, y, z]s; useful to pipe into mercantile shapes to visualize geometry:

untiler inspectdir <dir> -z 19 | mercantile shapes | fio collect | geojsonio

untiler's Issues

Use rasterio for tile reading

Using rasterio for input tile reading should be faster for reading input images. To implement:

• Use syntax:

   with rasterio.drivers():
       with rasterio.open(...) as src:
           imdata = src.read()
  

• Update tiler to handle (depth, height, width) arrays vs (height, width, depth) and (height, depth) arrays

cc @sgillies

Revisit command line options

At the risk of 🚲 🏡 ... @dnomadb we have some conflicts with traditional options (-c for one) and I'd like to suggest --foo-bar instead of --foobar.

How to get this to work with QGIS

So it doesn't seem to work with QGIS Tiles, and the things it works with output nonsense.

these were originally grey, and you can see they didn't even stitch properly, any insight on how to make this work?

KeyError: '\\d'

Hi, I recently installed untiler. When I run any of the untiler commands, I get the error attached below. I am using Python 3.9.1.

I'm not sure if it's an issue with the python version, I am currently trying to downgrade my python version, and will update this issue accordingly.

in __exit__
Traceback (most recent call last):
  File "/usr/lib/python3.9/sre_parse.py", line 1039, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\d'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/c_abraham/.local/bin/untiler", line 8, in <module>
    sys.exit(cli())
  File "/usr/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/cli.py", line 53, in streammbtiles
    untiler.stream_dir(input_tile_dir, output_dir, compositezoom, maxzoom, None, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/__init__.py", line 225, in stream_dir
    template, readTemplate, separator = tile_utils.parse_template("%s/%s" % (inputDir, read_template))
  File "/home/c_abraham/.local/lib/python3.9/site-packages/untiler/scripts/tile_utils.py", line 151, in parse_template
    return valPattern.sub('\d+', template), valPattern.sub('%s', template), separator[0]
  File "/usr/lib/python3.9/re.py", line 327, in _subx
    template = _compile_repl(template, pattern)
  File "/usr/lib/python3.9/re.py", line 318, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/usr/lib/python3.9/sre_parse.py", line 1042, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \d at position 0

pydocstyle reports 29 missing docstrings

$ python -m pydocstyle --count --select=D1 --add-ignore=D105 .
...
29

sudo: false with rasterio deps

As per chat w/ @yhahn (https://mapbox.slack.com/archives/satellite/p1441040393007215 + many other convos), we want to able to have modules such as this testable on travis w/ sudo: false. However, I am not sure how to accomplish this on a repo that uses rasterio, which has dependencies that are installed via sudo apt-get install ....

@sgillies can you help guide me in this?

Tests broken on master

Tests fail on Python 3.6 - see this Travis run.

    def test_extract_mbtiles():
        with TestTiler() as tt:
            testpath = tt.path
            testmbtiles = os.path.join(os.path.dirname(__file__), 'fixtures/testtiles.mbtiles')
            runner = CliRunner()
            result = runner.invoke(cli, [
                'streammbtiles', testmbtiles, testpath, '-z', '16', '-x', '-s',
                '{z}-{x}-{y}-mbtiles.tif', '--co', 'compress=lzw'])
            assert result.exit_code == 0
            expected_checksums = [[13858, 8288, 51489, 31223], [17927, 52775, 411, 9217]]
            for o, c in zip(result.output.rstrip().split('\n'), expected_checksums):
                with rio.open(o) as src:
                    checksums = [src.checksum(i) for i in src.indexes]
>                   assert checksums == c
E                   assert [17927, 52775, 411, 9217] == [13858, 8288, 51489, 31223]
E                     At index 0 diff: 17927 != 13858
E                     Use -v to get the full diff

Noticed this while working on the autodeploy branch. Initially thought this was failing due to changes I'd made, but I think this is some sort of config/dependency rot that's happening otherwise.

Open source tile-stitcher

Open Source is the Best Source

cc @camillacaros @sgillies @yhahn

Support for 512x512 tiles: ValueError: Source shape is inconsistent with given indexes

I am having trouble getting untiler to work with a directory of 512x512 tiles. Support for this does not seem to be documented, but it sounds like a typical use case given Mapbox's 512x512 default in Studio. Is this supported? Is it a rasterio issue, like this?

This is what happens when trying to untile with 512x512 images. (An equivalent run with 256x256 tiles succeeds.)

$ untiler streamdir -t {z}/{x}/{y}.png ./tiles ./untiles 
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
/usr/local/lib/python3.6/site-packages/rasterio/__init__.py:240: NotGeoreferencedWarning: Dataset has no geotransform set. Default transform will be applied (Affine.identity())
  s = DatasetReader(fp, driver=driver, **kwargs)
tiles/18/66179/97359.png errored
tiles/18/66179/97403.png errored
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 201, in streaming_tile_worker
    raise e
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 192, in streaming_tile_worker
    dst.write(imdata, window=window)
  File "rasterio/_io.pyx", line 1234, in rasterio._io.DatasetWriterBase.write
ValueError: Source shape is inconsistent with given indexes
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/untiler", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/untiler/scripts/cli.py", line 32, in streamdir
    untiler.stream_dir(input_dir, output_dir, compositezoom, maxzoom, logdir, readtemplate, scenetemplate, workers, creation_options, no_fill)
  File "/usr/local/lib/python3.6/site-packages/untiler/__init__.py", line 255, in stream_dir
    for p in pool.imap_unordered(streaming_tile_worker, tiler.get_sub_tiles(allTiles, superTiles)):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
ValueError: Source shape is inconsistent with given indexes

Python 3.6 compatibility

A few things to change (that I see now):

• A few integer division changes: https://github.com/mapbox/untiler/blob/master/untiler/scripts/tile_utils.py#L49
• Bigger prob: mbutil is (a lot farther) from 3.5 https://github.com/mapbox/untiler/blob/master/untiler/scripts/mbtiles_extract.py#L5. I think removing it entirely and reading tiles using sqlite3 is preferable.

cc @perrygeo