the-lay / tiler Goto Github PK
View Code? Open in Web Editor NEWN-dimensional NumPy array tiling and merging with overlapping, padding and tapering
Home Page: https://the-lay.github.io/tiler/
License: MIT License
N-dimensional NumPy array tiling and merging with overlapping, padding and tapering
Home Page: https://the-lay.github.io/tiler/
License: MIT License
Hiho,
thank you for this awesome project. I played a bit around with it and tried all tiling and windowing options, but it seems to me that in the stitched image the windowing functions do not sum up to 1 (=100%), but to 0. I tested it with the following code:
import numpy as np
from tiler import Tiler, Merger
import matplotlib.pyplot as plt
imsize = 64
im = np.reshape(np.arange(imsize*imsize),[imsize,imsize])
batch_size = 10
overlap = 5
tiler_mode = 'wrap'
windows_supported = ['boxcar', 'triang', 'blackman', 'hamming', 'hann', 'bartlett', 'flattop', 'parzen', 'bohman', 'blackmanharris', 'nuttall', 'barthann', 'overlap-tile']
tiler = Tiler(data_shape=im.shape,
tile_shape=(16,16),
channel_dimension=None,overlap=overlap,mode=tiler_mode,constant_value=1.0)
#Tiler.TILING_MODES == ['constant', 'drop', 'irregular', 'reflect', 'edge', 'wrap']
tiles_in_batches = [batch for _, batch in tiler(im, batch_size=batch_size)]
merged_images = []
for mwin in windows_supported:
merger = Merger(tiler,window=mwin)
#Merger.SUPPORTED_WINDOWS == ['boxcar', 'triang', 'blackman', 'hamming', 'hann', 'bartlett', 'flattop', 'parzen', 'bohman', 'blackmanharris', 'nuttall', 'barthann', 'overlap-tile']
for batch_id, batch in enumerate(tiles_in_batches):
merger.add_batch(batch_id, batch_size, batch)
imf = merger.merge(unpad=True,argmax=False)
merged_images.append(imf)
merged_images = np.array(merged_images)
fig,ax = plt.subplots(nrows=4,ncols=4,figsize=[14,14])
axm = ax.flatten()
for m,mim in enumerate(merged_images):
ima = axm[m].imshow(mim)
axm[m+1].imshow(im)
plt.show()
which results in
where the last image is the original (to-be-tiled and merged) image and the others are the different merges from all possible merging options.
What am I doing wrong here?
Is your feature request related to a problem? Please describe.
It is unclear how big is Tiler/Merger overhead, both in performance and memory usage. We should also create benchmarking scripts to compare how Tiler stands with other tiling/merging tools mentioned in the readme.
P.S. Baseline should probably be numpy's sliding window trick, numpy.lib.stride_tricks.sliding_window_view
. It's returning just views
tiler.get_tile_bbox_position
can return not just two opposite bbox corners, but all four corners.
For example, this can be helpful for napari-tiler to draw tiling preview in all dimensions.
Is your feature request related to a problem? Please describe.
I have been trying to use this library for the inference of TensorFlow binary and multiclass segmentation models. I am able to use the tiler object to perform the predictions. I have not been able to figure out how to leverage the merger for the following cases.
data_shape = 5000 x 3000 x 4
tile_shape = (256, 256, 4)
channel_dimension = 0
The output of the model can be either a batch of (N x 256 x 256 x 1) or (N x 256 x 256 x 6); where 6 is the number of classes.
ValueError: Passed data shape ([256 256 1]) does not fit expected tile shape ((256, 256, 4)).
Describe the solution you'd like
Would be great to have additional examples regarding similar use cases performing TensorFlow or PyTorch predictions.
Here is an example of what I have been trying:
model = tf.keras.models.load_model(model.hdf5)
image = rxr.open_rasterio(filename)
image = image.transpose("y", "x", "band")
print(image.shape)
tiler = Tiler(
data_shape=image.shape,
tile_shape=(256, 256, 4),
channel_dimension=2,
#overlap=0.50
)
# Calculate and apply extra padding, as well as adjust tiling parameters
#new_shape, padding = tiler.calculate_padding()
#tiler.recalculate(data_shape=new_shape)
#padded_image = np.pad(image, padding, mode="reflect")
merger = Merger(tiler=tiler)#, window="overlap-tile")
print(tiler)
for batch_id, batch in tiler(image, batch_size=512):
batch = model.predict(batch)
merger.add_batch(batch_id, 512, batch)
I am probably missing something, but would be nice to have it documented. Also, argmax option seems to be hardcoded for channel first images, which adds additional computational requirements when using channels last images. Any help would be appreciated.
Heads up that numpy >=1.20.0 doesn't support Python 3.6. numpy.typing
in merger.py was introduced in numpy 1.20.0. Tiler has been failing on any Python 3.6 work I try doing.
The two solutions I see are to remove numpy.typing
to allow Python 3.6 to work or update tiler to work 3.7 =<.
Is your feature request related to a problem? Please describe.
It would be cool to be able to create Tiler object that automatically calculates tile shape to obtain specific mosaic (collection of tiles) shape.
Describe the solution you'd like
I imagine it as a class method for Tiler class, that accepts all standard 'init' arguments, except tile_shape.
Describe alternatives you've considered
Currently something similar is already calculated in the script that generates teaser image, in misc/teaser. Once this feature is done, change the teaser script accordingly.
Is your feature request related to a problem? Please describe.
Windows for patches that are on the edges can be modified as described in Introducing Hann windows for reducing edge-effects in patch-based image segmentation (Pielawski and Wählby, March 2020).
It should not be too complicated and the API can be relatively simple, for example a keyword for Merger init.
Hi 👋
I just found your library as I was browsing for a tiler/ merger solution to replace my crappy custom code that cannot handle overlap.
I'm looking for a library that lets me split a source images (3-channel, 2048x2048px) into overlapping 3-channel tiles with size 256x256px (overlap width=32px in x and y direction).
I'm feeding my segmentation model batches of those tiles. The results are 1-channel images of size 256x256 (values are multi-class integer results: currently classes 0,1,2).
So the flow is:
If I understand the library correctly, overlapping tiles are merged and the values are averaged? In my case, I'd have to take the max value in an overlap area. Is this supported? Could I adapt the code to take the max instead of the mean? And if so: are there any examples how to do this?
Cheers and thanks for any hints
Is your feature request related to a problem? Please describe.
There should be more practical examples on how to use Tiler.
Is your feature request related to a problem? Please describe.
As described in Vooban/Smoothly-Blend-Image-Patches, final segmentation quality can significantly improve by blending multiple segmentations of the same patch but rotated/mirrored.
The solution that I see right now is to subclass Tiler object and handle rotations/blending.
ModuleNotFoundError Traceback (most recent call last)
in
1 import numpy as np
----> 2 from tiler import Tiler, Merger
ModuleNotFoundError: No module named 'tiler'
Encountered this problem when running this code:
tiler = Tiler(data_shape=(2, 180, 512, 512), tile_shape=(1, 64, 64, 64), overlap=32, mode='drop')
print(tiler.n_tiles) # 0
It should have raised an exception since overlap is bigger than data_shape on the first dimension.
Is your feature request related to a problem? Please describe.
As a pytorch user, it would be great to have a convenience TiledDataset
class that subclasses standard pytorch Dataset.
Describe alternatives you've considered
It's already possible, but you have to write a lot of boilerplate.
Is your feature request related to a problem? Please describe.
Pytorch's API is very numpy-like, but has differences in underlying memory model. It might be interesting to take a look what's happening there and whether it is possible to tile pytorch tensors too.
Hey Ilja @the-lay
Hope everything is going well!
Would you like to include a link in the README to the napari-tiler repo?
Btw, nice lib!
I already have horizontally tiled files with no overlap that I need to merge and use some of the windows using your lib.
Is there any guide to do this?
Tiler(data_shape=(3, 100), tile_shape=(3, 4), channel_dimension=0, overlap=3)
produces {ValueError}Overlap size much be smaller than tile_shape.
The code around tiler.py#110 should check for channel dimension and generate self._tile_overlap = [0, 3]
Something like tiler.get_all(axis=0)
that will join all tiles along a new axis.
That would support all tiling modes except 'irregular'.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.