Comments (5)
shrink-on-load: libjpeg supports shrink-on-load by factors of 2, 4 and 8. The library is able to avoid most decompression and this can give a huge speedup. For example:
$ time vips shrink wtc.jpg x.v 8 8 --vips-leak
memory: high-water mark 200.28 MB
real 0m0.822s
user 0m1.224s
sys 0m0.080s
$ header x.v
x.v: 1171x1171 uchar, 3 bands, srgb, jpegload
Here wtc.jpg is a 10,000 x 10,000 pixel RGB jpeg. This command is decoding the entire image, doing an 8x8 pixel block average, and writing the result to x.v. Memory use is high as this is a 12-core machine and each core needs a large buffer of pixels for the shrink.
With shrink-on-load:
$ time vips jpegload wtc.jpg x.v --shrink 8 --vips-leak
memory: high-water mark 8.17 MB
real 0m0.535s
user 0m0.520s
sys 0m0.040s
$ header x.v
x.v: 1172x1172 uchar, 3 bands, srgb, jpegload
sequential mode: see this blog post:
http://libvips.blogspot.co.uk/2012/06/how-libvips-opens-file.html
Currently sharp is decompressing the whole input image to a memory buffer if the decompressed image is under 100mb, or to a disc file for large files, then processing from that to the output.
If you turn on sequential mode when you open the file, vips will stream the image directly from the input file to the output file. This saves memory and disc traffic, but also gives a speedup, since the decompress of the input can run in parallel with the compress to the output.
The downside is that you can't use operations that need large coordinate changes, like 90 degree rotate, for example, and you can only process the input once. If you want to make several output images at different sizes it might be quicker to decompress once to a memory or disc buffer and then process from that.
from sharp.
Thanks @jcupitt - I owe you (at least) a pint.
from sharp.
With a 4x core CPU I still get the best throughput with the default VIPS_ACCESS_RANDOM so plan to leave this in place rather than switch to sequential mode.
VIPS_ACCESS_SEQUENTIAL_UNBUFFERED:
jpeg sharp-file-file x 95.35 ops/sec ±0.60% (79 runs sampled)
jpeg sharp-file-buffer x 97.51 ops/sec ±0.51% (78 runs sampled)
png sharp-file-file x 33.14 ops/sec ±1.15% (81 runs sampled)
VIPS_ACCESS_SEQUENTIAL:
jpeg sharp-file-file x 95.68 ops/sec ±0.42% (78 runs sampled)
jpeg sharp-file-buffer x 97.84 ops/sec ±0.45% (79 runs sampled)
png sharp-file-file x 32.12 ops/sec ±1.32% (78 runs sampled)
VIPS_ACCESS_RANDOM:
jpeg sharp-file-file x 97.77 ops/sec ±0.42% (79 runs sampled)
jpeg sharp-file-buffer x 99.94 ops/sec ±0.42% (81 runs sampled)
png sharp-file-file x 52.03 ops/sec ±8.02% (88 runs sampled)
from sharp.
Strange I see quite a large difference with jpeg here:
$ time vips jpegload --access random theo.jpg x.v --vips-leak
memory: high-water mark 80.57 MB
real 0m0.379s
user 0m0.287s
sys 0m0.154s
$ time vips jpegload --access sequential theo.jpg x.v --vips-leak
memory: high-water mark 27.00 MB
real 0m1.010s
user 0m0.290s
sys 0m0.178s
That's with a 6k x 4k RGB jpeg. Are you doing the shrink-on-load thing now? Perhaps that's masking the difference.
The time difference is small with a png, but the memory saving is useful:
$ time vips pngload --access random theo.png x.v --vips-leak
memory: high-water mark 79.01 MB
real 0m0.737s
user 0m0.721s
sys 0m0.176s
$ time vips pngload --access sequential theo.png x.v --vips-leak
memory: high-water mark 22.84 MB
real 0m0.727s
user 0m0.707s
sys 0m0.165s
from sharp.
Thanks John - like you I saw lower memory usage with sequential.
Use, or non-use, of mmap
for these access methods could explain the performance difference, especially between different architectures/OSes (I'm using x64 Ubuntu 13.10, Linux kernel 3.11.0-15).
I'll expose access method as a pass-through option, defaulting to random to stay in line with libvips.
l may also try firing up a few different EC2 instances to see which method performs best in a virtualised environment.
from sharp.
Related Issues (20)
- JPEG2000 HOT 3
- Error occurs when trying to resize webp images HOT 4
- [email protected] fails to download binary in docker build HOT 4
- Error: Could not load the "sharp" module using the linux-x64 runtime on AWS Lambda HOT 9
- I am facing error "heif: Unsupported feature: Unsupported codec (4.3000)" HOT 1
- Issue installing sharp glib-object.h: No such file or directory HOT 2
- Checksum and flle size difference on Apple chip Macs vs Linux and Windows HOT 2
- I Cannot make to work cropImage() as I want HOT 4
- Why is options.animated=true not the default? HOT 2
- Sharp Install script fails for node version 10,11, 12 HOT 1
- keepIccProfile - 16-bit PNG with P3 profile input converted to AVIF output results in sRGB gamut HOT 3
- Memory allocation failed HOT 3
- WebP "nearLossless" ignores "quality" value HOT 5
- Assigning metadata to color and non-color images HOT 5
- 0.33.2: pnpm: Error: Could not load the "sharp" module using the linux-x64 runtime HOT 4
- TypeError [ERR_INVALID_ARG_TYPE]: The "transform.writable" property must be an instance of WritableStream. Received an instance of Sharp HOT 3
- Sharp not compressing image with resolution HOT 3
- NodeJS Sharp - Change the color of an image while retaining the transparency of the background HOT 4
- No loader is configured for ".node" files: node_modules/sharp/build/Release/sharp-darwin-arm64v8.node HOT 5
- Usage of minSize in webp HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sharp.