Develop a webpack plugin [Help Wanted] about crunch

[helb]

I'll check if a similar package can be created for your fork.

So it seems to be easy, but i don't have MacOS or Windows to compile zopflipng for these platforms… :/ Works well on Linux though:

$ du -b suns-rays-478249_640*
66593	suns-rays-478249_640-crunch.png    # from the Crunch repo
66593	suns-rays-478249_640.png-chris     # your fork compiled on my laptop
66714	suns-rays-478249_640.png-google    # (Google's) zopflipng from a system package

[helb]

Thank you!

I've published a new version:

• it now depends on your zopfli fork packaged for NPM (just Linux and Mac, no Windows support yet… Mac binary comes from the Crunch repo)
• if pngquant returns 98 or 99, original input is passed to zopflipng and args are adjusted
• added some tests for this (the "unquantizable" image comes from the imagemin-pngquant repo)

[helb]

Ah, sorry, i didn't realize that Crunch uses pngquant & zopflipng internally (i thought it was doing some byte magic on the images by itself).

It probably makes no sense to put Python on top of the (already bloated) NPM dependecies then, but use the Crunch's crafted pngquant and zopfli configuration in JS instead.

So here's imagemin-crunch (which does not use @chrissimpkins' Crunch, but does the same thing):

• npmjs.com/package/imagemin-crunch
• github.com/helb/imagemin-crunch

@roy-k feel free to try it out – a basic example is included in the readme, and it can be used with webpack via image-webpack-loader or imagemin-webpack-plugin or imagemin-webpack and probably many others (as long as they use imagemin, like these three do)

[chrissimpkins]

I took a look at the documentation. This is currently implemented in Python and the source would need to be converted to JS to support a webpack plugin.

If we can recruit a JS developer who would like to pitch in to contribute this it looks like we might be able to make something like this happen. If that's you, feel free to pitch your ideas and show me how it works. If not, let's see if anyone comes along with an interest in this idea. Thanks for the suggestion.

[helb]

I think it's not necessary to rewrite Crunch to JS – it seems that many (most?) webpack image optimization plugins just lanch external tools on the image files.

Eg. image-webpack-loader uses pngquant (C/Rust) through imagemin-pngquant (and other tools for different image formats).

It shouldn't be too hard to create something like imagemin-crunch and it's webpack integration. I'll take a shot at it when i'll have more spare time (tomorrow, probably).

[chrissimpkins]

@helb excellent! look forward to seeing what you come up with

[chrissimpkins]

@helb 👍

which does not use @chrissimpkins' Crunch, but does the same thing

Note that we are using a modified fork of zopflipng in this project. It is in https://github.com/chrissimpkins/zopfli if anyone who uses the plugin linked above wants to use the same approach that we are using here. I assume that you are calling system installed zopflipng and pngquant dependencies there?

pngquant is at upstream release v2.12.0 without modifications as of Crunch v3.0.0

[helb]

Thanks :)

I assume that you are calling system installed zopflipng and pngquant dependencies there?

Imagemin calls zopflipng via imagemin/zopflipng-bin npm package, so it's not system installed, but it's Google's version of course… I'll check if a similar package can be created for your fork.

Any plans for merging the changes back to Google's repo?

pngquant is at release v2.12.0 as of Crunch v3.0.0

Yeah, pngquant in imagemin/pngquant-bin seems to be a bit outdated (2.4.1), and there are several issues requesting an update…

[chrissimpkins]

Any plans for merging the changes back to Google's repo

The upstream zopflipng repo is not terribly active and the changes that I made were defined with quantized PNG files so this is a special use case for lossy optimizations only. I don't know whether the changes would (1) be reviewed ; (2) would be acceptable in a general tool for all PNG files. I'd be happy to submit a PR if you'd like to give it a try. There are a number of other changes like implementation of version string reporting from the executable that are open PR's on the upstream repo and have not been accepted, in some cases for years.

[chrissimpkins]

Another nuance to the approach here is that you should catch the pngquant return status code for files that cannot be quantized due to quality issues or file size increases and consider modifying the zopflipng execution.

See error codes in this block of source:

Crunch/src/crunch.py

Lines 227 to 237 in 84ebd94

	except CalledProcessError as cpe:
	if cpe.returncode == 98:
	# this is the status code when file size increases with execution of pngquant.
	# ignore at this stage, original file copied at beginning of zopflipng processing
	# below if it is not present due to these errors
	pass
	elif cpe.returncode == 99:
	# this is the status code when the image quality falls below the set min value
	# ignore at this stage, original lfile copied at beginning of zopflipng processing
	# below if it is not present to these errors
	pass

If you are dealing with files that are not quantized, then you should remove the zopflipng --filters=0 option. Without the filters option, zopflipng executes several "mini-zopfli" runs to determine the most optimal PNG filter for use during the DEFLATE compression stages. I verified through testing that this appears to be only filter = 0 with quantized, palleted PNG files and serves as a speedup measure (eliminates multiple unnecessary cycles through zopfli DEFLATE compression to make this determination) when pngquant executes successfully on the first stage of file processing. This does not apply to files that are not quantized because pngquant recognizes that the file size increases with quantization or there are too many colors to maintain the quality definition that we use here. It is important to allow for these automated filter tests in zopflipng when you are dealing with files that are not quantized. Restricting to filter = 0 generally leads to larger than original files whereas use of the other filters can still result in (sometimes) significant DEFLATE compression of the files. This falls in the lossless range of compression rather than the lossy gains that you see when pngquant modifies the image file. I also add the lossy_transparent flag in those cases to remove any hidden, unnecessary color data in the binary (likely to be relatively rare edge case...).

Let me know if you have any other questions.

[chrissimpkins]

:/ Works well on Linux though

Part of my modifications include a more granular approach to the number of iterations through zopfli compression that are used at a given file size. I tried to achieve a reasonable speed/compression balance in quantized files. You will likely see better compression at most commonly used file sizes on the web and should have significantly reduced compression times at larger file sizes. In my experience, once you get to files approximately 1MB or larger you better get ready for a coffee break during the zopflipng stage. It can become an extremely lengthy compression process for larger files (on the order of several minutes for 1MB+ files, I have never had the time/desire to try very large files...). I tried to offset this with a reduction in the iterations so that you achieve more modest compression but within a timeframe that is semi-reasonable so that you will actually use the zopfli compression for larger files.

I also tinkered with the compression search frame size and some other modifications, but haven't found any other approach that improves upon the upstream with respect to time or size to date.

[chrissimpkins]

Also I would be happy to push a macOS compiled zopflipng binary on the downstream repo releases if that is helpful to you. Unfortunately I don't have an approach for cross compilation for 32/64 bit arch on Windows.

[chrissimpkins]

@helb 👍 excellent. Let me know when you reach a stage that you feel we can close this issue.

[chrissimpkins]

@helb this looks great! Let's return to the conversation about further development on your repository threads. Feel free to ping me if there is anything that I can do there. It would be worth having some fashion of a "other repositories that implement a similar strategy" section on the README here. Thoughts? Feel free to PR in a link into the README so that we can point to this tool if you'd like.

@roy-k closing this issue report. Please reopen if the above does not address your initial request.