jayxon / leanify Goto Github PK

As reported on FileOptimizer discussion (http://sourceforge.net/p/nikkhokkho/discussion/fileoptimizer/thread/666f945a/#2f64)

Setup: Win7x86 with FileOptimizer v7.60.1252

Steps:

Optimize a large LibreOffice ODT (Open Text Document) file (tested on 898k and 130k files)

Open file
Defect:
http://i.imgur.com/wqV9QHP.png

OpenOffice 4.1.1 had the same response. LibreOffice (v4.3.5.2 and v4.3.7.2) prompted me to repair it and it came back ok.

Tested both with 32 and 64-bit versions and operating systems.

File compression was excellent: 898k -> 751k (or 84%). This is remarkable because it's even better than a 7-zip "ultra"-level compression of the same file, which cut less than 1% off the file size.

As suggested by Jaff at http://encode.ru/threads/2036-Leanify?p=43423&viewfull=1#post43423

Would you like to add a small hack to optimise htm/html/css etc web based files wich contains base64 encoded images?
Detect base64 image, convert to binary file, optimise it, encode it back to base64 and replace original image in document.
It work with .png, .gif, .jpeg images. Thank you in advance!

some usefull info: http://en.wikipedia.org/wiki/Data_URI_scheme

There's a comment in zip.cpp:218 regarding files using the store method: "don't try to change it to deflate, it might break some file"
Can you elaborate on this? I note that the advzip program does compress stored files.

The reason is TinyXML2 does not support xml:space="preserve".

<w:t xml:space="preserve"> </w:t>

will become

<w:t xml:space="preserve"/>

But the author of TinyXML2 is not willing to fix it.
Should I switch to another xml library?

Here's one of my wedding photos. What happens when you try to Leanify it, please?

http://akb.me.uk/IMG_2575.jpg

When I try, the orientation changes from portrait to landscape. I think this might be a bug, and the orientation should stay the same?

Leanify removes spaces it shouldn't in the attached .pptx file. The title looks like this in the original:

And gets transformed to this with leanify on default settings:

This may be the result of poor formatting in the original document, but it is still a valid .pptx that was created by someone who was not specifically trying to find bugs in leanify.

MMD1019DiaposOligoéléments 2019 (2) - Copie - Copie.pptx

1. • key C don`t work after optimization (no fire)
2. • animation goes wild - fast changing

same goes to Nikkho`s zrecompress LZMA mode, I posted issue on encode.su

I found Leanify while doing some research and decided to use Radamsa to test the robustness of the application. It appears pretty solid. I grabbed a ton of test files and then ran this simple loop to generate some more files; for i in $(ls); do cat $i | radamsa; done.

I did note a few things that were interesting, the most of which was a crash noted by ASAN from mutated files that proved hard to debug. I've done the test case reduction already down from several hundred bytes to the smallest byte string I could find that still induced the crash.

I did note that by modifying the length of the input file, the resulting address of the crash could be changed. Note that in the reproduction steps below, amongst the output shows the READ attempt at crash time. Another interesting crash I noted was a WRITE of 1024 bytes in another crash file.
//Cleaning up stack traces.

Leanify lib
mozjpeg 3.2
tinyxml2 4.0.1
LZMA 16.04

when i try to leanify xmplay.exe it crash.

OpenDocument (.odt, .ods, .odp, .odb, .odg, .odf) contain images (.png, .jpg, .svg), are this optimized, or only ZIP container?

Minifying html, js and css would be a nice feature to add to Leanify.

Issue summarized in another issue. Closing this one.

It has been reported on FileOptimizer forums, that Leanify is corrupting APK files in some cases.

I have been able to reproduce it with daily binaries on both x86, and x64. Problem seems to be Leanify aborts with message "EOCD not found!" but does not generate an error errorlevel (it exits with code 0).

You have more details as well as sample files here: https://sourceforge.net/p/nikkhokkho/discussion/fileoptimizer/thread/74f4f5df/

Anyway, let me know if further details are needed.

知乎上看到的 很不錯的小工具 给个赞！

1.不过能加个多线程功能么？

批量处理几百张图片有点慢。。。。
i7 8700的性能发挥不出来呀

2.还有相同的相机拍出来 都是8MB多点单张

有些压缩成1MB多点 而有些6MB多 差异有点大？
这张压缩成6MB的图片直接扔到windows自带的画图工具另存为jpg 可以压缩到2MB
继续将这张2MB的图片扔到tinypng处理变成400K 我不指望能直接上400K 但也不至于8MB只压缩到了6MB

例子：https://imgchr.com/i/PG5dld
链接: https://pan.baidu.com/s/16lpqdHM41Ku8qIzg_Xn9Xw 密码: 4kza
这是被Leanify压缩后的 默认没有备份功能 所以没原图了

3. 增加忽略错误功能

比如某个文件优化失败了可以跳过继续优化别的

Processing: D:\Data\Desktop\DCIM\100D5500\Leanify.exe
Open file error!
????????????????????????????????????????

4.报错有点皮！

It would be a great improvement if Leanify could utilise the swix console program http://appnee.com/swix/ to convert swf to xml and then optimize embedded base64 streams then restore it back to swf with same program then optimize compression .. you need to include mp3packer support to optimize audio streams

Google Perceptual JPEG encoder
https://github.com/google/guetzli

subj
1.gz

http://encode.ru/threads/2036-Leanify?p=46274&viewfull=1#post46274

...by launching with any argument. I downloaded binary release, 0.4.3 and 0.4.2.

Manjaro 17.1.12 Hakoila

(gdb) run cryfs.png 
Starting program: /home/user/Downloads/leanify cryfs.png

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

core.leanify.1000.65596381a2c44c2a84fe4ab6a6122586.32726.1538230669000000.zip

A issue reported on FileOptimizer have come after some investigation on my side to be releated to Leanify 0.4.2 x86 and x64 precompiled binaries.
The problem happens on at least GZ and PNG files.

Launching it from pure command-line (cmd.exe) works OK, and properly returns errorlevel, but when launched using CreateProcess, GetExitCodeProcess code always returns 259 (STILL_ACTIVE), and what is more, asking it for completion with WaitForSingleObject INFINITE it gets stuck waiting.

Details are here: https://sourceforge.net/p/nikkhokkho/discussion/general/thread/d04bc925/

subj

CBZ and CBT are used far less than CBR these days, it seems to be the primary format. Is support for RARs a possibility?

Some docx embeds zip files inside OLEobjects that are not optimized by Leanify.

If you add an OLEobject codec, you should also support DOC and XLS old format.

Would you provide Win64 binaries for Leanify?

Microsoft Visual C++ Runtime Library
Runtime Error!
Program:
R6002

• floating point support not loaded

after hitting Details (Подробности) button

diskat-gui.exe.gz

https://www.w3.org/TR/WOFF/

• Recompress FontTables

  It's compressed with zlib.

• Support WOFF in DataURI search

  data:application/font-woff;charset=utf-8;base64,d09GRgABAAAAAU...

• Remove metadata?

https://gist.github.com/nuomi1/1de7077fa1ca76125f1b
https://gist.github.com/anonymous/469034ba6df1be56dfc109d1871c6cc9

clang: error: invalid linker name in argument '-fuse-ld=gold'
make: *** [leanify] Error 1

I see it in source, but how it works with default settings? Is it applied for all files or not? How do I force it to be always used?

It is documented that most metadata is removed from most formats, with the exception of keep-exif swith.

Would be nice having a global flag to not remove any metadata at all, or to selectively keep it as keep-iptc, ...

http://googledevelopers.blogspot.com/2013/02/compress-data-more-densely-with-zopfli.html
Typically used to re-encode ZIP & GZip, but Google uses it for their fonts for their Web Font API.
It is very slow (should be a flag to turn on), but does consistently make smaller archives.

Things to do:

• Format parsing
• Optimize FlateDecode
• Optimize DCTDecode
• Remove unused object
• Remove comments
• Remove unnecessary spaces
• Remove Metadata

The first is: error: ‘perror’ was not declared in this scope

I fixed it by adding "#include <stdio.h>" in fileio.h

The second is: formats/pe.cpp:376:25: error: ‘to_string’ is not a member of ‘std’
new_name += std::to_string(entry[i].Name);

Leanify.exe crash with RegistryChangesView.exe

I obtain a:
not a valid Win32 application
in xp sp3

It seems that Leanify does not support UNC path.

How can I make use of all the CPU cores?
Is there any simple way?

The filenames can not contain ñ, á, é, í, ó, ú, ü, Etc.
The filenames can not start with -

Controlled Length Out of Bounds Write

Using the corpus of test files from the previous crash I also included an XML file. The crash is relative to the XML library implementation of Leanify resulting in another issue with an OOB write independent of the fixed OOB Read/Write in ICO. I discovered this via fuzzing and did some light root cause analysis and another party heavily expanded on the exploitability of this issue. Details below:

A moderately controllable memory corruption vulnerability affects the implementation of xml_memory_writer in formats/xml.cpp:179 as shown below:

++
struct xml_memory_writer : pugi::xml_writer {
uint8_t* p_write;

void write(const void* data, size_t size) override {
memcpy(p_write, data, size); # Line 179
p_write += size;
}
};
} // namespace
pugi::xml_writer implementations should additionally track the capacity of the target buffer as shown in the pugixml documentation here: https://pugixml.org/docs/samples/save_custom_writer.cpp

It is possible to control the amount of data written past the end of the buffer by appending characters that require escaping as shown in the text_output_escaped function in pugixml.cpp:3906 as shown below:

++
PUGI__FN void text_output_escaped(xml_buffered_writer& writer, const char_t* s, chartypex_t type, unsigned int flags)
{
/* TRUNCATED */
switch (*s)
{
case 0: break;
case '&':
writer.write('&', 'a', 'm', 'p', ';');
++s;
break;
case '<':
writer.write('&', 'l', 't', ';');
++s;
break;
case '>':
writer.write('&', 'g', 't', ';');
++s;
break;
case '"':
writer.write('&', 'q', 'u', 'o', 't', ';');
++s;
break;
default: // s is not a usual symbol
{
unsigned int ch = static_cast(*s++);
assert(ch < 32);

                                    if (!(flags & format_skip_control_chars))
                                            writer.write('&', '#', static_cast<char_t>((ch / 10) + '0'), static_cast<char_t>((ch % 10) + '0'), ';');
                            }

/* TRUNCATED */

The following base64 test case will crash the target binary by expanding a number of & to the escaped & (a gain of 4 bytes in extension).

PHBlcnNvbiBpZD0iPgombmFtYW5nciBSb2FDL2RpdBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQECYm
JiYmJiYmJiYmJiZ5JnkmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JyYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJkhFQVBfT1ZFUkZMT1cmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
JiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYmJiYm
EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlp
aWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlsaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaWlpaXdpaWlp
aWlpEBgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGdhIFJvYUMvZGl0eT4KEBAQEBAQEBAQ
EBBpaWlpaWlpaWlpaWlpaWlDaXR5PkghbmdhYXN0ZXIgSXNhaWxzLiAsIj4KPC9wZXJzb24+Cg==
For full stack trace on the crash, see the ASAN output below:

=================================================================
==461==ERROR: AddressSanitizer: unknown-crash on address 0xb7fce000 at pc 0xb7af7a42 bp 0xbfffc8f8 sp 0xbfffc4cc
WRITE of size 518 at 0xb7fce000 thread T0
#0 0xb7af7a41 in __asan_memcpy (/usr/lib/i386-linux-gnu/libasan.so.2+0x8aa41)
#1 0xb7af7c2f in memcpy (/usr/lib/i386-linux-gnu/libasan.so.2+0x8ac2f)
#2 0x819a38c in (anonymous namespace)::xml_memory_writer::write(void const*, unsigned int) [clone .lto_priv.243] formats/xml.cpp:179
#3 0x80d927e in flush lib/pugixml/pugixml.cpp:3716
#4 0x80d9106 in pugi::impl::(anonymous namespace)::xml_buffered_writer::flush() [clone .lto_priv.561] lib/pugixml/pugixml.cpp:3705
#5 0x80bfed4 in pugi::xml_document::save(pugi::xml_writer&, char const*, unsigned int, pugi::xml_encoding) const lib/pugixml/pugixml.cpp:7168
#6 0x819b5fe in Xml::Leanify(unsigned int) formats/xml.cpp:307
#7 0x81af203 in LeanifyFile(void*, unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const
&) /root/Leanify/leanify.cpp:140
#8 0x81b08c2 in ProcessFile /root/Leanify/main.cpp:53
#9 0x81b1cde in main /root/Leanify/main.cpp:230
#10 0xb76e6636 in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x18636)
#11 0x804b650 (/root/Leanify/leanify+0x804b650)
The testfile was generated using the AFL fuzzer by lcamtuf.

it gives me:

g++ -Wall -O3 -msse2 -mfpmath=sse -flto --std=c++11 fileio.cpp utils.cpp leanify.cpp main.cpp formats/base64.cpp formats/ico.cpp formats/lua.cpp formats/dwf.cpp formats/swf.cpp formats/rdb.cpp formats/gft.cpp formats/xml.cpp formats/png.cpp formats/data_uri.cpp formats/pe.cpp formats/bmp.cpp formats/tar.cpp formats/jpeg.cpp formats/zip.cpp formats/gz.cpp lzma.a miniz.o mozjpeg.a tinyxml2.o zopfli.a zopflipng.a -s -o leanify
/tmp/cctGuFIz.ltrans0.ltrans.o: In function `Jpeg::Leanify(unsigned long)':
<artificial>:(.text+0x6d5e): undefined reference to `jpeg_std_error'
<artificial>:(.text+0x6d90): undefined reference to `jpeg_destroy_compress'
<artificial>:(.text+0x6da0): undefined reference to `jpeg_destroy_decompress'
<artificial>:(.text+0x6de3): undefined reference to `jpeg_CreateDecompress'
<artificial>:(.text+0x6ded): undefined reference to `jpeg_std_error'
<artificial>:(.text+0x6e15): undefined reference to `jpeg_CreateCompress'
<artificial>:(.text+0x6e4a): undefined reference to `jpeg_c_set_int_param'
<artificial>:(.text+0x6e63): undefined reference to `jpeg_mem_src'
<artificial>:(.text+0x6e83): undefined reference to `jpeg_save_markers'
<artificial>:(.text+0x6e95): undefined reference to `jpeg_read_header'
<artificial>:(.text+0x6ea2): undefined reference to `jpeg_read_coefficients'
<artificial>:(.text+0x6eba): undefined reference to `jpeg_copy_critical_parameters'
<artificial>:(.text+0x6f0a): undefined reference to `jpeg_mem_dest'
<artificial>:(.text+0x6f15): undefined reference to `jpeg_write_coefficients'
<artificial>:(.text+0x6f5e): undefined reference to `jpeg_write_marker'
<artificial>:(.text+0x6f79): undefined reference to `jpeg_finish_compress'
<artificial>:(.text+0x6f86): undefined reference to `jpeg_finish_decompress'
<artificial>:(.text+0x6f93): undefined reference to `jpeg_destroy_decompress'
<artificial>:(.text+0x6fca): undefined reference to `jpeg_destroy_compress'
/tmp/cctGuFIz.ltrans0.ltrans.o: In function `Swf::Leanify(unsigned long)':
<artificial>:(.text+0xd89e): undefined reference to `LzmaCompress'
<artificial>:(.text+0xda6e): undefined reference to `LzmaUncompress'
/tmp/cctGuFIz.ltrans0.ltrans.o: In function `Png::Leanify(unsigned long)':
<artificial>:(.text+0x10585): undefined reference to `ZopfliPNGOptions::ZopfliPNGOptions()'
<artificial>:(.text+0x1075c): undefined reference to `ZopfliPNGOptimize(std::vector<unsigned char, std::allocator<unsigned char> > const&, ZopfliPNGOptions const&, bool, std::vector<unsigned char, std::allocator<unsigned char> >*)'
collect2: error: ld returned 1 exit status
make: *** [Makefile:23: leanify] Error 1

on fedora x86-64 29. The git master builds fine. Please mint a new release.

Any chance you can add PDF to the file types? There are often multiple embedded images in pdfs which, I suspect, make up the bulk of the file size.

Leanify supports various kinds of image format except this. I think this format is common.

Process the test file with Leanify.exe -d 1 app-debug.apk, and get the below msg and an broken .apk file.

Central directory header magic mismatch!
EOCD not found!

app-debug.zip(rename to .apk or just process it)

Tested with nightly build 3a29d22 win64/32
v0.4.3 win stable also failed, but only output EOCD not found!

subj

Successfully compiled current master on Linux (8c1cc53).
It seems that the --keep-exif option does not work for me? Or does color profile info not count? Or am I using it wrong.

$ cp -a source.jpg original.jpg
$ cp -a source.jpg reencoded.jpg
$ leanify --keep-exif reencoded.jpg
Processing: reencoded.jpg
3.14 MB -> 2.94 MB  Leanified: 206.19 KB (6.40%)
$ file original.jpg reencoded.jpg 
original.jpg:  JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 10000x7514, frames 3
reencoded.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, progressive, precision 8, 10000x7514, frames 3
$ exiftool -a -u -g1 original.jpg reencoded.jpg 
======== original.jpg
---- ExifTool ----
ExifTool Version Number         : 9.74
---- System ----
File Name                       : original.jpg
Directory                       : .
File Size                       : 3.1 MB
File Modification Date/Time     : 2016:03:30 19:04:07+00:00
File Access Date/Time           : 2016:04:25 16:55:04+00:00
File Inode Change Date/Time     : 2016:04:25 16:53:47+00:00
File Permissions                : rw-r--r--
---- File ----
File Type                       : JPEG
MIME Type                       : image/jpeg
Image Width                     : 10000
Image Height                    : 7514
Encoding Process                : Baseline DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
---- JFIF ----
JFIF Version                    : 1.01
Resolution Unit                 : None
X Resolution                    : 1
Y Resolution                    : 1
---- ICC-header ----
Profile CMM Type                : ADBE
Profile Version                 : 2.1.0
Profile Class                   : Display Device Profile
Color Space Data                : RGB
Profile Connection Space        : XYZ
Profile Date Time               : 2000:08:11 19:51:59
Profile File Signature          : acsp
Primary Platform                : Apple Computer Inc.
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : none
Device Model                    : 
Device Attributes               : Reflective, Glossy, Positive, Color
Rendering Intent                : Perceptual
Connection Space Illuminant     : 0.9642 1 0.82491
Profile Creator                 : ADBE
Profile ID                      : 0
---- ICC_Profile ----
Profile Copyright               : Copyright 2000 Adobe Systems Incorporated
Profile Description             : Adobe RGB (1998)
Media White Point               : 0.95045 1 1.08905
Media Black Point               : 0 0 0
Red Tone Reproduction Curve     : (Binary data 14 bytes, use -b option to extract)
Green Tone Reproduction Curve   : (Binary data 14 bytes, use -b option to extract)
Blue Tone Reproduction Curve    : (Binary data 14 bytes, use -b option to extract)
Red Matrix Column               : 0.60974 0.31111 0.01947
Green Matrix Column             : 0.20528 0.62567 0.06087
Blue Matrix Column              : 0.14919 0.06322 0.74457
---- Composite ----
Image Size                      : 10000x7514
======== reencoded.jpg
---- ExifTool ----
ExifTool Version Number         : 9.74
---- System ----
File Name                       : reencoded.jpg
Directory                       : .
File Size                       : 2.9 MB
File Modification Date/Time     : 2016:04:25 16:54:49+00:00
File Access Date/Time           : 2016:04:25 16:55:01+00:00
File Inode Change Date/Time     : 2016:04:25 16:54:49+00:00
File Permissions                : rw-r--r--
---- File ----
File Type                       : JPEG
MIME Type                       : image/jpeg
Image Width                     : 10000
Image Height                    : 7514
Encoding Process                : Progressive DCT, Huffman coding
Bits Per Sample                 : 8
Color Components                : 3
Y Cb Cr Sub Sampling            : YCbCr4:4:4 (1 1)
---- JFIF ----
JFIF Version                    : 1.01
Resolution Unit                 : None
X Resolution                    : 1
Y Resolution                    : 1
---- Composite ----
Image Size                      : 10000x7514
    2 image files read
$ leanify -h
Unknown option: h
Leanify 0.4.3

Usage: leanify [options] paths
  -i, --iteration <iteration>   More iterations produce better result, but
                                  use more time, default is 15.
  -d, --max_depth <max depth>   Maximum recursive depth, unlimited by default.
                                  Set to 1 will disable recursive minifying.
  -f, --fastmode                Fast mode, no recompression.
  -q, --quiet                   No output to stdout.
  -v, --verbose                 Verbose output.
  --keep-exif                   Do not remove Exif.

Feature request: Add an option to keep file modification date&time.

Reason: For some files (e.g. .png) no other way to store create/modification time than in file attributes (when change the file name is undesirably).

Effect is: Make lossless optimization real lossless: keep important file information on the place.