assertion failed about libflate

[sile]

Is it possible to tell me how to reproduce the error?
If the problematic file is publicly accessible, it will be helpful if you let me know its URL.

I tried to decode some gzip files downloaded from s3 (e.g., emsdk-portable.tar.gz ), but no problem was found.

[dovahcrow]

Sorry there're some sensitive data inside the file so I cannot give it to you directly.
I added these to huffman.rs: 97 and saw some interesting results.

    for padding in 0..(1 << (self.max_bitwidth - code.width)) {
            let i = ((padding << code.width) | code_be.bits) as usize;
            if i == 0 {
                println!("
                symbol: {}
                code: Code {{ width: {}, bits: {:b} }}
                code_be.bits: {:b}
                i: {} 
                self.max_bitwidth: {}
                padding: {} 
                value: {}",
                         symbol,
                         code.width,
                         code.bits,
                         code_be.bits,
                         i,
                         self.max_bitwidth,
                         padding,
                         value)
            }
            debug_assert_eq!(self.table[i], MAX_BITWIDTH as u16 + 1);
            unsafe {
                *self.table.get_unchecked_mut(i) = value;
            }

Prints:

                symbol: 0
                code: Code { width: 3, bits: 0 }
                code_be.bits: 0
                i: 0
                self.max_bitwidth: 7
                padding: 0
                value: 3

                symbol: 48
                code: Code { width: 5, bits: 0 }
                code_be.bits: 0
                i: 0
                self.max_bitwidth: 15
                padding: 0
                value: 1541

                symbol: 230
                code: Code { width: 13, bits: 10000000000000 }
                code_be.bits: 0
                i: 0
                self.max_bitwidth: 15
                padding: 0
                value: 7373

It seems like the index 0 has been repeatedly used.

[sile]

Thank you for your information.

It seems like the index 0 has been repeatedly used.

Typically, it indicates that "the input data is broken".
DecoderBuilder::set_mapping method (and its callers) tries to reconstruct huffman codes from the canonical huffman codes stored in the input data.
But for some reason, the canonical huffman codes have used overlapped (conflicted) bits spaces for different symbols (so the assertion raised).

Of course, the cause may be simply a bug in the libflate.
But, unfortunately, there is no sufficient information to determine the exact reason of the error ... .

[dovahcrow]

But actually the archive can be successfully ungzipped by `flate2`, which is a binding to `miniz.c`. So I'm not sure what's exactly going on here.

On Sat, Jan 21, 2017 at 2:24 PM Takeru Ohta ***@***.***> wrote: Thank you for your information. It seems like the index 0 has been repeatedly used. Typically, it indicates that "the input data is broken". DecoderBuilder::set_mapping method (and its callers) tries to reconstruct huffman codes from the canonical huffman codes stored in the input data. But for some reason, the canonical huffman codes have used overlapped (conflicted) bits spaces for different symbols (so the assertion raised). Of course, the cause may be simply a bug in the libflate. But, unfortunately, there is no sufficient information to determine the exact reason of the error ... . — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA88zptHpuvxgW56X8Q4aViUqP_9ObZZks5rUaSygaJpZM4LgRbo> .

-- Young Wu *Data Scientist* Data Team, Strikingly a: 248 Daxue Road, Yangpu District, Shanghai w: wooya.me e: [email protected] <https://cn.linkedin.com/in/wooya>

[sile]

Hmm... .
I can not do it for a while because I do not have time, but if I can afford, I will try to reproduce it in my environment.

[dovahcrow]

I'll also take some time on your implementation to see where goes wrong.

On Sun, Jan 22, 2017 at 1:53 AM Takeru Ohta ***@***.***> wrote: Hmm... . I can not do it for a while because I do not have time, but if I can afford, I will try to reproduce it in my environment. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA88znmaKqnZEpIz-YqvKLdHm1CBMvNTks5rUkYMgaJpZM4LgRbo> .

-- Young Wu *Data Scientist* Data Team, Strikingly a: 248 Daxue Road, Yangpu District, Shanghai w: wooya.me e: [email protected] <https://cn.linkedin.com/in/wooya>

[sile]

Thanks!

[sile]

FYI, I fixed a bug (at the commit a03c2f3 ) which seems the cause of the problem reported by this issue.

[dovahcrow]

Okay I'll try that ASAP to confirm it.

On Wed, Apr 12, 2017 at 12:23 AM Takeru Ohta ***@***.***> wrote: FYI, I fixed a bug (at the commit a03c2f3 <a03c2f3> ) which seems the cause of the problem reported by this issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA88zjbF8V76w43Na3G7pwrw4PR2p_D8ks5ru6kBgaJpZM4LgRbo> .

-- Young Wu *Data Scientist* Data Team, Strikingly a: 248 Daxue Road, Yangpu District, Shanghai w: wooya.me e: [email protected] <https://cn.linkedin.com/in/wooya>

[sile]

Thanks!
I hope this will solve your problem.

[dovahcrow]

Great, I can confirm the bug have been fixed! Thanks for your help!

[sile]

Good!
Thank you for confirmation.
(And sorry for the inconvenience for a long time)