nomeata / haskell-leb128-cereal Goto Github PK

Thus allowing to reuse the coding functionality of cereal by providing instances for Serialize. E.g.,

-- | Wrapper that provides instances of Serialize 
-- that implement LEB128 encoding.
newtype Leb128 n = Leb128 n

instance Serialize (Leb128 Word64) where ​...
instance Serialize (Leb128 Natural) where ​...

The current go definitions like this:

    go :: Int -> Int -> G.Get Int
    go shift w = do
        byte <- G.getWord8 <|> fail "short encoding"
        let !byteVal = fromIntegral (clearBit byte 7)
        let !hasMore = testBit byte 7
        let !val = w .|. (byteVal `unsafeShiftL` shift)
        let !shift' = shift+7
        if hasMore
            then go shift' val
            else return $! val

result in intermediate Int's being allocated on each iteration.

toLazyBytestring preallocates a buffer of 4k bytes. This is highly execessive for this kind of use.

Using toLazyByteStringWith allows giving the size of the inital buffer and avoid this overhead. In practice I imagine something like 32 bytes will do. Numbers far larger than this are better served by a different encoding scheme so leb128 should not be optimized for this case.

You could go further and just compute the required length ahead of doing the encoding which would allow us to limit allocation to a single (strict) bytestring as well.