sjakobi / bsb-http-chunked Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 4.0 1.75 MB

Chunked HTTP transfer encoding for bytestring builders

Home Page: http://hackage.haskell.org/package/bsb-http-chunked

License: Other

Haskell 100.00%

bsb-http-chunked's People

Contributors

Stargazers

Watchers

Forkers

skress srinivas32 kronor-io sysfce2

bsb-http-chunked's Issues

chunkedTransferEncoding "" == "" !?

> toLazyByteString $ chunkedTransferEncoding $ byteString ""
""

That really surprised me. I would have expected to get "0\r\n\r\n" aka chunkedTransferTerminator.

Add a `ChunkSizePolicy` argument for limiting the transfer chunk size

Some applications don't accept large chunk sizes. E.g. snap-server has a limit of 256kB.

Add tests

Idea for a property test:

The parsed output should be the same as the input

Add reference to builder execution strategies

http://hackage.haskell.org/package/bytestring-0.10.8.2/docs/Data-ByteString-Builder-Extra.html#g:1

untrimmedStrategy sounds especially useful. Maybe I should benchmark it though…

How to handle monolithic bytestrings > (maxBound :: Word32)?

This is what currently happens:

λ let s = S.replicate (4 * 1024 * 1024 * 1024) 95
λ map (S.take 10) $ L.toChunks  $ B.toLazyByteString $ chunkedTransferEncoding $ B.byteString s
["0\r\n","__________","\r\n"]

The code contains comments like FIXME: assert(S.length bs < maxBound :: Word32). Given that we can't control the input though, I think we ought to correctly handle chunks larger than that too.

Maybe we should just use Word to handle the length.

Test handling of oversize inputs or buffers

Maybe create a wrapper with type MaxSize -> Builder -> Builder.

Tag a new release

I'm using this library with Nix/Nixpkgs and GHC 9.6.1. It works with sources from master HEAD, but not from v0.0.0.4.

Please tag a v0.0.0.5 at the current master and update it on Hackage.

Test GHC-8.8 in CI

Tests (at least tests) already work with --allow-newer. Dependencies we're blocked on:

doctest
tasty-hedgehog
tasty

Fix reserved space computation

From meiersi/blaze-builder#6:

The required space could be larger than expected after the buffer is handed over. This in turn increases the space to be reserved for encoding the chunk size, which in turn might lead to not enough bytes being handed over to the inner builder.

I should take a look at this once I understand the code a bit better.

Latest 0.0.0.3 upload to Hackage fails to build on Windows

The tar.gz file has files dated "Jan 1 1970" which may be the reason. Everything works okay with cabal on FreeBSD but Windows 10 generates an error.

cabal.exe: Failed to unpack bsb-http-chunked-0.0.0.3 (which is required by exe:transparent-server from transparent-bots-0.1.0.0). The exception was: D:\msys64\home\sumo\dev\transparent-bots\dist-newstyle\tmp\src-10136\bsb-http-chunked-0.0.0.3\: setModificationTime:setFileTimes: invalid argument (The parameter is incorrect.)

Cabal is:
cabal-install version 2.4.0.0 compiled using version 2.4.0.1 of the Cabal library
GHC is:
The Glorious Glasgow Haskell Compilation System, version 8.4.3

Use `Data.Bits.countLeadingZeros` in `word32HexLength`

https://hackage.haskell.org/package/base-4.11.0.0/docs/Data-Bits.html#v:countLeadingZeros

Why are there leading zeros in the chunk-size when the chunk is small?

λ g n = L.take 10 . B.toLazyByteString . chunkedTransferEncoding . B.byteString $ S.replicate n 95
λ g 10
"00A\r\n_____"
λ g 200
"0C8\r\n_____"
λ g 300
"12C\r\n_____"
λ g 40000
"9C40\r\n____"

Setup Travis

Add examples to docs

Add performance tests

doctests fail with GHC 9.2.1 and doctest 0.18.2

Test suite doctests: RUNNING...

Data/ByteString/Builder/HTTP/Chunked.hs:117:47: error:
    • Found type wildcard ‘_x’ standing for ‘a1’
      Where: ‘a1’ is a rigid type variable bound by
               the inferred type of
                 go :: (BufferRange -> IO (BuildSignal a1)) -> BuildStep a
               at Data/ByteString/Builder/HTTP/Chunked.hs:(118,9)-(177,64)
      To use the inferred type, enable PartialTypeSignatures
    • In the type signature:
        go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
      In an equation for ‘transferEncodingStep’:
          transferEncodingStep k
            = go (B.runBuilder innerBuilder)
            where
                go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
                go innerStep (BufferRange op ope)
                  | outRemaining < minimalBufferSize
                  = pure $ B.bufferFull minimalBufferSize op (go innerStep)
                  | otherwise
                  = B.fillWithBuildStep innerStep doneH fullH insertChunkH brInner
                  where
                      outRemaining = ope `F.minusPtr` op
                      maxChunkSizeLength = word32HexLength $ fromIntegral outRemaining
                      ....
      In an equation for ‘chunkedTransferEncoding’:
          chunkedTransferEncoding innerBuilder
            = B.builder transferEncodingStep
            where
                transferEncodingStep :: forall a. BuildStep a -> BuildStep a
                transferEncodingStep k
                  = go (B.runBuilder innerBuilder)
                  where
                      go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
                      go innerStep (BufferRange op ope)
                        | outRemaining < minimalBufferSize
                        = pure $ B.bufferFull minimalBufferSize op (go innerStep)
                        | otherwise
                        = B.fillWithBuildStep innerStep doneH fullH insertChunkH brInner
                        where
                            ...
    • Relevant bindings include
        k :: BuildStep a
          (bound at Data/ByteString/Builder/HTTP/Chunked.hs:114:26)
        transferEncodingStep :: BuildStep a -> BuildStep a
          (bound at Data/ByteString/Builder/HTTP/Chunked.hs:114:5)
        innerBuilder :: Builder
          (bound at Data/ByteString/Builder/HTTP/Chunked.hs:110:25)
        chunkedTransferEncoding :: Builder -> Builder
          (bound at Data/ByteString/Builder/HTTP/Chunked.hs:110:1)
Data/ByteString/Builder/HTTP/Chunked.hs:97: failure in expression `let f = B.toLazyByteString . chunkedTransferEncoding . B.lazyByteString'
expected:
 but got:
          ^
          <interactive>:27:30: error:
              Variable not in scope:
                chunkedTransferEncoding :: Builder -> Builder

Examples: 5  Tried: 3  Errors: 0  Failures: 1

Test suite doctests: FAIL
Test suite logged to: dist/test/bsb-http-chunked-0.0.0.4-doctests.log

Use BoundedPrim instead of Write!?

BoundedPrim

Not sure if it would work but the types are very similar.

Add CI with GHC 8.10

This is currently blocked on doctest: sol/doctest#246

Test failures with ghc-9.6.1

When building this library with the latest GHC, the tests are failing with:

bsb-http-chunked> Test suite doctests: RUNNING...
bsb-http-chunked> Data/ByteString/Builder/HTTP/Chunked.hs:117:47: error: [GHC-88464]
bsb-http-chunked>     • Found type wildcard ‘_x’ standing for ‘a1’
bsb-http-chunked>       Where: ‘a1’ is a rigid type variable bound by
bsb-http-chunked>                the inferred type of
bsb-http-chunked>                  go :: (BufferRange -> IO (BuildSignal a1)) -> BuildStep a
bsb-http-chunked>                at Data/ByteString/Builder/HTTP/Chunked.hs:(118,9)-(177,64)
bsb-http-chunked>       To use the inferred type, enable PartialTypeSignatures
bsb-http-chunked>     • In the type signature:
bsb-http-chunked>         go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
bsb-http-chunked>       In an equation for ‘transferEncodingStep’:
bsb-http-chunked>           transferEncodingStep k
bsb-http-chunked>             = go (B.runBuilder innerBuilder)
bsb-http-chunked>             where
bsb-http-chunked>                 go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
bsb-http-chunked>                 go innerStep (BufferRange op ope)
bsb-http-chunked>                   | outRemaining < minimalBufferSize
bsb-http-chunked>                   = pure $ B.bufferFull minimalBufferSize op (go innerStep)
bsb-http-chunked>                   | otherwise
bsb-http-chunked>                   = B.fillWithBuildStep innerStep doneH fullH insertChunkH brInner
bsb-http-chunked>                   where
bsb-http-chunked>                       outRemaining = ope `F.minusPtr` op
bsb-http-chunked>                       maxChunkSizeLength = word32HexLength $ fromIntegral outRemaining
bsb-http-chunked>                       ....
bsb-http-chunked>       In an equation for ‘chunkedTransferEncoding’:
bsb-http-chunked>           chunkedTransferEncoding innerBuilder
bsb-http-chunked>             = B.builder transferEncodingStep
bsb-http-chunked>             where
bsb-http-chunked>                 transferEncodingStep :: forall a. BuildStep a -> BuildStep a
bsb-http-chunked>                 transferEncodingStep k
bsb-http-chunked>                   = go (B.runBuilder innerBuilder)
bsb-http-chunked>                   where
bsb-http-chunked>                       go :: (BufferRange -> IO (BuildSignal _x)) -> BuildStep a
bsb-http-chunked>                       go innerStep (BufferRange op ope)
bsb-http-chunked>                         | outRemaining < minimalBufferSize
bsb-http-chunked>                         = pure $ B.bufferFull minimalBufferSize op (go innerStep)
bsb-http-chunked>                         | otherwise
bsb-http-chunked>                         = B.fillWithBuildStep innerStep doneH fullH insertChunkH brInner
bsb-http-chunked>                         where
bsb-http-chunked>                             ...
bsb-http-chunked>     • Relevant bindings include
bsb-http-chunked>         k :: BuildStep a
bsb-http-chunked>           (bound at Data/ByteString/Builder/HTTP/Chunked.hs:114:26)
bsb-http-chunked>         transferEncodingStep :: BuildStep a -> BuildStep a
bsb-http-chunked>           (bound at Data/ByteString/Builder/HTTP/Chunked.hs:114:5)
bsb-http-chunked>         innerBuilder :: Builder
bsb-http-chunked>           (bound at Data/ByteString/Builder/HTTP/Chunked.hs:110:25)
bsb-http-chunked>         chunkedTransferEncoding :: Builder -> Builder
bsb-http-chunked>           (bound at Data/ByteString/Builder/HTTP/Chunked.hs:110:1)
bsb-http-chunked> Data/ByteString/Builder/HTTP/Chunked.hs:97: failure in expression `let f = B.toLazyByteString . chunkedTransferEncoding . B.lazyByteString'
bsb-http-chunked> expected:
bsb-http-chunked>  but got:
bsb-http-chunked>           ^
bsb-http-chunked>           <interactive>:31:30: error: [GHC-88464]
bsb-http-chunked>               Variable not in scope:
bsb-http-chunked>                 chunkedTransferEncoding :: Builder -> Builder
bsb-http-chunked> Examples: 5  Tried: 3  Errors: 0  Failures: 1
bsb-http-chunked> Test suite doctests: FAIL

Run doctests with more GHC versions

Hackage releases via Travis

See https://github.com/tfausak/flow/blob/edec54ab1e78ae7ad1bcd17a98aabccc6714353e/.travis.yml.

bytestring-0.12 not accepted

This line excludes bytestring-0.12:

bsb-http-chunked/bsb-http-chunked.cabal

Line 42 in c0ecd72

bytestring >= 0.10.2 && < 0.12

I can send you a PR if you'd like.

Tested using

cabal repl -w ghc-9.8.1 -b bsb-http-chunked -c 'bytestring>=0.12' --allow-newer=bsb-http-chunked:bytestring

which passes.

Add -O2?

Needs benchmarks first.

Unexpected output with large inputs

See lpsmith/blaze-builder#18:

When splitting the output of chunkedTransferEncoding into strict bytestrings, the result usually looks somewhat like this:

λ> import qualified Data.ByteString as S
λ> import qualified Data.ByteString.Lazy as L
λ> import qualified Data.ByteString.Builder as B
λ> import Blaze.ByteString.Builder.HTTP 
λ> f n = L.toChunks . B.toLazyByteString . chunkedTransferEncoding . B.byteString $ S.replicate n 95
λ> f 12
["00C\r\n____________\r\n"]

Each bytestring forms a complete chunk in the sense of RFC 7230 Section 4.1. From this observation I got the assumption that the 1-to-1 correspondence between bytestrings in the output and transfer chunks should be an invariant of chunkedTransferEncoding.

As I found out during some property testing that (supposed) invariant doesn't hold for certain larger inputs – the smallest "monolithic" input breaking it being a bytestring of length 8161:

λ> f 8161
["1FE1\r\n","___<snip>___","\r\n"]

In other tests I also found chunks like this:

...___", "\r\n2276\r\n___...

So my question is: Is this supposed to be an invariant of chunkedTransferEncoding?

Investigate benchmark slowdown from GHC-8.6.5 -> GHC-8.8.1

GHC-8.6.5:

benchmarked clone village/bsbhc
time                 1.312 ms   (1.298 ms .. 1.330 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 1.308 ms   (1.301 ms .. 1.323 ms)
std dev              32.07 μs   (18.10 μs .. 59.81 μs)
variance introduced by outliers: 10% (moderately inflated)

benchmarked clone village/Blaze
time                 1.346 ms   (1.310 ms .. 1.381 ms)
                     0.997 R²   (0.996 R² .. 0.999 R²)
mean                 1.317 ms   (1.309 ms .. 1.328 ms)
std dev              34.37 μs   (26.15 μs .. 45.74 μs)
variance introduced by outliers: 11% (moderately inflated)

benchmarked 100 4kB chunks/bsbhc
time                 25.98 μs   (25.76 μs .. 26.26 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 26.00 μs   (25.91 μs .. 26.25 μs)
std dev              466.7 ns   (211.3 ns .. 930.0 ns)

benchmarked 100 4kB chunks/Blaze
time                 26.02 μs   (25.87 μs .. 26.20 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 26.09 μs   (26.01 μs .. 26.28 μs)
std dev              388.2 ns   (204.6 ns .. 739.0 ns)

benchmarked 200kB strict bytestring/bsbhc
time                 415.9 ns   (412.0 ns .. 420.3 ns)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 419.4 ns   (416.2 ns .. 432.6 ns)
std dev              18.16 ns   (4.885 ns .. 40.15 ns)
variance introduced by outliers: 24% (moderately inflated)

benchmarked 200kB strict bytestring/Blaze
time                 431.6 ns   (426.2 ns .. 439.0 ns)
                     0.999 R²   (0.997 R² .. 1.000 R²)
mean                 429.7 ns   (427.8 ns .. 434.0 ns)
std dev              9.081 ns   (4.172 ns .. 16.75 ns)

benchmarked 1000 small chunks/bsbhc
time                 48.30 μs   (47.76 μs .. 48.91 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 48.12 μs   (47.91 μs .. 48.64 μs)
std dev              1.064 μs   (644.6 ns .. 1.802 μs)

benchmarked 1000 small chunks/Blaze
time                 48.01 μs   (47.58 μs .. 48.54 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 48.01 μs   (47.81 μs .. 48.63 μs)
std dev              1.065 μs   (411.8 ns .. 2.251 μs)

benchmarked 1000 small chunks nocopy/bsbhc
time                 306.4 μs   (302.2 μs .. 311.2 μs)
                     0.998 R²   (0.997 R² .. 1.000 R²)
mean                 304.5 μs   (302.9 μs .. 307.4 μs)
std dev              7.075 μs   (3.975 μs .. 11.24 μs)

benchmarked 1000 small chunks nocopy/Blaze
time                 311.6 μs   (307.9 μs .. 315.9 μs)
                     0.998 R²   (0.996 R² .. 0.999 R²)
mean                 314.2 μs   (311.7 μs .. 318.7 μs)
std dev              11.28 μs   (5.600 μs .. 17.64 μs)
variance introduced by outliers: 17% (moderately inflated)

GHC-8.8.1:

benchmarked clone village/bsbhc
time                 1.404 ms   (1.388 ms .. 1.430 ms)
                     0.998 R²   (0.994 R² .. 1.000 R²)
mean                 1.423 ms   (1.407 ms .. 1.464 ms)
std dev              79.22 μs   (32.23 μs .. 182.7 μs)
variance introduced by outliers: 34% (moderately inflated)

benchmarked clone village/Blaze
time                 1.404 ms   (1.385 ms .. 1.422 ms)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 1.425 ms   (1.410 ms .. 1.465 ms)
std dev              74.33 μs   (29.23 μs .. 144.5 μs)
variance introduced by outliers: 30% (moderately inflated)

benchmarked 100 4kB chunks/bsbhc
time                 26.63 μs   (26.41 μs .. 26.91 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 26.62 μs   (26.53 μs .. 26.81 μs)
std dev              436.2 ns   (212.4 ns .. 768.2 ns)

benchmarked 100 4kB chunks/Blaze
time                 26.60 μs   (26.43 μs .. 26.80 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 26.70 μs   (26.60 μs .. 26.95 μs)
std dev              515.8 ns   (277.8 ns .. 923.2 ns)

benchmarked 200kB strict bytestring/bsbhc
time                 415.4 ns   (411.2 ns .. 420.6 ns)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 416.6 ns   (414.6 ns .. 420.3 ns)
std dev              8.485 ns   (5.649 ns .. 13.38 ns)

benchmarked 200kB strict bytestring/Blaze
time                 430.4 ns   (426.6 ns .. 434.7 ns)
                     0.998 R²   (0.996 R² .. 1.000 R²)
mean                 433.6 ns   (431.2 ns .. 441.0 ns)
std dev              12.36 ns   (5.608 ns .. 24.40 ns)
variance introduced by outliers: 13% (moderately inflated)

benchmarked 1000 small chunks/bsbhc
time                 55.63 μs   (54.82 μs .. 56.59 μs)
                     0.998 R²   (0.997 R² .. 1.000 R²)
mean                 55.50 μs   (55.16 μs .. 56.21 μs)
std dev              1.586 μs   (702.2 ns .. 2.673 μs)
variance introduced by outliers: 13% (moderately inflated)

benchmarked 1000 small chunks/Blaze
time                 55.99 μs   (55.35 μs .. 56.76 μs)
                     0.999 R²   (0.998 R² .. 0.999 R²)
mean                 55.79 μs   (55.53 μs .. 56.26 μs)
std dev              1.220 μs   (843.9 ns .. 1.956 μs)

benchmarked 1000 small chunks nocopy/bsbhc
time                 314.9 μs   (310.0 μs .. 320.9 μs)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 314.0 μs   (312.1 μs .. 317.1 μs)
std dev              7.985 μs   (5.842 μs .. 10.44 μs)
variance introduced by outliers: 11% (moderately inflated)

benchmarked 1000 small chunks nocopy/Blaze
time                 322.0 μs   (319.1 μs .. 325.3 μs)
                     0.999 R²   (0.997 R² .. 1.000 R²)
mean                 323.4 μs   (321.3 μs .. 328.1 μs)
std dev              10.29 μs   (5.229 μs .. 18.53 μs)
variance introduced by outliers: 15% (moderately inflated)

sjakobi / bsb-http-chunked Goto Github PK

bsb-http-chunked's People

Contributors

Stargazers

Watchers

Forkers

bsb-http-chunked's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs