An efficient compact, immutable byte string type (both strict and lazy) suitable for binary or 8-bit character data.
Home Page: http://hackage.haskell.org/package/bytestring
License: Other
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
I noticed that Data.ByteString.Builder has functions in it like word32LE :: Word32 -> Builder. Contrast this to Data.Binary.Builder which has word32le or getWord32le. Cereal has a similar naming scheme. Can we get lowercase suffixes in ByteString too?
As the author of mysql-haskell, i'm keen on formatting numeric values into texual values, the comment on doubleDec/floatDec says
-- TODO: Use Bryan O'Sullivan's double-conversion package to speed it up.
But i don't think we really need double-conversion here, since with Builder infrastructure we don't have to estimate texual length and allocate extra memory, we can format them in haskell pretty well IMHO, there's two reference native implementations: blaze-texual and text(Which use a similar Builder type). So is there any interest in merging some related code?
It's not as easy as it seems to be, but i still made some progress on it ; )
Data.ByteString.Internal implementsunsafeCreateUptoN' and createUptoN' but does not export them. These functions turned out to be useful for glguy/utf8-string#12. I see no reason to not export them from the internals.
The behaviour of Data.ByteString split/splitWith differs from Data.Text.split when it comes to zero-length arguments:
> Data.ByteString.split undefined ByteString.empty
[]
> Data.Text.split undefined Text.empty
[""]
Both results are perfectly reasonable, the only real problem is that they are different. The reasons I'm logging this bug here rather than with Text are:
length (split a b) == 1 + length (elemIndices a b)Seems like a fairly straightforward omission.
Is there any particular reason that some form of fusion (either foldr/build or full stream fusion) is not implemented for this module? It might even be useful to use Vector-style bundles so mutation and C functions (memcpy et al.) can be used where appropriate.
I'm willing to work on this if it's considered a worthwhile feature.
As per http://stackoverflow.com/questions/1398322/split-bytestring-on-a-bytestring-instead-of-a-word8-or-char this is a commonly requested piece of functionality. Here is one possible implementation for Data.ByteString:
splitOn :: Str -> Str -> [Str]
splitOn needle haystack0 = go haystack0
where
l = length needle
breaker = breakSubstring needle -- save preprocessing
go s = let (pre, post) = breaker s
in pre : if null post
then []
else go (drop l post)
However, for lazy bytestrings, we will need an implementation of breakSubstring for them.
This would be very convenient for debugging. I'll submit a pull request if deemed acceptable.
When several threads are writing lazy bytestring content to a Handle using hPut, the output will be mixed, which defeats the excluding behavior of the Handle. You can see the difference between the behaviors here (you just have to listen on 8050).
I don't see a use case where this would be useful, though there might be. Also the behavior differs from the strict bytestrings. Is this due to technical difficulties, for instance not wanting to rely Handle's implementation?
Here's the link to the relevant portion of Lazy. hPut is simply binding strict hPuts, releasing the Handle lock every time.
isSuffixOf available in Data.ByteString.Lazy, but not in Data.ByteString.Lazy.Char8 (it's commented out). Likewise isInfixOf is not available Data.ByteString.Lazy.
Similarly, there's no lazy version of breakSubstring
Is there any reason for this?
It would be nice if BS.getLine would use the handle's newline setting so that in CRLF mode strings are returned with both the CR (if present) and LF removed.
An example of code that demonstrates the behavior:
http://stackoverflow.com/questions/22417171/bs-getline-and-crlf-endings
I had a huge memory-leak in my application because when reading the documentation for BSL.ByteString suggests that copy should allow the "parent" to be released for GC. I thought this was a bit suspicious, but went with it anyways and it turned out quite bad... So I checked the source:
-- | /O(n)/ Make a copy of the 'ByteString' with its own storage.
-- This is mainly useful to allow the rest of the data pointed
-- to by the 'ByteString' to be garbage collected, for example
-- if a large string has been read in, and only a small part of it
-- is needed in the rest of the program.
copy :: ByteString -> ByteString
copy cs = foldrChunks (Chunk . S.copy) Empty cs
--TODO, we could coalese small blocks here
--FIXME: probably not strict enough, if we're doing this to avoid retaining
-- the parent blocks then we'd better copy strictly.Important part here is the FIXME which does not show up in the docs but is critical in my opinion. Can we move this there?
I wanted to see if I could reimplement the missing code described in #50, but some of the internals of the Builder modules are not exposed, so I'm stuck. This is a shame.
Please consider using
{-# OPTIONS_HADDOCK not-home #-}for the .Internal modules rather than
{-# OPTIONS_HADDOCK hide #-}as this would make it much easier for folks to browse inside of these modules, while still ensuring that the documentation links point to the current locations.
I just want to point out that on haskell.org, the haddock for 0.10.6.0, uses this function in an example without importing it, and the function doesn't seem to be exported by this library, or even to my knowledge, by any other library published to hackage.
The offending example is in the haddock for Data.ByteString.Builder.Prim.
I do see the source for this function here: http://hackage.haskell.org/package/bytestring-0.10.0.1/docs/src/Data-ByteString-Lazy-Builder-BasicEncoding.html
I'm using ShortByteStrings to convert from (STUArray Int Word8). Having hPut available would save the memory allocation and memcpy that happens when converting a ShortByteString into a ByteString.
Would you accept a patch that adds this function to ShortByteStrings?
The current Show instance output looks like this:
Chunk "John,27\r\nJane,28\r\n" EmptyI suggest we instead match the output of the strict version (e.g. "John,27\r\nJane,28\r\n").
The current instance is:
Show output include constructors.)I understand that cabal bench won't run since the suit isn't intended for "public release" as stated in the .cabal file under bench directory. But even building the bench suit on its own produces typing errors:
Building bench-bytestring-0.1.0.0...
Preprocessing executable 'bench-bytestring-builder' for
bench-bytestring-0.1.0.0...
[18 of 18] Compiling Main ( BenchAll.hs, dist/build/bench-bytestring-builder/bench-bytestring-builder-tmp/Main.o )
BenchAll.hs:133:46:
Couldn't match expected type ‘Benchmarkable’
with actual type ‘IO ()’
In the second argument of ‘($)’, namely ‘benchIntEncodingB nRepl e’
In the expression:
bench (name ++ " (" ++ show nRepl ++ ")")
$ benchIntEncodingB nRepl e
Should this be fixed or should the suit just disappear?
I just received two Travis build failures due to usage of the non-present <$> operator:
Data.ByteString.Builder pushes GHC's optimizer quite hard. This bug is intended to provide a place to take note of performance-related issues both in bytestring and GHC.
FixedPrims will encounter stack thrashing due to GHC's reluctance to float values into small case analysesBuilder implementations suffered from lacking constructor specialization optimization due to GHC's inability to match aritiesIMO it's a convenient verb to have. For reference, Data.Text provides both, Data.List provides stripPrefix
Something changed somewhat recently such that all links to the 'ByteString' type in a large amount of packages are all broken. Here's one example:
http://hackage.haskell.org/package/binary-0.7.2.1/docs/Data-Binary.html
Clicking on the 'ByteString' type anywhere in there will lead to a "There is no documentation for bytestring-0.10.0.2" error. I could've sworn this worked within the past few weeks, but now it doesn't. A lot of packages link to 'ByteString' in this manner, and now all those links all over are all broken.
Hiya,
I noticed that Data.ByteString.Lazy.foldl' calls S.foldl on each chunk. Is this intentional? Could we make it call S.foldl' instead? This is killing the performance of one of my benchmarks[1]. Making what should be a tight loop take 250 megs of memory.
Cheers,
David.
[1] CRC32 from http://mirror.seize.it/linux-ghc-7.8.2-2014.05.11.html
These programs segfault on a 32-bit system due to integer overflow.
import qualified Data.ByteString as B
main = B.concat (replicate (2^14) (B.replicate (2^18) 0)) `seq` return ()
import qualified Data.ByteString as B
import qualified Data.ByteString.Lazy as BL
main = BL.toStrict (BL.fromChunks (replicate (2^14) (B.replicate (2^18) 0))) `seq` return ()
On a 64-bit system, increase all the exponents by 16. By my reckoning they will then take around 40 GB to run, not an impossible amount.
See the implementation of concat in the text library for a correct way to handle integer overflow.
I'd like to write some Backpack signatures which help smooth over the difference between Char8 and Word8 ByteStrings; however, in order to write the signature, I need to declare a new type "data Elem" to parametrize over the element type of the ByteString:
signature Data.ByteString where
data ByteString
data Elem
singleton :: Elem -> ByteString
In order for a module to implement this signature, it needs to export Elem. So this is very disappointing, which is the literal Data.ByteString from bytestring doesn't implement this signature.
Could we make it implement it? Yes, very easily: just add a type synonym type Elem = Char/Word8 to each of the ByteString modules. Done. Can we? (It's an easy patch but I have to convince you guys to take it.)
Looking at the implementation of equality, I have a question...
-- Implementations for Eq, Ord and Monoid instances
eq :: ByteString -> ByteString -> Bool
eq a@(PS fp off len) b@(PS fp' off' len')
| len /= len' = False -- short cut on length
| fp == fp' && off == off' = True -- short cut for the same string
| otherwise = compareBytes a b == EQ
{-# INLINE eq #-}
-- ^ still needed
compareBytes :: ByteString -> ByteString -> Ordering
compareBytes (PS _ _ 0) (PS _ _ 0) = EQ -- short cut for empty strings
compareBytes (PS fp1 off1 len1) (PS fp2 off2 len2) =
accursedUnutterablePerformIO $
withForeignPtr fp1 $ \p1 ->
withForeignPtr fp2 $ \p2 -> do
i <- memcmp (p1 `plusPtr` off1) (p2 `plusPtr` off2) (min len1 len2)
return $! case i `compare` 0 of
EQ -> len1 `compare` len2
x -> xThere is a short cut for same string, good! This is probably optimal if ByteStrings cannot overlap in memory. If they can overlap, the more accurate test for "same string" would probably be
(p1 `plusPtr` off1) == (p2 `plusPtr` off2) Are ByteStrings guaranteed to be overlap free or would it make sense to change the code?
(analogous to what we have in Data.Text)
The subject says it all. The following code should be equivalent to S8.pack liftM readFile "/dev/null", but instead it throws an exception.
GHCi, version 7.10.3: http://www.haskell.org/ghc/ :? for help
Prelude> import qualified Data.ByteString as S
Prelude S> S.readFile "/dev/null"
*** Exception: /dev/null: hFileSize: inappropriate type (not a regular file)
Prelude S>
import qualified Data.ByteString.Lazy.Char8 as B8
main = do
(l:_) <- B8.lines `fmap` B8.getContents
B8.putStrLn l
The error occurs as soon as it executes.
HaskellPlatform-2014.2.0.0-x86_64 on Windows 7 x64
The following simple program (sucks in stdin in chunks of 32K and splits on newlines) demonstrates (I believe) some issues with break routine on GHC 7.10.3, 7.8.4, 7.6.3, 7.4.2, 7.2.2, and 7.0.4.
main :: IO ()
main = goEmpty stdin
where
goEmpty :: Handle -> IO ()
goEmpty handle = do
leftover <- DB.hGetSome handle 32768
case DB.null leftover of
True -> return ()
False -> goSome handle leftover
goUnsure :: Handle -> ByteString -> IO ()
goUnsure handle leftover = do
case DB.null leftover of
True -> goEmpty handle
False -> goSome handle leftover
goSome :: Handle -> ByteString -> IO ()
goSome handle leftover = assert (not (DB.null leftover)) $ do
let (prefix,suffix) = DB.break (== c2w '\n') leftover -- DB.breakByte (c2w '\n') leftover
case DB.null suffix of
True -> goEmpty handle
False -> goUnsure handle (DB.tail suffix)break (== c2w '\n') into breakByte (c2w '\n'). This is confirmed by -ddump-simpl or replacing break with breakByte and observing the performance difference.{-# INLINE [1] break #-}
#endif
{-# RULES
"ByteString specialise break (x==)" forall x.
break ((==) x) = breakByte x
"ByteString specialise break (==x)" forall x.
break (==x) = breakByte x
#-}findIndexOrEnd call in the non-breakByte implementation. This is confirmed by -ddump-simpl or viewing the memory allocation stats on a run over a large amount of input.break :: (Word8 -> Bool) -> ByteString -> (ByteString, ByteString)
break p ps = case findIndexOrEnd p ps of n -> (unsafeTake n ps, unsafeDrop n ps)
findIndexOrEnd :: (Word8 -> Bool) -> ByteString -> Int
findIndexOrEnd k (PS x s l) =
accursedUnutterablePerformIO $
withForeignPtr x $ \f ->
go (f `plusPtr` s) 0
where
go !ptr !n | n >= l = return l
| otherwise = do w <- peek ptr
if k w
then return n
else go (ptr `plusPtr` 1) (n+1)Note that the allocations from this boxing/unboxing can be quite significant. As an example, I have around 10.7GB of input data that my original code runs on. When split it on '\n', the system indicates the non-breakByte version does an additional 3GB of allocations over the 10.8GB used by the byteBreak version (which doesn't have the boxing/unboxing issues). Splitting on the more frequently occuring ':' pushes it up to 34GB of additional allocations (yes 34GB which is why I noticed it).
For completeness, here is the -O2 -ddump-simpl -dsuppress-all -dno-suppress-idinfo -output of the key loop for GHC 7.10.3 (ByteString 0.10.6.0). You can see it has failed to use the breakByte implementation. It has also failed to eliminate the boxing of the offset that findIndexOrEnd ($wa1_s3lI) returns (I# ww5_s3lG or l_a1r7 = I# ww3_s3lR) in the non-breakByte implementation`
$wa_r3nK
[GblId, Arity=6, Str=DmdType <L,U><L,U><L,U><L,U><L,U><L,U>]
$wa_r3nK =
\ w_s3lJ ww_s3lO ww1_s3lP ww2_s3lQ ww3_s3lR w1_s3lL [OS=OneShot] ->
let {
l_a1r7
[LclId, Str=DmdType m]
l_a1r7 = I# ww3_s3lR } in
letrec {
$wa1_s3lI [InlPrag=[0], Occ=LoopBreaker]
[LclId, Arity=3, Str=DmdType <L,U><L,U><L,U>]
$wa1_s3lI =
\ ww4_s3lC ww5_s3lG w2_s3lz [OS=OneShot] ->
case tagToEnum# (>=# ww5_s3lG ww3_s3lR) of _ [Occ=Dead] {
False ->
case readWord8OffAddr# ww4_s3lC 0 w2_s3lz
of _ [Occ=Dead] { (# ipv2_a1t8, ipv3_a1t9 #) ->
case ipv3_a1t9 of _ [Occ=Dead] {
__DEFAULT ->
$wa1_s3lI (plusAddr# ww4_s3lC 1) (+# ww5_s3lG 1) ipv2_a1t8;
__word 10 -> (# ipv2_a1t8, I# ww5_s3lG #)
}
};
True -> (# w2_s3lz, l_a1r7 #)
}; } in
case $wa1_s3lI (plusAddr# ww_s3lO ww2_s3lQ) 0 realWorld#
of _ [Occ=Dead] { (# ipv_a1tv, ipv1_a1tw #) ->
case touch# ww1_s3lP ipv_a1tv
of _ [Occ=Dead, OS=OneShot] { __DEFAULT ->
case ipv1_a1tw of _ [Occ=Dead] { I# y_a1xD ->
let {
dt3_a1r0
[LclId, Str=DmdType]
dt3_a1r0 = -# ww3_s3lR y_a1xD } in
case tagToEnum# (<=# dt3_a1r0 0) of _ [Occ=Dead] {
False ->
let {
dt1_X1rx
[LclId, Str=DmdType]
dt1_X1rx = -# dt3_a1r0 1 } in
case tagToEnum# (<=# dt1_X1rx 0) of _ [Occ=Dead] {
False ->
$wa_r3nK
w_s3lJ
ww_s3lO
ww1_s3lP
(+# (+# ww2_s3lQ y_a1xD) 1)
dt1_X1rx
w1_s3lL;
True -> main2 w_s3lJ w1_s3lL
};
True -> main2 w_s3lJ w1_s3lL
}
}
}
}Thanks! -Tyson
PS: I don't see how this boxing/unboxing survived. Both I# ww5_s3lG and l_a1r7 = I# ww3_s3lR clearly cannot be bottom, and the $wa1_s3lI call site immediately turns around and unboxes the returned value. I also don't see why all the DmdType values are <L,U> as the functions clearly force many of their arguments to at least WHNF regardless of the path taken through them.
Data.ByteString.readFile appears to always return an empty string when reading /proc nodes. It was recommended in #haskell that I file this as a bug report.
λ> import qualified Data.ByteString as B
λ> B.readFile "/proc/3437/statm"
""
λ> Prelude.readFile "/proc/3437/statm"
"703486 368855 28992 23 0 485400 0\n"
λ> B.readFile "/proc/cpuinfo"
""
λ> B.readFile "/proc/meminfo"
""
However, hGetContents works as expected
λ> h <- openFile "/proc/3437/statm" ReadMode
λ> B.hGetContents h
"708252 370944 28854 23 0 490509 0\n"
I've enabled the travis build behind the scenes but folks on github won't be able to see easily whether the project is building. This is clearly terrible. I propose translating README to README.md and adding the inimitable travis badge to the top.
Hi,
consider the following programme:
{-# OPTIONS_GHC -O2 #-}
module Main where
import qualified Data.ByteString.Lazy.Char8 as BS
import System.IO
main :: IO ()
main =
withBinaryFile "hugefile" ReadMode $ \handle -> do
size <- hFileSize handle
print . length . BS.lines =<< BS.hGet handle (fromIntegral size)This seems to load the entirety of hugefile (here 4.6GB) into memory, which defeats the purpose of using a lazy bytestring in the first place. When replacing BS.hGet ... with BS.readFile ..., it streams in constant memory as expected. Is this the intended behaviour (in which case I think it should be documented) or a bug?
The readFile function from Data.ByteString.Char8 will fail on files whose reported file size is zero, where the function from Data.ByteString will succeed, e.g. /proc/cpuinfo on Linux:
>>>import qualified Data.ByteString as B
>>>import qualified Data.ByteString.Char8 as C
>>>B.length <$> B.readFile "/proc/cpuinfo"
1592
>>>B.length <$> C.readFile "/proc/cpuinfo"
0The problem is obvious:
readFile :: FilePath -> IO ByteString
readFile f = bracket (openBinaryFile f ReadMode) hClose
(\h -> hFileSize h >>= hGet h . fromIntegral)We can't read any bytes from a handle for which hFileSize returns zero. Here's the definition in Data.ByteString:
readFile :: FilePath -> IO ByteString
readFile f =
bracket (openBinaryFile f ReadMode) hClose $ \h -> do
filesz <- hFileSize h
let readsz = (fromIntegral filesz `max` 0) + 1
hGetContentsSizeHint h readsz (readsz `max` 255)The use of an hGetContents-style function allows bytes to be read even if hFileSize returns zero.
Is there a good reason for this difference? If not, can the definition of readFile from Data.ByteString be used in Data.ByteString.Char8?
The work in glguy/utf8-string#13 suggests two optimisations to the lazy pack.
Firstly, packBytes and packChars in https://github.com/haskell/bytestring/blob/master/Data/ByteString/Lazy/Internal.hs#L129 should probably do a [] test before mallocing. This is probably a common case, and it's both time and memory efficient to do so.
Secondly, the inner loop in packUptoLenBytes and packUptoLenChars in https://github.com/haskell/bytestring/blob/master/Data/ByteString/Internal.hs#L296 loops over two variables - the ptr (which goes up) and the remaining length (which goes down). In the UTF8 pull request I calculate the end pointer upfront (plusPtr ptr len), and then only compare that against ptr, meaning I can delete the length variable. This gives a 10% performance improvement, even when I also pay the cost of doing unicode conversion, so I suspect for ByteString it would be even more.
It should be boundedPrim. See: https://github.com/haskell/bytestring/blob/master/Data/ByteString/Builder/Prim/Internal.hs#L45
The function unfoldrN calls the predicate one more time than you would expect. I would expect the invariants:
unfoldrN n f z makes at most n calls to the predicate f. In reality, unfoldrN calls it n+1 times.unfoldr f z == let (s,z2) = unfoldrN n f z in s2 <> maybe empty (unfoldr f) zNote that meeting the first invariant would change the return value in some cases. At the moment unfoldrN 0 (const Nothing) 0 returns ("",Nothing). I'm pretty sure this is logically incorrect - it should be ("", Just 0). Note that it doesn't break the unfoldrN is a prefix of unfoldr property, and I think is an improvement.
In Bytestring, String and Text EOL always '\n', but in windows its \r\n
how about separate version for lines, unlines?
lines can be universal, and unlines like
unlinesUniversal :: EOL -> [ByteString] -> ByteString
Its not critical, but can be useful.
The time complexity of unsafePackMallocCStringLen in Data.ByteString.Unsafe is documented as being O(n), while the implementation seems to be O(1).
Here's the current implementation for reference:
unsafePackMallocCStringLen :: CStringLen -> IO ByteString
unsafePackMallocCStringLen (cstr, len) = do
fp <- newForeignPtr c_free_finalizer (castPtr cstr)
return $! PS fp 0 len
Hi, would it be possible to update the test suite for bytestring to support the current version of QuickCheck? I tried building it (after editing the Cabal file), but I ran into this compiler error:
Preprocessing test suite 'prop-compiled' for bytestring-0.10.4.1...
[ 1 of 12] Compiling Data.ByteString.Internal ( Data/ByteString/Internal.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Internal.dyn_o )
[ 2 of 12] Compiling Data.ByteString.Unsafe ( Data/ByteString/Unsafe.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Unsafe.dyn_o )
[ 3 of 12] Compiling Data.ByteString.Short.Internal ( Data/ByteString/Short/Internal.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Short/Internal.dyn_o )
[ 4 of 12] Compiling Data.ByteString.Short ( Data/ByteString/Short.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Short.dyn_o )
[ 5 of 12] Compiling Data.ByteString ( Data/ByteString.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString.dyn_o )
[ 6 of 12] Compiling Data.ByteString.Lazy.Internal ( Data/ByteString/Lazy/Internal.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Lazy/Internal.dyn_o )
[ 7 of 12] Compiling Data.ByteString.Lazy ( Data/ByteString/Lazy.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Lazy.dyn_o )
[ 8 of 12] Compiling Data.ByteString.Char8 ( Data/ByteString/Char8.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Char8.dyn_o )
[ 9 of 12] Compiling Data.ByteString.Lazy.Char8 ( Data/ByteString/Lazy/Char8.hs, dist/build/prop-compiled/prop-compiled-tmp/Data/ByteString/Lazy/Char8.dyn_o )
[10 of 12] Compiling QuickCheckUtils ( tests/QuickCheckUtils.hs, dist/build/prop-compiled/prop-compiled-tmp/QuickCheckUtils.dyn_o )
tests/QuickCheckUtils.hs:31:18:
Couldn't match expected type ‘Property’ with actual type ‘Gen a0’
In the expression: sized $ \ sz -> resize (f sz) (property p)
In an equation for ‘adjustSize’:
adjustSize f p = sized $ \ sz -> resize (f sz) (property p)
tests/QuickCheckUtils.hs:31:48:
Couldn't match expected type ‘Gen a0’ with actual type ‘Property’
In the second argument of ‘resize’, namely ‘(property p)’
In the expression: resize (f sz) (property p)
This doesn't look like it would be hard to fix?
This issue relates to the Haddock API documentation. Is the following note about Data.ByteString.Lazy.hGetContents correct or is it out of date?
Note: the Handle should be placed in binary mode with hSetBinaryMode for hGetContents to work correctly.
If it is out of date, please consider adding a statement somewhere in the documentation that hSetBinaryMode is definitely not required, for avoidance of confusion.
This also affects the following issue in the process package.
Resolving dependencies...
Configuring deepseq-1.4.0.0...
Building deepseq-1.4.0.0...
Installed deepseq-1.4.0.0
Configuring bytestring-0.10.4.1...
Building bytestring-0.10.4.1...
Failed to install bytestring-0.10.4.1
Build log ( /home/local/ANT/shida/.cabal/logs/bytestring-0.10.4.1.log ):
Configuring bytestring-0.10.4.1...
Building bytestring-0.10.4.1...
Preprocessing library bytestring-0.10.4.1...
[ 1 of 23] Compiling Data.ByteString.Builder.Prim.Internal.UncheckedShifts ( Data/ByteString/Builder/Prim/Internal/UncheckedShifts.hs, dist/build/Data/ByteString/Builder/Prim/Internal/UncheckedShifts.o )
[ 2 of 23] Compiling Data.ByteString.Builder.Prim.Internal ( Data/ByteString/Builder/Prim/Internal.hs, dist/build/Data/ByteString/Builder/Prim/Internal.o )
[ 3 of 23] Compiling Data.ByteString.Builder.Prim.Internal.Floating ( Data/ByteString/Builder/Prim/Internal/Floating.hs, dist/build/Data/ByteString/Builder/Prim/Internal/Floating.o )
[ 4 of 23] Compiling Data.ByteString.Builder.Prim.Binary ( Data/ByteString/Builder/Prim/Binary.hs, dist/build/Data/ByteString/Builder/Prim/Binary.o )
[ 5 of 23] Compiling Data.ByteString.Internal ( Data/ByteString/Internal.hs, dist/build/Data/ByteString/Internal.o )
Data/ByteString/Internal.hs:212:10:
No instance for (Control.DeepSeq.GNFData
(GHC.Generics.Rep ByteString))
arising from a use of ‘Control.DeepSeq.$gdmrnf’
In the expression: Control.DeepSeq.$gdmrnf
In an equation for ‘rnf’: rnf = Control.DeepSeq.$gdmrnf
In the instance declaration for ‘NFData ByteString’
cabal: Error: some packages failed to install:
bytestring-0.10.4.1 failed during the building phase. The exception was:
ExitFailure 1
unsafeUseAsCString uses withForeignPtr and useAsCString uses alloca; both of these functions require the pointer to not be returned or stored in any way. This is not obvious from the documentation!
Why Vector Word8 is faster than ByteString in this benchmark?
{- compile with -O2 -}
module Main where
import Data.Word
import Data.List
import qualified Data.ByteString as B
import qualified Data.ByteString.Lazy as BL
import qualified Data.Vector as V
import qualified Data.Vector.Unboxed as VU
import Criterion.Main (bench, bgroup, defaultMain, whnf, whnfIO)
input = take 100000 [128,128..] :: [Word8]
inputVect = V.fromList input
inputVectU = VU.fromList input
inputBs = B.pack input
inputBsl = BL.pack input
plus a b = fromIntegral a + fromIntegral b
f = id
{-# INLINE f #-}
summ_list :: [Word8] -> Int
summ_list = foldl' plus 0 . map f
summ_bs :: B.ByteString -> Int
summ_bs = B.foldl' plus 0 . B.map f
summ_bs2 :: B.ByteString -> Int
summ_bs2 = B.foldl' plus 0
summ_bsl :: BL.ByteString -> Int
summ_bsl = BL.foldl' plus 0 . BL.map f
summ_bsl2 :: BL.ByteString -> Int
summ_bsl2 = BL.foldl' plus 0
summ_vect :: V.Vector Word8 -> Int
summ_vect = V.foldl' plus 0 . V.map f
summ_vectU :: VU.Vector Word8 -> Int
summ_vectU = VU.foldl' plus 0 . VU.map f
main :: IO ()
main = do
defaultMain
[ bgroup "bench"
[ bench "list" $ whnf summ_list input {- mean: 9.728395 ms -}
, bench "bs" $ whnf summ_bs inputBs {- mean: 1.604811 ms -}
, bench "bs2" $ whnf summ_bs2 inputBs {- mean: 4.934994 ms -}
, bench "bsl" $ whnf summ_bsl inputBsl {- mean: 4.314427 ms -}
, bench "bsl2" $ whnf summ_bsl2 inputBsl {- mean: 3.257671 ms -}
, bench "vector" $ whnf summ_vect inputVect {- mean: 1.165711 ms -}
, bench "vector u" $ whnf summ_vectU inputVectU {- mean: 496.3630 us -}
]
]Another question:
Why B.foldl' plus 0 . B.map f is faster than B.foldl' plus 0? Core output showed, that foldl' is not inlined. Perhaps a bug?
Having GHC 7.6.3, bytestring 0.10.4.0
Self explanatory.
I wrote a blog post about this issue, and @bos and @tibbe mentioned on Reddit that it should really be filed as a bug, which I'm doing now. The short version is: the strict readFile takes the reported file size as an upper bound, when in reality some files- in particular files in the /proc file system- do not in fact report their true file size.
I'd be happy to send a pull request to address this. My first stab at a solution would look like this:
hGetContents. Even if this buffer is unused, using a standard buffer size will mean that- in many cases- malloc will be able to simply reassign that same memory buffer to the next requester.hGet as we currently due, followed by a call to hGetContents. If that latter call returns something non-empty, then append it to the result of hGet.
hGet by 1, and then check if the returned buffer is smaller than what we requested. While in reality this will likely be a good indication that the file has been completed read, there is no guarantee that all requested bytes will in fact be provided.If this approach is acceptable, I'll send a PR for it. My other question is: where should a regression test be added for this? The active test suites seem to be for either properties (which this would not be) and for testing the builder implementation.
The docs are very cheeky and fun but aren't very helpful. At a minimum it would enumerate some precise cases where it's safe to use (e.g. can I always do accursedUnutterablePerformIO $ withForeignPtr fp ...), and better would be to also summarize the dangers; the linked issues are not particularly enlightening.
Also it's not clear to me if the intent is for accursedUnutterablePerformIO to be deprecated or not. Obviously it's in an internal module so users can expect less stability, but it would be helpful to know as people are using it.
Failed to build / install bytestring-builder 0.10.4.1.1
MacBook-Pro:~ xxx$ cabal install --verbose bytestring-builder
'/usr/bin/alex' '--version'
'/Users/xxx/Library/Haskell/bin/cpphs' '--version'
'/usr/bin/gcc' '-dumpversion'
'/usr/bin/haddock' '--version'
'/usr/bin/happy' '--version'
'/usr/bin/hpc' 'version'
looking for tool hsc2hs near compiler in /usr/bin
found hsc2hs in /usr/bin/hsc2hs
'/usr/bin/hsc2hs' '--version'
'/Users/andi/Library/Haskell/bin/HsColour' '-version'
'/usr/bin/ghc' '-c' '/var/folders/wg/g79zd2q15vqgl1f0h_b7gk0c0000gq/T/12271.c' '-o' '/var/folders/wg/g79zd2q15vqgl1f0h_b7gk0c0000gq/T/12271.o'
'/usr/bin/ld' '-x' '-r' '/var/folders/wg/g79zd2q15vqgl1f0h_b7gk0c0000gq/T/12271.o' '-o' '/var/folders/wg/g79zd2q15vqgl1f0h_b7gk0c0000gq/T/12272.o'
'/usr/bin/tar' '--help'
Reading available packages...
Choosing modular solver.
Resolving dependencies...
Ready to install bytestring-builder-0.10.4.1.1
Waiting for install task to finish...
Extracting
/Users/xxx/Library/Haskell/repo-cache/hackage.haskell.org/bytestring-builder/0.10.4.1.1/bytestring-builder-0.10.4.1.1.tar.gz
to
/var/folders/wg/g79zd2q15vqgl1f0h_b7gk0c0000gq/T/bytestring-builder-0.10.4.1.1-12271...
Updating bytestring-builder.cabal with the latest revision from the index.
Configuring bytestring-builder-0.10.4.1.1...
Flags chosen: bytestring_has_builder=False
Dependency base ==4.6.0.1: using base-4.6.0.1
Dependency bytestring ==0.10.0.2: using bytestring-0.10.0.2
Dependency deepseq ==1.3.0.1: using deepseq-1.3.0.1
Using Cabal-1.22.0.0 compiled by ghc-7.6
Using compiler: ghc-7.6.3
Using install prefix:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1
Binaries installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/bin
Libraries installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/lib
Private binaries installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/libexec
Data files installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/share
Documentation installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/doc
Configuration files installed in:
/Users/xxx/Library/Haskell/ghc-7.6.3/lib/bytestring-builder-0.10.4.1.1/etc
Using alex version 3.0.5 found on system at: /usr/bin/alex
Using ar found on system at: /usr/bin/ar
No c2hs found
Using cpphs version 1.18.1 found on system at:
/Users/xxx/Library/Haskell/bin/cpphs
Using gcc version 4.2.1 found on system at: /usr/bin/gcc
Using ghc version 7.6.3 found on system at: /usr/bin/ghc
Using ghc-pkg version 7.6.3 found on system at: /usr/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.13.2 found on system at: /usr/bin/haddock
Using happy version 1.18.10 found on system at: /usr/bin/happy
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.6 found on system at: /usr/bin/hpc
Using hsc2hs version 0.67 found on system at: /usr/bin/hsc2hs
Using hscolour version 1.20 found on system at:
/Users/xxx/Library/Haskell/bin/HsColour
No jhc found
Using ld found on system at: /usr/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using strip found on system at: /usr/bin/strip
Using tar found on system at: /usr/bin/tar
No uhc found
Component build order: library
creating dist/build
creating dist/build/autogen
Building bytestring-builder-0.10.4.1.1...
'/usr/bin/ghc-pkg' 'init' 'dist/package.conf.inplace'
Preprocessing library bytestring-builder-0.10.4.1.1...
Building library...
creating dist/build
'/usr/bin/ghc' '--make' '-fbuilding-cabal-package' '-O' '-outputdir' 'dist/build' '-odir' 'dist/build' '-hidir' 'dist/build' '-stubdir' 'dist/build' '-i' '-idist/build' '-isrc' '-idist/build/autogen' '-Idist/build/autogen' '-Idist/build' '-optP-include' '-optPdist/build/autogen/cabal_macros.h' '-package-name' 'bytestring-builder-0.10.4.1.1' '-hide-all-packages' '-package-db' 'dist/package.conf.inplace' '-package-id' 'base-4.6.0.1-6c351d70a24d3e96f315cba68f3acf57' '-package-id' 'bytestring-0.10.0.2-bf7aaeef3f644374cd7d5a9066cd9017' '-package-id' 'deepseq-1.3.0.1-5cc4cd89bdc2e8f6db1833d95ec36926' '-XHaskell98' 'Data.ByteString.Builder' 'Data.ByteString.Builder.Extra' 'Data.ByteString.Builder.Prim' 'Data.ByteString.Builder.Internal' 'Data.ByteString.Builder.Prim.Internal' 'Data.ByteString.Short' 'Data.ByteString.Short.Internal' 'Data.ByteString.Builder.ASCII' 'Data.ByteString.Builder.Prim.Binary' 'Data.ByteString.Builder.Prim.ASCII' 'Data.ByteString.Builder.Prim.Internal.Floating' 'Data.ByteString.Builder.Prim.Internal.UncheckedShifts' 'Data.ByteString.Builder.Prim.Internal.Base16'
src/Data/ByteString/Builder.hs:83:58:
warning: missing terminating ' character [-Winvalid-pp-token]
renderTable rs = 'mconcat' [renderRow r \<\> 'charUtf8' \'\\n\' | r <- rs]
^
src/Data/ByteString/Builder.hs:88:43:
warning: missing terminating ' character [-Winvalid-pp-token]
renderCell c \<\> mconcat [ charUtf8 \',\' \<\> renderCell c\' | c\' <- cs ]
^
src/Data/ByteString/Builder.hs:95:29:
warning: missing terminating ' character [-Winvalid-pp-token]
renderString cs = charUtf8 \'\"\' \<\> foldMap escape cs \<\> charUtf8 \'\"\'
^
src/Data/ByteString/Builder.hs:97:13:
warning: missing terminating ' character [-Winvalid-pp-token]
escape \'\\\\\' = charUtf8 \'\\\\\' \<\> charUtf8 \'\\\\\'
^
src/Data/ByteString/Builder.hs:98:13:
warning: missing terminating ' character [-Winvalid-pp-token]
escape \'\\\"\' = charUtf8 \'\\\\\' \<\> charUtf8 \'\\\"\'
^
src/Data/ByteString/Builder.hs:117:14:
warning: missing terminating '"' character [-Winvalid-pp-token]
strings = [\"hello\", \"\\\"1\\\"\", \"λ-wörld\"]
^
src/Data/ByteString/Builder.hs:128:70:
warning: missing terminating ' character [-Winvalid-pp-token]
We can clearly see that we are converting to a /binary/ format. The \'λ\'
^
src/Data/ByteString/Builder.hs:129:6:
warning: missing terminating ' character [-Winvalid-pp-token]
and \'ö\' characters, which have a Unicode codepoint above 127, are
^
src/Data/ByteString/Builder.hs:147:80:
warning: missing terminating ' character [-Winvalid-pp-token]
the above code takes 1ms to generate the 22'500 bytes long lazy 'L.ByteString'.
^
src/Data/ByteString/Builder.hs:171:10:
warning: missing terminating '"' character [-Winvalid-pp-token]
their \"inner loops\" using the functions in
^
10 warnings generated.
src/Data/ByteString/Builder/Extra.hs:81:43:
warning: missing terminating ' character [-Winvalid-pp-token]
-- * an IO action for writing the Builder's data into a user-supplied memory
^
src/Data/ByteString/Builder/Extra.hs:101:24:
warning: missing terminating ' character [-Winvalid-pp-token]
-- | This means we're all done. All the builder data has now been written.
^
src/Data/ByteString/Builder/Extra.hs:132:41:
warning: missing terminating ' character [-Winvalid-pp-token]
bufferFullH endPtr minReq step' =
^
src/Data/ByteString/Builder/Extra.hs:134:45:
warning: missing terminating ' character [-Winvalid-pp-token]
next = More minReq (run step')
^
src/Data/ByteString/Builder/Extra.hs:137:38:
warning: missing terminating ' character [-Winvalid-pp-token]
insertChunkH endPtr bs step' =
^
src/Data/ByteString/Builder/Extra.hs:139:42:
warning: missing terminating ' character [-Winvalid-pp-token]
next = Chunk bs (run step')
^
src/Data/ByteString/Builder/Extra.hs:153:77:
warning: missing terminating ' character [-Winvalid-pp-token]
-- host endian form, for the machine you're on. On a 64 bit machine the 'Int'
^
src/Data/ByteString/Builder/Extra.hs:178:78:
warning: missing terminating ' character [-Winvalid-pp-token]
-- host endian form, for the machine you're on. On a 64 bit machine the 'Word'
^
8 warnings generated.
src/Data/ByteString/Builder/Prim.hs:13:44:
warning: missing terminating ' character [-Winvalid-pp-token]
blocks for constructing 'Builder's. You don't need to go down to this level but
^
src/Data/ByteString/Builder/Prim.hs:66:33:
warning: missing terminating '"' character [-Winvalid-pp-token]
"Data.ByteString.Builder". Its \"hot code\" is the rendering of a table's
^
src/Data/ByteString/Builder/Prim.hs:78:31:
warning: missing terminating ' character [-Winvalid-pp-token]
renderString cs = B.charUtf8 \'\"\' \<\> foldMap escape cs \<\> B.charUtf8 \'\"\'
^
src/Data/ByteString/Builder/Prim.hs:80:13:
warning: missing terminating ' character [-Winvalid-pp-token]
escape \'\\\\\' = B.charUtf8 \'\\\\\' \<\> B.charUtf8 \'\\\\\'
^
src/Data/ByteString/Builder/Prim.hs:81:13:
warning: missing terminating ' character [-Winvalid-pp-token]
escape \'\\\"\' = B.charUtf8 \'\\\\\' \<\> B.charUtf8 \'\\\"\'
^
src/Data/ByteString/Builder/Prim.hs:99:46:
warning: missing terminating ' character [-Winvalid-pp-token]
B.charUtf8 \'\"\' \<\> E.'encodeListWithB' escape cs \<\> B.charUtf8 \'\"\'
^
src/Data/ByteString/Builder/Prim.hs:103:20:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\\\\\') (fixed2 (\'\\\\\', \'\\\\\')) $
^
src/Data/ByteString/Builder/Prim.hs:104:20:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\\\"\') (fixed2 (\'\\\\\', \'\\\"\')) $
^
src/Data/ByteString/Builder/Prim.hs:121:35:
warning: missing terminating '"' character [-Winvalid-pp-token]
maxiStrings = take 1000 $ cycle [\"hello\", \"\\\"1\\\"\", \"λ-wörld\"]
^
src/Data/ByteString/Builder/Prim.hs:135:76:
warning: missing terminating ' character [-Winvalid-pp-token]
exploits that the escaped character with the maximal Unicode codepoint is \'>\'.
^
src/Data/ByteString/Builder/Prim.hs:141:37:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (> \'\>\' ) E.'charUtf8' $
^
src/Data/ByteString/Builder/Prim.hs:142:18:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\<\' ) (fixed4 (\'&\',(\'l\',(\'t\',\';\')))) $ -- <
^
src/Data/ByteString/Builder/Prim.hs:143:18:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\>\' ) (fixed4 (\'&\',(\'g\',(\'t\',\';\')))) $ -- >
^
src/Data/ByteString/Builder/Prim.hs:144:18:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'&\' ) (fixed5 (\'&\',(\'a\',(\'m\',(\'p\',\';\'))))) $ -- &
^
src/Data/ByteString/Builder/Prim.hs:145:18:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\"\' ) (fixed5 (\'&\',(\'\#\',(\'3\',(\'4\',\';\'))))) $ -- &\#34;
^
src/Data/ByteString/Builder/Prim.hs:146:18:
warning: missing terminating ' character [-Winvalid-pp-token]
'condB' (== \'\\\'\') (fixed5 (\'&\',(\'\#\',(\'3\',(\'9\',\';\'))))) $ -- &\#39;
^
src/Data/ByteString/Builder/Prim.hs:147:84:
warning: missing terminating ' character [-Winvalid-pp-token]
('liftFixedToBounded' E.'char7') -- fallback for 'Char's smaller than \'\>\'
^
src/Data/ByteString/Builder/Prim.hs:203:26:
warning: missing terminating ' character [-Winvalid-pp-token]
-- combines escaping the ' and \\ characters with UTF-8 encoding. More
^
src/Data/ByteString/Builder/Prim.hs:317:70:
warning: missing terminating ' character [-Winvalid-pp-token]
A 'Builder' first checks if the buffer-range is large enough. If that's
^
src/Data/ByteString/Builder/Prim.hs:392:39:
warning: missing terminating '"' character [-Winvalid-pp-token]
-- Note that functions marked with \"/Heavy inlining./\" are forced to be
^
src/Data/ByteString/Builder/Prim.hs:520:78:
warning: missing terminating ' character [-Winvalid-pp-token]
-- inner loop. That's not possible when writing 'encodeListWith' as a 'foldr'.
^
src/Data/ByteString/Builder/Prim.hs:552:11:
warning: missing terminating ' character [-Winvalid-pp-token]
op' <- runB w x op
^
src/Data/ByteString/Builder/Prim.hs:554:13:
warning: missing terminating ' character [-Winvalid-pp-token]
k br'
^
src/Data/ByteString/Builder/Prim.hs:569:4:
error: invalid preprocessing directive
#-}
^
src/Data/ByteString/Builder/Prim.hs:618:52:
warning: missing terminating ' character [-Winvalid-pp-token]
go !Nothing !op = do let !br' = BufferRange op ope0
^
src/Data/ByteString/Builder/Prim.hs:619:49:
warning: missing terminating ' character [-Winvalid-pp-token]
k br'
^
src/Data/ByteString/Builder/Prim.hs:620:24:
warning: missing terminating ' character [-Winvalid-pp-token]
go !(Just (y, x')) !op
^
src/Data/ByteString/Builder/Prim.hs:621:65:
warning: missing terminating ' character [-Winvalid-pp-token]
| op `plusPtr` bound <= ope0 = runB w y op >>= go (f x')
^
src/Data/ByteString/Builder/Prim.hs:624:25:
warning: missing terminating ' character [-Winvalid-pp-token]
!opNew' <- runB w y opNew
^
src/Data/ByteString/Builder/Prim.hs:664:25:
warning: missing terminating ' character [-Winvalid-pp-token]
op' <- runB w x op
^
src/Data/ByteString/Builder/Prim.hs:665:45:
warning: missing terminating ' character [-Winvalid-pp-token]
go (ip `plusPtr` 1) op'
^
30 warnings and 1 error generated.
Failed to install bytestring-builder-0.10.4.1.1
World file is already up to date.
cabal: Error: some packages failed to install:
bytestring-builder-0.10.4.1.1 failed during the building phase. The exception
was:
ExitFailure 1
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.