different read blocksize result in failed checksum about blake2

[sneves]

Yes, that is strange and should not happen. Do you have a reproducible case?

[hajes]

In Python I used following and it always failed:

BLOCKSIZE = os.statvfs(os.getcwd()).f_bsize # determine optimal/prefered block size of file system

with open(f, 'rb') as afile:
buf = afile.read(BLOCKSIZE)
chk.update(buf)
l[f] = chk.hexdigest()
afile.close()

It automatically detects file system block size...my Apple SSD has got 1MB whereas external HDD has got 65kB for example. Unless, both blocksizes are not equal there is problem as confirmed by using official b2sum.

Ever since I use fixed blocksize all seems to be ok until I used b2sum. I noticed in b2sum source code there is buffer size of 37kB or so.

Hard to say what happen if you play with buffer/blocksize on different OS

Of course it may be some multiOS issue as well !

Blake2 is not official under macOS and it has been installed by brew install coreutils

Python Blake2 implementation is considered as extra compared to core hash functions. Various Python OS implantation give different names fro Blake2, which should be no issue.

[sneves]

I'm not sure how to reproduce this. Does the following script fail for you? (Feel free to adjust parameters or sizes as you please)

from hashlib import blake2b
import os

MAX_LEN = 1234
DATA = os.urandom(MAX_LEN)

reference = blake2b(DATA).digest()

for step in range(1,MAX_LEN):
  obj2 = blake2b()
  for chunk in [DATA[i:i+step] for i in range(0, len(DATA), step)]:
    obj2.update(chunk)
  assert(obj2.digest() == reference)

print("ok")

[hajes]

your script prints ok

where I use it is as following:

• read list of files inside source directory (usually on CF card from camera)
• create hash for each file as shown in function above by [still in source path]
• copy files to destination (usually SSD)
• create checksum fo copied files [destination path]
• compare checksum created in source and destination

In other words if I create checksum of file in source with read blocksize 1MB and check the very same file in destination path but read blocksize different than 1MB - checksum fails.

[sneves]

BLOCKSIZE = os.statvfs(os.getcwd()).f_bsize	# determine optimal/prefered block size of file system

with open(f, 'rb') as afile:
  buf = afile.read(BLOCKSIZE)
  chk.update(buf)
l[f] = chk.hexdigest()
afile.close()

I assume this is the correct indentation? If so, aren't you just reading the first BLOCKSIZE bytes of the file?

[hajes]

Last two lines false indentation, it suppose to be in cycle as above by.

I have no idea, I used this code from some guy on internet. It suppose to read file by chunks defined by blocksize. If you don't specify blocksize/buffer it reads whole file into memory...I often copy larger files than my 16GB RAM.

https://docs.python.org/3.4/library/io.html?highlight=io#io.BufferedIOBase.read

[sneves]

To hash the entire file, you have to do something like

with open(f, 'rb') as fd:
  chk = blake2b()
  while True:
    chunk = fd.read(BLOCKSIZE)
    if not chunk:
      break
    chk.update(chunk)
print(chk.hexdigest())

So I don't think blake2 is the problem here.

[hajes]

thanks for info, I will try.