Checksum calculation mismatch failure on python 3 about ngas

[rtobar]

That's because of a change on how crc32 is represented under the hood between ptyhon2 and 3. That's why we introduced the crc_info structures, which take this into account. For example, this is how we check the checksum in the ARCHIVE/QARCHIVE command:

ngas/src/ngamsServer/ngamsServer/ngamsArchiveUtils.py

Line 247 in eb83678

if not checksum_info.equals(checksum, result.crc):

Note also thatt this old crc32 checksum behavior means if a checksum was calculated and written into the database with python2, then it might cause the file checker thread to fail if you are running the system now with python3. This is why I named this variant CHECKSUM_CRC32_INCONSISTENT in the code, why you need to use these special equals method, and why I introduced the "crc32z" variant (which could have been better named probably). All details are here:

ngas/src/ngamsServer/ngamsServer/ngamsFileUtils.py

Lines 812 to 901 in eb83678

	CHECKSUM_NULL = -1
	CHECKSUM_CRC32_INCONSISTENT = 0
	CHECKSUM_CRC32C = 1
	CHECKSUM_CRC32Z = 2
	def _normalize_variant(variant_or_name):
	variant = variant_or_name
	if variant is None:
	variant = CHECKSUM_NULL
	# A plug-in name or variant name
	elif isinstance(variant, six.string_types):
	# In NGAS versions <= 8 the CRC was calculated as a separate step after
	# archiving a file, and therefore was loaded as a plugin that received a
	# filename when invoked.
	# These two are the names stored at the database of those plugins, although
	# the second one is simply a dummy name
	if variant in ('ngamsGenCrc32', 'StreamCrc32', 'crc32'):
	variant = CHECKSUM_CRC32_INCONSISTENT
	elif variant == 'crc32c':
	variant = CHECKSUM_CRC32C
	elif variant == 'crc32z':
	variant = CHECKSUM_CRC32Z
	else:
	variant = int(variant)
	return variant
	def _filter_none(cond):
	def wrapped(x, y):
	if x is None:
	return y is None
	elif y is None:
	return x is None
	return cond(x, y)
	return wrapped
	def get_checksum_info(variant_or_name):
	"""
	Given a CRC variant, this method returns the method that should be
	continuously called to calculate the CRC of a given byte stream.
	The variant_or_name argument can be a number, where 0 is python's binascii
	crc32 implementation and 1 is Intel's SSE 4.2 CRC32c implementation, or a
	name indicating one of the old NGAMS plug-in names for performing CRC.
	The special value -1 means that no checksum is performed, and thus this
	method returns None
	"""
	variant = _normalize_variant(variant_or_name)
	if variant == CHECKSUM_NULL:
	return None
	if variant == CHECKSUM_CRC32_INCONSISTENT:
	# This version of the crc is inconsistent because depending on the
	# python version binascii.crc32 returns signed or unsigned values.
	# python version <2.6 returned signed/unsigned depending on the platform,
	# 2.6+ returns always signed, 3+ returns always unsigned).
	fmt = '!i' if six.PY2 else '!I'
	return checksum_info(0, binascii.crc32, lambda x: x, lambda x: struct.unpack(fmt, x)[0], _filter_none(lambda x, y: (int(x) & 0xffffffff) == (int(y) & 0xffffffff)))
	elif variant == CHECKSUM_CRC32C:
	if not _crc32c_available:
	raise Exception('Intel SSE 4.2 CRC32c instruction is not available')
	return checksum_info(0, crc32c.crc32, lambda x: x & 0xffffffff, lambda x: struct.unpack('!I', x)[0], _filter_none(lambda x, y: int(x) == int(y)))
	elif variant == CHECKSUM_CRC32Z:
	# A consistent way of using binascii.crc32.
	return checksum_info(0, binascii.crc32, lambda x: x & 0xffffffff, lambda x: struct.unpack('!I', x)[0], _filter_none(lambda x, y: int(x) == int(y)))
	raise Exception('Unknown CRC variant: %r' % (variant_or_name,))
	def get_checksum_name(variant_or_name):
	"""
	Given a CRC variant, this method returns the name used to denote that
	variant.
	The variant_or_name argument can be a number, where 0 is python's binascii
	crc32 implementation and 1 is Intel's SSE 4.2 CRC32c implementation, or a
	name indicating one of the old NGAMS plug-in names for performing CRC.
	The special value -1 means that no checksum is performed, and thus this
	method returns 'nocrc'
	"""
	variant = _normalize_variant(variant_or_name)
	if variant == CHECKSUM_NULL:
	return None
	if variant == CHECKSUM_CRC32_INCONSISTENT:
	return 'crc32'
	elif variant == CHECKSUM_CRC32C:
	return 'crc32c'
	elif variant == CHECKSUM_CRC32Z:
	return 'crc32z'
	raise Exception('Unknown CRC variant: %d' % (variant_or_name,))

[smclay]

Thanks @rtobar. I have applied a fix following your suggestion. It now passes my test.

[rtobar]

Merged, thanks!