centerforsecureenergyinformatics / data-compressor Goto Github PK

DCCLI reads and writes full bytes from and to input and output files, respectively. If only a fraction of the last byte is used, this may lead to decoding errors. Adding, e.g., a module which adds/parses a file header can solve this issue by reserving additional space to store the number of used bits.

From the documentation:

If a fractional number of bytes (i.e., a number of bits not divisible by eight) is written to the output file, decoding said output file later may lead to errors at the last byte when processing the superfluous bits at the end of the file.

The diff encoder currently only supports unsigned input values. Similarly, the diff decoder only supports unsigned output values.

An option, e.g., signed, should be added which allows switching between signed and unsigned values. It should default to unsigned to retain backwards compatibility.

Add automated tests for different platforms (x86, x86-64, ARM) to validate compilation and encoding/decoding test data with IO_SIZE_BITS of 16, 32 and 64.

Either add an additional solution for newer Visual Studio versions (2017, 2019) or verify that the current solution also works fine with these versions.

buffer_t and file_buffter_t support peek operations, but bit_file_buffer_t does not (yet):

Rename all documentation currently in .txt format to .md and make use of Markdown formatting, e.g., links and code/call formatting.

Add a .gitignore file so that .o files, .a files and the like are not tracked in version control.

Create a smaller version of the testdata file and verify that all example calls still work.

When sizeof(io_int_t) is exactly sizeof(size_t), i.e., as large as IO_SIZE_BITS in bytes, range checks and other operations do not work properly anymore, some of them without any warnings or errors. As documented in the common documentation:

If IO_SIZE_BITS is the same size as size_t, the Read/Write functions in dependent libraries do not work properly if the MSB of a size_t variable specifying the size to be read/written is used. For example, if IO_SIZE_BITS is 32 and sizeof(size_t) is 4, the maximum size (parameter value) that the Read/Write function can work with is 2^31 - 1, i.e., the 32nd bit cannot be used. If it is used, the return value of the functions will be interpreted as an error (since it is interpreted as a negative number)

Two solutions are possible:

1. Disallow parameters, e.g., valuesize to be as large as IO_SIZE_BITS and issue an error. This may have undesired side effects as the machine word size could not be used for I/O operations on the machine, e.g., 64-bit reads would not be supported on 64-bit machines.
2. (preferred) Add more sophisticated range checks to issue warnings or errors whenever there are overflows/underflows. This might require a change in architecture (e.g., additional out parameters instead of return values to reserve the latters for errors only).

Verify that the code files and Makefiles work fine with newer gcc versions (8.x).