orkon / base-x-rs Goto Github PK

No_alloc support seems to be possible, but would require a change in the API. Instead of constructing Strings or Vec, just use mutable slice passed in. This change would either require a seperate section of the API dedicated to no_alloc, or causing breaking changes in the current one.

Considering the nature of base-x, it might be worth considering to limit the encoder input and return an error if limit is reached (yes, I know I just had a PR merged that removed Result type from encode, bare with me).

The way encoder works now is continously reallocating a Vec as new digits are pushed onto it (Rust will grow the vec exponentially in this case, so it's not super terrible), then reverse the Vec to obtain the encoded value. Since what we are trying to do is, effectively, treat the input byte buffer as a single integer and modulo/divide it by base to obtain digits, it might be worth considering an optimization usually done when stringifying integers:

• Instead of a heap allocated Vec, create a reasonably big byte array buffer on stack. When stringifying u64 to decimal the buffer would need to be 20 bytes long since that's how many decimal digits u64 can produce. In our case a buffer of 1024 bytes could be enough for virtually all use-cases. The buffer can be uninitialized via unsafe { mem::uinitialized() }, at which point it really doesn't matter how big it is.
• Instead writing digits from the beginning and reversing, write the incoming digits from the end (I know, right).
• Once done, take a slice of the buffer from the end equal to the number of digits produces and convert to a Vec.

Pros:

• No expensive reversing.
• Only a single heap allocation performed during the whole process, and only allocating the precise amount of bytes necessary.
• Operating on stack is generally faster since you don't have to worry so much about going out of CPU caches.

Cons:

• Can only produce outputs no longer than the buffer (1024 bytes). Since we know the size of the input and the base of the alphabet, we can predict an upper bound for which the buffer would be overflowed and return a meaningful error should that be the case.

Per https://github.com/cryptocoinjs/base-x

WARNING: This module is NOT RFC3548 compliant, it cannot be used for base16 (hex), base32, or base64 encoding in a standards compliant manner.

This is because standards-compliant base64 aligns the start of processing with the start of the data (hence implicitly padding the bottom with zeroes), whereas base-x aligns with the end (therefore padding the top) - it will work sometimes, if the number of bits in the data is divisible by 6 (for base64) as there are no bits to pad. I found https://www.lucidchart.com/techblog/2017/10/23/base64-encoding-a-visual-explanation/ useful to help my understanding)

(as it happens, I think it might be ok for base16 because bytes are always divisible into 4 bits, but 5 bits with base32 surely has the same problem)

Concretely, encoding 'A' with a standards compliant base64 encoder results in QQ, but results in BB with base-x.

Using an array of bytes as the alphabet is fine if all we are using is ASCII characters that are byte each, but however Rust strings are UTF-8 strings by default, meaning the characters in strings can be multiple bytes (char type is 4 bytes).

Also, not all Unicode characters (unicode codepoints) can be represented by a char, that is just a 'unicode scalar value'.

I might do a pull request on this later.

This line:

Needs to be:

#[cfg(features = "std")]