Input range checking about pysodium

[stef]

i mean both nonces are very sensitive, just as much as keys. handling these must be done with care. and if errors like the ones we capture here with these exceptions are made, then the whole protocol above is of very questionable quality. but i'm getting convinced that we should indeed do type/valueerrors, defense-in-depth makes sense, with some documentation that puts the devs into a position to explain how they even got into this code-path.

[stef]

nice find, thanks.

[jvarho]

As a user of the library I'd much prefer getting a (useful, with description!) ValueError or TypeError than an assert fail if I pass an argument of the wrong type or size.

[stef]

interesting question. i think what you are doing is analogous to division by zero.

i believe you are abusing the libsodium api. its clearly documented what the parameters should be. both nonces and keys are not something that you should provide arbitrarily. keys should never be provided like you do, they should be somehow derived either using a Key derivation function or by using a DH like key exchange. nonces are similar, they have much stricter requirements than only being a byte array of a certain size, they also need to be used only once.

if you do what you do in your examples you have much bigger problems than segfaulting python interpreters.

i would like to keep the wrapper a thin layer. and if you use this thin layer correctly these problems should never occur.

you could achieve the same crashing results by using c and calling the libsodium functions incorrectly.

shall div check for 0 divisor? shall div also check for other edgecases?

maybe we should check these things, but i'm not sure the costs and the benefits are in relation.

[robehickman]

Yes, I know that the key should be derived from a key derivation function and nonces need to be created with a CSPRNG or a counter depending on the algorithm. I accidentally passed a null key to it while experimenting and was surprised that it worked, then tried a few other things like passing an incorrect nonce.

Considering nonces do not have to be kept private unlike keys it is conceivable that a 3rd party could be in a position to provide an invalid nonce, if that crashes an application it creates a denial of service.

Say, encrypt a message with crypto_secretbox using a key created from a password using crypto_pwhash. Both of these functions need nonces which must be transmitted with the message (or otherwise transmitted) in order to decrypt it.

So you think that range checks in such cases should be done by the application?

[stef]

i hope this not one of those lame vendor responses i always loathed.

[stef]

nonces don't have to be always transmitted, there is also nonces that are derived from the message, or simply counters that are implicit.

[jedisct1]

There's a difference between a division by zero and not checking the length before calling an external API. The outcome of a division by zero is well defined. The outcome of reading uninitialized memory is way less predicable.

Everything may seem to work as expected, and under some hard-to-define, hard-to-reproduce conditions, the application crashes or may expose a vulnerability. This is not great.

Even if implemented as simple assert statements, I'd suggest checking parameters lengths.

[jedisct1]

(but I know adding these checks is not fun)

[stef]

3rd party could be in a position to provide an invalid nonce, if that crashes an application it creates a denial of service.

the only thing a 3rd party could do is to truncate a sole nonce. changing it would only make the decryption fail. truncation could be indeed a problem. and it is depending on the implementation if that is feasible. imagine a simple network packet which is "nonce | ciphertext | mac" in this case truncation of the nonce should be ok. however in a different implementation receiving a json structure {nonce: "xxx", ciphertext: "..."} the truncation is a problem.

[stef]

but i guess we also have a test if the ciphertext is long enough to hold the mac. so i guess checking for the nonce length makes also sense.

[jedisct1]

we also have a test if the ciphertext is long enough to hold the mac

This is not strictly required. Decryption functions in libsodium will reject short ciphertexts.

[stef]

right the check was about the allocation of memory for the message without the mac. i was inaccurate in my phrasing.

[robehickman]

I don't understand your first response, and wasn't aware of deriving a nonce from a message. I know that changing the nonce would cause decryption to fail. An attacker truncating a message and causing a crash, while it does not expose the message, is still a nuisance. If this was on a web-exposed system it would allow anyone to crash your server...

The reason that I raised this concern is because, while a C developer is aware of these issues, Python developers don't tend to think about range checks as the language does it most of the time. Access past the end of an array in C you get a segfault or read uninitialized memory. If you access past the end of a python list you get a catchable exception.

[stef]

wasn't aware of deriving a nonce from a message.

imagine some protocol where each message is encrypted with a new shared secret (because there might be a parallel DH with every message as in signal/otr), then you could derive the nonce by hash(key) and since both parties have the key, they can also derive the nonce. or both parties count the messages they exchanged and use that counter as the nonce.

[stef]

range checks as the language does it most of the time.

the language i don't think does any range checks.

Access past the end of an array in C you get a segfault or corrupt the stack, the classic buffer overflow.

this nonce stuff would be a read past the stack, not a write, a classic buffer overflow would be writing past the stack.

[robehickman]

Thanks for explaining re nonces and yes I misused the term buffer overflow.

All I meant is that Python generally validates parameters and raises an exception if something is wrong, segfaulting is unusual. If you don't want to implement this in your library that's fine but I'd at least mention it in the documentation.

[stef]

i guess i'm gonna add a bunch of asserts, but only after the holidays. i guess it can't hurt, but prevents some shooting in foot scenarios.

[robehickman]

I have started to add some asserts starting from the bottom of init.py working upwards.

[robehickman]

I have added asserts between line 550 and the end of the file. Adding the following one (to the second function below) causes a test to fail and I'm not sure why as it's allocated in the first function below using the same constant. Is there a reason for 'out.value' here where most use 'out.raw'?

def crypto_pwhash_scryptsalsa208sha256_str(passwd, opslimit, memlimit):
    if None in (passwd, opslimit, memlimit):
        raise ValueError
    out = ctypes.create_string_buffer(crypto_pwhash_scryptsalsa208sha256_STRBYTES)
    __check(sodium.crypto_pwhash_scryptsalsa208sha256_str(out, passwd, ctypes.c_ulonglong(len(passwd)), ctypes.c_ulonglong(opslimit), ctypes.c_size_t(memlimit)))
    return out.value

def crypto_pwhash_scryptsalsa208sha256_str_verify(stored, passwd):
    if stored is None or passwd is None:
       raise ValueError
    #assert len(stored) == crypto_pwhash_scryptsalsa208sha256_STRBYTES, "Truncated stored password"
 __check(sodium.crypto_pwhash_scryptsalsa208sha256_str_verify(stored, passwd, ctypes.c_ulonglong(len(passwd))))

[stef]

thanks! can you make it into a branch? then i can pull you and check it out myself.

[robehickman]

https://github.com/robehickman/pysodium/tree/dev

In the existing code there where a number of functions which check the length of the passed argument and raise ValueError. There where also a few asserts doing the same. In python a failing assert is the same as using an if statement raising AssertionError. Is there any consensus on what to use where?

[stef]

asserts are removed when compiling with optimizations. so it's a good question which to use, i guess the assert is better than the if not raise valueerrors for this reason.

[robehickman]

Good point, I wouldn't be inclined to change any of the existing checks raising ValueError though as it would break any application code that catches them.

[robehickman]

Note that the assert in the branch linked above for 'crypto_pwhash_scryptsalsa208sha256_str_verify' is commented out.

[robehickman]

Another thought, considering that most error handling that exists raises ValueError, would sticking with that be preferable for consistency?

[stef]

@jvarho the asserts contain information regarding the problem which was asserted. the backtrace is informative, gives you all the information you need. i think these should help during development, but when deployed there should be no need for them as the calling code should be well tested and ensure that there cannot be wrong sized/typed parameters. i don't see a difference between assert exceptions and type or value error exceptions. with one caveat when optimized py code is written the asserts are eliminated. can you give an example of why the newly added asserts are not as good as type/value error exceptions, and how should those look instead? i'm kinda undecided about all this, happy to hear more elaborate reasoning.

[robehickman]

Yet another thing that came to mind regarding this is the C type checker. I don't know what protection it would provide in this situation as I haven't used C for a long time. From my limited experience python CTypes feels kind of like pointer to void, no effective type checking.

[jvarho]

@stef the difference is that Value/TypeErrors are easy to catch and present to the user (or next layer) when they are due to user input. They allow you to avoid writing another layer that checks everything and instead write pythonic "EAFP" code.

[stef]

why is catching Value/TypeErrors easier than AssertionErrors? they are also just exceptions. and why are they easier to represent than AssertionError? and what use-case protocol do you have in mind where the user input should be directly used as keys/nonces without some sanity checks on them? both nonces and keys should be something that users should never really come in direct contact with. what does EAFP mean?

try: assert False, 'doomed'
except AssertionError as e:
   print e

is quite equivalent to

try: raise TypeError( 'doomed')
except TypeError as e:
   print e

[stef]

usually you want to use a hash, DH, or kdf on what ever is gonna be used as a key, and probably some kind of counter, randombytes or hash for the nonces. and nonces are especially sensitive when reused and stuff, having users control them sounds dangerous.

[jvarho]

@stef Like you wrote above, assertions can be disabled at compile time, so they are not equivalent. Relying on catching an AssertionError can lead to bugs.

EAFP = Easier to Ask for Forgiveness than Permission

[stef]

Relying on catching an AssertionError can lead to bugs.

i agree on that. however i also believe that this is more for helping devs, and if you have a protocol where you can raise such an assertion during production, then you have a huge problem. in production such exceptions should never be possible. irrespective of type/valueerrors or assertionerrors.

[stef]

maybe we should opt for the type/value exceptions and note that if any of these are ever triggered the devs should seriously reconsider their protocol. ;)

[jvarho]

I agree that they should never "happen", i.e. be passed on, but there's nothing wrong with having code that triggers and handles them. That is, I think code such as this is fine:

try:
    plaintext = crypto_secretbox_open(c, nonce, k)
except ValueError:
    log("Invalid cryptotext")
    # ...handle it

If c comes from the user or the network it can be whatever. Similarly, if some parameter where length matters comes from the user/network/file it could be handled similarly.

[stef]

sure, that is equivalent to what i posted above. but c is irrelevant here, most of the checks are related to kand nonce. c is in fact checked by libsodium, e.g.

    if (clen < crypto_box_curve25519xchacha20poly1305_MACBYTES) {
        return -1;
    }

in libsodium/src/libsodium/crypto_box/curve25519xchacha20poly1305/box_curve25519xchacha20poly1305.c:151

[stef]

check out ecc7adb and close this issue if you agree that this solves this.

[stef]

some more commits on this fix some bugs for older libsodiums and make the whole thing a bit cleaner. check it out and if acceptable please close this issue.

[robehickman]

Looks fine.