sha2 API hash_words does not support iterated hashing about risc0

[intoverflow]

hash_words() and iterated hashing

Our SHA2 API is based on u32 words: our Digest type provides an easy way to get a &[u32] out of the digest, and our SHA2 API provides a convenient hash_words(&[u32]) -> Digest function.

Now suppose someone wants to compute sha256(sha256(x)). Intuitively, one should be able to do something like hash_words(hash_words(x)). However, due to the way endianness is treated by the current implementation, this does not work as expected.

Sample code

To illustrate the issue, I've created a repo that computes iterated SHA2 hashes, using both the sha2 crate as well as the R0 implementation of SHA2: https://github.com/intoverflow/r0-sha-wishlist

You can run the code by cloning the repo and running cargo run --release. The main() function compares iterated SHA2 output for various numbers of iterations (currently 1 and 2), and using different methods of computing SHA2.

Notably, when the iteration count is just 1, all of the tests pass. This means I am basically using the functions correctly. However, when the iteration count is 2 or greater, the hash_words() test case fails.

I've included the relevant source citations below:

Iterated SHA2 using the sha2 crate

https://github.com/intoverflow/r0-sha-wishlist/blob/main/host/src/main.rs#L25

        let host_sha2crate_output: Vec<u8> = {
            let mut host_sha2crate_output = Vec::from([0u8; 32]);

            for _i in 0..num_iter {
                let mut hasher = Sha256::new();
                hasher.update(&host_sha2crate_output);
                host_sha2crate_output = hasher.finalize().to_vec();
            }

            host_sha2crate_output
        };

Iterated SHA2 using R0's byte-level interface

https://github.com/intoverflow/r0-sha-wishlist/blob/main/methods/guest/src/bin/iter_sha2_bytes.rs

    let hasher = sha::Impl { };
    let guest_input: Vec<u32> = env::read();

    let num_iter: u32 = guest_input[0];

    let mut hash = &guest_input[1..9];
    for _i in 0 .. num_iter {
        let bytes: Vec<u8> = hash.iter().map(|x| x.to_be_bytes()).flatten().collect();
        hash = hasher.hash_bytes(&bytes).get();
    }

    env::commit(&Vec::from(hash))

Iterated SHA2 using R0's word-level interface (does not work!)

https://github.com/intoverflow/r0-sha-wishlist/blob/main/methods/guest/src/bin/iter_sha2_words.rs

#![no_main]

use risc0_zkvm_guest::{env, sha};
use risc0_zkp::core::sha::Sha;

risc0_zkvm_guest::entry!(main);

pub fn main() {
    let hasher = sha::Impl { };
    let guest_input: Vec<u32> = env::read();

    let num_iter: u32 = guest_input[0];

    let mut hash = &guest_input[1..9];
    for _i in 0 .. num_iter {
        hash = hasher.hash_words(hash).get();
    }

    env::commit(&Vec::from(hash))
}

Proposal

Replace hash_words() with hash_words_le() and hash_words_be()

[intoverflow]

Output from running the test program:

(Note that the assertion fails at this line: https://github.com/intoverflow/r0-sha-wishlist/blob/main/host/src/main.rs#L65)

tcarstens@tcarstens-dev1:~/risc0/r0-sha-wishlist$ cargo run --release
   Compiling methods v0.1.0 (/home/tcarstens/risc0/r0-sha-wishlist/methods)
methods-guest: Starting build for riscv32im-risc0-zkvm-elf   ld)                                                                                                                          
methods-guest:     Updating crates.io index   
methods-guest:     Updating git repository `https://github.com/risc0/risc0`   
methods-guest:     Finished release [optimized] target(s) in 0.33s   
   Compiling starter v0.1.0 (/home/tcarstens/risc0/r0-sha-wishlist/host)
    Finished release [optimized] target(s) in 1.54s
     Running `target/release/starter`
num_iter = 1
+ iter_sha2_bytes
  ... checking host_sha2crate_output == guest_output
+ iter_sha2_words
  ... checking host_r0_output == guest_output
  ... checking host_sha2crate_output == guest_output
num_iter = 2
+ iter_sha2_bytes
  ... checking host_sha2crate_output == guest_output
+ iter_sha2_words
  ... checking host_r0_output == guest_output
  ... checking host_sha2crate_output == guest_output
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[43, 50, 219, 108, 44, 10, 98, 53, 251, 19, 151, 232, 34, 94, 168, 94, 15, 14, 110, 140, 123, 18, 109, 0, 22, 204, 189, 224, 230, 103, 21, 30]`,
 right: `[246, 251, 169, 146, 9, 145, 101, 49, 108, 72, 136, 220, 180, 206, 167, 250, 218, 229, 123, 69, 195, 148, 221, 144, 131, 191, 19, 13, 65, 148, 199, 102]`', host/src/main.rs:65:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
tcarstens@tcarstens-dev1:~/risc0/r0-sha-wishlist$

[flaub]

I think this has now been resolved, such that running hash_words in a loop now works, but if it doesn't please feel free to re-open this.