Private Key (Secret Key) length is 2544, It's not match to 2528 (serialize) about dilithium-java

[eidola7]

Crystal Dilithium Ref code (in C ) :
uses tr[SEEDBYTES],

void pack_sk(uint8_t sk[CRYPTO_SECRETKEYBYTES],
             const uint8_t rho[SEEDBYTES],
             const uint8_t tr[SEEDBYTES],
             const uint8_t key[SEEDBYTES],
             const polyveck *t0,
             const polyvecl *s1,
             const polyveck *s2)
{
  unsigned int i;

  for(i = 0; i < SEEDBYTES; ++i)
    sk[i] = rho[i];
  sk += SEEDBYTES;

  for(i = 0; i < SEEDBYTES; ++i)
    sk[i] = key[i];
  sk += SEEDBYTES;

  for(i = 0; i < SEEDBYTES; ++i)
    sk[i] = tr[i];
  sk += SEEDBYTES;

  for(i = 0; i < L; ++i)
    polyeta_pack(sk + i*POLYETA_PACKEDBYTES, &s1->vec[i]);
  sk += L*POLYETA_PACKEDBYTES;

  for(i = 0; i < K; ++i)
    polyeta_pack(sk + i*POLYETA_PACKEDBYTES, &s2->vec[i]);
  sk += K*POLYETA_PACKEDBYTES;

  for(i = 0; i < K; ++i)
    polyt0_pack(sk + i*POLYT0_PACKEDBYTES, &t0->vec[i]);
}

but, in Dilithium-java,
uses tr[Dilithium.CRHBYTES] instead of Dilithium.SEEDBYTES
and, it's different 32 vs. 48

		public final static int SEEDBYTES = 32;
		public final static int CRHBYTES = 48;

		final int CRYPTO_SECRETKEYBYTES = (2 * Dilithium.SEEDBYTES + Dilithium.CRHBYTES + s1.length() * POLYETA_PACKEDBYTES
				+ s2.length() * POLYETA_PACKEDBYTES + s2.length() * Dilithium.POLYT0_PACKEDBYTES);
		byte[] buf = new byte[CRYPTO_SECRETKEYBYTES];
	
		for (int i = 0; i < Dilithium.SEEDBYTES; i++)
			buf[off + i] = rho[i];
		off += Dilithium.SEEDBYTES;
	
		for (int i = 0; i < Dilithium.SEEDBYTES; i++)
			buf[off + i] = K[i];
		off += Dilithium.SEEDBYTES;
	
		for (int i = 0; i < Dilithium.CRHBYTES; i++)
			buf[off + i] = tr[i];
		off += Dilithium.CRHBYTES;
	
		for (int i = 0; i < s1.length(); i++) {
			s1.poly[i].etapack(eta, buf, off);
			off += POLYETA_PACKEDBYTES;
		}
	
		for (int i = 0; i < s2.length(); i++) {
			s2.poly[i].etapack(eta, buf, off);
			off += POLYETA_PACKEDBYTES;
		}
	
		for (int i = 0; i < t0.length(); i++) {
			t0.poly[i].t0pack(buf, off);
			off += Dilithium.POLYT0_PACKEDBYTES;
		}
		return buf;

[eidola7]

OK, now it operate well. Maybe some parameteric difference here.

Cross sign/verify with Crystal Dilithium_ref code

impl/Dilithium.java :

	// public final static int CRHBYTES = 48;
	public final static int CRHBYTES = 64;

impl/PackingUtils.java :

static byte[] packPrvKey(int eta, byte[] rho, byte[] tr, byte[] K, PolyVec t0, PolyVec s1, PolyVec s2) {
...
		// final int CRYPTO_SECRETKEYBYTES = (2 * Dilithium.SEEDBYTES + Dilithium.CRHBYTES + s1.length() * POLYETA_PACKEDBYTES
		// 		+ s2.length() * POLYETA_PACKEDBYTES + s2.length() * Dilithium.POLYT0_PACKEDBYTES);
		final int CRYPTO_SECRETKEYBYTES = (3 * Dilithium.SEEDBYTES + s1.length() * POLYETA_PACKEDBYTES
				+ s2.length() * POLYETA_PACKEDBYTES + s2.length() * Dilithium.POLYT0_PACKEDBYTES);

		// for (int i = 0; i < Dilithium.CRHBYTES; i++)
		// 	buf[off + i] = tr[i];
		// off += Dilithium.CRHBYTES;
		for (int i = 0; i < Dilithium.SEEDBYTES; i++)
			buf[off + i] = tr[i];
		off += Dilithium.SEEDBYTES;
...
	public static PrivateKey unpackPrivateKey(DilithiumParameterSpec parameterSpec, byte[] bytes) {
...
		// byte[] tr = new byte[Dilithium.CRHBYTES];
		// for(int i = 0; i < Dilithium.CRHBYTES; i++) {
		// 	tr[i] = bytes[off+i];
		// }
		// off += Dilithium.CRHBYTES;
		byte[] tr = new byte[Dilithium.SEEDBYTES];
		for(int i = 0; i < Dilithium.SEEDBYTES; i++) {
			tr[i] = bytes[off+i];
		}
		off += Dilithium.SEEDBYTES;

[eidola7]

But, now then,,, Dilithium-java self sign & verify failed....

[eidola7]

It's OK now, after change below matching to Crystal Dilithium_ref (.c)

int crypto_sign_verify(const uint8_t *sig, size_t siglen, const uint8_t *m, size_t mlen, const uint8_t *pk) {
...
  /* Compute CRH(H(rho, t1), msg) */
  shake256(mu, SEEDBYTES, pk, CRYPTO_PUBLICKEYBYTES);

For Dilithium-java
impl/Dilithium.java

	public static boolean verify(DilithiumPublicKey pk, byte[] sig, byte[] M) {
...
		// byte[] mu = Utils.crh(pk.getEncoded()); // use CRHBYTES
		byte[] mu = Utils.getSHAKE256Digest(SEEDBYTES, pk.getEncoded());

[mthiim]

Hi thanks for writing! I based my implementation on the last (round 3) submission to NIST since I supposed that's the one that was approved: https://pq-crystals.org/dilithium/resources.shtml. This does it the sake way my implementation does (2xSEED + CRH) both with respect to their PDF (see algorithm "Gen" on page 13 where "tr" is defined as a 384-bit = 48 bytes output of the "CRH" - collission resistant hash - this is what causes 32 + 32 + 48) and their reference C code. My implementation also passed all the KAT tests in that package, which also verifies the encoding of the private key. So I think there must be multiple variants around: maybe changes were made during the final round or after they won. I guess you are referring to their github with a reference impl? Whatever the case, it all seems to be boil down to the size of "tr". It would be preferable to have the change validated via an official "PDF" like their round 3 submission.
I'm on vacation but when returning in ~2wks I will look into this and figure out which version is the most correct. If you figure out what is correct feel free to write a pull request with either a clarification or a code change and I can approve it.
Of course once NIST publishes Dilithium in an actual standard this will be solved as well. And maybe the encoding might ultimately be a PKCS#8 style encoding (ASN.1 DER encoded etc.), e.g. same as RSA and ECC keys "getEncoded()" rather than the more 'flat' format in Dilithium.

[eidola7]

OK, I understand that there may be some difference in basement between C and Java

The C source I referenced is: Crystal-Dilithium Home's Public Repository

And, change some code in Dilithium-java ( check difference between C and Java ),
I can successfully perform the following tasks:

1. KeyPair generate with Dilithium-Java ( self Sign/Verify using Dilithium-java work well with this keyPair )
2. Serialize KeyPair ( just use getEncoded() for both privateKey and publicKey )
3. Create Signature with Dilithium-Java
4. load PublicKey with Dilithium-C
5. Verify Signature with Dilithium-C

Code Changed (Java) :

diff --git a/src/main/java/net/thiim/dilithium/impl/Dilithium.java b/src/main/java/net/thiim/dilithium/impl/Dilithium.java
index cde1f55..defe641 100644
--- a/src/main/java/net/thiim/dilithium/impl/Dilithium.java
+++ b/src/main/java/net/thiim/dilithium/impl/Dilithium.java
@@ -17,7 +17,7 @@ public class Dilithium {
        public final static int POLYT0_PACKEDBYTES = 416;
        public final static int POLYT1_PACKEDBYTES = 320;
        public final static int SEEDBYTES = 32;
-       public final static int CRHBYTES = 48;
+       public final static int CRHBYTES = 64;
        public final static int SHAKE128_RATE = 168;
        public final static int SHAKE256_RATE = 136;
        public final static int STREAM128_BLOCKBYTES = Dilithium.SHAKE128_RATE;
@@ -52,7 +52,7 @@ public class Dilithium {
        public static KeyPair generateKeyPair(DilithiumParameterSpec spec, byte[] seed) {
                byte[] zeta = seed;
 
-               byte[] o = Utils.getSHAKE256Digest(3*32, zeta);
+               byte[] o = Utils.getSHAKE256Digest(2*SEEDBYTES+CRHBYTES, zeta);
                byte[] rho = new byte[32];
                byte[] sigma = new byte[32];
                byte[] K = new byte[32];
@@ -78,7 +78,7 @@ public class Dilithium {
                PolyVec[] res = t1.powerRound();
                byte[] pubbytes = PackingUtils.packPubKey(rho, res[1]);
 
-               byte[] tr = Utils.crh(pubbytes);
+               byte[] tr = Utils.getSHAKE256Digest(SEEDBYTES, pubbytes);
 
                byte[] prvbytes = PackingUtils.packPrvKey(spec.eta, rho, tr, K, res[0], s1, s2);
 
@@ -228,7 +228,7 @@ public class Dilithium {
                        throw new RuntimeException("Bad signature");
                }
 
-               byte[] mu = Utils.crh(pk.getEncoded());
+               byte[] mu = Utils.getSHAKE256Digest(SEEDBYTES, pk.getEncoded());
                mu = Utils.getSHAKE256Digest(CRHBYTES,  mu, M);
 
                Poly cp = generateChallenge(spec.tau, c);
diff --git a/src/main/java/net/thiim/dilithium/impl/PackingUtils.java b/src/main/java/net/thiim/dilithium/impl/PackingUtils.java
index 63dfca4..591ab71 100644
--- a/src/main/java/net/thiim/dilithium/impl/PackingUtils.java
+++ b/src/main/java/net/thiim/dilithium/impl/PackingUtils.java
@@ -97,7 +97,7 @@ public class PackingUtils {
                        throw new RuntimeException("Illegal eta");
                }
 
-               final int CRYPTO_SECRETKEYBYTES = (2 * Dilithium.SEEDBYTES + Dilithium.CRHBYTES + s1.length() * POLYETA_PACKEDBYTES
+               final int CRYPTO_SECRETKEYBYTES = (3 * Dilithium.SEEDBYTES + s1.length() * POLYETA_PACKEDBYTES
                                + s2.length() * POLYETA_PACKEDBYTES + s2.length() * Dilithium.POLYT0_PACKEDBYTES);
                byte[] buf = new byte[CRYPTO_SECRETKEYBYTES];
 
@@ -109,9 +109,9 @@ public class PackingUtils {
                        buf[off + i] = K[i];
                off += Dilithium.SEEDBYTES;
 
-               for (int i = 0; i < Dilithium.CRHBYTES; i++)
+               for (int i = 0; i < Dilithium.SEEDBYTES; i++)
                        buf[off + i] = tr[i];
-               off += Dilithium.CRHBYTES;
+               off += Dilithium.SEEDBYTES;
 
                for (int i = 0; i < s1.length(); i++) {
                        s1.poly[i].etapack(eta, buf, off);
@@ -261,12 +261,12 @@ public class PackingUtils {
                }
                off += Dilithium.SEEDBYTES;
 
-               byte[] tr = new byte[Dilithium.CRHBYTES];
-               for(int i = 0; i < Dilithium.CRHBYTES; i++) {
+               byte[] tr = new byte[Dilithium.SEEDBYTES];
+               for(int i = 0; i < Dilithium.SEEDBYTES; i++) {
                        tr[i] = bytes[off+i];
                }
-               off += Dilithium.CRHBYTES;
-
+               off += Dilithium.SEEDBYTES;
+
                PolyVec s1 = new PolyVec(parameterSpec.l);
                for(int i=0; i < parameterSpec.l; i++) {
                          s1.poly[i] = etaunpack(parameterSpec.eta, bytes, off);

No Code Change in C .

Anyway, Your work is very helpful. Many thanks.

[mthiim]

Hi, I've written a bit about it in the README file. The Dilithium team made changes to their implementation, that version is now referred to as 3.1 (where my implementation was compatible with 3.0). I've now updated my implementation to that as well. There's more changes on the way in the FIPS / ML-DSA standardization process. You can see an overview of the differences between all versions here: https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.ipd.pdf. I've not implemented the ML-DSA changes yet, but will do so later :-) I've verified that with the committed changes the code gives the same results on all KAT tests as the 3.1 reference implementation.