Race condition between keygen and update, resulting in "Key derivation key not available!" about tang

[cbiedl]

Strike that, I missed the fact the dependencies have been reworked in commit 7778512
So I'll have to re-check why this still happens.

[sergio-correia]

Strike that, I missed the fact the dependencies have been reworked in commit 7778512
So I'll have to re-check why this still happens.

Did you have that commit in your build? It's not part of any release, so it's likely you don't have it applied there.

[cbiedl]

Yeah, I realized too late this bug report was based on the latest release. However, it still happens if I include that commit, so it's certainly still an issue. At the moment however I'm stuck figuring what triggers tangd-update invocation now. It does happen, but why?

[sergio-correia]

Could you try #53 and report back, please?

[cbiedl]

Will do in a moment, just want to share the analysis I did:

Starting with @jwkdir@ and @cachedir@ empty.

Requesting an advertisement using

echo foo | clevis encrypt tang '{"url": "http://<tang-server>/"}' > bar.txt

which results in an error:

Unable to fetch advertisement: 'http://<tang-server>/adv/'!

This is the output of "journalctl -o short-precise", with timestamps of the created files merged, ordered by time.

Nov 28 11:38:25.272465 freekeh systemd[1]: Starting Tang Server key generation script...

-rw-r--r-- 1 root root 354 2020-11-28 11:38:25.401147740 +0100 @dbdir@/OCWpHA3_4-E9mUpyW_0N3-Crcoc.jwk

Nov 28 11:38:25.457995 freekeh systemd[1]: Starting Tang Server key update script...

So update is started ...

-rw-r--r-- 1 root root 349 2020-11-28 11:38:25.531138284 +0100 @dbdir@/tPfTOakfqz0qMjjloGn6v989ttc.jwk

... before keygen finished the job. Also, update will ignore the second .jwk (not shown here)

Nov 28 11:38:25.557283 freekeh systemd[1]: tangd-keygen.service: Succeeded.
Nov 28 11:38:25.575785 freekeh systemd[1]: Started Tang Server key generation script.
Nov 28 11:38:25.597302 freekeh systemd[1]: Started Tang Server (<client>:54812).

And now the server is started although update is still running.

Nov 28 11:38:25.630481 freekeh tangd[2709]: <client> GET /adv/ => 404 (src/tangd.c:70)

... hence the 404

Nov 28 11:38:25.639115 freekeh systemd[1]: tangd@1-<server>:80-<client>:54812.service: Succeeded.

-rw-r--r-- 1 root root 618 2020-11-28 11:38:25.761121554 +0100 @cachedir@/default.jws

The update script needed 180ms to parse the .jwk and create the first .jws. This is the earliest moment where the server may be run. But to be safe, this should rather happen after update finished the job which took ...

(...)
lrwxrwxrwx 1 root root  44 2020-11-28 11:38:28.280938310 +0100 @cachedir@/MQHhVWoc4arWBQYK1wHf69l4dnXsX7fC2nZ7JAem1-3zyU53bUBjcOqtMCaB8WFRrvXX6g9pSz2AcgWRpWMAbw.jwk -> @dbdir@/tPfTOakfqz0qMjjloGn6v989ttc.jwk

... another 2520ms.

Nov 28 11:38:28.558664 freekeh systemd[1]: tangd-update.service: Succeeded.
Nov 28 11:38:28.561176 freekeh systemd[1]: Started Tang Server key update script.

So far I'm not convinced using the systemd semantics to resolve dependencies are the best idea.

[cbiedl]

Sergio Correia wrote...

Could you try #53 and report back, please?

Doesn't look good on armhf: ``` 1/3 adv TIMEOUT 90.13s

…

--- command --- 22:10:54 SD_ACTIVATE='/usr/bin/systemd-socket-activate --inetd' PATH='/<<PKGBUILDDIR>>/src:/<<PKGBUILDDIR>>/obj-arm-linux-gnueabihf/src: /usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games' /<<PKGBUILDDIR>>/tests/adv --- stdout --- {"payload": "eyJrZXlzIjogW3siYWxnIjogIkVTNTEyIiwgImNydiI6ICJQLTUyMSIsICJrZXlfb3BzIjogWyJ2ZXJpZnkiXSwgImt0eSI6ICJFQyIsICJ4IjogIkFVLWZZc19 HZHd5Q2g0N0tDOEU5U0tkWERxMU9nYU9CcVNiQ1JPMlNYN1JvMnBxWVoyZ0llZzBiRU9COFowTzk3TlJjcjFpLS12dHljTE9TQTNQaFBtY3AiLCAieSI6ICJBQVdjRXVsRlkxTGN TZzZJUk9xQ25DMkZIWW9tTEVrd2hkU3BhTDlXZTJvUi1RY1l4a1RobFcxb01NX2loR1lVc3Z6cnhXeVdWd2Ewa191LWpSYmdNa2p3In0sIHsiYWxnIjogIkVDTVIiLCAiY3J2Ijo gIlAtNTIxIiwgImtleV9vcHMiOiBbImRlcml2ZUtleSJdLCAia3R5IjogIkVDIiwgIngiOiAiQUV1Mjd5dm9KUjNxN2xHaDY1VFBNZWxzMy1Rc3hxTmJRajA3SzQxMkJlVjlxRzF uODhYX1hhSV9wQTMxTU1Ua2prV2pSbW8yMnhOX3YwRi1jVjBvZjk3YSIsICJ5IjogIkFjMkNvRnQ2X19uR1pXam9GamFwdWYyOEdpY29HaTJWaDVOc3JsYjBkQ1BMOW12VlFOMVF jaWNueUNhd2hGUW9KcGxXUmtVYWstSkZpczdLaVlaRWVENnkifV19", "protected": "eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9", "signature": "A FmazBHspqbzUUIrSbRA5NiIB1ePk1K69bFcuCbMZ0QHfeGv7GlsWrd6Y78I464pPYuYGTL6StQ_ojTrvZcbR3hGAAf18u7KesoT0efZhbgYqd0UU3ZqmYyLZrbztjPT2Pfua2xDV PPF-L_rpYxoZXnu9FPyW69a4W1HmKcwaJxzGV_3"} --- stderr --- + trap on_exit EXIT + trap exit ERR ++ mktemp -d + export TMP=/tmp/tmp.gwHry8iGOS + TMP=/tmp/tmp.gwHry8iGOS + mkdir -p /tmp/tmp.gwHry8iGOS/db + tangd-keygen /tmp/tmp.gwHry8iGOS/db sig exc + jose jwk gen -i '{"alg": "ES512"}' -o /tmp/tmp.gwHry8iGOS/db/.sig.jwk + jose jwk gen -i '{"alg": "ES512"}' -o /tmp/tmp.gwHry8iGOS/db/.oth.jwk ++ shuf -i 1024-65536 -n 1 + export PORT=50243 + PORT=50243 + export PID=301670 + PID=301670 + sleep 0.5 + /usr/bin/systemd-socket-activate --inetd -l 127.0.0.1:50243 -a tangd /tmp/tmp.gwHry8iGOS/db Listening on 127.0.0.1:50243 as 3. + fetch / + curl -sfg http://127.0.0.1:50243/ Communication attempt on fd 3. Connection from 127.0.0.1:57526 to 127.0.0.1:50243 Spawned tangd (tangd /tmp/tmp.gwHry8iGOS/db) as PID 301678. Execing tangd (tangd /tmp/tmp.gwHry8iGOS/db) <unknown> GET / => 404 (../src/http.c:128) Child 301678 died with code 0 ``` Will check further.

[sergio-correia]

@cbiedl: can close this, now that #53 has been merged?

[cbiedl]

ACK, none of this still applies.