`rauc install` hangs about rauc

[jktjkt]

I am no longer hitting this problem. I suspect that https://patchwork.ozlabs.org/patch/876150/ in Buildroot might have fixed that, but it could have been some random update as well. Sorry for not letting you know earlier -- I managed to forgot about this one and my local patches are long lost by now.

[ejoerns]

Hi, thanks for the report!

Seems like

Feb 08 13:38:34 czechlight rauc[394]: Invalid byte sequence in conversion input

is the bug and

Feb 08 13:38:34 czechlight rauc[394]: g_error_new_literal: assertion 'message != NULL' failed
Feb 08 13:38:34 czechlight rauc[394]: g_task_return_error: assertion 'error != NULL' failed

is a follow-up issue in error handling.

I am not sure what could have went wrong yet, but at least I noted you have a misspelled config option in your system.conf. Inspired by that and because it also hit me some times, I opened #225.

[ejoerns]

Mh... from the error messages I would assume it fails in the g_subprocess_communicate() call of uboot_env_get().

Do you have any possibility to verify this on your setup?

As it says

g_error_new_literal: assertion 'message != NULL' failed

this could be a bug in glib itself. Which version do you use?

[jktjkt]

Thanks for getting back to me so quickly. Yup, it's indeed in g_subprocess_communicate() (verified by some printf-s).

This is on yesterday's Buildroot git master which has libglib2-2.54.2. I've never worked with glib. Do you have any pointers on what I should check? The system that I'm generating is pretty minimal (even though it has bash and systemd), so there's not even a locale shell command. However, I can easily build anything you'd like me to build. HW platform is armv7l, Solidrun Clearfog with Marvell Armada A388 SoC.

[jktjkt]

I was able to get a backtrace. Stuff breaks in g_convert_with_iconv when handling EILSEQ. I see some interesting bits in their master branch for that file...

bt 0: /lib/libglib-2.0.so.0(g_convert_with_iconv+0x228) [0xb6b85bb8]
bt 1: /lib/libglib-2.0.so.0(g_convert+0x108) [0xb6b85df4]
bt 2: /lib/libglib-2.0.so.0(g_locale_to_utf8+0x90) [0xb6b863bc]
bt 3: /lib/libglib-2.0.so.0(g_strerror+0xa4) [0xb6bbe314]
bt 4: /lib/libglib-2.0.so.0(g_key_file_load_from_file+0xe8) [0xb6b9b108]
bt 5: rauc() [0x2bd04] // rauc-0.3/src/config_file.c:407
bt 6: rauc() [0x2be64] // rauc-0.3/src/config_file.c:472
bt 7: rauc() [0x20100] // rauc-0.3/src/install.c:875
bt 8: rauc() [0x21248] // rauc-0.3/src/install.c:1443
bt 9: /lib/libglib-2.0.so.0(+0x68818) [0xb6bc6818]

As I said earlier, I have no clue how glib works. Is there a test that I can run to check if my system image provides the necessary environment?

[jktjkt]

Hmm, that stuff above was a different failure (weird -- how come that I was getting that, and why am I not getting it anymore?).

Anyway, after several rebuild and extra calls to g_warning, here's what I get:

bt 0: /lib/libglib-2.0.so.0(g_convert_with_iconv+0x228) [0xb6b68bb8]
bt 1: /lib/libglib-2.0.so.0(g_convert+0x108) [0xb6b68df4]
bt 2: /lib/libglib-2.0.so.0(g_locale_to_utf8+0x90) [0xb6b693bc]
bt 3: /lib/libglib-2.0.so.0(g_strerror+0xa4) [0xb6ba1314]
bt 4: /lib/libgio-2.0.so.0(+0x6b2d0) [0xb6e052d0]
bt 5: /lib/libgio-2.0.so.0(+0x595f0) [0xb6df35f0]
bt 6: /lib/libgio-2.0.so.0(+0x5b698) [0xb6df5698]
bt 7: /lib/libgio-2.0.so.0(g_input_stream_read_async+0x214) [0xb6df4664]
bt 8: /lib/libgio-2.0.so.0(g_output_stream_splice_async+0x200) [0xb6e03250]
bt 9: /lib/libgio-2.0.so.0(+0x81ec8) [0xb6e1bec8]
bt 10: /lib/libgio-2.0.so.0(g_subprocess_communicate+0x1b4) [0xb6e1dd30]
bt 11: rauc() [0x1b610] // src/bootchooser.c:472
bt 12: rauc() [0x20324] // src/install.c:969
bt 13: rauc() [0x21128] // src/install.c:1443
bt 14: /lib/libglib-2.0.so.0(+0x68818) [0xb6ba9818]

[ejoerns]

Well, I dove a bit into the corresponding glib code.

What basically seems to happen here is that some kind of error occurs, for example in g_input_stream_read_async() or g_key_file_load_from_file() as in the two examples above.

The error then triggers a call to g_strerror() to output an error message with a string representation of an error code. Unfortunately, g_strerror() does not return a string representation but simply NULL in your case (despite the documentation does not define this to be a valid return value of g_strerror()).

This is what triggers the error message we see:

g_error_new_literal: assertion 'message != NULL' failed

Thus, the remaining question is why g_strerror() returns NULL here.

It seems to be this code:

  if (!g_get_charset (NULL))
    {
      msg = g_locale_to_utf8 (msg, -1, NULL, NULL, &error);
      if (error)
        g_print ("%s\n", error->message);
    }

If g_locale_to_utf8() fails (the logs strongly indicate it to do so), g_locale_to_utf8() returns NULL. As a consequence, the error is printed out, what we actually see in your log:

Feb 08 13:38:34 czechlight rauc[394]: Invalid byte sequence in conversion input

Finally, the function returns msg which is NULL at this moment...

Now, the reasons that triggers this seems to be the missing character set for the current locale on your system. BUT, nevertheless, from my point of view the handling in g_strerror() I described above is wrong!

Haven't found a definitive fix for this in recent glib, yet.

[jktjkt]

FYI, the target system did not have any gconv runtime files, so it makes sense that these operations fail. I wasn't able to persuade buildroot to install them without a full rebuild -- I'll report back once I get around to rebuilding everything.

[ejoerns]

Ok, thanks for feedback. Meanwhile I submitted a Bug to glib to discuss what we've found

https://bugzilla.gnome.org/show_bug.cgi?id=793377

[jktjkt]

Hmmm, so it doesn't appear to be due to missing gconv runtime libraries. Configuring Buildroot with BR2_TOOLCHAIN_GLIBC_GCONV_LIBS_COPY=y did not change the situation :(.

[ejoerns]

I assume this to be unresolved and thus reopen it, maybe you just picked the wrong button?

Do you have the ability to test with a patched glib in your setup?

What if you apply this patch?

diff --git a/glib/gstrfuncs.c b/glib/gstrfuncs.c
index e7240b6..7cea77b 100644
--- a/glib/gstrfuncs.c
+++ b/glib/gstrfuncs.c
@@ -1300,8 +1300,10 @@ g_strerror (gint errnum)
       if (!g_get_charset (NULL))
         {
           msg = g_locale_to_utf8 (msg, -1, NULL, NULL, &error);
-          if (error)
+          if (error) {
             g_print ("%s\n", error->message);
+            msg = g_strdup("(null)");
+          }
         }
       else if (msg == (const gchar *)buf)
         msg = g_strdup (buf);

[jktjkt]

maybe you just picked the wrong button?

Gah, sorry, long day.

What if you apply this patch?

Yup, thanks, it works! What puzzles me is that there's no (null) anywhere in the rauc's log. Almost as if the return value was not used anywhere.

-- Logs begin at Tue 2018-02-13 12:54:50 UTC. --
Feb 13 12:54:53 czechlight systemd[1]: Starting Rauc Update Service...
Feb 13 12:54:53 czechlight systemd[1]: Started Rauc Update Service.
Feb 13 12:54:53 czechlight rauc[211]: rauc mark: marked slot rootfs.0 as good
Feb 13 12:56:07 czechlight rauc[211]: input bundle: /tmp/update.raucb
Feb 13 12:56:07 czechlight rauc[211]: Active slot bootname: A
Feb 13 12:56:07 czechlight rauc[211]: installing /tmp/update.raucb: started
Feb 13 12:56:07 czechlight rauc[211]: installing /tmp/update.raucb: Checking and mounting bundle...
Feb 13 12:56:07 czechlight rauc[211]: Reading bundle: /tmp/update.raucb
Feb 13 12:56:07 czechlight rauc[211]: Verifying bundle... 
Feb 13 12:56:08 czechlight rauc[211]: Mounting bundle '/tmp/update.raucb' to '/run/rauc/bundle'
Feb 13 12:56:08 czechlight rauc[211]: Target Group:
Feb 13 12:56:08 czechlight rauc[211]:   cfg -> cfg.1
Feb 13 12:56:08 czechlight rauc[211]:   rootfs -> rootfs.1
Feb 13 12:56:08 czechlight rauc[211]: Using default handler
Feb 13 12:56:08 czechlight rauc[211]: Marking target slot as non-bootable...
Feb 13 12:56:08 czechlight rauc[211]: installing /tmp/update.raucb: Updating slots...
Feb 13 12:56:08 czechlight rauc[211]: Checking image type for slot type: ext4
Feb 13 12:56:08 czechlight rauc[211]: Image detected as type: *.tar*
Feb 13 12:56:08 czechlight rauc[211]: installing /tmp/update.raucb: Checking slot rootfs.1
Feb 13 12:56:08 czechlight rauc[211]: mounting slot /dev/mmcblk0p3
Feb 13 12:56:08 czechlight rauc[211]: Slot needs to be updated with /run/rauc/bundle/rootfs.tar.xz
Feb 13 12:56:08 czechlight rauc[211]: Updating /dev/mmcblk0p3 with /run/rauc/bundle/rootfs.tar.xz
Feb 13 12:56:08 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot rootfs.1
Feb 13 12:56:08 czechlight rauc[211]: Formatting ext4 slot /dev/mmcblk0p3
Feb 13 12:56:08 czechlight rauc[211]: mke2fs 1.43.8 (1-Jan-2018)
Feb 13 12:56:10 czechlight rauc[211]: [82B blob data]
Feb 13 12:56:10 czechlight rauc[211]: Creating filesystem with 476928 4k blocks and 119280 inodes
Feb 13 12:56:10 czechlight rauc[211]: Filesystem UUID: 78b0b409-553a-45ba-9382-aba3f4d25016
Feb 13 12:56:10 czechlight rauc[211]: Superblock backups stored on blocks:
Feb 13 12:56:10 czechlight rauc[211]:         32768, 98304, 163840, 229376, 294912
Feb 13 12:56:10 czechlight rauc[211]: [49B blob data]
Feb 13 12:56:10 czechlight rauc[211]: [46B blob data]
Feb 13 12:56:10 czechlight rauc[211]: Creating journal (8192 blocks): done
Feb 13 12:56:12 czechlight rauc[211]: [83B blob data]
Feb 13 12:56:12 czechlight rauc[211]: Mounting ext4 slot /dev/mmcblk0p3
Feb 13 12:56:12 czechlight rauc[211]: Extracting /run/rauc/bundle/rootfs.tar.xz to /run/rauc/rootfs.1
Feb 13 12:56:16 czechlight rauc[211]: Unmounting ext4 slot /dev/mmcblk0p3
Feb 13 12:56:20 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot rootfs.1 status
Feb 13 12:56:20 czechlight rauc[211]: Updating slot file /run/rauc/rootfs.1/slot.raucs
Feb 13 12:56:20 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot rootfs.1 done
Feb 13 12:56:20 czechlight rauc[211]: Checking image type for slot type: ext4
Feb 13 12:56:20 czechlight rauc[211]: Image detected as type: *.tar*
Feb 13 12:56:20 czechlight rauc[211]: installing /tmp/update.raucb: Checking slot cfg.1
Feb 13 12:56:20 czechlight rauc[211]: mounting slot /dev/mmcblk0p4
Feb 13 12:56:20 czechlight rauc[211]: Slot needs to be updated with /run/rauc/bundle/cfg.tar.xz
Feb 13 12:56:20 czechlight rauc[211]: Updating /dev/mmcblk0p4 with /run/rauc/bundle/cfg.tar.xz
Feb 13 12:56:20 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot cfg.1
Feb 13 12:56:20 czechlight rauc[211]: Formatting ext4 slot /dev/mmcblk0p4
Feb 13 12:56:20 czechlight rauc[211]: mke2fs 1.43.8 (1-Jan-2018)
Feb 13 12:56:22 czechlight rauc[211]: [82B blob data]
Feb 13 12:56:22 czechlight rauc[211]: Creating filesystem with 477184 4k blocks and 119520 inodes
Feb 13 12:56:22 czechlight rauc[211]: Filesystem UUID: d1521a58-5071-4ea1-af25-8a5acb6daa7b
Feb 13 12:56:22 czechlight rauc[211]: Superblock backups stored on blocks:
Feb 13 12:56:22 czechlight rauc[211]:         32768, 98304, 163840, 229376, 294912
Feb 13 12:56:22 czechlight rauc[211]: [49B blob data]
Feb 13 12:56:22 czechlight rauc[211]: [46B blob data]
Feb 13 12:56:22 czechlight rauc[211]: Creating journal (8192 blocks): done
Feb 13 12:56:24 czechlight rauc[211]: [83B blob data]
Feb 13 12:56:24 czechlight rauc[211]: Mounting ext4 slot /dev/mmcblk0p4
Feb 13 12:56:24 czechlight rauc[211]: Extracting /run/rauc/bundle/cfg.tar.xz to /run/rauc/cfg.1
Feb 13 12:56:24 czechlight rauc[211]: Running slot hook slot-post-install for cfg.1
Feb 13 12:56:24 czechlight rauc[211]: Unmounting ext4 slot /dev/mmcblk0p4
Feb 13 12:56:24 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot cfg.1 status
Feb 13 12:56:24 czechlight rauc[211]: Updating slot file /run/rauc/cfg.1/slot.raucs
Feb 13 12:56:24 czechlight rauc[211]: Marking slots as bootable...
Feb 13 12:56:24 czechlight rauc[211]: installing /tmp/update.raucb: Updating slot cfg.1 done
Feb 13 12:56:24 czechlight rauc[211]: Invalid byte sequence in conversion input
Feb 13 12:56:24 czechlight rauc[211]: installing /tmp/update.raucb: All slots updated

[ejoerns]

Indeed a little strange. My assumption is that the error we see is an 'expected' glib internal one that is handled by upper layers or somehow got lost during asynchronous execution in glib...

[ejoerns]

@jktjkt Do you think this error is still relevant or did you have more recent findings on what went wrong?