fedora-iot / iot-distro Goto Github PK

Outcomes

• frequency of bootc base image layer composes is determined
• bootc base image layer is composed automatically
• documentation updates provided to Fedora docs on how to find/use base image layer
• define what is included in the Fedora base image and where the configs should reside.

Depends on #17

Describe the bug

Since moving to osbuild for our installer, we no longer produce the network installer bits needed for things like PXE boot or HTTP boot.

Example in Fedora 39:
https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-39-20240118.0/compose/IoT/x86_64/os/

Now in Fedora 40:
https://kojipkgs.fedoraproject.org/compose/iot/Fedora-IoT-40-20240118.0/compose/IoT/x86_64/

Included on the osbuild produced installer iso:

EFI
└── BOOT
    ├── BOOTAA64.EFI
    ├── fonts
    │   └── unicode.pf2
    ├── grubaa64.efi
    ├── grub.cfg
    └── mmaa64.efi
images
├── efiboot.img
├── install.img
└── pxeboot
    ├── initrd.img
    └── vmlinuz

Missing from F40 is the boot.iso. F39 boot.iso.

Due to issues with Anaconda, we may need to revert bootupd in Fedora 40 IoT.

Ongoing discussion with more details:
rhinstaller/anaconda#5508
https://bugzilla.redhat.com/show_bug.cgi?id=2268505
ostreedev/ostree#3150 (comment)

We'll need to revert :
https://pagure.io/fedora-iot/ostree/c/58d264b7669547e07a9ed90c9255885c28f297fc?branch=f40

And check on impacts to simplified-provisioning and bootc deliverables.

Based on #24 we need to provide the network installer artifacts for things like PXE boot or HTTP boot: Note the missing os directory in Fedora 40 vs Fedora 39.

Goal

• Create new image type for the Anaconda Network Installer to provide the missing artifacts.
• The new image type should generate, if possible, a size optimized installer image that might require a new stage in osbuild to allow the removal of files within the resulting tree (Please see: osbuild/images#409) .

Acceptance Criteria

• It's possible to generate the missing artifacts with the new osbuild image type.
• The documentation is updated accordingly.
• The new artifact have its corresponding CI tests.

Outcomes

• using the Fedora bootc base image, additional required packages are determined for Fedora IoT
• using the Fedora bootc base image, additional required configuration is determined for Fedora IoT
• required changes to Fedora pungi config are made to support above items
• Fedora IoT bootc image is composed regularly
• Fedora IoT bootc image is pushed to a publicly available registry regularly
• gating tests are enabled to prevent a broken Fedora IoT bootc image from being pushed to the registry
• documentation updates are made to the Fedora IoT docs describing how to find and use the Fedora IoT bootc image

Depends on #18

Describe the bug
When attempting to boot after rebasing to version 40/41, no volumes can be found by dracut and the system fails to boot.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Install Fedora IoT 39 (In this case to a GHF51 SBC)
2. Rebase to Fedora IoT 40 or 41
3. Restart

Expected behavior
Fedora IoT boots successfully.

Screenshots

OS version:

  fedora-iot:fedora/rawhide/x86_64/iot
                  Version: 41.20240315.0 (2024-03-15T16:10:39Z)
                   Commit: ae3236721a484ff13d39ce5cad7f9e934b307e3016848747f7596031e5c3cba5
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
                     Diff: 348 upgraded, 10 removed, 13 added

● fedora-iot:fedora/stable/x86_64/iot
                  Version: 39.20231103.0 (2023-11-03T15:32:03Z)
                   Commit: 257e2c9d3fe02d977859f01a4e8187181e7cf24d4d755fd89a06beb7c78bec61
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464

Describe the bug
After upgrading to F39, the parsec.service fails to start

To Reproduce

1. Staring from a previous version of Fedora IoT, rebase to the F39 release (i.e. rpm-ostree rebase fedora-iot:fedora/devel/x86_64/iot)
2. Reboot and observe the journal

Expected behavior
parsec.service starts successfully

OS version:

$ rpm-ostree status -b
State: idle
BootedDeployment:
● fedora-iot:fedora/devel/x86_64/iot
                  Version: 39.20231026.0 (2023-10-26T12:27:40Z)
                   Commit: 0599c27fe88ed2aaeb8144c7b604aaa69e31a94cbc384c894e29b27a077bdb6a
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

Additional context

This is a carry-over from the F37 and F38 release, see https://bugzilla.redhat.com/show_bug.cgi?id=2170957

Failed service:

$ sudo journalctl -b -u parsec.service
[sudo] password for core: 
Oct 27 09:21:13 localhost systemd[1]: Starting parsec.service - Parsec Service...
Oct 27 09:21:13 localhost parsec[773]: Error: Permission denied (os error 13)
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:13 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 1.
Oct 27 09:21:13 localhost systemd[1]: Starting parsec.service - Parsec Service...
Oct 27 09:21:13 localhost parsec[798]: Error: Permission denied (os error 13)
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:13 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 2.
Oct 27 09:21:13 localhost systemd[1]: Starting parsec.service - Parsec Service...
Oct 27 09:21:13 localhost parsec[813]: Error: Permission denied (os error 13)
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 09:21:13 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:13 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 3.
Oct 27 09:21:14 localhost systemd[1]: Starting parsec.service - Parsec Service...
Oct 27 09:21:14 localhost parsec[815]: Error: Permission denied (os error 13)
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:14 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 4.
Oct 27 09:21:14 localhost systemd[1]: Starting parsec.service - Parsec Service...
Oct 27 09:21:14 localhost parsec[817]: Error: Permission denied (os error 13)
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:14 localhost systemd[1]: Failed to start parsec.service - Parsec Service.
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Scheduled restart job, restart counter is at 5.
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Start request repeated too quickly.
Oct 27 09:21:14 localhost systemd[1]: parsec.service: Failed with result 'exit-code'.
Oct 27 09:21:14 localhost systemd[1]: Failed to start parsec.service - Parsec Service.

The IoT project should start evaluating changes to Fedora that will impact IoT, similar to how the CoreOS folks do it.

It looks like we could basically use a similar fork of the pgm_scripts repo that @dustymabe is using to generate the tracker issue.

I reached out to Dusty for some operational knowledge on using the scripts; we'll meet sometime next week to discuss.

(Migrated from coreos/fedora-coreos-tracker#1615 -> https://pagure.io/fedora-iot/issue/53 -> to here)

Oringally reported by @gabrieleturchi

Describe the bug

Apparently any symlink created or removed under /etc/systemd/system (like "systemctl disable ModemManager" or "systemctl set-default graphical.target" - I haven't checked under other /etc folders) is not kept for the next reboot.
Creating a file (like a new service) works. I'm pretty sure that worked well under Fedora IoT 38.

My current workaround in fact is to make a service who force the right target and usind predefined services configuration to enable my new services.

Of course I installed xfce before to run “systemctl set-default graphical.target”.

GT

Reproduction steps

1. Install graphical environment
2. systemctl set-default graphical.target

Expected behavior

Graphical environment running at boot

Actual behavior

multi-user text environment running at boot

System details

Raspberry Pi 3

Describe the bug

boot loader install failed while trying to install Fedora-IoT-ostree-x86_64-39-20231026.0.iso using GNOME Boxes.

To Reproduce

1. Download Fedora-IoT-ostree-x86_64-39-20231026.0.iso
2. Select the iso in Boxes
3. Add user, select disk (default), and after the installation starts the error will show up.

Expected behavior
Fedora is installed
Screenshots

OS version:

Not applicable.

Additional context
Add any other context about the problem here.

Osbuild created Installer images do not include fedora-release-common license files in the root of the iso. We should include Fedora-Legal-README.txt and LICENSE files for Fedora.

This should likely be included for all ISOs and all distro specific license files.

qemu-system-x86_64 \
-machine q35 \
-M accel=hvf \
-m 4096 \
-cpu Broadwell \
-drive file=image.raw,format=raw,index=0,media=disk,if=virtio \
-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd \
-device virtio-net-pci,netdev=n0,mac=FE:30:2E:2B:71:95 \
-netdev user,id=n0,net=10.0.2.0/24,hostfwd=tcp::8080-:8080,hostfwd=tcp::8082-:8082,hostfwd=tcp::2223-:22 \
-rtc base=localtime

worked for Rhel for edge but not with fedora IoT

Phase 1 - Container image

• #16
• #17
• #18
• #19

Phase 2 - Disk images

• Create disk images with osbuild to produce disk images (Anaconda, installer etc)

Describe the bug

Using disk re-encryption FDO features with Fedora 38/39 gets a selinux denial. We cannot use it.

To Reproduce

1. Generate a F38/39 simplified installer with fdo options, sample blueprint:

name = "fedora-si-fdo"
description = ""
version = "0.0.1"
packages = []
modules = []
groups = []
distro = ""

[customizations]
installation_device = "/dev/vda"

[[customizations.user]]
name = "admin"
password = "$6$vBo.9c8SeguWtjmu$8cj9HGn6nX6rPQvWh.pbdqaD.8FvLuIEToMOh9vHIQjjM.7PGZFWHYGxEO1dxuQ7ajjzzyuLI4EH.W6/ndXrV0"
groups = ["wheel"]
[customizations.fdo]
manufacturing_server_url = "http://192.168.122.180:8080"
diun_pub_key_insecure = "true"

2. Run the FDO infrastructure: fdo-admin-tool aio --directory=./aio run, the serviceinfo-api-server must have some diskencryption_clevis config, such as:

  diskencryption_clevis:
    - disk_label: /dev/vda3
      reencrypt: true
      binding:
        pin: tpm2
        config: '{}'

Expected behavior

I expect the disk to be re-encrypted.

Screenshots
If applicable, add screenshots to help explain your problem.

OS version:

Fedora 38/39

bash-5.2# rpm-ostree status -b
State: idle
BootedDeployment:
● fedora-iot:fedora/38/x86_64/iot
                  Version: 38 (2023-10-19T07:54:31Z)
                   Commit: e190257652791f6214518765f5e6ccee4969d67f9f5004adfb7401c101a291b1

Additional context
These are the logs:

ct 19 15:30:21 localhost.localdomain systemd[1]: Starting fdo-client-linuxapp.service - FDO client...
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.300Z INFO  fdo_client_linuxapp > Found device credential at FileSystemPath { path: "/boot/device-credentials", deactivation_method: None }
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.443Z INFO  fdo_client_linuxapp > Got TO2 addresses: ["http://192.168.122.180:8081", "http://fe80::97e2:1716:6aa8:88ba:8081"]
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.443Z INFO  fdo_client_linuxapp > Performing TO2 protocol, URL: "http://192.168.122.180:8081"
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.564Z INFO  fdo_client_linuxapp::serviceinfo > Initiating disk re-encryption, disk-label: /dev/vda3, pin: tpm2, config: {}, reencrypt: true
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain audit[1228]: AVC avc:  denied  { search } for  pid=1228 comm="pwmake" name="cracklib" dev="dm-1" ino=37010 scontext=system_u:system_r:fdo_t:s0 tcontext=system_u:object_r:crack_db_t:s0 tclass=dir permissive=0
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.777Z ERROR fdo_client_linuxapp              > ServiceInfo failed, error: Error processing returned serviceinfo
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]: Caused by:
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     0: Error executing clevis
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     1: Error executing disk encryption for disk label /dev/vda3
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     2: Error rebinding clevis
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     3: Error binding clevis
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     4: Failed to bind clevis: ExitStatus(unix_wait_status(256)), stdout: , stderr:
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        /usr/share/cracklib/pw_dict.pwd.gz: Permission denied
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        Error: Password generation failed - required entropy too low for settings
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        Unable to generate a new key
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        Error adding new binding to /dev/vda3
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:        
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:  2023-10-19T15:30:21.779Z ERROR fdo_client_linuxapp              > Error performing TO2 ownership protocol
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]: Caused by:
Oct 19 15:30:21 localhost.localdomain fdo-client-linuxapp[1116]:     Error performing the ServiceInfo roundtrips with TO2 address http://192.168.122.180:8081

The OpenQA test for Clevis fails on aarch64, investigate why it is currently failing (looks like a display issue) and add a PR to the fix the issue in OpenQA.

Please try to answer the following questions about the package you are requesting:

1. Is the package installed by default in a Fedora Edition? If yes, which one.
   Silverblue, possibly others.

2. What, if any, are the additional dependencies on the package? What is the output of this command on a system without overrides or locally installed packages:

rpm-ostree install --dry-run <package>

Installing 2 packages:
  passt-0^20240220.g1e6f92b-1.fc39.aarch64 (updates)
  passt-selinux-0^20240220.g1e6f92b-1.fc39.noarch (updates)

3. What is the size of the package and its dependencies?

rpm -qi <package>

passt: 817939
passt-selinux: 197986

4. What problem are you trying to solve with this package? Or what functionality does the package provide?

pasta is the new default tool for rootless networking in Podman 5 (replacing slirp4netns)

6. Can the software provided by the package be run from a container? Explain why or why not.

It is used to run containers.

8. Can the tool(s) provided by the package be helpful in debugging container runtime issues?

Probably not.

9. Can the tool(s) provided by the package be helpful in debugging networking issues?

Probably not.

10. Is it possible to layer the package locally via rpm-ostree install <package>? Explain why or why not.

It is possible but all basic Podman functionality should be provided out of the box IMHO.

Based on #24 document how to use PXE boot: Note the missing os directory in Fedora 40 vs Fedora 39.

(split from #27 )

Goal

• Document how to use PXE boot

Acceptance Criteria

• Docs are updated
• Add a test case that tests that PXE boot works

With the upcoming release of Fedora 40, we will have a simplified-installer artifact that can leverage Ignition to land some configuration on a Fedora IoT system.

We should have docs that explain how to do this.

As part of Fedora 40, Fedora IoT will have another means of building, installing, configuring a system via bootable containers.

https://fedoraproject.org/wiki/Changes/Fedora_IoT_Bootable_Container

As we roll out this new addition, we need to document how to use it:

• how can we customize the container?
• how can we create disk images from the container?
• how can users update their IoT systems using a container?
• etc

This issue tracker is intended only for IoT specific issues. Please try to reproduce the issue on a relevant Fedora release to determine if the issue is specific to IoT or a general issue in Fedora. If is a general issue in Fedora, please report it in Red Hat Bugzilla (see How to file a bug) or in an upstream project and not in this issue tracker.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. ..
2. ..

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

OS version:

Please replace this line with output of `rpm-ostree status -b`

Additional context
Add any other context about the problem here.

Describe the bug
Cockpit has started to fail with PR_CONNECT_RESET_ERROR when navigating to it in Firefox. When I try to navigate to the HTTP version of the site, it just redirects me to the HTTPS one and gives the same error. HTTP-Only Mode is disabled in Firefox.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Install Cockpit
2. Start it with sudo systemctl start cockpit.service
3. Navigate to http://<SERVER_IP>:9090 in browser

Expected behavior
Cockpit loads

Screenshots

OS version:

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 15h ago
BootedDeployment:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 39.20240105.0 (2024-01-05T14:26:16Z)
               BaseCommit: 5a9c39f47c18ef49633e131da4225229b2f727f89ef4d2fb5a7efdc98e408964
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: borgmatic btrfs-progs cockpit cockpit-pcp cockpit-podman distrobox fish git ncurses smartmontools wireguard-tools

Is your feature request related to a problem? Please describe.

When we update Fedora IoT via rpm-ostree upgrade, the bootloader binaries remains unchanged due the fact that the ostree model does not cover changes to the bootloader. Thus, users need to manually intervene to workaround this limitation and ensure the GRUB + UEFI shim firmware are up-to-date.

Describe the solution you'd like

Fedora IoT should be produced in a way that allows the use of bootupd to update the bootloader successfully.

Describe alternatives you've considered

Current alternative is a manual intervention by the user to copy the GRUB + UEFI shim from the ostree commit to the /boot partition. See fedora-silverblue/issue-tracker#120 (comment)

Additional context

Supporting bootupd requires that Fedora IoT is built using "unified-core" mode, which is not yet supported by osbuild.

See also: https://gitlab.com/CentOS/cloud/issue-tracker/-/issues/6

After Fedora has signed off on the release and declared Go! we need to do a final compose switch the branch from devel to stable

• PR's changing streams
• Ensure the final release deliverables are signed with the correct key
• Ensure the release is copied to the staging location for the mirrors

The Fedora Robotics SIG is quite active and are working to package up ROS2 into Fedora and are interested in how they could work with us to support it on Fedora IoT. It's also a target in EPEL for the enterprise use case as well as it's used in factory automation and all sorts of industrial robotics, not just fun cars :)

Describe the bug
When I try to boot a Rawhide deployment (41.20240315.0) on a Raspberry Pi 4 Model B Rev 1.4 8 GB, the device becomes unresponsive.

To Reproduce

1. Flash Fedora IoT 39 or 40 raw image to microSD card with arm-image-installer.
2. rpm-ostree rebase fedora-iot:fedora/rawhide/aarch64/iot

Expected behavior
The 41.20240315.0 (2024-03-15T16:11:04Z) deployment should boot.

Screen output after the bootloader

  Booting `Fedora Linux 41.20240315.0 (IoT Edition Prerelease) (ostree:0)'
EFI stub: Deconpressing Linux Kernel...
EFT stub: Using DTB from configuration table
EFI stub: Exiting boot services...

Then the device freezes.

OS version:
The output of the rpm-ostree status command after rebase and before reboot.

[root@localhost ~]# rpm-ostree rebase fedora-iot:fedora/rawhide/aarch64/iot
⠠ Receiving objects; 99% (24951/24961) 1.2 MB/s 686.7 MB
3499 metadata, 21465 content objects fetched; 671510 KiB transferred in 557 seconds; 1.8 GB content written
Receiving objects; 99% (24951/24961) 1.2 MB/s 686.7 MB... done
Staging deployment... done

...

Changes queued for next boot. Run "systemctl reboot" to start a reboot
[root@localhost ~]# rpm-ostree status 
State: idle
Deployments:
  fedora-iot:fedora/rawhide/aarch64/iot
                  Version: 41.20240315.0 (2024-03-15T16:11:04Z)
                   Commit: d2c8b1f04509fa610e6a70adf4a915ca4297a05358588ccf10fb1cda5679c362
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
                     Diff: 356 upgraded, 11 removed, 13 added

● fedora-iot:fedora/stable/aarch64/iot
                  Version: 39.20231103.1 (2023-11-03T18:17:43Z)
                   Commit: cc8d419f72d84ac24d0a95e235c0cdf72844f73d9d6f42a41fcddf23dfb34f7d
             GPGSignature: Valid signature by 6A51BBABBA3D5467B6171221809A8D7CEB10B464
[root@localhost ~]# reboot 
[root@localhost ~]# Connection to 192.168.30.14 closed by remote host.
Connection to 192.168.30.14 closed.

Additional context

[root@localhost ~]# hostnamectl status 
...
     Hardware Vendor: raspberrypi
      Hardware Model: Raspberry Pi 4 Model B Rev 1.4
...

Describe the bug
Create the Fedora 40 Beta IoT compose, ensuring all blockers and Freeze Exceptions have been pulled in.

Upstream request with blocker and freeze exceptions:
https://pagure.io/releng/issue/12007

Add an SOP to the Fedora Quick start docs

Describe the bug

Recent disk images no longer include the '/etc/ostree/remotes.d/fedora-iot.conf' file parsed by greenboot.

As a result when logging into a system:

Script '01_update_platforms_check.sh' FAILURE (exit code '1'). Continuing...
Boot Status is GREEN - Health Check SUCCESS

To Reproduce

Please describe the steps needed to reproduce the bug:

1. Boot recent Fedora 39 IoT disk image
2. SSH in

Expected behavior

No script failure.

Additional context

The system can still be updated, the remote repository information is now included in:

cat /sysroot/ostree/repo/config
[core]
repo_version=1
mode=bare

[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot
gpgkeypath=/etc/pki/rpm-gpg/
gpg-verify=true
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

[sysroot]
bootloader=none
readonly=true

Reported by @AdamWill on the IoT channel:

btw the last few f39 IoT nightlies do not install with an error Multiple specifications found for remote "fedora-iot"
https://openqa.fedoraproject.org/tests/2454758#step/_do_install_and_reboot/115
same on x86_64 and aarch64
has been happening since Fedora-IoT-39-20240226.0

Describe the bug
pmlogger_daily.service is failing:

Jan 24 15:19:27 haddock systemd[1]: Starting pmlogger_daily.service - Process archives...
Jan 24 15:19:27 haddock systemd[1]: Started pmlogger_daily.service - Process archives.
Jan 24 15:19:28 haddock systemd[1]: pmlogger_daily.service: Main process exited, code=exited, status=1/FAILURE
Jan 24 15:19:28 haddock systemd[1]: pmlogger_daily.service: Failed with result 'exit-code'.

I tried running the command that it runs sudo /usr/libexec/pcp/bin/pmlogger_daily -E, and that failed with an exit code of 1.

To Reproduce

1. Run sudo systemctl start pmlogger_daily.service or sudo /usr/libexec/pcp/bin/pmlogger_daily -E

Expected behavior
pmlogger_daily.service doesn't fail

OS version:

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
BootedDeployment:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 39.20240122.0 (2024-01-22T15:18:27Z)
               BaseCommit: 23398f5f1fc27cc8e6e56f967420eeb142f3295d4999251a1cc6f223497ef509
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: borgmatic btrfs-progs cockpit cockpit-pcp cockpit-podman distrobox fish git ncurses smartmontools wireguard-tools

Outcomes

• understand how to compose bootc bootable containers in existing Fedora build infra
• understand any changes necessary changes to configuration, infrastructure to enable above
• additional issues tracking follow-up work/implementation

Describe the bug
The fedora iot docs still need to be updated for osbuild components.

• Need docs on how to compose Fedora IoT artifacts locally using osbuild
• Need docs on the newly available artifacts produced by osbuild (and how to use them)

Create the Fedora 40 Final IoT compose, ensuring all blockers and Freeze Exceptions have been pulled in.

Upstream request with blocker and freeze exceptions:
https://pagure.io/releng/issue/12060

Add an SOP to the Fedora Quick start docs

Currently our docs point to using the IRC for our weekly meetings[1]. We recently decided to move over to use Matrix, so update the doc's accordingly and book the meeting location to avoid conflicts like we had this week.

1- https://docs.fedoraproject.org/en-US/websites/rvmp/meetings/

Describe the bug
Osbuild created deliverables use the wrong naming format:

ISO - Fedora-IoT-ostree-39.20231003.0-20231003.0.aarch64.iso
DISK - Fedora-IoT-39.20231003.0-20231003.0.aarch64.raw.xz

Expected behavior

These should follow the Fedora's image naming policy.

Expected format:

• '%(release_short)s-%(disc_type)s-%(arch)s-%(version)s-%(date)s%(type_suffix)s.%(respin)s.iso'

Example- Fedora-IoT-ostree-aarch64-39-20231003.0.iso

Describe the bug
After installing Fedora IOT 40 beta by using the Simplified Installer and passing the FDO manufacturing configuration, the installation finishes correctly, the manufacturing process takes place and the device is powered off.

However, in the first boot the device does not try to perform the FIDO onboarding protocol as expected.
Taking a deeper look it looks like the systemd service is disabled:

systemctl list-unit-files fdo-client-linuxapp.service
UNIT FILE                   STATE    PRESET  
fdo-client-linuxapp.service disabled disabled

1 unit files listed.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Install Fedora IoT 40 Beta:

virt-install --connect qemu:///system \
             --name fedora-iot-simplified-installer \
             --os-variant fedora-rawhide \
             --boot uefi,loader.secure=false \
             --noautoconsole \
             --vcpus 1 --memory 3072 \
             --network network=default,model=virtio \
             --disk pool=default,size=30 \
             --cdrom Fedora-IoT-provisioner-40-20240319.2.x86_64.iso

2. Reclaim and provision the system from https://provision.fedoraproject.org/portal/devices/
3. Connect to the system and check the FDO client application:

systemctl status fdo-client-linuxapp.service 
○ fdo-client-linuxapp.service - FDO client
     Loaded: loaded (/usr/lib/systemd/system/fdo-client-linuxapp.service; disabled; preset: disabled)
     Drop-In: /usr/lib/systemd/system/service.d
              └─10-timeout-abort.conf
     Active: inactive (dead)

Expected behavior
The onboarding process is triggered in firstboot after the installation.

Screenshots
If applicable, add screenshots to help explain your problem.

OS version:
Fedora-IoT-provisioner-40-20240407.0.x86_64.iso

rpm-ostree status -b
State: idle
BootedDeployment:
● fedora-iot:fedora/devel/x86_64/iot
                  Version: 40.20240319.2 (2024-03-19T17:58:04Z)
                   Commit: 4761cfc7d5c969b7a8c2b36d5c9980424af0a0127358b48dcb775857ec587406
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC

xref: https://gitlab.com/fedora/websites-apps/fedora-websites/fedora-websites-3.0/-/issues/296

We should be clear in our docs about what is a "raw image" vs. "installer ISOs" etc. These are the choices that are going to be given to users when they navigate to the IoT download page from getfedora.org, so we should be able to guide them in more detail about why they would choose one format over the other.

(There's another issue here where https://fedoraproject.org/iot/download should probably list the simplified-provisioner, too)

With the upcoming release of Fedora 40, we will have a simplified-installer artifact that can leverage FDO perform onboarding and configuration.

We should have docs that explain how to do this.

Describe the bug
As described above, installing fish doesn't pull in ncurses as a dependency, which causes the clear command to fail. This only seems to be the case on IoT, and not Kinoite, unless ncurses is installed by default on Kinoite so I didn't notice.

To Reproduce
Please describe the steps needed to reproduce the bug:

1. $ rpm-ostree install fish
2. Reboot
3. $ fish
4. $ clear

Expected behavior
Running clear run successfully, and clears the screen.

OS version:

State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: last run 20h ago
BootedDeployment:
● fedora-iot:fedora/stable/x86_64/iot
                  Version: 39.20240122.0 (2024-01-22T15:18:27Z)
               BaseCommit: 23398f5f1fc27cc8e6e56f967420eeb142f3295d4999251a1cc6f223497ef509
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: borgmatic btrfs-progs cockpit cockpit-pcp cockpit-podman distrobox fish git ncurses smartmontools wireguard-tools

Describe the bug

When using a Fedora 39 Simplified-Installer to install to a device, there is a failure and the system drops to a dracut emergency shell- then continues on and performs a successful installation:

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Create a Fedora simplified-installer
2. Provision a device.

Expected behavior
No error, no dracut shell.

Screenshots

From the logs:

Oct 13 16:50:02 fedora dracut-mount[714]: Warning: Can't mount root filesystem
Oct 13 16:50:02 fedora systemd[1]: Starting dracut-emergency.service - Dracut Emergency Shell...

Additional context

After installation, using coreos.inst.skip_reboot to stop the system from rebooting :

Is your feature request related to a problem? Please describe.
ostree is moving to support unified core as the one true way to generate ostrees. We should migrate to that to ensure we consume the path most used as it makes support of the ostree components more straight forward.

Describe the solution you'd like
User ostree unified core to build the FIoT ostrees

Describe alternatives you've considered
Remain on the current path but less groups are using the code paths so we may end up with other issues.

Additional context
Need to add various upstream trackers

The docs for the simplified-provisioner point to iot.fedoraproject.org to download the artifact, but there is no link to the artifact there.

I suspect this requires a change outside of docs.fedoraproject.org, but maybe in the short-term we provide a dirty link to https://dl.fedoraproject.org/pub/alt/iot/40/IoT/x86_64/ (and aarch64 version) for folks to discover the artifact.

Outcomes

• definition of base packages that go into the bootc base image layer
• changes made to Fedora build infrastructure/tooling/configuration to produce base image layer
• base image layer pushed to publicly available image registry

Depends on the completion of #16

This issue tracker is intended only for IoT specific issues. Please try to reproduce the issue on a relevant Fedora release to determine if the issue is specific to IoT or a general issue in Fedora. If is a general issue in Fedora, please report it in Red Hat Bugzilla (see How to file a bug) or in an upstream project and not in this issue tracker.

Describe the bug
The docs are not clear how to use a custom ignition file with the simplified provision installer.

Can I use a local file or just a remote one over http?

To Reproduce
Please describe the steps needed to reproduce the bug:

1. Download the simplified provision iso
2. Run it using qemu-system
3. How to mount an ignition file ...?

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

OS version:

Using Fedora IoT 40, I can't login into it.

Additional context

As the "bug" description says, the docs is not very clear how to mount an ignition file into the image.