dell / dell-recovery

Dell Recovery for Ubuntu

Dell Recovery Media Creator

The Dell Recovery media creation tool supports two different usage modes, "End User Mode" and "Builder Mode". In End User mode, the tool will simply create an image from an existing recovery partition with no customizations. In builder mode, the tool allows modifying the source of the base image, the source of the framework, as well as injection of additional content.

Tool Modes

End User Mode

When a customer receives a Dell machine that has been factory shipped with Linux, there will be an icon available in GNOME shell to launch this tool. They will be prompted for what type of media they would like to create.

Out of box experience mode

If a DVD burner or USB port is found on the machine, customers will be offered to create media at the end of the out of box experience.

Media builder mode

In builder mode, the user will be offered a variety of options that allow them to create ISOs based upon different snapshots of release upon standard Ubuntu media.

The latest information on how to use builder mode will be documented within the integrated help dialog.

Flow

Due to the nature of including a recovery partition, the installation flow varies from the standard Ubuntu installation. It is further documented here.

Ubiquity

Dell recovery is built with an integrated Ubiquity plugin. It is branched with each Ubuntu LTS release and has code that will tightly integrate with Ubiquity for factory installation. Documentation for all of the features in Ubiquity mode and how to create packages to support it are stored outside of the dell-recovery tree.

Modifying a factory image

Information about how to modify a factory image are available here.

dell-recovery's Issues

build error with proposed pyflakes3 2.2.0-1~20.04.1

./tests/run-pyflakes
Dell/recovery_backend.py:563:21 local variable 'timestamp' is assigned to but never used
Dell/recovery_backend.py:564:21 local variable 'distributor_string' is assigned to but never used
Dell/recovery_common.py:653:25 local variable 'type' is assigned to but never used
Dell/recovery_basic_gtk.py:157:72 local variable 'wfd' is assigned to but never used
make[1]: *** [debian/rules:17: check] Error 1
make[1]: Leaving directory '/home/alextu/workspace/PC/somerville/tasks/recovery/dell-recovery'
make: *** [debian/rules:7: clean] Error 2
dpkg-buildpackage: error: fakeroot debian/rules clean subprocess returned exit status 2

The messages will disappear in systemd.

dell-recovery/late/scripts/fifuncs

Line 30 in c8be7db

echo $* > /dev/tty1 2>&1

We need to find some way else to show the messages.

When "full disk encryption" is enabled, no partition is mounted on /cdrom during OOBE.

In normal installation, the recovery partition will be mounted on /cdrom during OOBE.
But, when FDE is enabled, no partition is mounted on /cdrom during OOBE.

/var/log/installer/casper-md5check.json has a failed result.

When I used somerville-jammy-amd64-20220504-33.iso or dell-bto-jammy-jellyfish-X07-20220504-12.iso based on somerville-jammy-amd64-20220504-33.iso, both of them will generate the failed result in /var/log/installer/casper-md5check.json.
But there is no problem to execute md5sum -c md5sum.txt --quiet to check the base image, i.e somerville-jammy-amd64-20220504-33.iso, directly.
So dell-recovery needs to fix the problem when it generated the recovery image or created the recovery partition.

'late/scripts/chroot.sh' missing space before ']'

$ shellcheck ./late/scripts/chroot.sh

In ./late/scripts/chroot.sh line 84:
if [ ! -e "$TARGET/pts"]; then
^-- SC1009: The mentioned syntax error was in this if expression.
^-- SC1073: Couldn't parse this test expression. Fix to allow more checks.
^-- SC1019: Expected this to be an argument to the unary condition.
^-- SC1020: You need a space before the ].
^-- SC1072: Missing space before ]. Fix any mentioned problems and try again.

For more information:
https://www.shellcheck.net/wiki/SC1020 -- You need a space before the ].
https://www.shellcheck.net/wiki/SC1019 -- Expected this to be an argument t...
https://www.shellcheck.net/wiki/SC1072 -- Missing space before ]. Fix any m...

Travis-ci: Error while enabling ppc64le build

Hi ,

I am enabling ppc64le build on travis-ci, and updated the .travis.yml and added arch: ppc64le along with amd64. But ppc64le build is failing and below error is coming:

"E: Unable to locate package gnu-efi
The command '/bin/sh -c apt-get install -yq --no-install-recommends build-essential debhelper dh-python dpkg-dev python3 python3-distutils-extra po-debconf pyflakes3 lsb-release gnu-efi' returned a non-zero code: 100
The command "docker build -t dell-recovery-ubuntu -f docker/Dockerfile-ubuntu ." failed and exited with 100 during .
Your build has been stopped."

Full error log can be tracked on this link 👍 https://travis-ci.com/github/sanjay-cpu/dell-recovery/jobs/386367642

Please have a look into the issue.

Thanks !!

BIOS ID should be identified by tag instead string number.

from the commit fe685de , the BIOS ID came from string 2, but it should came from tag ID '1' instead.

late/chroot_scripts/03-ubuntu-drivers.sh:
https://paste.ubuntu.com/p/BqnHVFWGzt/

dell-recovery shows no error when the target disk space is insufficient

There's a feature request to warn people that the space of target storage is not enough.

LP: #1842325 dell-recovery shows no error when the target disk space is insufficient

Exception encountered ‘server:0’

Hi, I’m trying to restore the factory image onto a new drive and I’m getting the above error pretty much as soon as the installer gui shows up. I can only click on OK, after which system reboots. There is nothing in the log. Media verifies as OK during startup… What could be the cause? The only difference between the factory system and the current config is the 2x nvme drives that have a leftover MD based raid setup. Could this be an issue?

.disk/casper-uuid is missing and conf/uuid.conf in initrd is not updated for jammy

I made a customized Ubuntu jammy image with dell-recovery.
I can boot from Ubuntu system on the USB stick to execute dell-recovery to install the recovery partition. (stage1)
However .disk/casper-uuid is missing and conf/uuid.conf in initrd is not updated in the recovery partition. (stage2)
So it won't boot from the recovery partition to install Ubuntu system. (It can not enter stage2.)

Dell recovery shows an "Operation was cancelled" prompt

[Steps]

1. Install image
2. Launch Dash and enter "Dell"
3. Select Dell recovery tool
4. Select "Restore System" and enter password
5. A shut down prompt will pop up, with an exception prompt seen in the background.
6. Close the purple shutdown prompt and check the exception prompt that is displayed

[Expected result]
No exception prompt should be seen

[Actual result]
An exception prompt is seen after step 4 and can be brought to foreground after doing step 6

Note: If user directly selects reboot in the shutdown prompt and ignore the exception prompt, Dell recovery can still be completed.

[Test details]
machine: Constantine i7 EVT board, CONS-DVT1-C2 (i5) FORD-DVT1-C2
image: somerville-xenial-amd64-iso-hybrid-20160303-3.iso

Can't see storage that once used as RSTe.

On a machine that once installed windows and RAID on in bios, windows will use the storage in RSTe mode.

Given the status of the storage, the storage won't be display in stage 1 of the OEM image.

Found the code affected is:

    id_type = block.get_cached_property("IdType").get_string()
    if id_type == 'isw_raid_member':
        continue

And the original commit that introduced above is (been refactored by other commits though): 38630b8

Shall we either revert the whole commit, since it seems we are not going to either

1. revert the whole commit
2. comment out the major blocking code
3. keep as is

Unable to detect valid initrd

Machine:
Dell XPS 13 9380 - Ubuntu 18.04 (Preinstalled)

Steps:

1. Select Base OS Image: ubuntu-19.04-desktop-amd64.iso
2. Select FID Content:
   • Choose package
   • Build dell-recovery package from this system
3. Leave Driver Packages as default
4. Leave Application Packages as default
5. Media Type: No media, create image
6. Confirm Selections
7. Apply

I've tried this with different images, all vanilla images downloaded direct from Ubuntu and I get the same error on each.

I mounted the ISO and confirmed that these paths exist:

/path/to/iso/casper/initrd
/path/to/iso/.disk/casper-uuid-generic

to do shellcheck for all shell scripts

so that we can prevent issue like 379b789
It can be picked up by shellcheck

╰─>shellcheck late/scripts/chroot.sh 

In late/scripts/chroot.sh line 68:
    . /usr/share/dell/scripts/FAIL-SCRIPT
      ^-- SC1091: Not following: /usr/share/dell/scripts/FAIL-SCRIPT was not specified as input (see shellcheck -x).


In late/scripts/chroot.sh line 84:
    if ! -e "$TARGET/pts"; then
         ^-- SC2215: This flag is used as a command name. Bad line break or missing [ .. ]?

swap file instead of swap partition

For Ubuntu 17.04 there are some changes potentially going into Ubiquity that will change the defaults to create swap files instead of swap partitions.

This should be monitored, if that change remains for Ubuntu 18.04 dell-recovery should make a similar change.

dell-recovery 1.63 depends on python 3.7 unavailable in bionic

dell-recovery 1.63 does not work on bionic-based Recovery medias as the image for Dell XPS 9380, which is based on ubuntu bionic where the latest available python version is 3.7.

Refer also to b8dce58#r35569513

failed record in recovery partition will be copy to created iso.

reproduce step:

1. do recovery from usb disk and select chinese language
2. after 1. finished execute 'dell-recovery' to do recovery again.
3. after 2. finished , execute 'dell-recovery' to create recovery iso.
4. use the created iso to do recovery again and get red error grub menu.

Because of #47, the environment in 2 have ${recovery-partition}/factory/grubenv with error bit. So, the ISO generated by 3 will also have it. So that introduce the error in 4.

dell-recovery should stop process and warn user if current environment is not correct. ex. ${recovery-partition}/factory/grubenv should not be there, otherwise the red grub menu will confuse user because there are no error in creating iso.

automatic mode enhancement

The idea is: in development phase, install from usb still is the most stable way.
For engineer, save time and less interactivity in repeated process helps a lot.

So I'd like a way to easily and enable full-automatic installation/recovery mode.

Full picture:

1. modify oobe so that it's full-automatic if automatic mode in ubiquity is enable.
2. enable full-automatic mode in stage 1, 2 and recovery from hdd
3. a way to preseed oobe. either make sure we can preseed stage 1, 2 can properly carry those to oobe (by ubiquity), or we can add oobe specific preseed.
4. enable automatic mode in oobe.

We current have other way to do 3 and 4. Even better if that's built-in in dell-recovery.

`ubuntu-report show | grep DCD` shows many redundant suffixes.

$ ubuntu-report show | grep DCD
    "DCD": "canonical-oem-somerville-jammy-amd64-20220504-33+jellyfish+X07+jellyfi+X07.1+jellyfi+X07.2+jellyfi+X07.3+jellyfi+X07.4+jellyfi+X07.5+jellyfi+X07.6"

It is expected to be "DCD": "canonical-oem-somerville-jammy-amd64-20220504-33+jellyfish+X07.6" instead.

The installation on Intel RSTe SATA RAID failed.

When the volume name of RSTe SATA RAID has no number, such as 'RSTe' and 'raid', it will fail to install the system.

This tool is incomprehensible to end users, especially when attempting to create dell installation media from a stock standard ubuntu installation.

There needs to be an explanation of where to get this file:

As a user, I have no idea what this thing is asking me for, and I have no course of action to take. Looking it up hasnt helped much.

Enhanced sorting storage

In https://github.com/dell/dell-recovery/blob/master/ubiquity/dell-bootstrap.py#L651
It just do disks.sort(), and the result seems not quite end-user prefer.
IMHO, for today, we could just put nvme in front of sdX. If we have two nvme disks, put largest disk first seems to be reasonable.

Given above idea, we still need the answer on the sorting order of (include nvme and sd):

/dev/dm
/dev/md
/dev/pmem
/dev/nvme
/dev/sd

Support for multiple storage.

There are several different combinations, notably:

• 1 SSD + 1 HDD

• 2 HDDs

• 2 SSDs

Current dell-recovery only uses one storage device, but having LVM RAID, btrfs, zfs can be really helpful on combinations above.

Is wake_network necessary in installation stage 2 ?

Commit 8223cce sleep the network in stage 2 but will wake up at the end.

There is a case that waking up network will cause package be upgraded through network during stage 2.
e.g.
google-chrome-stable installed at early stage 2 and the package generate sources list in /etc/apt/sources.list.d/google-chrome.list.
deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main

The chroot-script 89-upgrade.sh it will execute apt-get update;apt-get upgrade and this script is executed after network waked up. After execute 89-upgrade.sh, the browser package will be upgraded.

Wondering if wake_network is necessary ?

Encoding issue when using /usr/share/dell/bin/dell-bto-autobuilder

Received org.freedesktop.DBus.Python.TypeError: Traceback (most recent call last): |
File "/usr/lib/python3/dist-packages/dbus/service.py", line 707, in _message_cb
retval = candidate_method(self, *args, **keywords)
File "/usr/lib/python3/dist-packages/Dell/recovery_backend.py", line 464, in assemble_image
self._process_driver_fish(driver_fish, assembly_tmp)
File "/usr/lib/python3/dist-packages/Dell/recovery_backend.py", line 350, in _process_driver_fish
name = member.get_info(encoding='UTF-8', errors='strict')['name']
TypeError: get_info() got an unexpected keyword argument 'encoding'
when closing recovery-media-backend DBus service

dell-recovery treat the removed ubiquity as error when there's new version of ubiquity in debs folder

regarding to the behavior of ubiquity during image installation, the ubiquity debian package will be uninstalled in the end.
Then because dell-recovery set a guard to treat packages removing which installed by ubiquity operations as a failure [1].

So, dell-recovery treat the removed ubiquity as error when there's new version of ubiquity in debs folder.
The failure checking should filter out the ubiquity packages.

[1] https://github.com/dell/dell-recovery/blob/master/late/scripts/chroot.sh#L170

default to LVM in disk recipe

If possible, it would be ideal to switch from ext4 to LVM.
Switching to LVM will allow later expansion and support of multiple disks to be easier.

Typo in recovery_common.py

Attempting to build a custom image for bionic, I encountered the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/dbus/service.py", line 707, in _message_cb
    retval = candidate_method(self, *args, **keywords)
  File "/usr/lib/python3/dist-packages/Dell/recovery_backend.py", line 456, in assemble_image
    white_tree("copy", white_pattern, base_mnt, assembly_tmp)
  File "/usr/lib/python3/dist-packages/Dell/recovery_common.py", line 91, in white_tree
    return _tree(action, whitelist, src, dst, base, True)
  File "/usr/lib/python3/dist-packages/Dell/recovery_common.py", line 120, in _tree
    _tree(action, list, src_name, dst_name, base, white))
  File "/usr/lib/python3/dist-packages/Dell/recovery_common.py", line 132, in _tree
    shtuil.copy(src_name, dst_name)
NameError: name 'shtuil' is not defined

There is a typo shtuil should be shutil - fixing this in the file alleviated the error.

/usr/share/dell/bin/dell-bto-autobuilder generates ISO images with wrong partition ID.

There is no obvious side effect from this issue so far.

$ file italia_X80.iso
italia_X80.iso: DOS/MBR boot sector ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'ISOIMAGE' (bootable); partition 1 : ID=0x83, start-CHS (0x0,0,1), end-CHS (0x3fb,128,32), startsector 0, 4210560 sectors; partition 2 : ID=0xef, start-CHS (0x3fc,0,1), end-CHS (0x3fd,128,32), startsector 4210560, 8256 sectors

$ fdisk -l italia_X80.iso 
Disk italia_X80.iso: 2 GiB, 2160033792 bytes, 4218816 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device          Boot   Start     End Sectors Size Id Type
italia_X80.iso1            0 4210559 4210560   2G 83 Linux
italia_X80.iso2      4210560 4218815    8256   4M ef EFI (FAT-12/16/32)

$ file ubuntu-16.04.3-desktop-amd64.iso
ubuntu-16.04.3-desktop-amd64.iso: DOS/MBR boot sector ISO 9660 CD-ROM filesystem data (DOS/MBR boot sector) 'Ubuntu 16.04.3 LTS amd64' (bootable); partition 2 : ID=0xef, start-CHS (0x3ff,254,63), end-CHS (0x3ff,254,63), startsector 3006684, 4672 sectors

$ fdisk -l ubuntu-16.04.3-desktop-amd64.iso
Disk ubuntu-16.04.3-desktop-amd64.iso: 1.5 GiB, 1587609600 bytes, 3100800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0d66cd15

Device                            Boot   Start     End Sectors  Size Id Type
ubuntu-16.04.3-desktop-amd64.iso1 *          0 3100799 3100800  1.5G  0 Empty
ubuntu-16.04.3-desktop-amd64.iso2      3006684 3011355    4672  2.3M ef EFI (FAT-12/16/32)

(Edited to place in triple tick code blocks)

Have english only in "Create Dell Recovery Media Page"

The downstream bug: https://bugs.launchpad.net/somerville/+bug/1967550

Step to reproduce:

1. Install Ubuntu Dell OEM image and select language other than "English" during OOBE
2. Open "Dell Recovery" app.
3. The translation is still english which is not changed with the language setup.

update package which installed in base image cause error.

the commit "f352bd" for #28 will break base image packages updating.

example:

1. create a usb disk with ubuntu recovery iso content.
2. collect newer mesa stuff from x-staging[1] and x-swat[2] and put them to /debs for update.

• ex. one of the stuff is libdrm2_2.4.83-1~16.04.1_amd64.deb

3. do recovery from usb disk
4. an older libdrm2 has been installed in base image, so /var/lib/ubiquity/installed-packages exists libdrm2:amd64
5. then it will treat "libdrm2" differ to "libdrm2:amd64" then get a failed result in the code of commit "f352bd".

Then the failure cause showing a red error grub menu in next usb recovery booting.

Should we filter out ":amd64" from comparing?

[1] https://launchpad.net/~canonical-x/+archive/ubuntu/x-staging
[2] https://launchpad.net/~ubuntu-x-swat/+archive/ubuntu/updates/+packages

to support Intel and AMD SDP (dev board)

To support cutting-edge hardware, it would be more efficient if Intel and AMD can verify the latest OEM image on their dev board.
Can we also cover Intel and AMD SDP in the allowing list of dell-recovery?

The line to get this extending should be https://github.com/dell/dell-recovery/blob/master/Dell/recovery_common.py#L216

DVD recovery media creator failed

Some wodim commands need the root privilege to work properly on Ubuntu 18.04.

DBus error when more than 2 dell-bto-autobuilder run together on the same host

When there are more than 2 dell-bto-autobuilder execute at the same time on the same host. Only first one can successfully completed. The following ones would encounter following error.

---

13:30:54 Done
13:30:54
13:30:54 Generating BTO - executing /usr/share/dell/bin/dell-bto-autobuilder...
13:30:54
13:30:54
13:31:20 ERROR:dbus.proxies:Introspect error on :1.3021:/RecoveryMedia: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
13:32:10 Received org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken. when closing recovery-media-backend DBus service
13:32:10 Traceback (most recent call last):
13:32:10 File "/usr/share/dell/bin/dell-bto-autobuilder", line 319, in
13:32:10 iface.request_exit() # will call atexit in dell backend
13:32:10 File "/usr/lib/python3/dist-packages/dbus/proxies.py", line 145, in call
13:32:10 **keywords)
13:32:10 File "/usr/lib/python3/dist-packages/dbus/connection.py", line 651, in call_blocking
13:32:10 message, timeout)
13:32:10 dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.

Please help to create the jammy branch.

I would like to set up some CI jobs for jammy just like what I did for focal.