pop-os / distinst Goto Github PK

It's possible for the install process to fail due to running out of space on a partition, if the partition is smaller than what the image requires. The simplest way to go about this is to ensure that the root partition is at least 3x the size of the image (maybe we should include a file that shows the actual file size of the compressed image?). More complex checking may not be required, but would be welcomed (ie: recursively checking if the contents of the image will fit onto each partition that will be mounted).

At the end of install, when it finishes the bootloader step, distinst sits while the drive syncs.

Unsure on what's happening, but the generated data is empty now.

• Ext usage via dumpe2fs -h DEVICE
• Fat usage via fsck.fat -n -v DEVICE
• Btrfs usage via btrfs filesystem show DEVICE
• NTFS usage via ntfsresize --info DEVICE

XFS, F2FS, etc. don't support shrinking, so it's not useful to report usage.

If this is a bug, please use the template below. If this is a question, or a general discussion topic - please post on our subreddit https://reddit.com/r/pop_os or start a conversation in our chat https://chat.pop-os.org/ - as these are the proper forums for those types of posts.

Reporter info

Due to an issue with keyfile support implementation, this testing was done from commit: c2773bf62129bf0288207ab3ee03a0bdf014177d

 * Distribution - (run ```cat /etc/os-release```)
 * Related Application and/or Package Version - apt policy $PACKAGE NAME

Pop!_OS 18.04

Issue/Bug Description
Attempting to run the tests/partition-lvm.sh script without the --test parameter does not result in a working, installed system.

The output of the installation process:
lvminstalloutput.txt

When booting after the installation an initramfs shell appears:

Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
 - Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=None does not exist. Dropping to a shell!

BusyBox v1.27.2 (Ubuntu 1:1.27-2-2ubuntu3) built-in shell (ash)
Enter 'help for a list of built-in commands. 

(initramfs)

Steps to reproduce (if you know)
The partition-lvm.sh was modified in a few ways:

RUNS=1

    sudo env RUST_BACKTRACE=1 target/debug/distinst \
        -s "${FS}" \
        -r "${REMOVE}" \
        -h "pop-testing" \
        -k "us" \
        -l "en_US.UTF-8" \
        -b "$1" \
        -t "$1:gpt" \
        -n "$1:primary:start:512M:fat32:/boot/efi:esp" \ #--test removed
        #line removed  to eliminate unnecessary complications -n "$1:primary:512M:2048M:lvm=data" \
        -n "$1:primary:512M:end:lvm=data" \ #moved partition up to end of efi partition
        --logical "data:root:-4096M:ext4:mount=/" \
        --logical "data:swap:4096M:swap"

It appears moz-cheddar and its relatives (including rusty-cheddar) are dead.

Would you consider switching to the more actively-maintained cbindgen crate instead?

Currently, distinst shells out to the mount and umount commands, but it would be more efficient if it directly called the libc::mount and libc::umount functions instead, so that an external process would not need to be spawned and observed. Not a high priority.

• libc::mount
• libc::umount
• libc::swapoff
• Mounts with flags (ie: bind)

It seems like lots of similar work is already done in https://github.com/storaged-project/blivet used in Fedora installing. It might make sense to check whether it's viable alternative or addition to libparted.

If this is a bug, please use the template below. If this is a question, or a general discussion topic - please post on our subreddit https://reddit.com/r/pop_os - as that is the proper forum for those types of posts.

Pop!_OS 18.04 Build 13

Issue/Bug Description
After installation, when selecting the boot device, the Pop!_OS option is labeled:
UEFI OS ([drive name])

Expected behavior
Pop!_OS 18.04

The README still includes:

git submodule update --init libparted

Which is not present as a submodule anymore

Just filling a bug so that you can close it

partprobe sometimes returns before the kernel updates /sys/block and /sys/class/block to match the new partition table.

This may be fixed by using blockdev --rereadpt or by sleeping and checking for the partitions to match what is expected.

I believe this is caused by passing the device path to umoun2 instead of the mountpoint

Add a check to prevent distinst from attempting to partition and install with a '/' partition format of ntfs or FAT(16/32)

Existing unit tests may be found here. I'd like to have many more unit tests, however, as partition management is a particularly dangerous part of the installation process. All that needs to be done is to simply create more test units in the same fashion that test more of the code base.

As libparted (C) does not natively supporting resizing partitions on the disk itself, I am currently working on a means to combine dd and file system-specific commands for resizing and moving partitions. As much as I hate to call out to external commands, it seems to be the only way to achieve this. Due to the complexity of this, I will be prioritizing Ext/Btrfs/Fat/Swap at first.

• Logic implemented within in-memory partition management
• Resizing Logic
  • Shrinking
  • Growing
• Moving Logic
  • Backwards
  • Forward
• Filesystem Support
  • Swap (simply delete and recreate the partition)
  • Ext2/3/4 (resize2fs)
  • Fat16/32 (fatresize)
  • Btrfs (btrfs resize filesystem)
  • XFS (xfs_growfs) (NOTE: Shrinking not supported!)
  • NTFS (ntfsresize)
  • F2FS (resize.f2fs) (NOTE: Shrinking not supported!)
  • ExFat
• Ensure that flags are carried over
• Ensure that labels are set
• Ensure that partition types are carried over (Primary / Logical)

File systems can have restrictions on the minimum and maximum size of the partition.

• Btrfs
• Fat16
• Fat32
• Ext2
• Ext3
• Ext4
• NTFS
• XFS
• F2FS

• Ability to define that a partition belongs to a volume group, and whether it has encryption or not.
• Refactor external module to use a shared exec function
• Ability to create volume groups via vgcreate
• Ability to create physical volumes from encrypted partitions with pvcreate
• Ability to create partitions on volume groups with lvcreate
• Ability to format logical partitions on volume groups via mkfs
• Obtain a list of physical volumes and their associated volume groups via pvs
• Obtain a list of logical volumes on a volume group via lvs
• Ability to deactivate logical volumes with vgchange -an
• Ability to remove volume groups with vgremove
• Ability to remove physical volumes with pvremove
• Ability to encrypt partitions
  • Password support
  • Keyfile support
• Optional encryption
• Ability to define LVM + LUKS configurations in the Distinst CLI
• Remove logical device map volumes before partitioning
• Remove physical device map volumes before partitioning
• Remove volume groups before partitioning
• Close opened LUKS partitions before partitioning
• Unmount mounted logical volumes (currently working on)
• Obtain UUIDs of logical volumes
• /etc/fstab support
• /etc/crypttab generation
• Verify that paths are set
• LVM on LUKS
• LUKS on LVM
• Use entire disk as LVM / LUKS volume
• Integration Tests
  • LVM on LUKS
  • LUKS on LVM
  • LVM without LUKS
• C API Integration
• Vala API Integration

Will there be plans to support selecting systemd-boot as a simpler/faster alternative to GRUB for EFI installs?

Partitioning will depend upon the libparted library for obtaining information about disks and their partitions, as well as the creation and manipulation these partitions.

There shall be an in-memory partition management system that the user will interface with to make changes and be notified of incompatible changes, and from which disk operations will be constructed from using a diff of the original partition scheme and the partition scheme to deploy.

As partitions are created and manipulated, the mkfs command will be used to format the unformatted partitions, or to reformat existing partitions. In addition, the udevadm command will be used to get the serial model of disks being manipulated, to ensure that we are writing changes to the correct disk in the event that the user changes the location of the disk mid-install.

Current Progress

• Use mkfs to format newly-created partitions (already done)
  • Add a variant for each possible file system type.
  • mkfs routines for all possible file system types.
• Create libparted Rust bindings from libparted_sys
• Create examples to test the libparted bindings
  • list example
  • mkpart example
  • rmpart example
  • mvpart example
• Integrate libparted bindings into distinst (awaiting review: #24)
• In-memory partition management (awaiting review: #24)
  • Unit tests for the in-memory partition management logic (awaiting review: #24)
• Disk operations as a state machine (awaiting review: #24)
  • Partition creation disk operations (awaiting review: #24)
  • Partition move/resize disk operations (awaiting review: #24)
  • Partition remove disk operations (awaiting review: #24)
  • Partition format logic (awaiting review: #24)
• Ability to probe and obtain device information from public API
• Ability to create partitions from public API
• Ability to move/resize partitions from public API
• Ability to remove partitions from public API
• Ability to format partitions from public API
• Handle restrictions surrounding msdos and primary/logical/extended partitions
• Check partitions after writing

latest rust version being an example

libparted doesn't report the correct partition names, so we will have to do as gparted does, and get the proper partition names through other means.

It can be useful for installers to know if there are operating system's installed on partitions.

• Linux
• BSDs
• Mac OS
• Windows
• Btrfs subvolume-based detection
• LVM-based detection

Integration tests are being created surrounding the distisnt CLI command. The following partition-related integration tests are in the process of being completed, or already are implemented.

• Table Creation & New Partitions
• Partition Removal
• Partition Reuse
  • Keep Filesystem
  • Reformat
• Partition Resizing
  • Growing
  • Shrinking
• Partition Moving
  • Backward
  • Forward

I'm curious about the scope of this repo, what it does now, and what it's intended to do in the future. Would it be possible to add that to the readme? It should help discoverability too.

Thanks!

Mounts need to be dropped in reverse in the event of failure, and this can be done by wrapping the Vec<Mount> returned by Installer::mount() into a new Mounts structure which implements the Drop trait to iterate and drop mounts in reverse.

We support LUKS on LVM, and LVM without LUKS, so next will be supporting LUKS on LVM partitions. Difficulty may arise in the implementation due to the recursive nature of creating logical devices from logical volumes, which may themselves create more logical devices from their logical volumes.

Users will want the custom targets that they have specified in the installer to be written to the file system tab by distinst. To do this, all of the code involving mounting will need to be rewritten so that mount points in the chrooted system are created and mounted before the mounting process begins. With this implemented, it will no longer be required for the configure.sh file to generate the fstab file.

In the event that Ctrl + C is pressed, the program (distinst) should be able to clean up after itself before exiting. In particular, unmounting everything that's been mounted before exiting.

• Implement kill switch in the lib with an AtomicBool
• Implement signal handler in the distinst bin that controls that kill switch.

This causes the install to fail when it attempts to swapoff the partition. A fix would be to check it is swapped using /proc/swaps before attempting to swapoff

Source Code: https://github.com/pop-os/distinst/blob/master_bionic/src/disk/operations/move_and_resize.rs#L306:L327

There's a lot of room for some optimization of partition moving here. The current code is very basic. We could easily improve performance by utilizing chunks to perform buffered read/write operations (no more than 64 MB at a time, of course -- 16 MB is probably a good number).

I'll create this one for you

Right now distinst only removes a small handful of hardcoded packages, so the elementary OS installation ends up with gparted (and probably some more stuff) still present.

If this is a bug, please use the template below. If this is a question, or a general discussion topic - please post on our subreddit https://reddit.com/r/pop_os or start a conversation in our chat https://chat.pop-os.org/ - as these are the proper forums for those types of posts.

Pop!_OS 18.04

Issue/Bug Description
F2fs partitioning works but actually installing does not. It seems that the root partition is not added to /etc/fstab and therefore not found.

If this is a bug, please use the template below. If this is a question, or a general discussion topic - please post on our subreddit https://reddit.com/r/pop_os or start a conversation in our chat https://chat.pop-os.org/ - as these are the proper forums for those types of posts.

 * Distribution - (run ```cat /etc/os-release```)
 * Related Application and/or Package Version - apt policy $PACKAGE NAME

Pop!_OS 18.04

Due to an issue with keyfile support implementation, this testing was done from commit: c2773bf

Issue/Bug Description
Attempting to partition and install Pop!_OS 18.04 using the partition-encryption.sh script without the --test parameter fails.

Output of the installation process:
encryption-installation.txt

When booting after the installation:

Please unlock disk cryptdata:

types in password

cryptsetup (cryptdata): unknown fstype, bad password or options?
Please unlock disk cryptdata:

Steps to reproduce (if you know)
The only change made to the partition-encryption.sh script was removing --test

Partition Magic?

Set up /etc/hostname and /etc/hosts