bjohnson05 / cmsi3510

Course Materials for LMU CMSI 3510 ~ Operating Systems

CMSI387

Notes on building Linux

Remember that the kernel and the rest of the operating system are two different things! The kernel is one option of a thing you will need to build and/or modify in this class; the other is what is known as the common utilities. Here are some notes from my past build experiences that will help you along the way. REMEMBER that a very large part of this process is researching information, especially when things don't work right. You will find that operations RARELY go right with this process the first time through, so BE PREPARED TO LOOK THINGS UP!

1. Download a copy of the free VM from Oracle called Virtual Box and install it. Versions are available for both Windows [I used version 6.0.4-128413-Win.exe] and Mac [I used 6.0.0-127566-OSX.dmg], so you should be able to do this without regard to which platform you use. There may be later versions available, and they should work. I have found that when I started up the installed versions, I got a notification [on mac] that there is a later version available. If that is the case for you, go ahead and install the later version – it won't corrupt anything you've done previously. Virtual Box is available for free from: this site
2. Download a copy of Ubuntu Linux from the Ubuntu website. I used version 18.10, which is the SECOND green button on the page. The "LTS" button is for "Long Term Support" and the 18.10 version still has 9 months of support if you want it. Ubuntu is available for free from: here
3. When you've downloaded it [it's an iso file] you need to make a new machine on your Virtual Box VM. Start up the VM, then click the new icon to make a new machine. Call it whatever you like [I used ubuntuv1 so I can make more with successive numbers in case of a mistake]. Make sure you select the Linux install [which appears automatically if you name your VM instance with the word ubuntu in it] and click next. Reportedly you should use at least 4GB of RAM for this instance, so set the slider accordingly, and click next. You will also need to give yourself a 40GB hard drive to be able to successfully compile. I tried it with 10, 20, and 30 GB drives and it failed on all of them. You can take the defaults for the rest of the selections.
4. Start the VM you just built, and us it to install Ubuntu. There are a number of screens that will be required on this first start up and install, but once you have it up and running, it will start Linux automatically every time you start this particular VM instance. DON'T WORRY about the prompt that says something about erasing your hard disk… this is just erasing the VIRTUAL MACHINE'S HARD DISK, not the real one on your computer. Also, REMEMBER YOUR PASSWORD as this will be used to log in and to do sudo tasks later.
5. Start Ubuntu, if it isn't already started, and wait for it to load completely and display the desktop. Click on the little arrow in the upper right corner of the Ubuntu screen, then click the tool icon. Click the PRIVACY tab and turn screen lock off. Then click the POWER tab and make sure power saving is set to NEVER blank the screen.
6. Check if GCC is loaded: open a terminal window in Ubuntu and run the command gcc to check. If it's not there, you can get it by doing sudo apt install gcc to get it. When it is finished, type gcc again to make sure it's working. NOTE THAT THIS IS A DIFFERENT INSTANCE than the one you installed on your HOST machine to do homework and exercises!
7. Check out askubuntu.com for help – this site can be your best friend
8. Check out the build your own kernel page for some help and advice on the overall process of downloading the source and building the project; make sure you build it BEFORE you modify it, to prove the build process works, THEN do your modification
9. Note that building a NEW kernel takes several hours to compile; however, building the EXISTING kernel only takes a few minutes, since you didn't change any code [yet].
10. The sudo command will give you a temporary access as if you were the root user; you will need the account password you set up when you installed Ubuntu on your VM
11. The fakeroot command is also very helpful, and sometimes you need to use them together to get things to work. I've had success with sudo fakeroot when the command didn't work for either of them independently
12. Run sudo software-properties-gtk to get a dialog box that has a check box which will enable downloading the source code
13. Run sudo apt-get update to update the apt-get utility
14. Run sudo apt-get install dpkg-dev to install the dpkg-dev package if it is missing
15. Run sudo apt-get dist-upgrade to upgrade the distribution manager utility; this will NOT update the UBUNTU O/S, it will ONLY do the distribution manager
16. Check out Dr. Forney's homework 3 page for more helpful information, including some fun XKCD comics; there are also some helpful links and advice
17. Remember that the frustration is part of the fun…

The Build Process that helped me the most with Linux

1. Here is a list of things to do and to know that helped my get my build completed:
   

2. LTS = "Long Term Service" ~ you don't really need, but is slightly more stable than non-LTS version

3. The kernel is a file that is in directory "/boot". The file is called vmlinuz<version>"

4. Find the linux kernel archives at "www.kernel.org" and download the tarball for the version of interest. I used 4.19.34 which is from April 5th 2019

5. The "xz" extension is yet another type of "zip" archive compression. The Linux "tar" program knows how to unzip it with the correct command line switch option "J" <note: uppercase!>

6. Start firefox and navigate to www.kernel.org

7. Get kernel 4.19.34 tar ball – goes into Downloads folder when using FireFox

8. Use the mv command to move the zipped file to your home directory

9. Untar with tar xJvf linux-4.19.34.tar.xz: the gibberish xJvf means:

   • x to extract
   • J because it's a zipped file with XZ format
   • v for verbose so you can see the listings
   • f for the file name which follows

10. Change into the resulting top level directory linux-4.19.34

11. Do sudo apt install gcc to make sure that gcc is installed and updated

12. Do sudo apt install make to make sure that the make utility is installed and updated

13. During these operations you may get an error as follows: E: Could not get lock /var/lib/dpkg/lock – open (11: Resource temporarily unavailable)
    E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?

14. Do ps -eaf | grep -i apt :: this shows running apt processes ~~ there is a daily check for updates that locks apt ~~ you can kill the processes using kill -9 <process id>, or you can just wait for them to finish, they don't take TOO long...

15. You can use the ps -eaf | grep -i apt command to get the processes using the lock

16. You can use the kill -9 <process ID> command to kill them off

17. Do sudo apt-get install dpkg-dev to install the dpkg manager for the development environment

18. Do sudo apt-get dist-upgrade to update the apt-get distribution

19. Do make menuconfig to see options ~ make sure you SAVE so there is a config file for the build

20. Execute the command make && make modules_install && make install to run the build
    The && means concatenate the commands, but only continues if the previous one is successful
    You might have trouble with some of the required libraries being missing
    I had to get the ssl lib with sudo apt-get install libssl-dev
       the -dev makes sure to put the headers where they belong
    I had to get the ncurses lib with sudo apt install ncurses-dev
    I had to install bison with sudo apt-get install bison
    I had to install flex with sudo apt-get install flex
    

21. Then I had to do the make line again:
    make && make modules_install && make install

22. I did this on a VM that had 4096 RAM and 30 GB hard disk and ran out of space; I tried again with 8192 RAM and 40 GB hard disk

23. I started the build at 08:43; build failed at about 12:45 because I couldn't make a directory.

24. I switched to root account and tried again. Just did make and that seemed to work

25. I then did make module_install && make install and make skipped the compilation and went directly to "INSTALL"

26. Finished the build, then restarted Ubuntu

27. FINALLY SUCCESS!! I did a "uname -a" and saw the kernel version is my new 4.19 and the created date is today.

Adding a New System Call to the Kernel

In the directory where you put your source code; e.g., /home/beej/linux-4.19.34

• NOTE: you will need to write the code for your addition first because you need the "entry point"
  
• change to the directory arch/x86/entry/syscals
  
• find the file "syscall_64.tbl" [or "syscall_32.tbl" if you are on a 32-bit machine]
  
• you can list it out with more
  
• there are [on my build] 547 entries in there
  
• edit the file using vi or vim
  
• add your new command to the end with a new number
  The list of calls in the table will show you what you can modify if you are changing something
  

Semester Project Actions and Deliverables

• Come up with an idea for a modification or addition to the Linux kernel, and get it approved with the instructor. This might be an addition to an existing command, a new command, or an entirely new application that runs on Linux.
• Design the modification; don't do this in the dark. You might start by implementing some version of your idea in a stand-alone way so that you can more easily build and de-bug it. Doing this on a prototype can be much faster than having to wait three hours for compilation each time you re-build the Linux kernel
• Here is a list of the Spring 2019 project teams with an idea of what needs to be modified or implemented to accomplish the project tasks
  

1. Chu, Edmiston, Keba: drive automounting ~ module[s]: mount
2. Crowther, Rajavasireddy, Simmons: user account creation ~ module[s]: useradd
3. Moini, Lizarda, Zafiris: "hog" system utility that lists CPU hogs ~ new module: hog
4. Santander, Namba, C. Nguyen: "mbob it" file name case randomizer utility ~ module[s]: "mv" and "ls" utilities
5. K. Nguyen, Ruiz, Tolliver: "processInfo" module interfaces to kernel log ~ new module: processInfo
6. Higgins, Fletcher, KPatterson: fun inefficient task scheduler ~ module[s]: sched.c
7. Lopez, Byrne, Boyac: new scheduling algorithm for the I/O scheduler ~ module[s]: sched.c
8. Prochnow, Flora, Jay: new output to "w" module ~ module[s]: [[unsure which is required]]
9. Ochsner, Peters, West: cursor movement without mouse ~ module[s]: mousedev, keydev
10. Braekman, Kern, Martin: combination utility, mimics "find" and "grep" in one ~ new module: search
11. Arteaga, Wroblewski, Persily: "top" utility modification for priorities ~ module[s]: top [[might get from http://procps.sourceforge.net/index.html]](http://procps.sourceforge.net/index.html)
12. Patterson, Richardson: timedatectl modification ~ modules: set-time

* Some project resources: Consider using "apt-get source <module_name>" to get the source code for specific commands or modules. You might get them from [http://procps.sourceforge.net/index.html](http://procps.sourceforge.net/index.html) or possibly from [http://lxr.linux.no/#linux+v4.15.14/kernel/sched.c#L3566'](http://lxr.linux.no/#linux+v4.15.14/kernel/sched.c#L3566). Don't forget you may need to sudo the apt-get command. * note that not all of these are technically kernel modifications or addiitons. However, in my view, if you are creating or modifying a utility level program, you will still be making use of the kernel system calls, and you will have had to build your own version of the kernel as part of the class, so it still counts for credit!