In the previous module we generated a cross-toolchain that allows us to compile programs for the BeagleBone Black. We did this (almost) from scratch, in tune with the spirit of this training. In this module we will continue going down the rabbit hole, learning about the components that make up a Linux-based system, and in a similar way to what we did with the toolchain, we will generate some of these components by directly building them from the source code distribution.

Linux-based systems

At its core, a Linux-based system is composed of three main pieces:

  1. a bootloader,
  2. a kernel (Linux), and
  3. a root filesystem.

All three must be present in one way or another for a Linux-based embedded device to actually do something after it is energized.

A bootloader can be thought of “the first thing” that is executed right after the device is turned on. It doesn’t necessarily have to be a single binary, and in most cases, it is in fact a sequence of them – in the sense that one runs after the other. This is referred to as a “multi-stage” bootloader. Each platform has its own idiosyncrasies when it comes to the boot sequence, but in general it works like this:

  • Boot ROM
    All modern embedded systems include a boot ROM of some sort whose only goal is to load the Initial Program Loader (IPL). On x86 devices this is contained in the BIOS, which tends to try to find the first stage boot loader from a series of different choices. On other sorts of systems-on-chip (e.g., ARM-based) these tend to be built into the SoC itself. In these cases, there are typically boot pins that dictate which of several options will be consulted to find the code for the next stage. Typically, SoCs can boot from NOR flash, NAND/eMMC, SD/MMC, SPI, USB, etc.
  • IPL
    The Initial Program Loader is usually very small and typically written in assembler. In all cases other than NOR flash (which is memory mapped), the IPL is usually the first page or block read from the selected boot device, typically this is something like 1-4 KiB of code. The IPL is responsible for initializing the CPU, memory, and enough of the selected boot drive in order to load the code for the next stage.
  • SPL
    The Secondary Program Loader (SPL) can in fact be the bootloader in some platforms. However, in the case where it is not, the SPL has the simple task of loading the bootloader, typically from the same boot device from which the SPL was loaded. A separate SPL allows the bootloader to be upgraded without potentially breaking the base boot code which is contained within the SPL.
  • Bootloader
    The bootloader proper is usually more complex and is often designed to support booting the kernel from multiple sources. It offers network booting if the appropriate peripherals are available. The bootloader will load the kernel into memory along with an optional initrd rootfs, and a flattened device tree.
In the interest of time, we won’t discuss the concept of initial ramdisks (or initrd rootfs as it is commonly known). In a proper, hardened Linux-based system, it is beneficial to make use of it.

In the case of the kernel itself, there is not that much to say at the moment. After it has been loaded by the bootloader, it will start its own initialization sequence, which usually revolves around probing the peripherals available and loading the appropriate drivers. After that, Linux will give control to an executable contained the root filesystem (which will become the process with PID 1).

The root filesystem is, basically, the rest of the files that make up the system: everything under /. It’s a collection of executables, configuration files, documents, movies, etc. Linux does not require a specific structure for the root filesystem, but there are conventions that should be followed if one hopes for the system to be used by others. Namely, it is expected, for example, that /bin contains executables that are available system-wide, and that /dev contains device nodes. As mentioned, however, this does not have to be the case, and Linux will happily coexist with a root filesystem that exhibits a non-standard structure.

We will discuss device nodes in detail when we talk about drivers.

Booting the BeagleBoard Black with a custom Linux-based system

In the last module we used a prebuilt image to flash onto the uSD card and subsequently boot from. We will proceed to replace this image with one where the three components we just talked about are more readily apparent. Throughout the training we will incrementally replace each of the components (i.e., bootloader, kernel and root filesystem) with our own versions.

For this it is also necessary to have more control over the way the uSD card is partitioned, so we will proceed to do it “by hand” from now on.

  • Download the archive hosted here. It contains a minimal set of files needed to create a complete Linux-based system that can boot on the BeagleBone Black. If you are using a virtual machine, make sure that this file is accessible from there. (You could copy it from Windows to the VM, share it, etc.)
  • Extract the contents of the archive and navigate to the corresponding directory. It should contain a shell script named prepare_card.sh and a directory named image_files, which in turn contains the following files:
  • Take some time to read through the shell script and get an idea of what it is doing.
  • Insert the uSD card in the host computer and make sure that it is not mounted. This could happen automatically if you are still using a copy of the image we were working with in the last module.You can check the output of lsblk and see if the uSD card has any partition currently mounted.
    • In this case, sdb represents the uSD card. It has two partitions, and both are currently mounted.
    • The output of lsblk on your system does not have to match this, though. The important point is to make sure that the uSD card is not being used.
      • If it were, you can do something like $ sudo umount /media/training/root, for example. Again, the path does not have to be the same for you.
  • Execute the contents of prepare-card.sh.The script is not marked as executable, but you can do that if you prefer. Otherwise simply run $ shprepare-card.sh.This could take a couple of minutes, but it should succeed. If that is not the case, try and debug the issue, otherwise feel free to reach out to the instructor.
  • After the script finishes, safely eject the uSD card (e.g., $ sudo eject /dev/sdb) and insert it into the appropriate slot in the BeagleBone Black.
  • Connect to the serial line from the host computer.You can continue using screen for this, but if you are encountering persistent issues, consider an alternative.It is also possible to interact through the serial connection from Windows. Something like Tera Term is a good choice.
  • Hold the BOOT button and then energize the board. This will force the device to boot from the card, as opposed to the on-board eMMC.
    • It may be necessary to remove the expansion cap that comes with the BeagleBone Black. Be careful not to break the pins, though!
  • You should start seeing logs like these:
  • The boot process should go on as usual until you hit the login prompt.At this point the most important thing is to verify that the timestamps at the start of the boot log match what you are seeing here. This will indicate that the bootloader being used is the one that we just copied to the uSD card.A perceptive person could realize that we are not using the kernel and root filesystem from the uSD card, but this is not relevant at the moment.

Configuring and building the bootloader

We will start by replacing the bootloader in the previous image with one generated by ourselves. The overall process should be familiar given that it is very similar to what we did while generating the cross-toolchain.

The bootloader we will be working with is U-Boot. It is by no means the only choice, but we do need to focus on one, and U-Boot is certainly one of the most actively developed projects in this area. It is also (and declares an intent to remain) patent-free and completely under the GPL.

In principle, building U-Boot is quite simple. It uses make as the build system and generally speaking consists of two steps:

  1. First, we need to do a configuration step, where we end up with a text file named .config that contains entries of the form CONFIG_KEY=value.
    1. These entries determine the “flavor” of the final image, and the presence or not of certain features.
    2. For example, an entry like CONFIG_SYS_ARCH="arm" indicates that the target architecture is ARM.
    3. All of these settings can be handled visually through a build target called menuconfig (just like with crosstool-NG).
    4. A lot of “prebaked” configurations are available from U-Boot itself in the form of build targets whose names follow the pattern name_defconfig, where name refers to an architecture, a board, etc.
  2. Finally, we build the image following the usual make-based approach.
    • The default build rule generates the SPL and the bootloader proper.

Let us start getting ready:

  • Install some build prerequisites:
$ sudo apt install flex swig
  • Obtain a copy of the source code:
$ git clone -b v2022.04 https://source.denx.de/u-boot/u-boot.git --depth 1
  • The BeagleBoard requires a couple of patches to U-Boot. These are officially maintained by the community.
$ cd u-boot
$ git pull --no-edit https://git.beagleboard.org/beagleboard/u-boot.git v2022.04-bbb.io-am335x-am57x

U-Boot needs to be cross-compiled, however, so we must use the toolchain generated in the previous module. To make things easier, let us add the relevant entries to PATH:

  • Open the shell profile using your text editor of choice. Depending on the system, it can have different names, but usually is something like .bashrc or .profile.
  • Add the following line: export PATH=$HOME/cross-toolchain/arm-unknown-linux-gnueabi/bin/:$PATH
  • Reload the configuration by, for example, closing and reopening the terminal emulator.

The Makefiles used by U-Boot rely on an environment value named ARCH, which should contain the name of the architecture for which the bootloader is being configured and built. In our case, the value would be arm, so all invocations of make in the context of U-Boot would look like this:

$ make ARCH=arm <target>
Obviously, you can specify a different value for ARCH. If you take a look in the source tree, you will see a directory appropriately named arch, which in turn contains subdirectories like arm, powerpc, riscv, x86, etc. These contain platform-specific code for each architecture, and their names are valid values for ARCH, so you could do, for example $ make ARCH=riscv <target> if you wanted to build U-Boot for a system based on RISC-V.

There is one more environment variable that is relevant for cross-compiling U-Boot (and Linux as well), and it is named CROSS_COMPILE. Its need stems from the fact that U-Boot cannot know which toolchain is being used to build its source code. In particular, let us assume that you are writing a simple build rule for U-Boot, and you want to generate a relocatable object file out of a file named foo.c. In principle one could write something like the following:

CC := gcc

foo.o: foo.c
  $(CC) foo.c -c -o foo.o

However, what if we want to build two versions of U-Boot, one with a toolchain based on the triplet arm-none-eabi and another based on aarch64-linux-gnueabihf? The Makefile cannot then use a hard-coded name for the compiler (it would be named arm-none-eabi-gcc and aarch64-linux-gnueabihf-gcc in the previous examples).

This is solved by introducing an environment variable named CROSS_COMPILE that gets prepended to the compiler (and every other component of the toolchain) every time it is referenced in a Makefile. So the previous file would then look like this:

CC := gcc

foo.o: foo.c
  $(CROSS_COMPILE)$(CC) foo.c -c -o foo.o

We can then call make with the appropriate value for the variable:

$ make CROSS_COMPILE=arm-none-eabi- foo.o
$ make CROSS_COMPILE=aarch64-linux-gnueabihf- foo.o
Notice that the value of the variable ends with a - character. This is important! Make sure you understand why.

We can now finally configure and build U-Boot.

  • Start with a default configuration for the SoC present on the BeagleBoard Black:
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- am335x_evm_defconfig
  • Perform a couple of modifications to the configuration:
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi- menuconfig
    • Boot options
      • Autoboot options
        • Change delay in seconds before automatically booting to 5
          • It can be whatever you want, just make it something large enough so that you have time to interrupt the autoboot process.
    • Environment
      • Deactivate Environment is in a EXT4 filesystem
      • Activate Environment is in a FAT filesystem
  • Exit the menu and save the configuration.
  • Build the SPL and the bootloader.
$ make ARCH=arm CROSS_COMPILE=arm-unknown-linux-gnueabi-
      • It is recommended that if your computer has more than one processor (which is almost surely the case) then you use all of them to speed up the process. To that end, pass -j $(nproc) as an argument to the previous command.
      • Feel free to do some research if you are unsure about what this is doing. Alternatively, disregard the previous advice.

Once this is done, you should have both the SPL and the bootloader proper. The former is named MLO, and the latter u-boot.img.

Booting the BeagleBoard Black with the custom bootloader

You can now use the two files generated in the previous section as replacements for the bootloader we have been using so far. By this point you should be familiar with the booting process for the BeagleBone Black so we will not waste time going over it again.

When it comes to copying the new files to the uSD card, you could in principle replace the ones inside custom_image/image_files and execute prepare_card.sh again. That would certainly work, although it would also do a lot of unnecessary work, since we are not interested in replacing any of the other files that were already copied. Find a way to do the least amount of work necessary to only copy the two new files.

Once you have done that and are ready to boot the board, remember to press the BOOT button so you actually boot from the card. Pause the autoboot countdown and make sure that the timestamp at the top of the boot log matches the time around which the build step for U-Boot finished, which proves that we are in fact using the file generated in the previous section.

In the interest of understanding how things work under the hood, we will proceed to manually load the kernel into memory and then pass control to it.

U-Boot provides a shell-like interface after the autoboot process has been stopped. To get an idea of the commands available, you can type help:

=> help
?         - alias for 'help'
askenv    - get environment variables from stdin
base      - print or set address offset
bdinfo    - print Board Info structure
blkcache  - block cache diagnostics and control
boot      - boot default, i.e., run 'bootcmd'
bootd     - boot default, i.e., run 'bootcmd'
bootefi   - Boots an EFI payload from memory
bootelf   - Boot from an ELF image in memory
bootm     - boot application image from memory
bootp     - boot image via network using BOOTP/TFTP protocol
bootvx    - Boot vxWorks from an ELF image
bootz     - boot Linux zImage image from memory
btrsubvol - list subvolumes of a BTRFS filesystem
clone     - simple storage cloning
cmp       - memory compare
coninfo   - print console devices and information
cp        - memory copy
crc32     - checksum calculation
dfu       - Device Firmware Upgrade
dhcp      - boot image via network using DHCP/TFTP protocol
dm        - Driver model low level access
echo      - echo args to console
editenv   - edit environment variable
eeprom    - EEPROM sub-system
env       - environment handling commands
exit      - exit script
ext2load  - load binary file from a Ext2 filesystem
ext2ls    - list files in a directory (default /)
ext4load  - load binary file from a Ext4 filesystem
ext4ls    - list files in a directory (default /)
ext4size  - determine a file's size
ext4write - create a file in the root directory
(...)
Each of these basically corresponds to one of the files in u-boot/cmd. Commands are C functions with a specific signature that are invoked by the shell-like interface depending on the user’s input. For a simple example, take a look at u-boot/cmd/echo.c.

The process of booting a Linux kernel is highly platform-specific, and therefore it is adequate to mention that the exact sequence of steps we will consider in a minute is not necessarily applicable to other embedded systems. However, the general approach should be similar in the sense that we roughly want to do the following:

  1. Load the contents of file vmlinuz (the Linux kernel) into main memory at a specific address.
  2. Load the contents of file am335x-boneblack.dtb (the device tree blob used by the BeagleBone Black) into main memory at a specific address.
    • We will talk about device trees in a future module. For the moment, think of them as “something” that is required to boot a Linux-based system on ARM.
  3. Specify the arguments to pass to the kernel.
    • Keep in mind that, at its core, the Linux kernel is “just” a program, and as such it can be given arguments.
  4. Pass control to the kernel that now resides in main memory.

One of the commands provided by U-Boot is called lsblk, a version of which we have been using to list the available block devices. If we execute it we can see the following output:

Block Driver          Devices
-----------------------------
efi_blk             : <none>
mmc_blk             : mmc 0, mmc 1
usb_storage_blk     : <none>

Here, mmc_blk refers to the uSD card, and mmc 0 and mmc 1 to each of its partitions.

If you recall from before, when we copy the system image to the uSD card, we place the kernel and device tree blob in the first partition. Fortunately, U-Boot also provides a way to list the files in a device through the ls command.

=> help ls
ls <interface> [<dev[:part]> [directory]]
    - List files in directory 'directory' of partition 'part' on
      device type 'interface' instance 'dev'.

We can then check that the kernel and device tree blob are indeed there:

=> ls mmc 0:1 /
 11698688   vmlinuz
    99231   am335x-boneblack.dtb

The first partition on the uSD card is formatted as FAT32. To deal with these filesystems, we can use the fatload command.

=> help fatload
fatload - load binary file from a dos filesystem

Usage:
fatload <interface> [<dev[:part]> [<addr> [<filename> [bytes [pos]]]]]
    - Load binary file 'filename' from 'dev' on 'interface'
      to address 'addr' from dos filesystem.
      'pos' gives the file position to start loading from.
      If 'pos' is omitted, 0 is used. 'pos' requires 'bytes'.
      'bytes' gives the size to load. If 'bytes' is 0 or omitted,
      the load stops on end of file.
      If either 'pos' or 'bytes' are not aligned to
      ARCH_DMA_MINALIGN then a misaligned buffer warning will
      be printed and performance will suffer for the load.

For the BeagleBone Black, the kernel must be loaded at address 0x82000000, and the device tree blob at 0x88000000. Armed with this knowledge, we can finally load them both:

=> fatload mmc 0:1 0x82000000 /vmlinuz
11698688 bytes read in 769 ms (14.5 MiB/s)
=> fatload mmc 0:1 0x88000000 /am335x-boneblack.dtb
99231 bytes read in 9 ms (10.5 MiB/s)

For the kernel arguments, we can take advantage of the fact that U-Boot allows to define environment variables, that can later be “read” using the familiar shell syntax ${variable}. The command used to set the value of an environment variable is setenv.

=> help setenv
setenv - set environment variables

Usage:
setenv setenv [-f] name value ...
    - [forcibly] set environment variable 'name' to 'value ...'
setenv [-f] name
    - [forcibly] delete environment variable 'name'

For the moment we want the following arguments:

=> setenv bootargs console=${console} root=/dev/mmcblk0p2 ro rootfstype=${mmcrootfstype}
In the interest of time, we will not go through each of these and see what they indicate. A guide to the parameters recognized by the kernel can be found at https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

Finally, we can tell U-Boot to pass control to the kernel image in main memory. For this we make use of the bootz command:

=> help bootz
bootz - boot Linux zImage image from memory

Usage:
bootz [addr [initrd[:size]] [fdt]]
    - boot Linux zImage stored in memory
        The argument 'initrd' is optional and specifies the address
        of the initrd in memory. The optional argument ':size' allows
        specifying the size of RAW initrd.
        When booting a Linux kernel which requires a flat device-tree
        a third argument is required which is the address of the
        device-tree blob. To boot that kernel without an initrd image,
        use a '-' for the second argument. If you do not pass a third
        a bd_info struct will be passed instead

=> bootz 0x82000000 - 0x88000000
Kernel image @ 0x82000000 [ 0x000000 - 0xb28200 ]
## Flattened Device Tree blob at 88000000
   Booting using the fdt blob at 0x88000000
   Loading Device Tree to 8ffe4000, end 8ffff39e ... OK

Starting kernel ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 6.11.0-bone7 (training@modularmx) (arm-linux-gnueabi-gcc (GCC) 13.3.0, GNU ld (GNU Binutils) 2.42) #1 PREEMPT Thu Sep 26 05:42:53 JST 2024
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=50c5387d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: TI AM335x BeagleBone Black
(...)

Notice how the kernel logs refer a hostname that mentions the username and hostname under which the former was built. This kernel image was prepared by the instructor following an approach very similar to what we did for U-Boot. This is precisely what we will be doing for the next module.

Also note that U-Boot already provides environment variables that contain the addresses at which these two files must be loaded in main memory. They are called loadaddr and fdtaddr, for the kernel and device tree blob, respectively.

Labs

Write a U-Boot command. You can use cmd/echo.c as a basis.

  • The command must be “registered” by adding it to cmd/Makefile and cmd/Kconfig. As mentioned before, use echo as a basis.
  • Be sure to copy the new version of the bootloader to the uSD card. It is not necessary to copy MLO again, only u-boot.img.
  • Is it listed under help? Try running it!
  • Familiarize yourself with saving the U-Boot environment through saveenv.Identify the settings in the menuconfig-based interface that are associated with this feature.
  • Add support for uEnv.txt to our environment.
    • This is a way to “inject” variable definitions to the U-Boot environment.
    • In particular, follow the definition of bootcmd and notice how at some U-Boot tries to import a file into its environment with env import.
    • This file, usually named uEnv.txt, contains settings that can override those in the default environment. In particular, it is possible to override the default build sequence by providing a definition for the uenvcmd variable.
    • Make it so that we can load a definition like this into the environment:
uenvcmd=echo Executing custom boot command...; \
  fatload mmc 0:1 $loadaddr /vmlinuz; \
  fatload mmc 0:1 $fdtaddr /am335x-boneblack.dtb; \
  setenv bootargs console=${console} root=/dev/mmcblk0p2 ro rootfstype=${mmcrootfstype}; \
  bootz $loadaddr - $fdtaddr
    • This would provide a way to either do run uenvcmd or simply let U-Boot proceed with the autoboot process.

Closing thoughts

We only scratched the surface of what U-Boot can do. You are encouraged to play with it and push the limits of our development environment.

In the next module we will continue the trend of replacing the components of the system by configuring and building our own version of the Linux kernel.